Shared scratchpad memory with parallel load-store

Abstract

Methods, systems, and apparatus, including computer-readable media, are described for hardware circuits configured to implement neural networks. The circuits include a first memory, respective first and second processor cores, and a shared memory. The first memory provides data for performing computations to generate an output of a neural network layer. Each of the first and second cores includes a vector memory for storing vector values derived from the data provided from the first memory. The shared memory is generally disposed intermediate the first memory and at least one core and includes i) a direct memory storage (DMA) data path configured to route data between the shared memory and respective vector memories of the first and second cores, and ii) a load-store data path configured to route data between the shared memory and respective vector registers of the first and second cores.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Utility Application No. 15 / 931,970, filed May 14, 2020, and U.S. Provisional Application No. 62 / 966,518, filed January 27, 2020, the contents of which are incorporated herein by reference. Background Technology

[0003] This specification generally relates to circuitry used for hardware circuitry to perform neural network calculations.

[0004] A neural network is a machine learning model that uses one or more layers of nodes to generate outputs—such as classifications—from received inputs. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as input to one or more other layers in the network—such as other hidden layers or output layers of the network. Some layers of the network generate outputs from the received inputs based on the current values ​​of their respective parameter sets. Some neural networks are convolutional neural networks (CNNs) configured for image processing or recurrent neural networks (RNNs) configured for speech and language processing.

[0005] A CNN's neural network layer can have an associated set of kernels that can correspond to parameters or weights. This associated set of kernels is used to process inputs (e.g., batches of input) through the neural network layer to generate the corresponding output of that layer for computing neural network inferences. The sets of input batches and kernels can be represented as tensors of inputs and weights, respectively. Tensors can correspond to multidimensional arrays. The hardware circuitry implementing the neural network includes memory with locations identified by address values. Memory locations can correspond to elements of tensors, and tensor elements can be traversed or accessed using the circuit's control logic. For example, the control logic can determine or compute the memory address value of an element to load or store the corresponding data value of that element. Summary of the Invention

[0006] This document describes a hardware circuit with an improved architecture that includes resources allocated for sharing static random access memory (SPAM) among multiple corresponding processor cores of the circuit. More specifically, the circuit includes shared memory, which is typically located between high-bandwidth memory (HBM) and at least two processor cores of a dedicated processor. Shared memory includes resources such as memory banks and registers that are physically and logically arranged relative to certain computational resources of the processor cores. The physical and logical arrangement of shared memory resources allows for increased speed and throughput during data communication between the HBM and the corresponding processor cores of the hardware circuit, as well as between different resources of a single processor core or between corresponding resources of two different processor cores.

[0007] One aspect of the subject matter described in this specification can be embodied in circuitry configured to implement a neural network comprising multiple neural network layers. The circuitry includes: a first memory configured to provide data for performing computations to generate the outputs of layers in the neural network; a first processor core including a first vector memory configured to store first vector values ​​derived from data provided from the first memory; a second processor core including a second vector memory configured to store second vector values ​​derived from data provided from the first memory; and a shared memory disposed between at least one of the first or second processor cores and the first memory. The shared memory includes: a direct memory access (DMA) data path configured to route data communication between the shared memory and the first or second vector memory; and a load-store data path configured to route data communication between the shared memory and corresponding vector registers of the first or second processor core.

[0008] These and other implementations may optionally include one or more of the following features. For example, in some implementations, the circuitry includes a plurality of processor cores, with a first processor core and a second processor core among the plurality of processor cores; and shared memory includes a plurality of memory resources physically distributed around the circuitry to exchange data with each of the plurality of processor cores at the circuitry.

[0009] In some embodiments, the shared memory includes a shared memory control unit configured to execute software instructions that cause a first portion of a plurality of memory resources to act as DMA memory cells operable to move data between a first memory and each of a first processor core and a second processor core. The plurality of memory resources include a second portion of the resources configured to receive data values ​​routed along a load-store data path and to temporarily store the data values ​​for a duration of a threshold number of processor cycles.

[0010] The second portion of the resource can be configured to provide data values ​​to the corresponding vector registers of the first or second processor core in response to a processor cycle lasting for a threshold number of temporary storage data values. In some embodiments, the shared memory includes a software-controlled staging resource formed from a subset of the shared memory's memory resources, which is used to manage the flow of data values ​​from the first memory to the corresponding vector registers of the first or second processor core.

[0011] In some embodiments, the circuit includes a matrix computation unit configured to perform a subset of computations to generate accumulated values ​​used to generate the output of a neural network layer. Software-controlled architectural resources can be used to manage data flow corresponding to a vector array from a first memory to the matrix computation unit, wherein the vector array is derived from data values ​​provided from the first memory. In some embodiments, the circuit includes a vector processing unit communicating with the first memory; the vector processing unit is configured to generate a vector of activation values ​​from the accumulated values ​​generated at the circuit; the vector of activation values ​​corresponds to the output of the neural network layer.

[0012] The software-controlled architectural resources can be a first-in, first-out (FIFO) memory structure along the load portion of the load-store data path; and the FIFO memory structure is configured to temporarily store the value vector for a threshold number of processor cycles before routing the value vector to the corresponding vector register of a first or second processor core. In some embodiments, shared memory is configured to act as a shared global memory space, which includes memory resources corresponding to a group of memories shared among one or more processor cores in a plurality of processor cores.

[0013] In some implementations, the data used to perform computations to generate the output of the first layer includes: inputs to be processed by the first layer of the neural network; a set of corresponding weights for the first layer of the neural network; and instructions for using the set of corresponding weights of the first layer to process one or more of the inputs through the first layer to generate the output of the first layer.

[0014] One aspect of the subject matter described in this specification can be embodied in a method for performing computations to generate outputs for layers of a neural network comprising multiple neural network layers using circuitry configured to implement a neural network. The method includes: providing data from a first memory for generating the outputs of the neural network layers; and storing a value vector in the circuitry using a first vector memory of a first processor core, wherein the first vector memory is configured to store a first vector value derived from the data provided from the first memory.

[0015] The method further includes routing data communication including at least a first vector value between the shared memory and the first vector memory using a direct memory access (DMA) data path of the shared memory in the circuit; routing data communication including a second vector value between the shared memory and the corresponding vector registers of the first processor core using a load-store data path of the shared memory; and generating an accumulated value corresponding to the output of the neural network layer by using the corresponding first and second vector values ​​routed in parallel to the corresponding load-store data path and DMA data path of the shared memory to the matrix computation unit.

[0016] Other embodiments of this and other aspects include corresponding systems, apparatuses, and computer programs configured to perform actions of the method, encoded on a computer storage device. A system of one or more computers can be configured by means of software, firmware, hardware, or combinations thereof installed on the system, which, in operation, causes the system to perform actions. One or more computer programs can be configured by means of instructions that, when executed by a data processing device, cause the device to perform actions.

[0017] The subject matter described in this specification can be implemented in certain embodiments to achieve one or more of the following advantages.

[0018] Shared memory provides a memory structure that can be efficiently shared between two or more processor cores in a hardware circuit, improving the acceleration and simultaneous execution of neural network computations at the hardware circuit compared to previous methods. Shared memory can be constructed with some resources residing substantially outside the processor core, such that some (or all) of the lines used for moving data for direct memory access and for load / store operations from shared memory to the processor core are shared. This improves bandwidth utilization and efficiency compared to other methods that require dedicated lines for various data transfers.

[0019] Shared memory provides a load-store data path that delivers higher peak performance because it eliminates the need to route data through vector memory macros to load / write vector arrays to the memory locations of on-chip vector registers. With the improved architecture and resources of shared memory, loads and stores for vector registers can be performed in parallel with vector memory loads and stores, thanks to the additional available register ports that can be integrated at the vector registers.

[0020] Circuit architectures and techniques can be used to integrate and efficiently construct the shared SRAM resources of neural network processors into a single circuit package. This integration and efficient construction of shared SRAM resources allows for computational inference to obtain additional data values ​​without incurring penalties for off-chip communication.

[0021] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other potential features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0022] Figure 1 It is a block diagram of a computing system having hardware circuitry including example shared memory.

[0023] Figure 2This is a block diagram illustrating an example processor core communicating with an example shared memory in relation to hardware circuitry.

[0024] Figure 3 This is a block diagram illustrating an example vector processor communicating with an example matrix computation unit of hardware circuitry.

[0025] Figure 4 An example line connection for coupling shared memory to one or more vector processors in hardware circuitry is shown.

[0026] Figure 5 It shows the method for use Figure 1 The flowchart shows an example process of using shared memory to accelerate neural network computation.

[0027] The same reference numerals and names in different figures indicate the same elements. Detailed Implementation

[0028] This specification describes hardware circuitry and data processing techniques implemented using an improved hardware circuitry architecture compared to previous designs. The hardware circuitry can be a dedicated processor, such as a neural network processor, an application-specific integrated circuit (ASIC), or a hardware accelerator. The hardware circuitry is configured to implement a neural network comprising multiple neural network layers. The improved architecture and data processing techniques described in this document allow the circuitry representing the hardware accelerator to achieve increased speed and bandwidth when accelerating the computation of example neural network models, including computations that process inputs through layers of CNNs, RNNs, or other neural network types to generate outputs corresponding to neural network inferences.

[0029] This circuit typically includes a first memory, corresponding first and second processor cores, and shared memory. The first memory provides data for performing computations to generate the output of a neural network layer. Each of the first and second processor cores can be a tensor processor core, configured to perform neural network computations using inputs and weights derived from the data provided from the first memory. For example, the first and second processor cores can perform neural network computations using input vectors and weight vectors derived from corresponding input tensors and weight / parameter tensors provided from the first memory, respectively.

[0030] Each of the first and second cores includes a corresponding vector memory configured to store vector values ​​passed to the vector memory. The vector values ​​are derived from data provided by the first memory and passed to the vector memory via shared memory. The shared memory is typically located between the first memory and at least one of the processor cores. The shared memory includes: i) a direct memory access (DMA) data path configured to route data between the shared memory and the corresponding vector memories of the first and second cores, and ii) a load-store data path configured to route data between the shared memory and the corresponding vector registers of the first and second cores. The combination of the DMA data path and the load-store data path allows for increased bandwidth relative to previous designs by providing parallel data paths that can be used simultaneously to provide vector values ​​to a computational array of circuitry performing neural network computations.

[0031] Figure 1 This is a block diagram of a computing system 100 including example hardware circuitry 101. As described above, hardware circuitry 101 can represent a hardware accelerator or some other dedicated processor. In some cases, system 100 is an example computing system for accelerating tensor or neural network computations associated with artificial deep neural networks (DNNs) such as RNNs or CNNs. For example, system 100 is configured to implement a CNN on an example hardware accelerator and pass data values ​​to the hardware accelerator to generate outputs for computational inference.

[0032] In some implementations, system 100 is a system-on-a-chip. For example, a system-on-a-chip can include hardware circuitry 101 and some (or all) other components and devices described herein as included in system 100.

[0033] Hardware circuitry 101 may be a hardware accelerator configured to accelerate the execution and / or performance of a neural network model. For example, the execution of a neural network model may be accelerated relative to execution on a general-purpose machine such as a central processing unit (CPU). Similarly, the performance and execution of a neural network model may be accelerated relative to an implementation of the model on another hardware accelerator, such as a graphics processing unit (GPU), that does not have the improved hardware features and techniques described in this specification.

[0034] System 100, including circuitry 101, includes system memory 102 and shared memory 104. System memory 102 can represent a high-bandwidth memory (“HBM 102”) or an input / output (I / O) device that exchanges data with processor cores 105-1, 105-2 of hardware circuitry 101. Data communication can typically include writing data values ​​to or reading data from vector memories 106, 108 located in or from vector memories 106, 108 of a particular processor core. For example, HBM 102 can exchange data with processor core 105-1 to pass input to the core and receive output generated by one or more computing resources of the core. In some embodiments, system memory 102 represents different types of memory systems and memory technologies, including HBM, DRAM, DIMMS, or non-DRAM technologies.

[0035] Data values ​​can represent vector elements or arrays of vector values. For example, a first vector array can represent a batch of inputs to be processed by a neural network layer, while a second vector array can represent a set of weights for that layer. Relatedly, a third vector array can represent a vector of accumulated values ​​corresponding to the output generated at processor core 105-1, and a fourth vector array can represent a vector of activation values ​​representing the output generated at processor core 105-2.

[0036] HBM 102 can be a dynamic random access memory (DRAM) asset of system 100. In some embodiments, HBM 102 is an external or off-chip memory relative to circuit 101 and is configured to exchange data with the on-chip vector memory bank of system 100 (described below). For example, HBM 102 can be located at a physical location outside the integrated circuit die representing circuit 101. Therefore, HBM 102 can be remote or non-local relative to computing resources located within the integrated circuit die. Alternatively, HBM 102 or a portion of its resources can be located within the integrated circuit die representing circuit 101, such that HBM 102 is local to or co-located with the computing resources of the circuit.

[0037] System 100 may include one or more processor cores 105-1, 105-2. In some embodiments, system 100 includes multiple processor cores 105-n, where n is an integer greater than or equal to one. Figure 1 and the following description Figure 2 and Figure 3In the examples, system 100 is shown as including two processor cores; however, system 100 including the hardware circuitry 101 described herein may have more or fewer processor cores. In some examples, the techniques described, including its memory resources and different data paths, associated with shared memory 104, can be shared relative to the components and resources of a single processor core. Typically, processor cores 105-n are discrete, self-contained processing / computing units of system 100 (or hardware circuitry 101).

[0038] Each processor core 105 is configured to independently perform the computations required for one or more layers of a multi-layer neural network (e.g., neural network computations). The computations may need to process data for a machine learning workload or perform a specific task of the workload. Computations performed at the processor core to process inputs through one or more neural network layers may include multiplying a first set of data values ​​(e.g., inputs or activations) with a second set of data values ​​(e.g., weights). For example, the computations could include multiplying input or activation values ​​with weight values ​​over one or more cycles and performing the accumulation of the products over multiple cycles. Each core 105 is typically capable of including matrix computation units and optionally vector processing units for performing these computations. These units are referenced below. Figure 3 A more detailed description was provided.

[0039] The distinct values ​​in the first and second sets of data values ​​are stored at specific memory locations within the memory configuration of the processor core of the hardware circuit 101. In some embodiments, individual values ​​in the first set of data values ​​may correspond to corresponding elements of an input tensor, while individual values ​​in the second set of data values ​​may correspond to corresponding elements of a weight (or parameter) tensor. For example, a neural network layer in a layer sequence is capable of processing a set of inputs, such as image pixel data or activation values ​​generated by another neural network layer in the layer sequence.

[0040] The input set or activation value set can be represented as a one-dimensional (1D) or multi-dimensional tensor (e.g., 2D or 3D) in which multiple elements are along their respective dimensions. Each memory location storing data values ​​can be mapped to a corresponding element of the one-dimensional or multi-dimensional tensor, and the tensor elements can be traversed or accessed using the control logic of a circuit. For example, the control logic can determine or compute the memory address value mapped to an element to load or store the corresponding data value of that element.

[0041] Hardware circuit 101 has a dedicated memory hierarchy including different memory configurations. Each of these memory configurations has different bandwidth and latency characteristics relative to the others, and their physical locations within hardware circuit 101 may also differ. Example memory configurations include shared memory 104, vector memories 106, 108, and vector registers 110, 112. Generally, memory configurations are operable to store data values ​​such as vector values ​​related to inputs, activations, or gain values ​​to be processed by neural network layers, as well as output activations generated by the neural network layers in response to processing inputs or activations through that layer. (See below for reference.) Figure 2 and Figure 3 The generation and storage of output activations, as well as the various memory constructions used to perform these operations, are described in more detail.

[0042] Figure 2 This is a block diagram 200 illustrating an example of how resources or portions of shared memory 104 are arranged at hardware circuitry 101 to facilitate data communication between various components of the hardware circuitry. As indicated above, shared memory 104 provides the basis for improved hardware architecture and data processing techniques of system 100. Shared memory 104 is capable of representing on-chip SRAM resources of an example neural network processor chip. In some embodiments, shared memory 104 exists (e.g., typically or substantially) between HBM 102 and the corresponding vector memories 106, 108 of the corresponding processor cores 105-1, 105-2.

[0043] Shared memory 104 can represent a shared central space on chip or circuit 101. For example, shared memory 104 is configured to act as a shared global memory space including memory resources corresponding to memory banks and registers, which are shared among one or more processor cores 105-1, 105-2 that may exist in system 100 and / or included in hardware circuit 101. As described in more detail below, shared memory 104 is configured to act as software-controlled note-like memory. In some embodiments, some (or all) of the resources of shared memory 104 are configured to act as software-controlled notes (architectural resources) rather than hardware-managed caches. In other embodiments, some memory resources of shared memory 104 are configured to act as hardware-managed caches or operations, while some memory resources of shared memory 104 (e.g., the remainder of the resources) are configured to act as software-controlled notes. For example, system 100 can allocate a portion of the resources that allow operation as hardware-managed caches, while using the remainder of shared memory 104 as software notes.

[0044] System 100 is configured to expose at least two programming interfaces to a user to utilize the data transfer capabilities provided by shared memory 104. A first interface exposes programmable DMA data transfer capabilities and operations, while a second, distinct interface exposes programmable load / store data transfer capabilities and operations. Each of these interface capabilities represents a logical attribute of shared memory 104, as described in more detail below.

[0045] As described above, the memory architecture of system 100 has varying bandwidth and latency characteristics. For example, shared memory 104 may have higher bandwidth and lower latency than DRAM access in HBM 102, but lower bandwidth and higher latency than access in vector memories 106 and 108. In some examples, shared memory 104 has lower data capacity than DRAM assets in HBM 102, but higher data capacity than the corresponding vector memory of the processor core. Generally, these different bandwidth and latency attributes represent standard memory tier tradeoffs.

[0046] The memory configuration of system 100, particularly shared memory 104, may differ in its physical placement within hardware circuitry 101. Shared memory 104 includes resources such as memory banks and registers that can be physically and logically arranged relative to the placement of certain computational resources of processor cores 105-1 and 105-2. In this context, the characteristics of shared memory 104 can generally be described with reference to its physical structure and its logical structure. The physical structure of shared memory 104 will be discussed initially, while its logical structure is described below.

[0047] Regarding its physical structure, the resources of shared memory 104 can be physically distributed across hardware circuitry 101. For example, different subsets, portions, or segments of the resources forming shared memory 104 can be physically distributed across various locations within circuitry 101 to enable different types of data transfer operations and processing techniques to be implemented in system 100. In some embodiments, a portion of the resources of shared memory 104 can reside within the processor core of circuitry 101, while another portion can reside outside the processor core of circuitry 101. Figure 2 In the example, a portion of shared memory 104 is located outside each of processor cores 105-1 and 105-2 to enable DMA operations that move large blocks of data between memory locations of HBM 102 and shared memory 104.

[0048] Referring again briefly to HBM 102, this type of system memory can be an external memory structure used by system 100 to provide and / or exchange high-bandwidth data with the vector memory of the corresponding processor core. In some embodiments, HBM 102 is configured for various direct memory access (DMA) operations to retrieve data from or provide data to memory address locations in the vector memory of the processor core of circuit 101. More specifically, DMA operations involving HBM 102 exchanging data with vector memories 106, 108 are enabled by the example control scheme and the memory resources of shared memory 104.

[0049] exist Figure 2 , Figure 3 In the example described below, shared memory 104 includes a shared memory control unit 201 (“control unit 201”). Control unit 201 is configured to generate control signals 114 for controlling memory access operations involving each of HBM 102, shared memory 104, vector memories 106, 108, and vector registers 110, 112.

[0050] Control unit 201 implements a control scheme distributed across different memories (e.g., HBM 102, shared memory 104, vector memories 106, 108, and vector registers 110, 112) of system 100. In some embodiments, this control implementation is distributed across different memories based on communication between control unit 201 and the corresponding control unit of each memory. For example, the control scheme can be distributed across memories based on control signals provided by control unit 201 and processed locally by the corresponding control units of these different memories.

[0051] The sharing of data paths enables the movement of data between the HBM 102 and the corresponding vector memories of processor cores 105-1 and 105-2. When this occurs, any (and all) necessary control units of system 100 for a given memory or data path are activated to manage the data switching that needs to occur at the appropriate contact point. Control unit 201 is configured to execute software instructions and generate control signals that cause a first portion of the memory resources of shared memory 104 to act as DMA memory cells.

[0052] The first portion of the resource can be represented by shared core data path 204 of reference processor core 105-1 and shared core data path 224 of reference processor core 105-2. This representative DMA memory cell is operable to move data between HBM 102 and each of the first processor core 105-1 and the second processor core 105-2 based on control signals generated by control unit 201.

[0053] For example, control signals can be generated to perform DMA operations to move data blocks (e.g., vectors) between memory locations of shared memory 104 and vector memory 106 using data path 202, shared core data path 204, or data path 206, and b) move data blocks (e.g., vectors) between memory locations of shared memory 104 and vector memory 108 using data path 222, shared core data path 224, or data path 226. In some embodiments, shared memory 104 may alternatively be referred to as shared CMEM 104.

[0054] As used in this document, CMEM typically corresponds to a block of Physically Contiguous Memory (CMEM) that provides useful configuration as a data buffer and on-chip SRAM storage. As described in more detail below, in system 100, CMEM resource blocks are physically distributed at hardware circuitry 101 and arranged for sharing among components of a processor core that can be configured as a hardware accelerator or other type of dedicated processor. Each of the shared core data paths 204 and 224 is an example node capable of indicating potential static contention on the shared data path for the movement of vector data through these points in the system.

[0055] like Figure 2 As shown in the example, hardware circuitry 101 and system 100 are configured to include multiple load-store data paths 202, 206, multiple CMEM load data paths 208, 214, 228, 234, and multiple CMEM store data paths 215, 235. Hardware circuitry 101 and system 100 also include multiple shared architecture blocks 210, 230 (described below). Figure 2 In the example, each of data paths 202 and 222 can be configured as a data path for routing data (e.g., vector or scalar values) in response to performing a DMA operation, a data path for routing data in response to performing a CMEM load / store operation, or both. The DMA operations supported by shared memory 104 and data paths 202, 206, 222, and 226 can be used to move data between different memory structures with reference to certain memory offsets and stride parameters.

[0056] For example, system 100 is configured to use shared memory 104 to perform a DMA operation involving moving 1 megabyte of data from one set of memory locations to another set of memory locations with an offset of 0x04. Shared memory 104 and system 100 are operable to support various strut functions when performing the DMA operation. For example, a DMA operation for moving 1 megabyte of data can include strut operations to insert address intervals every 200 kilobytes relative to the address base.

[0057] In some implementations, stride operations are used to insert address intervals based on a desired read sequence that will be executed later, after the data has been moved to its destination location, to read 1 megabyte of data. For example, a 1-megabyte block of data can be stored based on stride operations that correspond to how the data will be read or retrieved for processing at different layers of a neural network or across different sets of filters or weights in a particular neural network layer.

[0058] The control unit 201 of the shared memory 104 is also configured to cause various load-store operations to be performed. For example, the control unit 201 generates control signals to perform load-store operations that move various amounts of data (e.g., vectors or vector values) between memory locations of the shared memory 104 using data path 202, shared core data path 204, or data path 208 (for load operations at core 105-1) and memory locations of the shared architecture block 210, and b) move various amounts of data (e.g., vectors or vector values) between memory locations of the shared memory 104 using data path 222, shared core data path 224, or data path 228 (for load operations at core 105-2) and memory locations of the shared architecture block 230.

[0059] Similarly, control signals can be generated to perform load-store operations, which involve: a) moving various amounts of data (e.g., vectors or vector values) between memory locations of shared memory 104 and vector register 110 using data path 202, shared core data path 204, or data path 215 (for storage operations at core 105-1), and b) moving various amounts of data (e.g., vectors or vector values) between memory locations of shared memory 104 and vector register 112 using data path 222, shared core data path 224, or data path 235 (for storage operations at core 105-2).

[0060] Referring now to the logical structure of shared memory 104, as described above, system 100 is configured to expose at least two programming interfaces to the user to utilize the data transfer capabilities provided by shared memory 104. At least one interface exposes programmable DMA functionality and another interface exposes programmable CMEM load / store functionality, each capable of representing a logical attribute of shared memory 104. For load / store purposes, shared memory 104 is logically exposed as a parallel memory to vector memories 106, 108. In this way, each load-store data path is operable to provide additional (or parallel) data paths for moving blocks of data or specific data blocks through the memory system, such as through vector registers of the respective processor cores 105-1, 105-2, or multiple cores of circuit 101. For example, load-store operations can be performed on the memory resources of shared memory 104 simultaneously with DMA operations.

[0061] More specifically, a DMA operation can be performed to move value vectors between memory locations in shared memory 104 and vector memory 106 using DMA data path 206, while a load-store operation can be performed concurrently with the DMA operation to move different value vectors between memory locations in shared memory 104 and shared architecture block 210. Similar concurrent operations can occur in processor core 105-2 (or other cores) using resources corresponding to those of processor core 105-1.

[0062] Compared to DMA operations, load / store operations performed using the CMEM resources of shared memory 104 can represent the high-performance capabilities of shared memory 104, or a high-performance method using shared memory 104. In some embodiments, control unit 201 is configured to execute software instructions and generate control signals that cause a second portion of the memory resources of shared memory 104 to act as an architectural resource for software control of the load / store operations.

[0063] The second portion of the resources can be represented by shared architecture block 210 of reference processor core 105-1 and shared architecture block 230 of reference processor core 105-2. Therefore, each of shared architecture blocks 210, 230 can represent a software-controlled architecture resource (or notepad) formed from a subset of the memory resources of shared memory 104. In some examples, the software-controlled architecture resource of system 100 is configured to manage the flow of vector data values ​​from HBM 102 to the corresponding vector registers 110 or 112 of the first processor core 105-1 or the second processor core 105-2.

[0064] Shared memory 104 and its resources have the characteristic of being uniquely configurable as DMA memory, for example, for moving data between memory constructs such as HBM 102 or vector memories 106, 108, and also have load / store memory for moving data directly to corresponding vector registers 110, 112 on each processor core 105-1, 105-2. These configurable aspects of shared memory 104 allow its resources and addressing to be scheduled in a fine-grained manner by software running on the core. For example, shared memory 104 can be a software-managed (not hardware-managed) SRAM resource, where the processor core's compiler specifically manages its memory addressing, including data types that may or may not exist at memory address locations in shared memory 104.

[0065] In some implementations, the software-controlled architectural resources of shared memory 104 are configured as a first-in-first-out (FIFO) memory structure (e.g., shared architecture block 210 or 230) along the load portion of the load-store data path of the processor cores including CMEM storage data path 215 or 235, for routing data stored at shared CMEM 203 or HBM 102. The FIFO memory structure is configured to temporarily store a set of data values ​​for a duration of a threshold number of processor cycles before routing the set of values ​​to the corresponding vector registers 110, 112 of the first processor core 105-1 or the second processor core 105-2. The FIFO memory structure is used to mitigate register pressure and scheduling complexity that may result from CMEM load operations with specific load latency.

[0066] In some implementations, the threshold number of clock cycles is determined based on an example high-latency (e.g., 50 cycles) CMEM load operation that could result in register pressure and scheduling complexity associated with reserving a given register for all 50 cycles. To offset or mitigate concerns about register pressure, the CMEM result FIFO (“CRF”) is physically instantiated at hardware circuitry 100 using resources of shared memory 104. Figure 2 In the example, the first CRF is represented by architecture block 210 of processor core 105-1, while the second CRF is represented by architecture block 230. Each CRF allows the example CMEM load operation to be divided into at least two stages: i) the CMEM to CRF stage, in which CMEM address information is provided, and ii) the CRF to register stage, in which the vector register target is provided.

[0067] For example, each of shared architecture blocks 210 and 230 is configured to receive data values ​​(e.g., scalar or vector values) and temporarily store the data values ​​for a duration of a threshold number of processor cycles. In processor core 105-1, data values ​​are routed to shared architecture block 210 along load data path 208 (and shared core data path 204) that connects architecture block 210 to other memory locations in shared memory 104. In processor core 105-2, data values ​​are routed to shared architecture block 230 along load data path 228 (and shared core data path 224) that connects architecture block 230 to other memory locations in shared memory 104.

[0068] Shared architecture block 210 is configured to provide data values ​​to vector register 110 of processor core 105-1 in response to a processor cycle that temporarily stores data values ​​for a duration of a threshold number of times. Similarly, shared architecture block 230 is configured to provide data values ​​to vector register 112 of processor core 105-2 in response to a processor cycle that temporarily stores data values ​​for a duration of a threshold number of times.

[0069] System 100 is configured to issue multiple CMEM load instructions in the same cycle. For example, system 100 can issue a CMEM load instruction executed using data path 208 (or 214) and shared architecture block 210, and in the same cycle, issue a load instruction executed using data path 212 for vector memory 106. In some examples, from a software control perspective, a Cmem load operation traversing data path 214 between resource 210 and vector register 110 and a Vmem load operation traversing data path 212 between vector memory 106 and vector register 110 can each be issued and executed in the same cycle. In some implementations, vector registers 110, 112 are adapted, relative to a previous design, to include additional ports allowing vector registers 110, 112 to receive simultaneous load operations.

[0070] For example, vector register 112 is configured to include an additional port that allows the register to receive corresponding vector payloads from vector memory 108 and shared architecture block 230 during a simultaneous load operation performed at processor core 105-2. In some examples, a single data block of payload loaded into each of vector registers 110, 112 is based on up to 128 data items that can be moved to vector register 110 or vector register 112 during a single load operation, involving 128 discrete loads.

[0071] The CMEM load / store functionality of shared memory 104 can provide higher peak performance compared to previous designs because it does not require routing data through vector memory macros. For example, load and store (along data paths 215, 235) can be performed in parallel with vector memory load and store, partly due to the additional available register ports at vector registers 110, 112.

[0072] In some implementations, system 100 includes an example load-store interface that provides a parallel interface to each of the sharded architecture blocks 210, 230, bypassing some (or all) bandwidth limitations that may exist when traversing data paths through vector memories 106, 108. This example load-store interface can effectively provide higher memory bandwidth that allows additional performance to be extracted from the example workload. For example, system 100 is configured to use the resources of shared memory 104 (e.g., software-controlled architecture resources) to perform various load / store operations, wherein load / store operations can be performed to bypass moving data through vector memory at the processor core.

[0073] For example, components of hardware circuitry 101 can communicate with shared memory 104 to read data from a single address location in a memory bank or register file of shared memory 104. In some examples, data stored at a single address in memory is read, and this single data block can be moved to a register file or architecture block located within the processor core. For example, a single data block can be read from an address location in shared CMEM 104, moved via shared core data path 224, and moved to an address location in shared architecture block 230 in processor core 105-2 for further processing. This operation can be performed to bypass moving data through the memory system via vector memory 108, thereby saving processor clock cycles at core 105-2 and bandwidth at the data path connected to vector memory 108.

[0074] Figure 3 This is a block diagram 300 illustrating an example vector processor communicating with an example matrix computation unit of hardware circuitry 101. More specifically, in some embodiments, an example tensor processor core 302-1 of hardware circuitry 101 includes a vector processing unit 304 (“vector processor 304”) and a matrix computation unit 308 coupled to the vector processor 304. Similarly, another tensor processor core 302-2 of hardware circuitry 101 includes a vector processor 306 and a matrix computation unit 308 coupled to the vector processor 306.

[0075] Typically, hardware circuitry 101 is configured to perform computations to generate outputs for neural network layers. Each of the matrix computation units 308 and 310 included in circuitry 101 is configured to perform a subset of computations to generate accumulated values ​​for generating the outputs of the neural network layers. In some embodiments, the software-controlled architectural resources described above (e.g., architectural blocks 210, 230) are configured to manage the data flow corresponding to vector operands from HBM 102 to each of the matrix computation units 308, 310. In some cases, the operands are inputs and weights provided by HBM 102. The operands may be constructed as vector arrays based on data operations performed using the arithmetic logic unit (ALU) of vector processors 304 or 306.

[0076] exist Figure 3 In the example, control unit 201 generates control signals to manage operations for retrieving (or reading) a set of multiple input batches and weights from memory locations of shared memory 104, vector memories 106, 108, and vector registers 110, 112. The retrieved inputs and weights can be processed by neural network layers to compute accumulated values ​​based on calculations performed at matrix computation units 308, 310. The accumulated values ​​can be processed at vector processors 304, 306 to generate activation values ​​corresponding to the outputs of the neural network layers. The control signals generated by control unit 201 are used to store (or write) the multiple outputs or sets of output activations generated by vector processors 304, 306 to other memory locations of HBM 102 or hardware circuitry 101 for processing in one or more other neural network layers.

[0077] Figure 4An example wired connection 400 is shown that couples shared memory 104 to one or more vector processors 304, 306 of corresponding processor cores 302-1, 302-2 at hardware circuitry 101. The wired connection 400 can represent a physical structure with certain physical constraints corresponding to the total number of wired or data bus connections that may be available at a given hardware circuitry. For example, a first CMEM resource block 402 (bank_0) of shared memory 104 can be coupled to a corresponding VPU resource block 406 via line connection 404. VPU resource 406 is a memory resource of an example vector processing unit, such as a vector register or vector memory, of a vector processor 304 in processor core 302-1. Similarly, a second CMEM resource block 412 (bank_1) of shared memory 104 can be coupled to a corresponding VPU resource block 416 via line connection 414. VPU resource 416 is a memory resource of an example vector processing unit, such as a vector register or vector memory, of a vector processor 306 in processor core 302-2. In some cases, when a given CMEM resource block is connected to a VPU block on two separate cores, there is a constraint that requires separating the CMEM resource and the VPU block. In particular, when the CMEM resource is configured to connect to a separate core, this constraint requires the CMEM and VPU blocks to be separated across the physical space of the hardware circuitry.

[0078] In some implementations, shared memory 104 has a given size, and the resource blocks of shared memory 104 and processor resources must reside within certain physical areas of chip or circuitry 101. Example operations involving moving data within or between processor cores require that data be able to traverse a certain physical distance at hardware circuitry 101 within a threshold number of processor cycles. Because some portions of the resources of shared memory 104 reside outside cores 105-1, 105-2, and implementing the wiring can be expensive, the wiring used for moving data for DMA operations (e.g., wiring connections 404, 414) and load / store operations from the resources of shared memory 104 to processor core 105 are shared. These wirings can also be used for DMA operations between HBM 102 and vector memories 106, 108. This sharing of wired connections improves bandwidth utilization and efficiency.

[0079] In some implementations, to save wiring in a manner compliant with a specific workload, the data path routing data from shared CMEM 203 to the processor core's resources can be twice the width of the data path routing data from the processor core's resources to shared CMEM 203. Therefore, the techniques described in this document provide example hardware layout options that balance the complexity of implementing the logical advantages of shared memory 104 with certain physical constraints and limitations regarding the number of wired connections and options for the physical placement of hardware components at circuit 101.

[0080] Figure 5 It shows the method for use Figure 1 A flowchart of an example process 500 for accelerating neural network computation using shared memory is provided. Process 500 can be implemented or executed using the system 100 described above. A description of process 500 can be made with reference to the computational resources of system 100 described above. The steps or actions of process 500 can be implemented by programmed firmware or software instructions, which can be executed by one or more processors of the devices and resources described in this document. In some embodiments, the steps of process 500 correspond to methods for performing computations to generate the output of neural network layers using hardware circuitry configured to implement a neural network.

[0081] Referring now to process 500, data is provided from the memory of system 100 to the system's hardware circuitry (502). For example, system 100 provides data from HBM 102 to hardware circuitry 101. In some embodiments, the data includes specific data values ​​and instructions for generating the output of a neural network layer. HBM 102 is capable of representing a system memory or input / output (I / O) device that exchanges data with the tensor processor core of hardware circuitry 101. For example, HBM 102 can exchange data with the tensor processor core to pass input to the tensor core and receive output generated by the tensor core.

[0082] System 100 uses the vector memory of the first processor core to store value vectors at the first processor core of the circuit (504). For example, system 100 can use the memory address location of vector memory 106 located in tensor processor core 105-1 to store input vectors corresponding to batches of inputs to be processed in the neural network layer. Vector memory 106 is configured to store a first vector value in response to processor core 105-1 identifying or deriving a first vector value from input data provided by HBM 102. In some embodiments, vector memory 106 stores the first vector value in response to first processor core 105 determining that the data provided by the first memory includes a subset of vector values ​​assigned to be stored in vector memory 106.

[0083] The first vector value can be used or derived directly from the set of elements of various elements representing the data values ​​of the multidimensional input tensor. As described above, the multidimensional input tensor can be provided as input data from HBM 102. In some embodiments, the first vector value represents a set of weights for a neural network layer, which is obtained, extracted, or otherwise derived from data provided by HBM 102. The first vector value can be stored in a memory location of vector memory 106 in response to a value being routed to vector processing unit 304, which includes vector memory 106. Each of vector processing unit 304 and vector memory 106 can be included as part of processor core 105-1.

[0084] Data provided by system memory is routed between shared memory and vector memory in the circuit (506). For example, data communication including a first vector value is routed between shared memory 104 and vector memory 106 of processor core 105-1 using at least DMA data path 206 of shared memory 104. More specifically, data communication including the first vector value is initially provided by HBM 102 and then routed between shared memory 104 and vector memory 106 using a first portion of the resources of shared memory 104. In some examples, the first vector value is routed between vector memory 106 and a specific portion (or sub-portal) of shared memory 104.

[0085] A first portion of the resources of shared memory 104 is configured to act as a DMA memory unit, operable to perform DMA-type operations to load data into memory locations of vector memory 106. DMA-type operations can be performed on individual memory address locations of the first portion of the resources to obtain a vector value array. The vector value array is routed to vector memory 106 using DMA data path 206 of shared memory 104.

[0086] Data provided by system memory is routed between shared memory in the circuit and the corresponding vector register of the circuit (508). For example, data communication including the second vector value is routed between shared memory 104 and vector register 110 of processor core 105-1 using at least load-store data paths 208, 214 of shared memory 104. More specifically, data communication including the second vector value is initially provided by HBM 102 and then routed between shared memory 104 and vector register 110 using a second portion of the resources of shared memory 104. In some examples, the second vector value is routed between vector register 112 and a specific portion (or sub-portal) of shared memory 104.

[0087] In some implementations, a second portion of the resources of shared memory 104 is configured to act as a software-controlled architectural resource, which is formed from a subset of the memory resources of shared memory 104. This software-controlled architectural resource is configured to: i) receive data values ​​(e.g., a second vector value) routed along load-store data path 208; ii) temporarily store the second vector value for a duration threshold number of processor cycles; and iii) provide the second vector value to a corresponding vector register 110 of at least processor core 105-1 in response to the processor cycles temporarily storing the vector value for a duration threshold number. The threshold number can be a software-defined threshold. For example, the threshold number can be specifically defined by software when scheduling instructions in an example instruction stream of the processor.

[0088] Shared memory 104 is operable to load an array of vector values ​​corresponding to the second vector values ​​into a memory location of the architectural resource included in the second part of the resource. Read operations can be performed for individual memory address locations within the second part of the resource to obtain the array of vector values ​​corresponding to the second vector values. These vector arrays are routed to vector register 110 using, for example, load-store data path 214 of shared memory 104. Alternatively, these vector arrays can be routed to vector register 112 using load-store data path 234 of shared memory 104.

[0089] System 100 uses a set of vector values ​​to generate accumulated values ​​(510) corresponding to the outputs of neural network layers. For example, hardware circuitry 101 includes a matrix computation unit 308 coupled to a vector processing unit 304 of processor core 105-1. System 100 causes matrix computation unit 308 to generate multiple accumulated values ​​using corresponding first and second vector values ​​routed to matrix computation unit 308. In some embodiments, the first and second vector values ​​are simultaneously routed to matrix computation unit 308 along different data paths of the circuitry. For example, the first set of vector values ​​and the second set of vector values ​​can be routed to matrix computation unit 308 in parallel using load-store data path 214 and DMA data path 206, respectively.

[0090] The embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, or in computer hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more computer program instruction modules encoded on a tangible, non-transitory program carrier, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus.

[0091] Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device. Computer storage media can be machine-readable storage devices, machine-readable storage substrates, random or serial access memory devices, or combinations thereof.

[0092] The term "computing system" encompasses all types of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device can include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device can also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0093] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment.

[0094] Computer programs can, but do not necessarily, correspond to files in a file system. A program can be stored as a portion of a file containing other programs or data; for example, as a markup language document, as a single file dedicated to the program in question, or as one or more scripts in multiple coordinating files, such as files storing one or more modules, subroutines, or portions of code. A computer program can be deployed to execute on one or more computers located at a single site or distributed across multiple sites and interconnected via a communication network.

[0095] The processes and logic flows described in this specification can be implemented by one or more programmable computers executing one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuits, and the devices can also be implemented as dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), or GPGPUs (General-Purpose Graphics Processing Units).

[0096] Computers suitable for executing computer programs include, for example, those based on general-purpose or special-purpose microprocessors or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. Some components of a computer are the central processing unit for executing or carrying out instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, receiving data from or transferring data to one or more mass storage devices (e.g., magneto-optical, magneto-optical, or optical disc) for storing data. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, global positioning system (GPS) receiver, or portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0097] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example: semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be supplemented by or contained within dedicated logic circuitry.

[0098] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user—such as an LCD (liquid crystal display) monitor—and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0099] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes, for example, a back-end component as a data server, or a middleware component including, for example, an application server, or a front-end component including, for example, a client computer with a graphical user interface or a web browser that a user can interact with embodiments of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0100] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other.

[0101] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features described in this specification within the context of individual embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases, one or more features from a claimed combination can be removed from the combination, and a claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0102] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or to perform all of the shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0103] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A circuit configured to implement a neural network including multiple neural network layers, the circuit comprising: A first memory, configured to provide data for performing computations to generate the outputs of the layers of the neural network; A first processor core, the first processor core including a first vector register, the first vector register being located within the first processor core and configured to at least load data from or store data in a first vector memory; A second processor core, the second processor core including a second vector register, the second vector register being located within the second processor core and configured to at least load data from or store data in the second vector memory; as well as A shared memory, disposed between at least one of the first processor cores or the second processor core and the first memory, wherein the shared memory and the first memory are communicatively coupled via Direct Memory Access (DMA), wherein the shared memory includes software-controlled architectural resources formed by a subset of the memory resources of the shared memory, wherein the software-controlled architectural resources are configured to load data from the shared memory in a first phase and provide the loaded data to the first vector register or the second vector register in a second phase; and A matrix computation unit within the first processor core or the second processor core is configured to perform a computational subset of the outputs of the layers of the neural network to generate an accumulated value, wherein software-controlled architectural resources are used to manage the flow of data values ​​corresponding to a vector array between the first memory and the matrix computation unit, wherein the vector array is derived from the data values ​​provided by the first memory.

2. The circuit according to claim 1, wherein, The first vector memory is located within the first processor core and is configured to store a first vector value derived from the data provided by the first memory, and the second vector memory is located within the second processor core and is configured to store a second vector value derived from the data provided by the first memory.

3. The circuit according to claim 1, wherein, The shared memory further includes: A first direct memory access (DMA) data path is configured to route data communication between the shared memory and the first vector memory included in the first processor core. The second direct memory access DMA data path is configured to route data communication between the shared memory and the second vector memory included in the second processor core. A first load-store data path, configured to route data communication between the shared memory and the first vector register included in the first processor core; and A second load-store data path is configured to route data communication between the shared memory and the second vector register included in the second processor core.

4. The circuit according to claim 1, wherein: The circuit includes multiple processor cores, among which the first processor core and the second processor core are; and The shared memory includes multiple memory resources physically distributed around the circuit to exchange data with each of the multiple processor cores at the circuit.

5. The circuit according to claim 4, wherein, The shared memory includes a shared memory control unit, which is configured to: Execute software instructions that cause a first portion of the plurality of memory resources to act as a DMA memory cell, the DMA memory cell being operable to move data between the first memory and each of the first processor core and the second processor core.

6. The circuit according to claim 3, wherein, The shared memory includes a second portion of memory resources, which is configured as follows: Receive data values ​​routed along the first load-store data path or the second load-store data path; and The number of processor cycles required to temporarily store the data value for a given threshold duration.

7. The circuit according to claim 6, wherein, The second part of the resource is configured as follows: In response to temporarily storing the data value for a number of processor cycles for the threshold duration, the data value is provided to the first vector register of the first processor core or the second vector register of the second processor core.

8. The circuit according to claim 1, wherein, The software-controlled architecture resources are used to manage the flow of data values ​​from the first memory to the corresponding vector registers of the first processor core or the second processor core.

9. The circuit according to claim 1, wherein: The circuit includes a vector processing unit that communicates with the first memory; The vector processing unit is configured to generate an activation value vector from the accumulated value generated at the circuit; and The activation value vector corresponds to the output of the layer of the neural network.

10. The circuit according to claim 3, wherein: The software-controlled architectural resources are a first-in-first-out (FIFO) memory structure along the loading portion of the first load-store data path or the second load-store data path; and The FIFO memory structure is configured to temporarily store the vector of the value for a duration of a threshold number of processor cycles before routing the vector of the value to the corresponding vector register of the first processor core or the second processor core.

11. The circuit according to claim 1, wherein, The shared memory is configured to act as a shared global memory space, which includes memory resources corresponding to a group of memory shared among one or more processor cores in the processor core.

12. The circuit according to claim 1, wherein, The data used to perform calculations to generate the output of the first layer of the neural network includes: The input to be processed by the first layer of the neural network; The corresponding set of weights of the first layer of the neural network; and Instructions for processing one or more of the inputs through the first layer using the corresponding set of weights of the first layer to generate the output of the first layer.

13. The circuit according to claim 3, wherein, The first direct memory access (DMA) data path and the second direct memory access (DMA) data path, as well as the first load-store data path and the second load-store data path, are assigned to corresponding blocks of the shared memory in a plurality of independent blocks of the shared memory.

14. A method for using circuitry to perform computations to generate the output of a layer in a neural network, the neural network comprising a plurality of neural network layers, the circuitry being configured to implement the neural network, the method comprising: Data for generating the output of the neural network layers is provided from the first memory; Using a first vector memory of a first processor core, a vector of values ​​is stored in the first processor core of the circuit, wherein the first processor core includes a first vector register, the first vector register being located within the first processor core and configured to at least load data from or store data in the first vector memory; Using a second vector memory of a second processor core, a vector of values ​​is stored in the second processor core of the circuit, wherein the second processor core includes a second vector register, which is located within the second processor core and configured to at least load data from or store data in the second vector memory; Data communication including a fourth vector value is routed between shared memory and a second vector register included in the second processor core, wherein the shared memory further includes software-controlled architectural resources formed by a subset of the memory resources of the shared memory, wherein the software-controlled architectural resources are configured to load data from the shared memory in a first phase and provide the loaded data to the first vector register or the second vector register in a second phase; and In response to executing a subset of computations for generating the output of the neural network layer, a matrix computation unit within the first or second processor core generates an accumulated value corresponding to the output of the neural network layer using corresponding first and third vector values ​​routed in parallel to the matrix computation unit along a first load-store data path and a first DMA data path of the shared memory, respectively; wherein the software-controlled architecture resources include data flow between the first memory and the matrix computation unit, and the data flow includes a vector array derived from the data provided by the first memory.

15. The method according to claim 14, wherein, The first vector memory is located within the first processor core and is configured to store a first vector value derived from the data provided by the first memory, and the second vector memory is located within the second processor core and is configured to store a second vector value derived from the data provided by the first memory.

16. The method of claim 15, further comprising: Using a first direct memory access DMA data path of shared memory in the circuit, data communication including at least the first vector value is routed between the shared memory and the first vector memory included in the first processor core, wherein the shared memory and the first memory are coupled communicatively via direct memory access DMA. Using the second direct memory access DMA data path of the shared memory in the circuit, data communication including at least the second vector value is routed between the shared memory and the second vector memory included in the second processor core; Using the first load-store data path of the shared memory, data communication including a third vector value is routed between the shared memory and the first vector register included in the first processor core; and The data communication is routed between the shared memory and the second vector register using the second load-store data path of the shared memory.

17. The method of claim 16, comprising: Using the second vector memory of the second processor core, a vector of values ​​is stored in the second processor core of the circuit, wherein the second vector memory is configured to store a third vector value derived from the data provided by the first memory.

18. The method of claim 17, comprising: Using the DMA data path of the shared memory, data communication including the third vector value is routed between the shared memory and the second vector memory; as well as Using the second load-store data path of the shared memory, data communication including a fourth vector value is routed between the shared memory and the corresponding vector registers of the second processor core.

19. The method of claim 14, wherein, The circuit includes multiple processor cores, and the shared memory includes multiple memory resources physically distributed around the circuit, and the method includes: The shared memory resources are used to exchange data communication with the first memory in each of the plurality of processor cores.

20. The method according to claim 19, wherein, The shared memory includes a shared memory control unit, and the method includes: Based on instructions executed by the shared memory control unit, a first resource portion of the plurality of memory resources is made to act as a DMA memory unit; and The representative DMA function of the first resource portion is used to move data between the first processor core and the first memory in each of the first and second processor cores.

21. The method of claim 18, comprising: The third vector value and the fourth vector value, which are respectively routed along the first load-store data path and the second load-store data path, are received by the second resource portion of the multiple memory resources included in the shared memory; The second resource portion is used to temporarily store the number of processor cycles for the duration threshold of the third vector value; as well as The second resource portion is used to temporarily store the number of processor cycles for the duration threshold of the fourth vector value.

22. The method of claim 21, comprising: In response to the temporary storage of the third vector value for the number of processor cycles of the threshold, the third vector value is provided to the first vector register of the first processor core using the second resource portion; as well as In response to the temporary storage of the fourth vector value for the number of processor cycles specified in the threshold, the fourth vector value is provided to the second vector register of the second processor core using the second resource portion.

23. The method of claim 14, further comprising: The software-controlled architecture resources are used to manage the data flow from the first memory to the corresponding vector registers of the first processor core and the data flow from the first memory to the corresponding vector registers of the second processor core.

24. The method according to claim 14, wherein, The circuit includes a matrix calculation unit, and the method includes: In response to performing a subset of the computations for generating the output of the neural network layer, an accumulated value is generated using the matrix computation unit.

25. The method of claim 24, wherein: The circuit includes a vector processing unit located between the first memory and the matrix calculation unit; The method includes: generating an activation value vector from the accumulated value generated by the matrix calculation unit using the vector processing unit; and The activation value vector corresponds to the output of the neural network layer.

26. A non-transitory machine-readable storage device for implementing a neural network having multiple neural network layers on circuitry used to perform neural network computations and for storing instructions executable by a processing device to cause the execution of operations, the operations including: Data for generating the output of the neural network layers is provided from the first memory; Using a first vector memory of a first processor core, a vector of values ​​is stored in the first processor core of the circuit, wherein the first processor core includes a first vector register, the first vector register being located within the first processor core and configured to at least load data from or store data in the first vector memory; Using a second vector memory of a second processor core, a vector of values ​​is stored in the second processor core of the circuit, wherein the second processor core includes a second vector register, which is located within the second processor core and configured to at least load data from or store data in the second vector memory; Data communication including a fourth vector value is routed between shared memory and a second vector register included in the second processor core, wherein the shared memory further includes software-controlled architectural resources formed by a subset of the memory resources of the shared memory, wherein the software-controlled architectural resources are configured to load data from the shared memory in a first phase and provide the loaded data to the first vector register or the second vector register in a second phase; and In response to executing a subset of computations for generating the output of the neural network layer, a matrix computation unit within the first or second processor core generates an accumulated value corresponding to the output of the neural network layer using corresponding first and third vector values ​​routed in parallel to the matrix computation unit along a first load-store data path and a first DMA data path of the shared memory, respectively; wherein the software-controlled architecture resources include data flow between the first memory and the matrix computation unit, and the data flow includes a vector array derived from the data provided by the first memory.

27. The non-transitory machine-readable storage device according to claim 26, wherein, The first vector memory is located within the first processor core and is configured to store a first vector value derived from the data provided by the first memory, and the second vector memory is located within the second processor core and is configured to store a second vector value derived from the data provided by the first memory.

28. The non-transitory machine-readable storage device of claim 27, wherein the operation further comprises: Using a first direct memory access DMA data path of shared memory in the circuit, data communication including at least the first vector value is routed between the shared memory and the first vector memory included in the first processor core, wherein the shared memory and the first memory are coupled communicatively via direct memory access DMA. Using the second direct memory access DMA data path of the shared memory in the circuit, data communication including at least the second vector value is routed between the shared memory and the second vector memory included in the second processor core; Using the first load-store data path of the shared memory, data communication including a third vector value is routed between the shared memory and the first vector register included in the first processor core; and The data communication is routed between the shared memory and the second vector register using the second load-store data path of the shared memory.

29. The non-transitory machine-readable storage device according to claim 28, wherein, The operation includes: Using the second vector memory of the second processor core, a vector of values ​​is stored in the second processor core of the circuit, wherein the second vector memory is configured to store a third vector value derived from the data provided by the first memory; Using the DMA data path of the shared memory, data communication including the third vector value is routed between the shared memory and the second vector memory; and Using the second load-store data path of the shared memory, data communication including a fourth vector value is routed between the shared memory and the corresponding vector register of the second processor core.

30. The non-transitory machine-readable storage device according to claim 26, wherein, The circuit includes multiple processor cores, and the shared memory includes multiple memory resources physically distributed around the circuit, and the operation includes: The shared memory resources are used to exchange data communication with the first memory in each of the plurality of processor cores.

31. The non-transitory machine-readable storage device according to claim 30, wherein, The shared memory includes a shared memory control unit, and the operation includes: Based on instructions executed by the shared memory control unit, a first resource portion of the plurality of memory resources is made to act as a DMA memory unit; and The representative DMA function of the first resource portion is used to move data between the first processor core and the first memory in each of the first and second processor cores.

32. The non-transitory machine-readable storage device of claim 29, wherein the operation further comprises: The third vector value and the fourth vector value, which are respectively routed along the first load-store data path and the second load-store data path, are received by the second resource portion of the multiple memory resources included in the shared memory; The second resource portion is used to temporarily store the number of processor cycles for the duration threshold of the third vector value; as well as The second resource portion is used to temporarily store the number of processor cycles for the duration threshold of the fourth vector value.

33. The non-transitory machine-readable storage device of claim 32, wherein the operation further comprises: In response to the temporary storage of the third vector value for the number of processor cycles of the threshold, the third vector value is provided to the first vector register of the first processor core using the second resource portion; as well as In response to the temporary storage of the fourth vector value for the number of processor cycles specified in the threshold, the fourth vector value is provided to the second vector register of the second processor core using the second resource portion.

34. The non-transitory machine-readable storage device according to claim 26, wherein: The circuit includes a vector processing unit located between the first memory and the matrix calculation unit; The operation further includes: generating an activation value vector from the accumulated value generated by the matrix calculation unit using the vector processing unit; and... The activation value vector corresponds to the output of the neural network layer.

Images