Matrix Processing Unit

Abstract

To provide an IC chip, a program, and a machine-readable medium that provide a device for executing programmable matrix processing.SOLUTION: A method includes: executing a matrix operation on one or more matrix operands to receive matrix data from a multidimensional memory, the matrix data associated with the one or more matrix operands; extracting one or more matrix operands from the matrix data; identifying a matrix routine associated with the matrix operation; executing the matrix routine on a matrix processor using the one or more matrix operands; and acquiring a result of the matrix operation based on the matrix routine executed by the matrix processor.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates generally to the field of computer processing, and more particularly, but not exclusively, to programmable matrix processing. [Background technology]

[0002] Matrix operations, such as matrix multiplication and matrix convolution, often involve complex operations on large, multidimensional matrix operands and can be very processor-intensive and memory-intensive. Therefore, the performance of complex matrix operations can be limited by processing and / or memory latency. As matrix operations are increasingly utilized with ever-growing data sets in a variety of applications (from graphics and image processing to machine learning and artificial intelligence), there is an increasing demand for high-performance and flexible processing of matrix operations. [Brief explanation of the drawings]

[0003] The present disclosure is best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be noted that, according to industry practice, various features are not necessarily drawn to scale and are used for illustrative purposes only. Where a scale is explicitly or implicitly indicated, that scale provides an illustrative example only. In other embodiments, the dimensions of various features may be arbitrarily increased or decreased for clarity of presentation.

[0004] [Figure 1] FIG. 1 illustrates a schematic diagram of an exemplary computing system in accordance with certain embodiments.

[0005] [Figure 2A] FIG. 1 shows a block diagram of an exemplary embodiment of a matrix processing architecture. [Figure 2B] FIG. 1 shows a block diagram of an exemplary embodiment of a matrix processing architecture. [Figure 2C] FIG. 1 shows a block diagram of an exemplary embodiment of a matrix processing architecture.

[0006] [Figure 3] 1 illustrates a block diagram of an exemplary embodiment of a computer processor. [Figure 4] 1 illustrates a block diagram of an exemplary embodiment of a computer processor.

[0007] [Figure 5] 1 illustrates an exemplary embodiment of a programmable matrix processing engine.

[0008] [Figure 6] 1 shows a flowchart of an exemplary embodiment of programmable matrix processing. DETAILED DESCRIPTION OF THE INVENTION

[0009] In the following disclosure, numerous different embodiments or examples are provided for implementing different features of the present disclosure. Specific examples of components and arrangements are described below to facilitate understanding of the present disclosure. These are, of course, merely examples and are not intended to be limiting. Furthermore, the present disclosure may repeat reference numerals and / or characters in various examples. This repetition is for the purposes of brevity and clarity, and the repetition itself does not dictate a relationship between the various embodiments and / or configurations discussed. Different embodiments may have different advantages, and a particular advantage is not necessarily required for every embodiment.

[0010] Matrix processing operations (e.g., linear algebraic operations involving matrix and / or vector operands) have a wide range of applications in computing systems, from graphics processing to machine learning and artificial intelligence, among other examples. For example, complex matrix operations may be used to implement artificial neural networks that provide artificial intelligence and machine learning capabilities, including computer vision, autonomous navigation, speech and voice recognition, and natural language processing, among other examples. These complex matrix operations (e.g., matrix multiplication and matrix convolution) can be used to implement fundamental operations of neural networks, such as forward propagation, back propagation, and weight updates. However, these matrix operations can be very processor- and memory-intensive because they often involve complex operations on large, multidimensional matrix operands. Thus, the performance of these matrix operations can be limited by processing and / or memory latency. Furthermore, these rigid matrix operations are often implemented without any flexibility to implement new types or variations of matrix operations and / or to modify the behavior of existing operations. As matrix operations are increasingly utilized in various applications, such as artificial intelligence and machine learning, with ever-growing datasets, there is an increasing demand for both high-performance processing and flexible implementation of matrix operations.

[0011] Existing matrix processing techniques suffer from various inefficiencies, particularly when used to implement artificial intelligence and machine learning in artificial neural networks. For example, while central processing units (CPUs) may be used to perform matrix operations, many CPU architectures are designed for low arithmetic intensity operations (i.e., a low ratio of arithmetic operations compared to memory operations) and are therefore not designed for efficient execution of matrix operations. Furthermore, many CPU architectures utilize complex local or cache memory management routines, which can increase the processing overhead and execution complexity of operations involving large matrix operands. Graphics processing units (GPUs) may also be used to perform matrix operations. However, GPUs are often designed for high-precision calculations and may provide a level of precision unnecessary for certain matrix operations, thus reducing the amount of matrix operations that can be performed. Therefore, existing matrix processing techniques are inefficient for certain matrix operations, such as matrix multiplication or convolution operations involving large matrix operands and / or matrix operands with certain dimensions, among other examples. Furthermore, existing techniques cannot be efficiently scaled to perform these matrix operations in parallel across additional processing resources. Thus, existing techniques do not achieve 100% processing efficiency when scaling and / or distributing these matrix operations. Furthermore, existing techniques are often rigid and inflexible, with limited or no ability to define new matrix operations, modify existing matrix operations, etc.

[0012] The matrix processing functionality described throughout this disclosure provides a flexible or "programmable" approach for defining or implementing specific matrix operations. For example, certain embodiments may include a matrix processor capable of executing programmable matrix routines. For example, a matrix routine may be a programmable routine designed to perform a specific matrix operation when executed by a matrix processor. For example, a matrix routine may include a set of instructions and / or commands supported by a particular matrix processor and designed to perform a desired matrix operation when executed by that matrix processor. In some embodiments, for example, a matrix processor may be designed to support a set of instructions and / or commands for performing various basic operations. In this manner, matrix routines for more complex matrix operations may be programmed using the basic instructions and / or commands supported by the matrix processor. In some embodiments, these matrix routines may be stored in a matrix routine memory associated with the matrix processor. Then, when a specific matrix operation needs to be performed, the matrix processor can retrieve the corresponding matrix routine from the matrix routine memory and then execute the instructions and / or commands of that routine to perform the desired matrix operation.

[0013] The programmable matrix processing functions described throughout this disclosure provide numerous technical advantages, including mitigating inefficiencies in existing approaches and allowing flexible matrix operations to be efficiently defined and implemented using programmable matrix routines. These programmable matrix routines allow a wide range of matrix processing functions to be programmatically implemented in a matrix processor rather than via inefficient, time-consuming, and costly hardware-based implementations.

[0014] Exemplary embodiments that may be used to implement the matrix processing functionality of the present disclosure will now be described in more detail with reference to the accompanying figures.

[0015] FIG. 1 illustrates a schematic diagram of an exemplary computing system 100 in accordance with certain embodiments.

[0016] In some embodiments, the matrix processing functionality described throughout this disclosure may be implemented in system 100. The matrix processing functionality may be used in system 100 for a wide range of applications and / or use cases, ranging from graphics processing to machine learning and artificial intelligence, among other examples, that involve matrix operations. For example, in some embodiments, the matrix processing functionality may be used to implement artificial intelligence and machine learning in artificial neural networks. Furthermore, the matrix processing functionality may be implemented by any component of system 100. For example, in the illustrated embodiment, system 100 includes edge device 110, cloud service 120, matrix processing node 130, and network 150. Matrix processing node 130 may include any component or device with matrix processing functionality, including any component of system 100. For example, matrix processing nodes 130 may include cloud services 120 and / or servers (e.g., application servers in a data center) on which matrix processing functionality is implemented, edge devices 110 (e.g., end user devices 112, Internet of Things devices 114, gateways 116) on which matrix processing functionality is implemented, etc. These various components of system 100 are discussed further below.

[0017] Edge devices 110 may include any equipment and / or devices located near or connected to the “edge” of communication system 100. Edge devices 110 may communicate with each other and / or with other remote networks and services (e.g., cloud services 120) over one or more networks and / or communication protocols, such as network 150. In some embodiments, certain edge devices 110 may include matrix processing functionality as described throughout this disclosure and may therefore be used as matrix processing nodes 130. In the illustrated embodiment, edge devices 110 include end-user devices 112 (e.g., desktops, laptops, mobile devices), Internet of Things (IoT) devices 114, and gateways and / or routers 116, among other examples.

[0018] End user devices 112 may include any device that enables or facilitates user interaction with computing system 100, including, for example, desktop computers, laptops, tablets, mobile phones and other mobile devices, and wearable devices (e.g., smart watches, smart glasses, headsets), among other examples.

[0019] IoT devices 114 may include any device capable of communicating with and / or participating in an Internet of Things (IoT) system or network. IoT systems may refer to new or improved ad hoc systems and networks composed of multiple different devices (e.g., IoT devices 114) that interoperate and synergize for a specific application or use case. Such ad hoc systems are emerging as more and more products and devices become “smart.” “Smart” means that these products and devices are controlled or monitored by a computer processor and can communicate with other devices. For example, IoT devices 114 may include a computer processor and / or communication interfaces that enable them to interoperate with other components of the system 100, such as cloud services 120 and / or other edge devices 110. IoT devices 114 may be “greenfield” devices, which are developed with IoT capabilities from the outset, or “brownfield” devices, which are created by integrating IoT capabilities into existing legacy devices that were originally developed without IoT capabilities. For example, in some cases, IoT devices 114 may consist of sensors and communication modules integrated into or attached to "things," such as appliances, toys, tools, vehicles, living things (e.g., plants, animals, humans), etc. Alternatively, or in addition, certain IoT devices 114 may rely on intermediate components, such as edge gateways or routers 116, to communicate with various components of the system 100.

[0020] IoT devices 114 may include various types of sensors to monitor, detect, measure, and generate sensor data and signals associated with characteristics of their environment. For example, a given sensor may be configured to detect one or more respective characteristics such as motion, weight, physical contact, temperature, wind, noise, light, position, humidity, radiation, liquids, specific chemical compounds, battery life, wireless signals, computer communications, and bandwidth, among other examples. Sensors may include physical sensors (e.g., physical monitoring components) and virtual sensors (e.g., software-based monitoring components). IoT devices 114 may also include actuators that perform various operations in their respective environments. For example, actuators may be used to selectively activate specific functions, such as turning on or off the power or operation of security systems (e.g., alarms, cameras, locks) or home appliances (e.g., audio systems, lighting fixtures, HVAC equipment, garage doors), among other examples.

[0021] Moreover, this disclosure contemplates a potentially limitless range of applications for IoT devices 114 and associated sensors / actuators. IoT devices 114 may include, for example, any type of equipment and / or devices associated with any type of system 100 and / or industry, including the transportation industry (e.g., automobiles, airlines), industrial manufacturing industry, energy industry (e.g., power plants), communications industry (e.g., internet, cellular, and television broadcasters), medical industry (e.g., healthcare, pharmaceuticals), food processing industry, and / or retail industry, among others. For example, in the transportation industry, IoT devices 114 may include equipment and devices such as navigation systems, autonomous flight or driving systems, traffic sensors and controllers, and / or any internal mechanical or electrical components (e.g., engines) monitored by sensors associated with aircraft, automobiles, or ships. IoT devices 114 may also include equipment, devices, and / or infrastructure associated with industrial manufacturing and production, transportation (e.g., cargo tracking), communication networks (e.g., gateways, routers, servers, cellular towers), server farms, power plants, wind farms, oil and gas pipelines, water treatment and distribution, wastewater collection and treatment, and weather observation (e.g., temperature sensors, wind sensors, and humidity sensors), among other examples. IoT devices 114 may also include, for example, any type of “smart” device or system, including, among other examples, smart entertainment systems (e.g., televisions, audio systems, video game systems), smart home or office appliances (e.g., heating, ventilation, and air conditioning (HVAC) appliances, refrigerators, washers and dryers, coffee makers), power control systems (e.g., automatic controls for electricity, lighting, and HVAC), security systems (e.g., alarms, locks, cameras, motion detectors, fingerprint scanners, facial recognition systems), and other home automation systems. The IoT device 114 may be statically located, such as attached to a building, wall, floor, ground, utility pole, sign, water tower, or any other stationary or fixed structure.The IoT devices 114 may be mobile, such as devices in vehicles or aircraft, drones, packages (e.g., for tracking cargo), mobile devices, and wearable devices, among other examples. Additionally, the IoT devices 114 may be any type of edge device 110, including end-user devices 112 and edge gateways and routers 116.

[0022] Edge gateways and / or routers 116 may be used to facilitate communications to and from edge devices 110. For example, gateways 116 may provide communications capabilities to existing legacy devices (e.g., “brownfield” IoT devices) that were initially developed without any such capabilities. Gateways 116 may also be used to extend the geographic reach of edge devices 110 with short-range, proprietary, or otherwise limited communications capabilities, such as IoT devices 114 with Bluetooth or ZigBee communications capabilities. For example, gateways 116 can act as intermediaries between IoT devices 114 and remote networks or services by providing fronthaul to IoT devices 114 using their own native communications capabilities (e.g., Bluetooth, ZigBee) and backhaul to other networks 150 and / or cloud services 120 using another wired or wireless communications medium (e.g., Ethernet, Wi-Fi, cellular). In some embodiments, the gateway 116 may be implemented by a dedicated gateway device or a general-purpose device, such as another IoT device 114, an end-user device 112, or other type of edge device 110.

[0023] In some examples, the gateway 116 may also implement particular network management and / or application functionality (e.g., IoT management and / or IoT application functionality for IoT devices 114), either individually or in conjunction with other components, such as cloud services 120 and / or other edge devices 110. For example, in some embodiments, configuration parameters and / or application logic may be pushed to or pulled from the gateway device 116, thereby enabling IoT devices 114 (or other edge devices 110) within range of or in proximity to the gateway 116 to be configured for a particular IoT application or use case.

[0024] Cloud services 120 may include services that are remotely hosted across network 150 or in the “cloud.” In some embodiments, for example, cloud services 120 may be remotely hosted on a server (e.g., an application server or a database server) in a data center. Cloud services 120 may include any service that may be utilized by or for edge devices 110, including, but not limited to, data storage, computational services (e.g., data analysis, search, diagnostics, and fault management), security services (e.g., monitoring, alarms, user authentication), mapping and navigation, geolocation services, network or infrastructure management, IoT applications and management services, payment processing, audio and video streaming, messaging, social networking, news, and weather, among other examples. In some embodiments, certain cloud services 120 may include the matrix processing functionality described throughout this disclosure and thus may be used as matrix processing nodes 130.

[0025] In general, edge devices 110 (and IoT devices 114 in particular) can generate extremely large amounts of diverse data. IoT edge devices 114 typically offload this data to the cloud (e.g., by cloud services 120) for processing and / or storage. However, cloud services 120 may not always be well-suited to handle the rapidly increasing volume, variety, and velocity of data generated by IoT devices 114 and other edge devices 110. For example, cloud-based processing may not be ideal in certain situations, such as processing time-sensitive or highly confidential data or facing network bandwidth constraints, among other examples. In some embodiments, cloud services 120 may utilize “edge”-based processing using edge devices 110 to improve the performance of the cloud service. Edge processing is an approach that involves processing certain data at the network edge (e.g., using edge devices 110), i.e., close to where the data is generated, rather than simply sending large amounts of data to the cloud for processing and storage. Certain data may still be sent to the cloud as needed, such as for deeper analysis and / or long-term storage. Edge processing may be used to compensate for the shortcomings of cloud-based processing (e.g., when cloud-based processing is inefficient, ineffective, and / or insecure) and thus better accommodate the increasing volume, variety, and velocity of data generated by IoT devices 114 and / or other edge devices 110. For example, in some cases, processing data closer to the data source (e.g., at the network edge) rather than in the cloud can improve performance and / or avoid system failure or disaster. Edge processing can also conserve network bandwidth, which can be particularly beneficial when faced with bandwidth constraints and / or limited network connectivity.

[0026] In some embodiments, edge devices 110 that provide edge-based processing to cloud services 120 may be collectively referred to as the “fog” because the edge devices 110 act to extend the “cloud” to the edge of the network, thus forming a “fog” on the network edge. In some embodiments, devices 110 in the “fog” may connect and / or communicate with each other using, for example, an interconnection standard or protocol. For example, in some embodiments, the interconnection of devices may be implemented using the Open Interconnect Consortium (OIC) Standard 1.0, published by the Open Connectivity Foundation™ (OCF) on December 23, 2015, which enables devices to discover and connect to each other. Another interconnection protocol that may be used is Thread, a networking protocol for Internet of Things (IoT) devices used in “smart” home automation and similar deployments, developed by a consortium named the “Thread Group.” Other interconnection protocols may also be used, including, for example, the Optimized Link State Routing (OLSR) protocol or the better approach to mobile ad-hoc networking (BATMAN), among others.

[0027] Network 150 may be used to facilitate communication between components of computing system 100. For example, edge devices 110, such as end user devices 112 and IoT devices 114, may use network 150 to communicate with each other and / or to access one or more remote cloud services 120. Network 150 may include any number or type of communication networks, including, for example, a local area network, a wide area network, a public network, the Internet, a cellular network, a Wi-Fi network, a short-range network (e.g., Bluetooth or ZigBee), and / or any other wired or wireless network or communication medium.

[0028] Any, all, or some of the computing devices of system 100 may be adapted to run any operating system, including Linux® or other UNIX®-based operating systems, Microsoft Windows®, Windows® Server, MacOS, Apple iOS®, Google Android®, or any customized and / or proprietary operating system, along with virtual machines adapted to virtualize the execution of a particular operating system.

[0029] Although Figure 1 is described as including or associated with multiple elements, not all elements illustrated in system 100 of Figure 1 may be utilized in each alternative embodiment of the present disclosure. Furthermore, one or more of the elements described in connection with the example of Figure 1 may be located external to system 100, and in other examples, particular elements may be included within or as part of one or more of the other described elements, as well as other elements not described in the illustrated embodiment. Furthermore, particular elements illustrated in Figure 1 may be combined with other components and may be used for alternative or additional purposes in addition to those described herein. [Example Matrix Processing Architecture]

[0030] 2A-2C show block diagrams of exemplary embodiments of matrix processing architectures.

[0031] In some embodiments, the matrix processing functions described throughout this disclosure may be implemented using a matrix processing architecture, such as the matrix processing architecture of FIGS. 2A-2C. A matrix processing architecture, such as the matrix processing architecture of FIGS. 2A-2C, may be implemented in or used in various systems, devices, and / or components, such as those described throughout this disclosure, including system 100 of FIG. 1 and / or any of its associated components (e.g., cloud service 120 / data center server, edge device 110, matrix processing node 130). In some embodiments, the matrix processing architecture of FIGS. 2A-2C may be used to implement artificial intelligence and machine learning in neural networks. The matrix processing architecture illustrated in FIGS. 2A-2C is merely one example embodiment for performing the matrix processing functions described throughout this disclosure. Other embodiments may use different types, arrangements, and / or numbers of components. For example, other embodiments may include any number of matrix processing chips 220, matrix processing clusters 230, matrix processing units (MPUs) 234, high-bandwidth memory (HBM) modules 240, and / or memory resource blocks (MRBs) 238. Additionally, all or a portion of any of the components of the matrix processing architectures of Figures 2A-2C (e.g., any of matrix processing system 200, matrix processing chips 220, and / or matrix processing clusters 230) may be implemented as separate or stand-alone components or chips, or may be integrated with other components or chips, such as a system-on-chip (SoC) that integrates various computer components into a single chip.

[0032] 2A shows a block diagram of an exemplary embodiment of a matrix processing system 200. In the illustrated embodiment, matrix processing system 200 includes a host processor 260, a host memory 270, matrix processing resources 210, and an interconnect bus 280.

[0033] The host processor 260 may be configured to control and / or manage the matrix processing system 200. For example, in some embodiments, the host processor 260 may use the matrix processing resources 210 to perform complex matrix operations. The host processor 260 may be any processing resource capable of controlling and / or managing the matrix processing functions of the matrix processing system 200. For example, in some embodiments, the host processor 260 may be implemented using the computer processors 300 or 400 of FIGS. 3 and 4, respectively. In some embodiments, the host processor 260 may be a separate or standalone component communicatively coupled to the matrix processing resources 210. Alternatively, in other embodiments, the host processor 260 and the matrix processing resources 210 may be integrated into the same component or chip. For example, in some embodiments, the components of the matrix processing system 200, including the host processor 260 and the matrix processing resources 210, may be implemented as a system-on-chip (SoC).

[0034] Host memory 270 may include any type or combination of volatile and / or non-volatile memory. Examples of volatile memory include various types of random access memory (RAM), such as dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and static random access memory (SRAM), among other examples. Examples of non-volatile memory include disk-based storage media (e.g., magnetic and / or optical storage media), solid-state storage (e.g., any form of persistent flash memory, including planar or three-dimensional (3D) NAND flash memory or NOR flash memory), 3D cross-point memory, electrically erasable programmable read-only memory (EEPROM), and / or other types of non-volatile random access memory (RAM), among other examples. For example, host memory 270 may be used to store information for host processor 260, such as code and / or data, during execution.

[0035] Interconnect bus 280, in some embodiments, may be used to communicatively couple host processor 260 and host memory 270 to matrix processing resources 210. Interconnect bus 280 may use any interconnect protocol, such as Peripheral Component Interconnect Express (PCIe), Universal Serial Bus (USB), or Small Computer System Interface (SCSI), among other examples.

[0036] The matrix processing resources 210 may include any processing resources configured to perform matrix operations. For example, the matrix processing resources 210 may be configured to perform matrix multiplication operations, convolution operations, element-wise matrix operations (e.g., +, ×, / , <, >, ==), dimension shuffle operations, and / or any combination thereof. In some embodiments, the matrix processing resources 210 may include processing resources designed and optimized for performing matrix operations. In some embodiments, the matrix processing resources 210 may be arranged hierarchically with multiple levels of processing resources. For example, in the illustrated embodiment, the matrix processing resources 210 include multiple matrix processing chips 220, each of which may include any processing resources. For example, as described below in connection with FIGS. 2B and 2C , each matrix processing chip 220 may include multiple high-bandwidth memory (HBM) modules 240 and multiple matrix processing clusters 230, each of which may include multiple matrix processing units 234. Thus, in some embodiments, the matrix processing resources 210 may include multiple matrix processing chips 220, multiple high bandwidth memory (HBM) modules 240, and multiple matrix processing clusters 230 in each matrix processing chip 220, and / or multiple matrix processing units 234 in each matrix processing cluster 230.

[0037] The matrix processing chip 220 may be, for example, any chip or other component configured to perform matrix operations. For example, in some embodiments, the matrix processing chip 220 may be a peripheral card or chip connected to the host processor 260 using any type of interconnect interface, such as a PCIe interface. In some embodiments, the matrix processing chip 220 may be implemented using an integrated circuit, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and / or any other type of circuit. In the illustrated embodiment, the matrix processing chips 220 are configured in a cyclical arrangement, with communication channels 215 provided between adjacent matrix processing chips 220. In some embodiments, the communication channels 215 may provide unidirectional communication between adjacent matrix processing chips 220. However, in other embodiments, the communication channels 215 may provide bidirectional communication between adjacent matrix processing chips 220. A cyclical arrangement using unidirectional communication between adjacent processing resources may be referred to as a “single cyclical” configuration, and a cyclical arrangement using bidirectional communication between adjacent processing resources may be referred to as a “double cyclical” configuration.

[0038] Furthermore, although not illustrated, in some embodiments, the matrix processing system 200 may include a communication interface for communicating over a communication network. For example, in some embodiments, the matrix processing system 200 may communicate over a network with one or more remote matrix processing chips to perform distributed matrix operations.

[0039] 2B shows a block diagram of an exemplary embodiment of matrix processing chip 220. In the illustrated embodiment, matrix processing chip 220 includes a controller 222, a host interface 224, an inter-chip link 225, a high-bandwidth memory (HBM) module 240, and a matrix processing cluster 230.

[0040] The controller 222 may be configured to control and / or manage the matrix operations performed by the matrix processing chip 220. In some embodiments, the controller 222 may control and / or manage the matrix operations in conjunction with the host processor 260 of FIG. 2A and / or the master control CPU (MCC) 232 of the matrix processing cluster 230 of FIG. 2C. For example, in some embodiments, the host processor 260, the controller 222, and / or the master control CPU (MCC) 232 may be configured to receive matrix operations or commands and distribute the matrix operations and matrix operands across the matrix processing cluster 230 and the high bandwidth memory (HBM) module 240. In some embodiments, the controller 222 may be a microprocessor, an integrated circuit, and / or any other type of circuitry and / or processing logic.

[0041] The host interface 224 may be a communications interface that allows the matrix processing chip 220 to communicate with the host processor 260 of Figure 2A. In some embodiments, for example, the controller 222 may use the host interface 224 to communicate with the host processor 260 of Figure 2A. The host interface 224 may use any type of interconnection protocol or interface, including Peripheral Component Interconnect Express (PCIe), Universal Serial Bus (USB), or Small Computer System Interface (SCSI), among other examples.

[0042] The inter-chip link (ICL) 225 may enable the matrix processing chip 220 to communicate with other matrix processing chips. For example, the inter-chip link 225 may be used to implement the communication channel 215 between the matrix processing chips 220 in FIG. 2A . The inter-chip link 225 may be, for example, any communication interface that enables the matrix processing chip 220 to communicate with another matrix processing chip. In some embodiments, the matrix processing chip 220 may include multiple inter-chip links 225 (e.g., 12 inter-chip links). In some embodiments, the inter-chip link 225 may be implemented using one or more serializer / deserializer (SerDes) interfaces. The SerDes interface may be a communication interface that converts data from serial to parallel and vice versa. For example, the transmitter of the SerDes interface may include a serial-to-parallel converter, and the receiver of the SerDes interface may include a parallel-to-serial converter. In some embodiments, the matrix processing chip 220 may use multiple SerDes interfaces for each connection to another matrix processing chip (e.g., four SerDes interfaces between each pair of connected matrix processing chips).

[0043] The high-bandwidth memory (HBM) module 240 may be a memory component used to store matrix operands and other matrix data associated with the matrix processing chip 220. In some embodiments, the high-bandwidth memory (HBM) module 240 may be designed to efficiently store and retrieve matrix data. In some embodiments, the high-bandwidth memory (HBM) module 240 may be a multi-dimensional memory component configured to store and retrieve data in multiple dimensions. For example, in some embodiments, the high-bandwidth memory (HBM) module 240 may be a memory component configured to store and retrieve data in two dimensions, such as rows and columns. However, other embodiments may use memory components configured to store and retrieve data using any other number of dimensions (e.g., one dimension, three dimensions, four dimensions, etc.). In the illustrated embodiment, the matrix processing chip 220 includes four high-bandwidth memory (HBM) modules 240a-d. In some embodiments, the high-bandwidth memory (HBM) module 240 may be shared by the matrix processing clusters 230 of the matrix processing chip 220.

[0044] The matrix processing clusters 230 may include processing resources configured to perform matrix operations such as matrix multiplication, convolution, and / or dimensional shuffling, among other examples. In some embodiments, the matrix processing clusters 230 may be collectively used to perform a particular matrix operation by performing matrix operations in parallel. In the illustrated embodiment, the matrix processing chip 220 includes 12 matrix processing clusters 230a-1. Furthermore, in the illustrated embodiment, the matrix processing clusters 230 are configured or arranged using a two-dimensional mesh-type interconnection topology. The interconnection topology of the matrix processing clusters 230 can facilitate circular communication between the matrix processing clusters 230. Furthermore, other embodiments may include any number and / or arrangement of matrix processing clusters 230.

[0045] 2C shows a block diagram of an example embodiment of matrix processing cluster 230. In the illustrated embodiment, matrix processing cluster 230 includes a master control CPU (MCC) 232, a matrix processing unit (MPU) 234, a slice engine 236, and a memory resource block (MRB) 238.

[0046] The master control CPU (MCC) 232 may be configured to control and / or manage the matrix operations performed by the matrix processing cluster 230. In some embodiments, the master control CPU 232 may be a microprocessor, an integrated circuit, and / or any other type of circuit and / or processing logic. In some embodiments, the master control CPU 232 may receive instructions from another component, such as the host processor 260 of FIG. 2A and / or the controller 222 of FIG. 2B. Based on the instructions, the master control CPU 232 may then use the matrix processing unit 234 to perform matrix operations such as matrix multiplication, convolution, and / or dimension shuffling, among other examples. For example, the master control CPU 232 may receive an instruction to perform a matrix multiplication operation such as C = A × B. The instruction may include a handle or identifier for each matrix and may also indicate how the matrices are to be stored in the memory resource block (MRB) 238. Matrices A and B may then be divided into a series of smaller matrices (e.g., 32 × 32 matrices). Matrix operations may then be performed on smaller matrices until the output matrix C is completely calculated, and partial results may be stored in memory resource blocks (MRBs) 238.

[0047] The matrix processing unit (MPU) 234 may be configured to perform matrix operations such as matrix multiplication, convolution, and / or dimension shuffling. In some embodiments, the matrix processing unit (MPU) 234 performs the matrix operations based on commands received from the master control CPU (MCC) 232. Furthermore, in some embodiments, each matrix processing cluster 230 may include multiple matrix processing units (MPUs) 234. For example, in the illustrated embodiment, the matrix processing cluster 230 includes two matrix processing units (MPUs) 234. The matrix processing unit (MPU) 234 may be capable of performing matrix operations such as matrix multiplication on sub-matrices (e.g., 32×32 matrices). In some cases, the matrix processing unit (MPU) 234 may be designed and / or optimized to perform matrix multiplication operations. The matrix processing unit (MPU) 234 may load matrix operands from the memory resource block (MRB) 238. In some embodiments, the matrix processing unit (MPU) 234 may support the following arithmetic operations: matrix multiplication, unary matrix operations, binary matrix operations (addition (+), subtraction (-), multiplication (x), division ( / ), bitwise XOR, AND, OR, logical and arithmetic left and right shifts, comparisons (>, <, >=, <=, ==, !=), etc.), and column-wise, row-wise, and whole matrix operations (such as sum, max, and min).

[0048] The slice engine 236 may be configured to slice matrix operands for a particular matrix operation into smaller sub-matrices. For example, in some embodiments, the master control CPU (MCC) 232 may use the slice engine 236 to divide matrix operands into smaller sub-matrices for the matrix processing unit (MPU) 234. In some embodiments, the slice engine 236 may include a convolution slice engine (CSE) that performs matrix slicing for a convolution operation. For example, in some embodiments, the convolution slice engine (CSE) may slice matrix operands in a manner that allows the convolution operation to be cast as a matrix multiplication operation, thus enabling the same processing logic to perform both matrix multiplication and matrix convolution operations. Furthermore, in some embodiments, the slice engine 236 and / or an associated convolution slice engine (CSE) may be used to perform a dimension shuffle operation that reorders the dimensions of a matrix.

[0049] The memory resource blocks (MRBs) 238 may be memory components on the matrix processing cluster 230 used to store matrix operands and other matrix data. In some embodiments, the memory resource blocks (MRBs) 238 may be designed to efficiently store and retrieve matrix data. In some embodiments, the memory resource blocks (MRBs) 238 may be multidimensional memory components configured to store and retrieve data in multiple dimensions. For example, in some embodiments, the memory resource blocks (MRBs) 238 may be memory components configured to store and retrieve data in two dimensions, such as rows and columns. In the illustrated embodiment, the matrix processing cluster 230 includes ten memory resource blocks (MRBs) 238. However, in other embodiments, a different number of memory resource blocks (MRBs) 238 may be included on the matrix processing cluster 230. In some embodiments, each memory resource block (MRB) 238 may be capable of storing a matrix of a particular size (e.g., a 256×512 matrix). In some embodiments, a memory resource block (MRB) 238 may be shared by the matrix processing units (MPUs) 234 of a particular matrix processing cluster 230.

[0050] In some embodiments, the matrix processing architecture of FIGS. 2A-2C may be used to implement the matrix processing functions described throughout this disclosure. For example, matrix processing system 200 may be used to perform matrix operations using a distributed approach that achieves 100% processing efficiency using available processing resources. For example, in some embodiments, matrix operations may be distributed across multiple processing resources 210 that are optimized for matrix processing, thus enabling full utilization of the processing resources 210 throughout the duration of the matrix operation. For example, matrix processing system 200 may include multiple processing resources 210 that are designed and optimized to perform matrix operations. In some embodiments, these processing resources 210 may be configured in a single-circuit or dual-circuit arrangement. Furthermore, processing resources 210 may be arranged hierarchically with multiple levels of processing resources. For example, in some embodiments, the processing resources 210 may include multiple matrix processing chips 220, multiple high-bandwidth memory (HBM) modules 240, and multiple matrix processing clusters 230 in each matrix processing chip 220, and / or multiple matrix processing units (MPUs) 234 in each matrix processing cluster 230. This processing architecture allows matrix operations to be distributed across multiple processing resources 210 and / or allows matrix operations to process hierarchies with 100% processing efficiency. Furthermore, this processing architecture allows matrix operations to be efficiently scaled across a variable number of processing resources 210 operating in parallel while achieving 100% processing efficiency. For example, scaling may be achieved by adjusting the number of processing resources 210 used to perform a particular matrix operation, such as the number of matrix processing systems 200 or servers, the number of matrix processing chips 220 within each matrix processing system 200 or server, etc.

[0051] 2A-2C may be used to implement matrix multiplication and / or convolution operations. For example, in some embodiments, a matrix multiplication operation may be distributed across multiple processing resources 210 in a manner such that the latency for transmitting matrix operands is less than the matrix processing time, thereby allowing the transmission of matrix operands to be completed while the matrix processing is being performed. For example, for certain matrix operations involving matrix operands with certain dimensions (e.g., matrix multiplication with "skinny" matrix operands), the time required to access and transmit the matrix operands may exceed the time required to perform the actual matrix calculation, resulting in idle processing time while the matrix operands are retrieved from memory and / or transmitted to the processing resources 210. For example, a single-circuit configuration (e.g., each processing resource 210 retrieves matrix operands and data from only one adjacent processing resource 210 at any given time) may not achieve 100% processing efficiency for these particular types of matrix operations and matrix operands. However, the dual-circulation configuration of processing resources 210 allows each processing resource to perform matrix calculations while simultaneously obtaining matrix operands and data from its neighboring processing resources 210, thereby significantly reducing the latency for transmitting matrix operands and avoiding any idle processing time. For example, the communication latency of a particular operation can be reduced by half when using the dual-circulation approach rather than the single-circulation approach. In this way, the latency for transmitting matrix operands and matrix data is completely masked by the matrix processing time, thus avoiding any wasted time or idle processing time and achieving 100% processing efficiency. Therefore, matrix operations (e.g., matrix multiplication or GEMM) can be efficiently performed even for large matrix operands and / or matrix operands with certain dimensions, such as large matrix operands that are neither square matrices nor single vectors (e.g., “skinny” matrices with heights much greater than widths).For example, matrix multiplication may be performed efficiently even when multiplying two elongated matrices, a elongated matrix with a square matrix, etc. Similarly, convolution operations may be distributed across multiple processing resources 210 in a manner that results in 100% processing efficiency using available processing resources.

[0052] As an example, when a matrix operation or command is received, the matrix operation may be distributed across the processing resources 210 of the matrix processing system 200. For example, the matrix operands (or input matrices) may be partitioned based on the number of available processing resources 210. Furthermore, in some embodiments, these partitions may span across rows of the matrix operands and / or across any other dimensions of the matrix operands. Each partition may then be distributed to a particular processing resource 210. Each processing resource 210 may then perform multiple submatrix operations. In some embodiments, multiple submatrix operations are performed in multiple stages. For example, each processing resource 210 may perform a particular stage of a submatrix operation while simultaneously transmitting and receiving submatrix data to and from an adjacent processing resource 210. For example, in a single-circulation configuration of the processing resources 210, each processing resource 210 transmits or receives submatrix data to and from each of its adjacent processing resources. Similarly, in a double-circulation configuration of the processing resources 210, each processing resource 210 may transmit and receive submatrix data to and from each of its adjacent processing resources 210.

[0053] Each processing resource 210 can then use the submatrix data for the next submatrix operation. The result of the matrix operation may then be determined based on the submatrix operations performed by the processing resources 210 collectively.

[0054] Furthermore, when the processing resources 210 are arranged hierarchically, matrix operations may be distributed in a hierarchical manner. For example, matrix operands (or input matrices) may first be partitioned based on the number of available matrix processing chips 220. Then, each partition and associated submatrix operations may be distributed to a particular matrix processing chip 220. The partitions and submatrix operations distributed to a particular matrix processing chip 220 may then be similarly partitioned and distributed across the matrix processing clusters 230 and / or high-bandwidth memory (HBM) modules 240 of the particular matrix processing chip 220. For example, for a particular matrix operation, the submatrix operations may be distributed to each matrix processing cluster 230. Alternatively, for a particular matrix operation, the submatrix operations may be distributed across various “logical processing nodes” (e.g., groups of matrix processing clusters 230 associated with high-bandwidth memory (HBM) modules 240) and then distributed to each matrix processing cluster 230 of the particular logical processing node. In some embodiments, the matrix processing clusters 230 (and / or logical processing nodes) may be circularly organized similar to the matrix processing chips 220. The partitions and submatrix operations distributed to a particular matrix processing cluster 230 may then be similarly partitioned and distributed across the matrix processing units (MPUs) 234 of a particular matrix processing cluster 230. Exemplary Computer Processor Architectures

[0055] 3 and 4 show block diagrams of exemplary embodiments of computer processors that may be used in accordance with embodiments disclosed herein. For example, the computer processors illustrated in FIGS. 3 and 4 may be used as host processors associated with matrix processing systems (e.g., host processor 260 of matrix processing system 200 of FIG. 2A) or as processors associated with other components and / or devices discussed throughout this disclosure (e.g., processors associated with components of system 100 of FIG. 1). Other processors and system designs and configurations known in the art for laptops, desktops, handheld PCs, personal digital assistants, engineering workstations, servers, network devices, network hubs, switches, embedded processors, digital signal processors (DSPs), graphics devices, video game devices, set-top boxes, microcontrollers, mobile phones, portable media players, handheld devices, and various other electronic devices are also suitable. In general, a wide variety of systems or electronic devices that may incorporate processors and / or other execution logic as disclosed herein are generally suitable.

[0056] FIG. 3 illustrates a block diagram of an exemplary embodiment of a processor 300. Processor 300 is one example of a type of hardware device that may be used in connection with embodiments described throughout this disclosure. Processor 300 may be any type of processor, such as a microprocessor, an embedded processor, a digital signal processor (DSP), a network processor, a multi-core processor, a single-core processor, or other device that executes code. While only one processor 300 is illustrated in FIG. 3, a processing element may alternatively include more than one processor 300 as illustrated in FIG. 3. Processor 300 may be a single-threaded core, or in at least one embodiment, processor 300 may be multi-threaded in that it may include more than one hardware thread context (or "logical processor") per core.

[0057] 3 also illustrates memory 302 coupled to processor 300, according to one embodiment. Memory 302 may be any of a wide variety of memory (including various layers of a memory hierarchy) known or otherwise available to those skilled in the art. Such memory elements may include, but are not limited to, random access memory (RAM), read-only memory (ROM), logic blocks of a field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), and electrically erasable programmable ROM (EEPROM).

[0058] Processor 300 may execute any type of instruction associated with an algorithm, process, or operation described herein. In general, processor 300 may transform elements or items (e.g., data) from one state or condition to another state or condition.

[0059] Code 304, which may be one or more instructions executed by processor 300, may be stored in memory 302 or may be stored in software, hardware, firmware, or any suitable combination thereof, or in any other internal or external component, device, element, or object as needed based on particular needs. In one example, processor 300 may follow a program sequence of instructions indicated by code 304. Each instruction enters front-end logic 306 and is processed by one or more decoders 308. The decoders may generate as their output micro-operations, such as fixed-width micro-operations in a predefined format, or may generate other instructions, micro-instructions, or control signals that reflect the original code instructions. Front-end logic 306 may also include register renaming logic and scheduling logic, which generally allocate resources and queue operations corresponding to instructions for execution.

[0060] The processor 300 may also include execution logic 314 having a set of execution units 316a, 316b, 316n, etc. Some embodiments may include multiple execution units dedicated to a particular function or set of functions. Other embodiments may include only one execution unit, or one execution unit capable of performing a particular function. The execution logic 314 performs operations specified by code instructions.

[0061] After completing execution of the operation specified by the code instruction, back-end logic 318 can retire the instruction of code 304. In one embodiment, processor 300 allows out-of-order execution but requires in-order retirement of instructions. Retirement logic 320 may take various known forms (e.g., a reorder buffer, etc.). In this manner, processor 300 is transformed during execution of code 304 with respect to outputs generated by at least the decoder, the hardware registers and tables utilized by register renaming logic 310, and any registers (not shown) modified by execution logic 314.

[0062] 3, a processing element may include other elements on a chip that includes processor 300. For example, a processing element may include memory control logic along with processor 300. A processing element may include I / O control logic and / or I / O control logic integrated with memory control logic. A processing element may also include one or more caches. In some embodiments, non-volatile memory (such as flash memory or fuses) may also be included on a chip that includes processor 300.

[0063] 4 illustrates a block diagram of an exemplary embodiment of a multiprocessor 400. As shown in FIG. 4, multiprocessor system 400 is a point-to-point interconnect system and includes a first processor 470 and a second processor 480 coupled via a point-to-point interconnect 450. In some embodiments, each of processors 470 and 480 may be some version of processor 300 of FIG. 3.

[0064] Processors 470 and 480 are shown including integrated memory controller (IMC) units 472 and 482, respectively. Processor 470 also includes point-to-point (PP) interfaces 476 and 478 as part of its bus controller unit; similarly, second processor 480 includes PP interfaces 486 and 488. Processors 470, 480 can exchange information using PP interface circuits 478, 488 via point-to-point (PP) interface 450. As shown in FIG. 4, IMCs 472 and 482 couple the processors to their respective memories, i.e., memory 432 and memory 434, which may be part of main memory locally attached to the respective processors.

[0065] Each of the processors 470, 480 can exchange information with a chipset 490 via respective PP interfaces 452, 454 using point-to-point interface circuits 476, 494, 486, 498. The chipset 490 can optionally exchange information with a coprocessor 438 via a high performance interface 439. In one embodiment, the coprocessor 438 is a special purpose processor, such as a high throughput MIC processor, a network or communications processor, a compression engine, a graphics processor, a GPGPU, an embedded processor, a matrix processor, or the like.

[0066] A shared cache (not shown) may be included in either processor or may be included outside of both processors, yet connected to the processors via the PP interconnect, so that local cache information of either or both processors can be stored in the shared cache when the processors are in a low power mode.

[0067] Chipset 490 may be coupled to a first bus 416 via an interface 496. In one embodiment, first bus 416 may be a bus such as a Peripheral Component Interconnect (PCI) bus, or a PCI Express bus or another third generation I / O interconnect bus, although the scope of the present disclosure is not so limited.

[0068] As shown in FIG. 4 , various I / O devices 414 may be coupled to the first bus 416, along with a bus bridge 418 that couples the first bus 416 to a second bus 420. In one embodiment, one or more additional processors 415, such as a coprocessor, a high-throughput MIC processor, a GPGPU accelerator (e.g., a graphics accelerator or digital signal processing (DSP) unit), a matrix processor, a field programmable gate array, or any other processor, are coupled to the first bus 416. In one embodiment, the second bus 420 may be a low pin count (LPC) bus. Various devices may be coupled to the second bus 420, and in one embodiment, these devices include, for example, a keyboard and / or mouse 422, a communication device 427, and a storage unit 428, such as a disk drive or other mass storage device, which may contain instructions / code and data 430. Additionally, audio I / O 424 may be coupled to the second bus 420. Note that other architectures are possible. For example, instead of the point-to-point architecture of FIG. 4, the system may implement a multi-drop bus or other such architecture.

[0069] All or part of any of the components of FIG. 4 may be implemented as separate or stand-alone components or chips, or may be integrated with other components or chips, such as a system-on-chip (SoC) that integrates various computer components into a single chip.

[0070] While embodiments of the mechanisms disclosed herein may be implemented in hardware, software, firmware, or a combination of such implementation techniques, certain embodiments may be implemented as a computer program or program code running on a programmable system that includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0071] 4 may be applied to input instructions that perform the functions described herein and generate output information. The output information may be applied to one or more output devices, in known fashion. For purposes of this application, a processing system includes any system that has a processor, such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0072] The program code may be implemented in a high-level procedural or object-oriented programming language to communicate with a processing system. The program code may also be implemented in assembly or machine language, if desired. That is, the mechanisms described herein are not limited in scope to any particular programming language. In either case, the language may be a compiled or interpreted language.

[0073] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium that represent various logic within a processor, which instructions, when read by a machine, cause the machine to produce logic that performs the techniques described herein. Such representations are known as "IP cores," and may be stored on tangible machine-readable media and supplied to various customers or manufacturing facilities for loading into manufacturing machines that actually produce the logic or processor.

[0074] Such machine-readable storage media may include, but are not limited to, non-transitory, tangible articles of construction manufactured or formed by a machine or apparatus, including storage media such as hard disks, floppy disks, optical disks, compact disk read-only memories (CD-ROMs), compact disk re-writeables (CD-RWs), and any other type of disk including magneto-optical disks, read-only memories (ROMs), random access memories (RAMs) such as dynamic random access memories (DRAMs) and static random access memories (SRAMs), semiconductor devices such as erasable programmable read-only memories (EPROMs), flash memories, electrically erasable programmable read-only memories (EEPROMs), and phase change memories (PCMs), magnetic or optical cards, or any other type of medium suitable for storing electronic instructions.

[0075] Accordingly, embodiments of the present disclosure also include non-transitory, tangible, machine-readable media that include instructions or design data, such as a hardware description language (HDL), that defines the characteristics of the structures, circuits, devices, processors, and / or systems described herein. Such embodiments may also be referred to as program products. [Programmable matrix processing engine]

[0076] FIG. 5 illustrates an exemplary embodiment of a programmable matrix processing engine 500. In some embodiments, the matrix processing engine 500 may be implemented by a matrix processing architecture, such as the matrix processing architectures of FIGS. 2A-2C. For example, in some embodiments, the matrix processing engine 500 may be implemented by a matrix processing cluster on a matrix processing chip (e.g., the matrix processing cluster 230 of the matrix processing chip 220 of FIGS. 2B and 2C). In these embodiments, a particular matrix processing cluster may use its associated matrix processing engine 500 to perform matrix-based processing and operations, such as sub-matrix operations associated with a particular matrix operation, distributed across multiple matrix processing resources (e.g., as described throughout this disclosure).

[0077] In the illustrated embodiment, matrix processing engine 500 includes a read engine 535, a slice engine 536, and an output engine 537, which are discussed further below. The illustrated embodiment also illustrates various components of the underlying matrix processing architecture that may be involved when performing matrix operations using matrix processing engine 500. For example, the illustrated embodiment illustrates a high-bandwidth memory (HBM) module 540, a master control CPU (MCC) 532, a matrix processing unit (MPU) 534, a memory resource block (MRB) 538, and a matrix routine memory 539. In the illustrated embodiment, for example, these various components are overlaid on matrix processing engine 500 to illustrate when and how these components are used by matrix processing engine 500, as described further below.

[0078] HBM module 540 may be a high-bandwidth memory (HBM) module designed to efficiently store and retrieve large amounts of matrix data. In some embodiments, for example, HBM module 540 may be a high-bandwidth memory (HBM) module on a matrix processing chip (e.g., HBM module 240 of matrix processing chip 220 in FIG. 2B).

[0079] MCC 532 may be a master control CPU (MCC) used to control and / or manage matrix operations. In some embodiments, for example, MCC 532 may be the master control CPU on a particular matrix processing cluster (e.g., MCC 232 of matrix processing cluster 230 in FIG. 2C). In these embodiments, for example, MCC 532 may be used to control and / or manage the matrix operations performed on that particular cluster.

[0080] The MPU 534 may be a matrix processing unit (MPU) used to perform matrix operations. In some embodiments, for example, the MPU 534 may be a matrix processing unit on a particular matrix processing cluster (e.g., the MPU 234 of the matrix processing cluster 230 in FIG. 2C ). For example, in some embodiments, a matrix processing cluster may include multiple matrix processing units (MPUs) to perform matrix operations. For example, the illustrated embodiment shows two matrix processing units (MPUs) 534a and 534b. In some embodiments, the MPU 534 may perform matrix operations based on commands or instructions from the master control CPU (MCC) 532.

[0081] Memory resource block (MRB) 538 may be a memory component designed to efficiently store and retrieve matrix data. In some embodiments, for example, MRB 538 may be a memory resource block on a particular matrix processing cluster (e.g., memory resource block 238 of matrix processing cluster 230 in FIG. 2C). In these embodiments, for example, MRB 538 may be used to store and retrieve matrix data associated with matrix operations performed on a particular cluster.

[0082] The matrix routine memory 539 may be a memory component used to store matrix routines. For example, a matrix routine may be a programmable routine for a matrix processor that, when executed by the matrix processor, is designed to perform a particular matrix operation. For example, a matrix routine may include a set of instructions and / or commands supported by a particular matrix processor and designed to perform a desired matrix operation when executed by the matrix processor. In some embodiments, for example, a matrix processor may be designed to support a set of instructions and / or commands for performing various basic operations. For example, in some embodiments, a matrix processor may support instructions for manipulating data, performing various arithmetic operations, and / or identifying matrix operands and outputs for various instructions and operations.

[0083] For example, certain instructions may be used to manipulate data, such as reading, writing, and / or copying data (e.g., matrix data) to and from different locations, slicing matrix data, extracting matrix data, sorting matrix data, etc.

[0084] As another example, certain instructions may be used to perform certain arithmetic operations including any of the following operations: matrix multiplication, convolution, unary matrix operations, binary matrix operations (addition (+), subtraction (-), multiplication (x), division ( / ), bitwise XOR, AND, OR, logical and arithmetic left and right shifts, comparisons (>, <, >=, <=, ==, !=), etc.), and column-wise, row-wise, and whole matrix operations (such as sum, max, and min).

[0085] Additionally, special "register operand" (REGOP) instructions may be used to identify the matrix operands and outputs of various supported instructions and operations. Register operand instructions may be used, for example, to specify the size and location of the operands and the output of a particular instruction or operation. For example, in some embodiments, register operand instructions may be used to identify a location in a high bandwidth memory (HBM) module or memory resource block (MRB) associated with a particular operand or output. As an example, a basic matrix multiplication operation may be programmed using a REGOP instruction that identifies the location of each operand and the location of the output, followed by instructions to perform the matrix multiplication operation.

[0086] In this way, the basic instructions and / or commands supported by the matrix processor can be used to program matrix routines for more complex matrix operations such as distributed matrix multiplication and / or convolution operations, dimension shuffle operations, reshape operations, etc.

[0087] The matrix routine memory 539 may be implemented in any part of the matrix processing architecture, such as the matrix processing chip, the matrix processing cluster, and / or the host computing system. In some embodiments, for example, a matrix processing chip (e.g., matrix processing chip 220 of FIG. 2B) may include a matrix routine memory 539 accessible to each cluster on the matrix processing chip. As another example, in some embodiments, a matrix processing cluster (e.g., matrix processing cluster 230 of FIG. 2C) may include its own matrix routine memory 539. As yet another example, in some embodiments, a host computing system of a matrix processing architecture may include a matrix routine memory 539 accessible to its associated matrix processing resources (e.g., in FIG. 2A, host memory 270 of matrix processing system 200 may include a matrix routine memory accessible to matrix processing resource 210).

[0088] Furthermore, the matrix routine memory 539 may be any component or mechanism capable of storing data, including any type or combination of volatile and / or non-volatile memory, such as random access memory (RAM) (e.g., dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), static random access memory (SRAM)), flash-based memory, read-only memory (ROM), logic blocks of a field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), electrically erasable programmable ROM (EEPROM), and / or any suitable combination of the foregoing.

[0089] As an example, in some embodiments, the matrix routine memory 539 may be implemented on the matrix processing chip using random access memory (e.g., SRAM). In these embodiments, for example, the matrix routine may be loaded into the matrix routine memory 539 by a host computing system (e.g., host processor 260 of FIG. 2A). For example, the host computing system may transfer the matrix routine to the matrix processing chip via an interconnect interface (e.g., a PCIe interface), and the matrix processing chip may then store the matrix routine in its associated matrix routine memory 539. In some embodiments, a software driver of the host computing system may be used to load the matrix routine. Furthermore, in some embodiments, any existing matrix routine may be loaded into the matrix routine memory 539 during system startup, and any additional matrix routines may be subsequently loaded after system startup, as needed.

[0090] In the illustrated example, the matrix processing engine 500 executes multiple matrix operations 501 and 502 in parallel. For example, as described above, in some embodiments, the matrix processing engine 500 may be implemented on a particular matrix processing cluster, which may include multiple MPUs 534. In the illustrated example, the matrix processing engine 500 is implemented on a cluster with two MPUs 534a-b. Thus, the matrix processing engine 500 can execute the two matrix operations 501 and 502 in parallel using the respective MPUs 534. The illustrated example shows the control flow of the matrix processing engine 500 for both the first matrix operation 501 and the second matrix operation 502.

[0091] In the illustrated example, the control flow for the first matrix operation 501 begins with the read engine 535 of the matrix processing engine 500. The read engine 535 may first retrieve matrix data (e.g., matrix data associated with the operands of the matrix operation 501) from a corresponding HBM module 540a of the matrix processing chip, and the read engine 535 may then store the matrix data in a particular MRB 538a of a particular cluster associated with the read engine 535. For example, as described above, the HBM module 540a may be a high-bandwidth memory module on a particular matrix processing chip (e.g., memory shared by matrix processing clusters of a particular matrix processing chip), and the MRB 538 may be a local memory resource block on a particular matrix processing cluster. Furthermore, in some embodiments, the read engine 535 may use the master control CPU (MCC) 532 on its respective cluster to store and retrieve data from the HBM 540 and MRB 538.

[0092] Slicing engine 536a can then "slice" the matrix data stored in MRB 538a to extract particular matrix operands associated with matrix operation 501. For example, in some cases, the associated matrix operands may include only a subset of the matrix data stored in MRB 538a, and / or the matrix operands may not be contiguously located within the matrix data stored in MRB 538a. Thus, slicing engine 536a can extract particular "slices" or portions of the matrix data stored in MRB 538a and then arrange these slices to form respective matrix operands. For example, in the illustrated example, matrix operation 501 is associated with a convolution operation, and therefore slicing engine 536a is used to extract sliced ​​matrix operands and filters from the matrix data stored in MRB 538a. The sliced ​​matrix operands and filters are then stored in MRBs 538b and 538c, respectively. In some cases, the particular slicing technique used by the slice engine 536a may depend on various factors, including the type of matrix operation 501, the number of available processing resources, the size of the operands, etc. Furthermore, in some embodiments, the particular slicing performed by the slice engine 536a for a particular operation may be programmed and / or defined using the set of instructions supported by the slice engine 536a.

[0093] The output engine 537 may then be used to calculate the result for the particular matrix operation 501. For example, the output engine 537 may use the matrix operands generated by the slice engine 536a (e.g., the matrix operands stored in the MRBs 538b and 538c) to perform the appropriate matrix operation 501. For example, in some embodiments, the output engine 537 may first identify an associated matrix routine corresponding to the particular matrix operation 501, and the output engine 537 may then retrieve that matrix routine from the matrix routine memory 539. In some embodiments, the output engine 537 may use the master control CPU (MCC) 532 on its respective cluster to retrieve the matrix routine from the matrix routine memory 539.

[0094] The output engine 537 can then specify or provide, if appropriate, specific information or fields used by the matrix routine. For example, in some embodiments, specific information and / or fields of a matrix routine, such as the size and / or location of certain operands of the matrix routine, may be insufficient or unspecified. Thus, in some embodiments, the output engine 537 can use the MCC 532 to specify or provide any remaining information and / or fields of a particular matrix routine (e.g., the size and / or location of matrix operands).

[0095] The output engine 537 can then execute the particular matrix routine. For example, the output engine 537 can use the MCC 532 and / or the MPU 534a to execute programmed instructions associated with the particular matrix routine. The MCC 532 can be used to perform particular tasks, such as reading and writing data and communicating with other resources, as specified by the instruction. The MPU 534a can be used to perform particular arithmetic operations, as specified by the instruction. Furthermore, in some cases, a particular matrix routine can be executed repeatedly, or looped, until the particular operation has been performed on all required data (e.g., all data for a particular matrix operand) and completed.

[0096] The output engine 537 can then store the output or result of the matrix routine in a particular MRB 538d of the cluster used to execute the matrix routine. In some cases, the output engine 537 can then provide the stored output of the MRB 538d to another component of the matrix processing architecture. For example, in some cases, the matrix operation 501 may be a submatrix operation associated with a larger matrix operation distributed across multiple processing resources, and thus the output of the matrix operation 501 may be a partial result associated with the distributed larger operation. Furthermore, the output of the submatrix operation 501 may be required by other processing resources involved in the distributed matrix operation. Thus, the output engine 537 can provide the output of the submatrix operation 501 to an appropriate resource, for example, for further processing and / or storage. In some cases, the appropriate resource may vary based on circumstances including the type of matrix operation being performed, the implementation of the associated matrix routine, the number and availability of processing resources, etc. For example, in some cases, the specific processing and / or output destination of the matrix operation may be programmed or defined by the associated matrix routine. Additionally, in some embodiments, the output engine 537 can use the master control CPU (MCC) 532 on its respective cluster to provide the output of the submatrix operation 501 to the appropriate destination.

[0097] In some cases, for example, the output engine 537 may provide the output of the sub-matrix operation 501 (e.g., the output stored in the MRB 538d) to a specific destination used to store the partial results of the "distributed" matrix operation. For example, in the case of a distributed matrix operation, each partial result determined by each processing resource may be consolidated on a specific memory component, such as a specific HBM 540b of the matrix processing chip. For example, in some cases, each partial result determined by each cluster of the matrix processing chip may be consolidated on a specific HBM 540b of the matrix processing chip. Furthermore, these partial results may be stored on the HBM 540b using a specific arrangement that collectively forms the complete result of the matrix operation.

[0098] As another example, in some cases, the output engine 537 may feed back the output of the sub-matrix operation 501 (e.g., the output stored in the MRB 538d) to the MPU 534a, e.g., to enable the MPU 534a to use the output as an operand of the next sub-operation. In some cases, for example, the output of a sub-operation at one stage of the distributed matrix operation may be used as an input or operand to a sub-operation at another stage of the distributed matrix operation.

[0099] As another example, in some cases, the output engine 537 may provide the output of the sub-matrix operation 501 (e.g., the output stored in the MRB 538d) to another matrix processing resource, such as another matrix processing cluster on the same matrix processing chip, or to another matrix processing chip entirely. For example, in some cases, a distributed matrix operation may be distributed across multiple clusters of a matrix processing chip and / or across multiple matrix processing chips. Furthermore, in some cases, the output of a partial operation performed by a particular matrix processing resource may be used as an operand for another partial operation performed by a different processing resource.

[0100] In the illustrated example, the second matrix operation 502 may be performed in parallel with the first matrix operation 501. Moreover, the control flow of the second matrix operation 502 may be similar to the control flow described above for the first matrix operation 501. However, the second matrix operation 502 may be a different matrix operation having different matrix operands and results (e.g., performed using a different matrix routine), using different memory locations in the HBM 540 and / or MRB 538, and the second matrix operation 502 may be performed using a different MPU 534b and associated slice engine 536b.

[0101] 6 shows a flowchart 600 of an example embodiment of programmable matrix processing. Flowchart 600 may be implemented in some embodiments by components described throughout this disclosure (e.g., the matrix processing architecture of FIGS. 2A-2C and / or the programmable matrix processing engine of FIG. 5).

[0102] The flowchart may begin at block 602 by receiving a command to perform a matrix operation. The matrix operation may include an operation on one or more matrix operands. For example, the matrix operation may include any matrix-based arithmetic operation, including element-wise matrix operations, matrix multiplication, convolution, and / or any combination of such operations.

[0103] Additionally, in some embodiments, matrix operations may be used to implement computer vision artificial intelligence and machine learning functions in artificial neural networks. For example, in some embodiments, the matrix operations of block 602 may be associated with operations such as forward propagation, back propagation, and / or weight update operations within an artificial neural network.

[0104] The flowchart next proceeds to block 604, where matrix data may be retrieved from memory. The matrix data may be associated with one or more matrix operands of a matrix operation, for example. In some embodiments, the matrix data may be retrieved from a multidimensional memory. The multidimensional memory may be, for example, a memory component designed to efficiently store and retrieve matrix data of multiple dimensions (e.g., two dimensions). In some embodiments, the matrix data may be retrieved by executing one or more instructions to retrieve the matrix data from one or more memory locations of the multidimensional memory.

[0105] The flowchart then proceeds to block 606, where matrix operands may be obtained from the matrix data. In some embodiments, for example, the matrix operands may be obtained by slicing the matrix data to extract the matrix operands from the matrix data. Further, in some embodiments, the matrix operands may be obtained by executing one or more instructions to slice or extract the matrix operands from the matrix data.

[0106] The flowchart may then proceed to block 608, where a matrix routine associated with the matrix operation may be identified. For example, a matrix routine may be a programmable routine for a matrix processor that, when executed by the matrix processor, is designed to perform a particular matrix operation. For example, a matrix routine may include a set of instructions and / or commands supported by a particular matrix processor and designed to perform a desired matrix operation when executed by the matrix processor. In some embodiments, for example, a matrix processor may be designed to support a set of instructions and / or commands for performing various basic matrix operations. For example, a matrix processor may include instructions and / or commands for identifying memory locations of matrix operands, retrieving matrix operands from memory, and / or performing particular arithmetic operations or calculations on matrix operands, among other examples. In this manner, basic instructions and / or commands supported by a matrix processor may be used to program matrix routines for more complex matrix operations.

[0107] In some embodiments, a matrix routine memory may be used to store matrix routines, and the matrix routines may be retrieved from the matrix routine memory as needed. For example, to perform a desired matrix operation, the corresponding matrix routine may first be retrieved from the matrix routine memory. However, in some cases, a particular matrix routine may not already be stored in the matrix routine memory. Therefore, the particular matrix routine may need to be loaded onto the matrix routine memory. Therefore, in some cases, the particular matrix routine may first be retrieved from a host computing system and then stored in the matrix routine memory.

[0108] The flowchart may then proceed to block 610, where a matrix routine may be executed. In some embodiments, for example, the matrix routine may be executed by a matrix processor using one or more matrix operands. The flowchart may then proceed to block 612, where a result of the matrix operation may be obtained based on the matrix routine executed by the matrix processor. For example, in some cases, a particular matrix routine may return a result determined by a series of instructions and / or commands executed by the matrix processor.

[0109] At this point, the flowchart may be complete. However, in some embodiments, the flowchart may resume and / or certain blocks may be repeated. For example, in some embodiments, the flowchart may resume at block 602 to continue receiving and processing commands to perform matrix operations.

[0110] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various aspects of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions shown in the blocks need not occur in the order shown in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may be executed in the reverse order or in an alternative manner, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or acts, or by a combination of dedicated hardware and computer instructions.

[0111] The foregoing disclosure outlines features of several embodiments so that those skilled in the art may more fully appreciate various aspects of the present disclosure. Those skilled in the art will appreciate that this disclosure may readily be used as a basis for designing or modifying other processes and structures to serve the same purposes and / or achieve the same advantages of the embodiments introduced herein. Those skilled in the art will also appreciate that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that such constructions may result in various changes, substitutions, and alterations being made herein without departing from the spirit and scope of the present disclosure.

[0112] All or a portion of any hardware element disclosed herein may be readily provided in a system-on-chip (SoC), including a central processing unit (CPU) package. An SoC refers to an integrated circuit (IC) that integrates components of a computer or other electronic system onto a single chip. An SoC may include digital, analog, mixed-signal, and radio frequency functions, all of which may be provided on a single chip substrate. Other embodiments may include a multi-chip module (MCM), in which multiple chips are provided within a single electronic package and configured to communicate closely with each other through the electronic package. In various other embodiments, the computing functionality disclosed herein may be implemented in one or more silicon cores within an application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), and other semiconductor chips.

[0113] As used throughout this specification, the terms "processor" or "microprocessor" should be understood to include not only traditional microprocessors (such as Intel's industry leading x86 and x64 architectures), but also matrix processors, graphics processors, and any ASIC, FPGA, microcontroller, digital signal processor (DSP), programmable logic device, programmable logic array (PLA), microcode, instruction set, emulation processor or virtual machine processor, or any similar "Turing complete" device, combination of devices, or logic element (hardware or software) that enables the execution of instructions.

[0114] It should also be noted that in certain embodiments, some components may be omitted or integrated. In a general sense, the configurations shown in the figures should be understood as logical divisions, but the physical architecture may include various permutations, combinations, and / or hybrids of these elements. It is imperative to note that a myriad of possible design configurations may be used to achieve the operational objectives outlined herein. Thus, the associated infrastructure has a myriad of alternative configurations, design choices, device possibilities, hardware configurations, software implementations, and equipment options.

[0115] In a general sense, any suitably configured processor can execute instructions associated with data or microcode to perform the operations detailed herein. Any processor disclosed herein may transform elements or items (e.g., data) from one state or condition to another. In another example, some functions outlined herein may be implemented in fixed logic or programmable logic (e.g., software and / or computer instructions executed by a processor), and elements identified herein may be some type of programmable processor, programmable digital logic (e.g., field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), ASIC containing digital logic, software, code, electronic instructions, flash memory, optical disk, CD-ROM, DVD-ROM, magnetic or optical card, other type of machine-readable medium suitable for storing electronic instructions, or any suitable combination thereof.

[0116] In operation, as needed and based on particular needs, storage may store information in any suitable type of tangible, non-transitory storage medium (e.g., random access memory (RAM), read-only memory (ROM), field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), electrically erasable programmable ROM (EEPROM), or microcode), software, hardware (e.g., processor instructions or microcode), or any other suitable component, device, element, or object. Furthermore, information tracked, transmitted, received, or stored in a processor may be provided in any database, register, table, cache, queue, control list, or storage structure, all of which may be referenced in any suitable time frame, based on particular needs and implementation. Any memory or storage elements disclosed herein should be construed as being included within the broad terms “memory” and “storage,” as appropriate. Non-transitory storage medium herein is expressly intended to include any non-transitory dedicated or programmable hardware configured to provide the disclosed operations or to cause a processor to perform the disclosed operations. Non-transitory storage media also expressly includes hardware-coded instructions, and optionally, a processor having stored thereon hardware, firmware, or software-encoded microcode instructions or sequences.

[0117] Computer program logic implementing all or a portion of the functionality described herein may be embodied in various forms, including, but not limited to, hardware description languages, source code forms, computer-executable forms, machine instructions or microcode, programmable hardware, and various intermediate forms (e.g., forms generated by an HDL processor, assembler, compiler, linker, or locator). In one example, source code includes a series of computer program instructions implemented in various programming languages, such as object code, assembly language, or high-level languages ​​such as OpenCL, FORTRAN, C++, Java, or HTML for use with various operating systems or operating environments, or implemented in hardware description languages ​​such as Spice, Verilog, and VHDL. The source code defines and can use various data structures and communication messages. The source code may be in computer-executable form (e.g., via an interpreter) or may be converted (e.g., via a translator, assembler, or compiler) into computer-executable form or into an intermediate form such as bytecode. Where appropriate, any of the foregoing may be used to fabricate or describe suitable discrete or integrated circuits, whether sequential, combinational, state machines, or otherwise.

[0118] In one example, any number of the illustrated electrical circuits may be implemented on a board of an associated electronic device. The board may be a general circuit board that holds various components of the electronic device's internal electronic system and may also provide connectors for other peripherals. More specifically, the board may provide electrical connections by which other components of the system can communicate electrically. Any appropriate processor and memory may be suitably coupled to the board based on particular configuration needs, processing requirements, and computing design. Other components, such as external storage, additional sensors, audio / video display controllers, and peripheral devices, may be attached to the board via cables as plug-in cards or may be integrated into the board itself. As another example, the illustrated electrical circuits may be implemented as stand-alone modules (e.g., devices having associated components and circuits configured to perform specific applications or functions) or may be implemented as plug-in modules in the application-specific hardware of the electronic device.

[0119] It should be noted that, using numerous examples provided herein, interactions may be described with respect to two, three, four, or more electrical components. However, this is done for purposes of clarity and illustration only. It should be understood that systems may be integrated or reconfigured in any suitable manner. Along similar design alternatives, any of the components, modules, and elements shown in the figures may be combined into a variety of possible configurations, all of which are within the broad scope of this specification. In some cases, it may be easier to describe one or more of the functions for a given set of flows with reference to only a limited number of electrical elements. It should be understood that the electrical circuits shown and their teachings can be readily scaled to accommodate numerous components and more complex / sophisticated arrangements and configurations. Thus, the examples provided should not limit the scope or preclude the broad teachings of the electrical circuits as potentially applied to countless other architectures.

[0120] Numerous other changes, substitutions, variations, alterations, and modifications may be ascertainable to those skilled in the art, and it is intended that this disclosure encompass all such changes, substitutions, variations, alterations, and modifications as fall within the scope of the appended claims. Exemplary Embodiments

[0121] The following examples relate to embodiments described throughout this disclosure.

[0122] One or more embodiments may include an apparatus comprising a multidimensional memory and a plurality of processing elements including one or more matrix processors that perform a matrix operation on one or more matrix operands, the plurality of processing elements being configured to receive matrix data associated with the one or more matrix operands from the multidimensional memory, retrieve the one or more matrix operands from the matrix data, identify a matrix routine associated with the matrix operation, execute the matrix routine using the one or more matrix operands, and obtain a result of the matrix operation based on the executed matrix routine.

[0123] In one exemplary embodiment of the apparatus, the apparatus further comprises a matrix routine memory, the matrix routine memory including one or more matrix routines associated with one or more matrix operations.

[0124] In one exemplary embodiment of the apparatus, the plurality of processing elements are further configured to receive the matrix routine from a matrix routine memory.

[0125] In one exemplary embodiment of the apparatus, the plurality of processing elements are further configured to receive the matrix routine from the host computing system and store the matrix routine in the matrix routine memory.

[0126] In one exemplary embodiment of the apparatus, the matrix routine includes one or more instructions associated with matrix operations.

[0127] In one exemplary embodiment of the apparatus, one or more instructions of a matrix routine include an indication of one or more memory locations associated with one or more matrix operands.

[0128] In one exemplary embodiment of the apparatus, the processing elements that extract one or more matrix operands from the matrix data are further configured to slice the matrix data to extract the one or more matrix operands.

[0129] In one exemplary embodiment of the apparatus, the processing elements are further configured to execute one or more instructions to extract one or more matrix operands from the matrix data.

[0130] In one exemplary embodiment of the apparatus, the plurality of processing elements are further configured to execute one or more instructions to retrieve matrix data from one or more memory locations of the multidimensional memory.

[0131] In one exemplary embodiment of the apparatus, the matrix operations include one or more matrix multiplication operations.

[0132] In one exemplary embodiment of the apparatus, the matrix operations include one or more convolution operations.

[0133] In one exemplary embodiment of the apparatus, matrix operations are associated with operations in a neural network.

[0134] One or more embodiments may include a method comprising performing a matrix operation on one or more matrix operands, the performing the matrix operation including receiving matrix data associated with the one or more matrix operands from a multidimensional memory; retrieving the one or more matrix operands from the matrix data; identifying a matrix routine associated with the matrix operation; executing the matrix routine with a matrix processor using the one or more matrix operands; and obtaining a result of the matrix operation based on the matrix routine executed by the matrix processor.

[0135] In one exemplary embodiment of the method, the method further comprises receiving a matrix routine from a matrix routine memory, the matrix routine memory including one or more matrix routines associated with one or more matrix operations.

[0136] In one exemplary embodiment of the method, the matrix routine includes one or more instructions associated with a matrix operation.

[0137] In one exemplary embodiment of the method, one or more instructions of the matrix routine include an indication of one or more memory locations associated with one or more matrix operands.

[0138] One or more embodiments may include a system comprising a plurality of memory elements including a multidimensional memory and a plurality of processing elements that perform a matrix operation on one or more matrix operands, the plurality of processing elements including a host processor and one or more matrix processing chips, the plurality of processing elements being configured to: receive matrix data associated with the one or more matrix operands from the multidimensional memory; retrieve the one or more matrix operands from the matrix data; identify a matrix routine associated with the matrix operation; execute the matrix routine using the one or more matrix operands; and obtain a result of the matrix operation based on the executed matrix routine.

[0139] In one exemplary embodiment of the system, each matrix processing chip includes multiple matrix processing clusters.

[0140] In one exemplary embodiment of the system, each matrix processing cluster includes multiple matrix processing units.

[0141] In one exemplary embodiment of the system, each matrix processing cluster includes multiple memory resource blocks.

[0142] One or more embodiments may include at least one machine-accessible storage medium having instructions stored thereon that, when executed by a machine, cause the machine to perform a matrix operation on one or more matrix operands, the instructions causing the machine to perform the matrix operation further causing the machine to receive matrix data associated with the one or more matrix operands from a multidimensional memory, retrieve the one or more matrix operands from the matrix data, identify a matrix routine associated with the matrix operation, execute the matrix routine with a matrix processor using the one or more matrix operands, and obtain a result of the matrix operation based on the matrix routine executed by the matrix processor.

[0143] In one exemplary embodiment relating to the storage medium, the instructions further cause the machine to receive a matrix routine from a matrix routine memory, the matrix routine memory including one or more matrix routines associated with one or more matrix operations.

[0144] In one exemplary embodiment of a storage medium, a matrix routine includes one or more instructions associated with matrix operations.

[0145] In one exemplary embodiment of a storage medium, a matrix routine includes an indication of one or more memory locations associated with one or more matrix operands.

[0146] In one exemplary embodiment of the storage medium, the instructions that cause the machine to receive the matrix data from the multidimensional memory further cause the machine to obtain the matrix data from one or more memory locations of the multidimensional memory.

[0147] One or more embodiments may include an apparatus having means for carrying out any of the methods described above.

[0148] One or more embodiments may include at least one machine-accessible storage medium having instructions stored thereon that, when executed by a machine, cause the machine to perform any of the methods or implement any of the apparatuses described above. [Item 1] a programmable matrix routine memory having circuitry for storing a plurality of programmable matrix routines, each of the plurality of programmable matrix routines including instructions for performing a corresponding one of a plurality of matrix operations, the programmable matrix routine memory being programmed with the plurality of programmable matrix routines executed to perform the plurality of matrix operations; one or more matrix processing units having circuitry for performing matrix calculations; A controller having circuitry, receiving a command to perform a particular matrix operation of the plurality of matrix operations; identifying a programmable matrix routine corresponding to the particular matrix operation, the programmable matrix routine being identified from the plurality of programmable matrix routines; receiving the programmable matrix routine from the programmable matrix routine memory; executing the programmable matrix routine, using the one or more matrix processing units to perform one or more matrix calculations associated with executing the programmable matrix routine; a controller having circuitry for determining a result of the particular matrix operation, the result being determined based on execution of the programmable matrix routine; Equipped with Matrix processing circuit. [Item 2] The circuitry for executing the programmable matrix routine comprises: receiving one or more matrix operands associated with the particular matrix operation; performing the one or more matrix calculations on the one or more matrix operands using the one or more matrix processing units; Item 1. The matrix processing circuit according to item 1. [Item 3] the matrix processing circuit further comprises a plurality of memory resource blocks; The circuitry for receiving the one or more matrix operands associated with the particular matrix operation further comprises: receiving matrix data associated with the one or more matrix operands from a memory; extracting the one or more matrix operands from the matrix data; storing the one or more matrix operands in one or more of the plurality of memory resource blocks; Item 2. The matrix processing circuit according to item 2. [Item 4] the programmable matrix routine has a set of instructions for performing the particular matrix operation; The set of instructions may include: a first subset of instructions for receiving the matrix data from the memory and extracting the one or more matrix operands from the matrix data; a second subset of instructions for performing the one or more matrix calculations on the one or more matrix operands; Including, Item 3. The matrix processing circuit according to item 3. [Item 5] the one or more matrix processing units include a plurality of matrix processing units; the matrix processing circuit further includes a matrix processing cluster, the matrix processing cluster having the plurality of matrix processing units and the plurality of memory resource blocks; Item 3. The matrix processing circuit according to item 3 or 4. [Item 6] The controller receiving the plurality of programmable matrix routines from a host computing system; storing the plurality of programmable matrix routines in the programmable matrix routine memory; further comprising circuitry for 6. A matrix processing circuit according to any one of items 1 to 5. [Item 7] 7. The matrix processing circuit of any one of items 1 to 6, wherein the one or more matrix calculations include one or more matrix multiplication calculations. [Item 8] 8. The matrix processing circuit of claim 1, wherein the one or more matrix calculations include one or more convolution calculations. [Item 9] 9. The matrix processing circuit of any one of items 1 to 8, wherein the specific matrix operation is associated with an operation in an artificial neural network. [Item 10] a processor for executing an application, the execution of the application comprising a plurality of matrix operations; a matrix processing circuit that performs the plurality of matrix operations, The matrix processing circuit a programmable matrix routine memory including circuitry for storing a plurality of programmable matrix routines, each of the plurality of programmable matrix routines including instructions for performing a corresponding one of the plurality of matrix operations, the programmable matrix routine memory being programmed with the plurality of programmable matrix routines executed to perform the plurality of matrix operations; one or more matrix processing clusters, each of the one or more matrix processing clusters including a plurality of matrix processing units, the plurality of matrix processing units having circuitry for performing matrix calculations; A controller having circuitry, the circuitry comprising: receiving a command to perform a particular matrix operation of the plurality of matrix operations; identifying a programmable matrix routine corresponding to the particular matrix operation, the programmable matrix routine being identified from the plurality of programmable matrix routines; receiving the programmable matrix routine from the programmable matrix routine memory; distributing execution of said programmable matrix routines across said one or more matrix processing clusters; a controller that determines a result of the particular matrix operation, the result being determined based on execution of the programmable matrix routine; having system. [Item 11] The one or more matrix processing clusters a plurality of matrix processing clusters; a multidimensional mesh interconnected to communicatively couple to the plurality of matrix processing clusters; Item 11. The system according to item 10, comprising: [Item 12] Each of the one or more matrix processing clusters comprises: receiving one or more matrix operands associated with the particular matrix operation; performing a plurality of matrix calculations on the one or more matrix operands using the plurality of matrix processing units; further comprising circuitry for Item 12. The system according to item 10 or 11. [Item 13] the matrix processing circuit further comprises one or more memory modules; each of the one or more matrix processing clusters further includes a plurality of memory resource blocks; The circuitry for receiving the one or more matrix operands associated with the particular matrix operation further comprises: receiving matrix data associated with the one or more matrix operands from the one or more memory modules; extracting the one or more matrix operands from the matrix data; storing the one or more matrix operands in one or more of the plurality of memory resource blocks; Item 13. The system according to item 12. [Item 14] the programmable matrix routine has a set of instructions for performing the particular matrix operation; The set of instructions may include: a first subset of instructions for receiving the matrix data from the one or more memory modules and extracting the one or more matrix operands from the matrix data; a second subset of instructions for performing the plurality of matrix calculations on the one or more matrix operands; Including, Item 14. The system according to item 13. [Item 15] The controller receiving the plurality of programmable matrix routines from the processor; storing the plurality of programmable matrix routines in the programmable matrix routine memory; further comprising circuitry for 15. The system according to any one of items 10 to 14. [Item 16] At least one non-transitory machine-accessible storage medium storing instructions that, when executed on a machine, cause the machine to: receiving a command to perform a particular matrix operation of the plurality of matrix operations; identifying a programmable matrix routine corresponding to the particular matrix operation, the programmable matrix routine being identified from a plurality of programmable matrix routines, each of the plurality of programmable matrix routines having programmable instructions for performing a corresponding matrix operation of the plurality of matrix operations; receiving the programmable matrix routines from a programmable matrix routine memory, the programmable matrix routine memory being programmed with the programmable matrix routines that are executed to perform the matrix operations; executing the programmable matrix routine and performing one or more matrix calculations associated with the execution of the programmable matrix routine using one or more matrix processing units, the one or more matrix processing units including circuitry for performing the matrix calculations; determining a result of the particular matrix operation, the result being determined based on execution of the programmable matrix routine; Non-transitory machine-accessible storage media. [Item 17] The instructions for causing the machine to execute the programmable matrix routine may further cause the machine to: receiving one or more matrix operands associated with the particular matrix operation; causing the one or more matrix processing units to perform the one or more matrix calculations on the one or more matrix operands; Item 17. The non-transitory machine-accessible storage medium of item 16. [Item 18] The instructions for causing the machine to receive the one or more matrix operands associated with the particular matrix operation may further cause the machine to: receiving matrix data associated with the one or more matrix operands from a memory; causing said one or more matrix operands to be derived from said matrix data; storing the one or more matrix operands in one or more memory resource blocks; Item 18. The non-transitory machine-accessible storage medium of item 17. [Item 19] the programmable matrix routine has a set of instructions for performing the particular matrix operation; The set of instructions may include: a first subset of instructions for receiving the matrix data from the memory and extracting the one or more matrix operands from the matrix data; a second subset of instructions for performing the one or more matrix calculations on the one or more matrix operands; Including, Item 19. The non-transitory machine-accessible storage medium of item 18. [Item 20] The instructions may further cause the machine to: receiving the plurality of programmable matrix routines from a host computing system; storing the plurality of programmable matrix routines in the programmable matrix routine memory; 20. The non-transitory machine-accessible storage medium of any one of items 16 to 19. [Item 21] receiving a command to perform a particular matrix operation of the plurality of matrix operations; identifying a programmable matrix routine corresponding to the particular matrix operation, the programmable matrix routine being identified from a plurality of programmable matrix routines, each of the plurality of programmable matrix routines having programmable instructions for performing a corresponding matrix operation of the plurality of matrix operations; receiving the programmable matrix routines from a programmable matrix routine memory, the programmable matrix routine memory being programmed with the programmable matrix routines to be executed to perform the matrix operations; executing the programmable matrix routine, wherein one or more matrix calculations associated with executing the programmable matrix routine are performed using one or more matrix processing units, the one or more matrix processing units including circuitry for performing matrix calculations; determining a result of the particular matrix operation, the result being determined based on execution of the programmable matrix routine; A method for providing the above. [Item 22] Executing the programmable matrix routine comprises: receiving one or more matrix operands associated with the particular matrix operation; performing the one or more matrix calculations on the one or more matrix operands using the one or more matrix processing units; having Item 21. The method according to item 21. [Item 23] receiving the one or more matrix operands associated with the particular matrix operation, receiving matrix data associated with the one or more matrix operands from a memory; deriving said one or more matrix operands from said matrix data; storing the one or more matrix operands in one or more memory resource blocks; Including, Item 23. The method according to item 22. [Item 24] the programmable matrix routine comprises a set of instructions for performing the particular matrix operation; The set of instructions may include: a first subset of instructions for receiving the matrix data from the memory and extracting the one or more matrix operands from the matrix data; a second subset of instructions for performing the one or more matrix calculations on the one or more matrix operands; having Item 24. The method according to item 23. [Item 25] receiving the plurality of programmable matrix routines from a host computing system; storing said plurality of programmable matrix routines in said programmable matrix routine memory; Further provided with 25. The method according to any one of items 21 to 24.

Claims

1. A plurality of processing units that collectively perform matrix multiplication operations with matrix data based on at least one matrix multiplication instruction from a specific programmable matrix routine among a plurality of programmable matrix routines that include instructions for performing matrix operations by performing programmable matrix processing at least partially in parallel, each processing unit of the plurality of processing units processing a portion of the matrix data to perform a corresponding sub-matrix operation; a plurality of memories, each memory storing a portion of the matrix data for processing by a corresponding one of the plurality of processing units; a plurality of interconnects, a subset of the plurality of interconnects coupling each processing unit of the plurality of processing units to a plurality of adjacent processing units, at least one of the plurality of processing units transmitting submatrix data to a first adjacent processing unit and receiving submatrix data from a second adjacent processing unit through a corresponding interconnect of the plurality of interconnects; a first controller, wherein in response to the first controller, the plurality of processing units collectively perform the matrix multiplication operation in response to the at least one matrix multiplication instruction specifying a first input matrix A and a second input matrix B, and the plurality of processing units generate an output matrix C by multiplying the first input matrix A and the second input matrix B; a plurality of second controllers, each of which is associated with a processing unit of the plurality of processing units, and each of which retrieves from a system memory the portion of the matrix data to be processed by a corresponding processing unit of the plurality of processing units, and stores the portion of the matrix data in a corresponding memory of the plurality of memories; An integrated circuit (IC) chip comprising:

2. The first controller includes a microprocessor. The IC chip according to claim 1 .

3. The plurality of processing units includes a plurality of processing clusters.

3. The IC chip according to claim 1 or 2.

4. and a plurality of local controllers, each local controller controlling a matrix multiplication operation within a corresponding processing cluster. The IC chip according to claim 3 .

5. a memory interface coupling the plurality of memories to a high bandwidth memory (HBM); The IC chip according to claim 1 .

6. The at least one matrix multiplication instruction is one of a plurality of instructions of the particular programmable matrix routine, the particular programmable matrix routine being executed by one or more of the plurality of processing units. The IC chip according to claim 1 .

7. The particular programmable matrix routine includes a distributed matrix multiplication routine, the distributed matrix multiplication routine being executed by a plurality of the plurality of processing units. The IC chip according to claim 6.

8. The instructions of the particular programmable matrix routine are executed to perform one or more convolution operations. The IC chip according to claim 6.

9. The specific programmable matrix routines are associated with operations in neural networks. The IC chip according to claim 8.

10. a host interface coupling the plurality of processing units to the first controller; The IC chip according to any one of claims 1 to 9.

11. The at least one of the plurality of processing units transmits the submatrix data to the first adjacent processing unit and receives the submatrix data from the second adjacent processing unit while performing the submatrix operation. The IC chip according to any one of claims 1 to 10.

12. When executed by a machine, the machine performing programmable matrix processing at least partially in parallel to collectively perform, on a plurality of processing units having matrix data, matrix multiplication operations based on a matrix multiplication instruction from a particular programmable matrix routine among a plurality of programmable matrix routines including instructions for performing matrix operations, wherein each processing unit of the plurality of processing units processes a portion of the matrix data to perform a corresponding sub-matrix operation, the collectively performing the matrix multiplication operations being performed under control of a first controller in response to the matrix multiplication instruction specifying a first input matrix A and a second input matrix B, and the plurality of processing units multiplying the first input matrix A by the second input matrix B to generate an output matrix C; storing said portions of said matrix data in respective memories of a plurality of memories for processing by corresponding ones of said plurality of processing units; transmitting and receiving submatrix data by each processing unit to and from one or more adjacent processing units, the transmitting and receiving being performed over a corresponding interconnect of the plurality of interconnects; retrieving, by a second controller, the portions of the matrix data to be processed by corresponding ones of the plurality of processing units from a system memory and storing the portions of the matrix data in corresponding ones of the plurality of memories; A program that performs an action including:

13. The transmitting and receiving step includes transmitting and receiving, by each processing unit, the submatrix data to and from the one or more adjacent processing units while performing the submatrix operation. The program according to claim 12.

14. A machine-readable medium storing the program according to claim 12 or 13.

Images