Programmable matrix processing engine

Abstract

This application discloses a programmable matrix processing engine. In one embodiment, a matrix operation can be performed on one or more matrix operands. For example, matrix data can be received from a multidimensional memory, where the matrix data is associated with one or more matrix operands. The one or more matrix operands can be extracted from the matrix data. A matrix routine associated with the matrix operation can be identified. The matrix routine can be executed on a matrix processor using the one or more matrix operands. A result of the matrix operation can be obtained based on the matrix routine executed by the matrix processor.

Description

[0001] This application is a divisional application of the invention patent application application with application number 201810002887.6, filed on January 2, 2018, and entitled "Programmable Matrix Processing Engine". Technical Field

[0002] The present disclosure relates generally to the field of computer processing and, more particularly, but not exclusively, to programmable matrix processing. Background Art

[0003] Matrix operations (such as matrix multiplication and convolution) can be highly processor-intensive or memory-intensive because they often involve complex operations on large multidimensional matrix operands. Accordingly, the performance of complex matrix operations can be limited by processing and / or memory latency. As matrix operations are increasingly used in a variety of applications and with growing data sets (from graphics and image processing to machine learning and artificial intelligence), the demand for high-performance and flexible processing of matrix operations is growing. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The present disclosure is best understood by reading the following detailed description in conjunction with the accompanying drawings. It is emphasized that, in accordance with standard industry practice, various features are not necessarily drawn to scale and are for illustration purposes only. Where scale is shown, either explicitly or implicitly, it is merely to provide an illustrative example. In other embodiments, the dimensions of various features may be arbitrarily expanded or reduced for clarity of discussion.

[0005] Figure 1 A schematic diagram illustrating an example computing system in accordance with certain embodiments is shown.

[0006] Figure 2A-2C A block diagram illustrating an example embodiment of a matrix processing architecture.

[0007] Figure 3 and Figure 4 A block diagram illustrating an example embodiment of a computer processor.

[0008] Figure 5 An example embodiment of a programmable matrix processing engine is shown.

[0009] Figure 6 A flow chart illustrating an example embodiment of programmable matrix processing. DETAILED DESCRIPTION

[0010] The following disclosure provides many different embodiments or examples for implementing the different features of the present disclosure. Specific examples of components and arrangements will be described below to simplify the present disclosure. Of course, these are merely examples and are not intended to be limiting. In addition, the present disclosure may repeat reference numbers and / or letters in various examples. This repetition is merely for simplicity and clarity and does not in itself define the relationship between the various embodiments and / or configurations discussed. Different embodiments may have different advantages, and no particular advantage is necessarily required for any embodiment.

[0011] Matrix processing operations (e.g., linear algebra operations involving matrices and / or vector operands) are widely used in computing systems, from graphics processing to machine learning and artificial intelligence, among other examples. For example, complex matrix operations can be used to implement artificial neural networks that provide artificial intelligence and machine learning capabilities, including computer vision, autonomous navigation, speech and audio recognition, and natural language processing, among other examples. These complex matrix operations (e.g., matrix multiplication and convolution) can be used to implement basic operations of neural networks, such as forward propagation, backward propagation, and weight updates. However, these matrix operations can be highly processor-intensive and memory-intensive because they involve complex operations on large multidimensional matrix operands. Accordingly, the performance of these matrix operations can be limited by processing and / or memory latency. In addition, these rigid matrix operations are often implemented without any flexibility to implement new types or variants of matrix operations and / or modify the behavior of existing operations. As matrix operations are increasingly used in various applications and with growing data sets (such as artificial intelligence and machine learning), the demand for high-performance processing and flexible implementation of matrix operations is growing.

[0012] Existing matrix processing methods suffer from various inefficiencies, especially when used to implement artificial intelligence and machine learning in artificial neural networks. For example, although central processing units (CPUs) can be used to perform matrix operations, many CPU architectures are designed for low arithmetic intensity operations (i.e., a low ratio of arithmetic operations to memory operations) and are therefore not designed for efficient execution of matrix operations. In addition, many CPU architectures utilize complex local or cache memory management routines, which increase processing overhead and execution complexity for operations involving large matrix operands. Graphics processing units (GPUs) can also be used to perform matrix operations. However, GPUs are generally designed for high-precision calculations and provide a level of accuracy that is not required for certain matrix operations, thereby reducing the amount of matrix operations that can be performed. Accordingly, existing matrix operation methods are inefficient for certain matrix operations (such as matrix multiplication or convolution operations involving large matrix operands and / or matrix operands with certain dimensions, etc.). In addition, existing methods may not be able to be efficiently scaled to perform these matrix operations in parallel across additional processing resources. Thus, existing approaches do not achieve 100% processing efficiency when scaling and / or distributing these matrix operations.Furthermore, existing approaches are generally rigid and inflexible, with limited or no ability to define new matrix operations, modify existing matrix operations, and so on.

[0013] The matrix processing functionality described throughout this disclosure provides a flexible or "programmable" way to define and / or implement specific matrix operations. For example, certain embodiments may include a matrix processor that can execute programmable matrix routines. A matrix routine, for example, may be a programmable routine designed to perform a specific matrix operation when executed by the matrix processor. For example, a matrix routine may include a series of instructions and / or commands that are supported by a specific matrix processor and designed to perform the desired matrix operation when executed by the matrix processor. For example, in some embodiments, the matrix processor may be designed to support a set of instructions and / or commands for performing various basic operations. In this way, matrix routines for more complex matrix operations can be programmed using the basic instructions and / or commands supported by the matrix processor. In some embodiments, these matrix routines may be stored in a matrix routine memory associated with the matrix processor. Subsequently, when a specific matrix operation needs to be performed, the matrix processor may retrieve the corresponding matrix routine from the matrix routine memory and then execute the instructions and / or commands of the routine to perform the desired matrix operation.

[0014] The programmable matrix processing functionality described throughout this disclosure provides numerous technical advantages, including alleviating the inefficiencies of existing approaches and enabling the efficient definition and implementation of flexible matrix operations using programmable matrix routines. These programmable matrix routines enable a wide variety of matrix processing functions to be implemented programmatically on a matrix processor rather than via inefficient, time-consuming, and expensive hardware-based implementations.

[0015] Example embodiments that may be used to implement the matrix processing functionality of the present disclosure will now be described in more detail with reference to the accompanying drawings.

[0016] Figure 1 A schematic diagram of an example computing system 100 is shown, in accordance with some embodiments.

[0017] In some embodiments, the matrix processing functionality described throughout this disclosure may be implemented in system 100. The matrix processing functionality may be used in system 100 for a wide range of applications and / or use cases involving matrix operations, ranging from graph processing to machine learning and artificial intelligence, among other examples. For example, in some embodiments, the matrix processing functionality may be used to implement artificial intelligence and machine learning in artificial neural networks. Furthermore, the matrix processing functionality may be implemented by any component of system 100. For example, in the illustrated embodiment, system 100 includes edge devices 110, cloud services 120, matrix processing nodes 130, and network 150. Matrix processing nodes 130 may include any component or device with matrix processing functionality, including any component of system 100. For example, matrix processing nodes 130 may include cloud services 120 and / or servers (e.g., application servers in a data center) that implement matrix processing functionality, edge devices 110 (e.g., end-user devices 112, IoT devices 114, gateways 116) that implement matrix processing functionality, and so forth. These various components of system 100 are discussed further below.

[0018] Edge devices 110 may include any equipment and / or devices deployed or connected near the "edge" of communication system 100. Edge devices 110 may communicate with each other and / or other remote networks and services (e.g., cloud services 120) via one or more networks and / or communication protocols (such as network 150). In some embodiments, certain edge devices 110 may include matrix processing functionality as described throughout this disclosure, and thus, they may serve as matrix processing nodes 130. In the illustrated embodiment, edge devices 110 include end-user devices 112 (e.g., desktops, laptops, mobile devices), Internet of Things (IoT) devices 114, and gateways and / or routers 116, to name a few.

[0019] End device 112 may include any device that allows or facilitates user interaction with computing system 100, including, for example, desktop computers, laptops, tablets, mobile phones and other mobile devices, and wearable devices (e.g., smart watches, smart glasses, headphones), among other such examples.

[0020] The IoT device 114 may include any device capable of communicating and / or participating in an Internet of Things (IoT) system or network. An IoT system may refer to an ad-hoc system and network composed of multiple different devices (e.g., IoT device 114) that interoperate or collaborate for a specific application or use case. As more and more products and equipment evolve to become "smart," such ad-hoc systems are emerging, meaning that they are controlled or monitored by a computer processor and are able to communicate with other devices. For example, the IoT device 114 may include a computer processor and / or a communication interface to allow interoperability with other components of the system 100 (such as with the cloud service 120 and / or with other edge devices 110). The IoT device 114 may be a "greenfield" device that has IoT capabilities developed from scratch, or a "brownfield" device that is created by integrating IoT capabilities into existing traditional devices that were not originally developed with IoT capabilities. For example, in some cases, IoT devices 114 may be built from sensors and communication modules integrated into or attached to “things,” such as equipment, toys, tools, vehicles, living things (e.g., plants, animals, people), etc. Alternatively or additionally, certain IoT devices 114 may rely on intermediary components (such as edge gateways or routers 116) to communicate with various components of system 100.

[0021] The IoT devices 114 may include various types of sensors for monitoring, detecting, measuring, and generating sensor data and signals associated with characteristics of their environments. For example, a given sensor may be configured to detect one or more corresponding characteristics, such as movement, weight, physical contact, temperature, wind, noise, light, position, humidity, radiation, liquid, specific chemical compounds, battery life, wireless signals, computer communications, and bandwidth, among other examples. Sensors may include physical sensors (e.g., physical monitoring components) and virtual sensors (e.g., software-based monitoring components). The IoT devices 114 may also include actuators for performing various actions in their corresponding environments. For example, actuators may be used to selectively activate certain functions, such as switching the power or operation of a security system (e.g., alarms, cameras, locks) or a home appliance (e.g., an audio system, lighting, HVAC equipment, a garage door), among other examples.

[0022] Indeed, the present disclosure contemplates the use of a potentially unlimited range of IoT devices 114 and associated sensors / actuators. The IoT devices 114 may include any type of equipment and / or devices associated with any type of system 100 and / or industry, including transportation (e.g., automotive, aviation), industrial manufacturing, energy (e.g., power plants), telecommunications (e.g., Internet, cellular, and television service providers), medical (e.g., healthcare, pharmaceuticals), food processing, and / or retail industries, among others. For example, in the transportation industry, the IoT devices 114 may include equipment and devices associated with an aircraft, automobile, or ship, such as navigation systems, autonomous flight or driving systems, traffic sensors and controllers, and / or any internal mechanical or electrical components (e.g., engines) monitored by sensors. IoT devices 114 may also include equipment, devices, and / or infrastructure associated with industrial manufacturing and production, shipping (e.g., cargo tracking), communication networks (e.g., gateways, routers, servers, cellular towers), server farms, power plants, wind farms, oil and gas pipelines, water treatment and distribution, wastewater collection and treatment, and weather monitoring (e.g., temperature, wind, and humidity sensors), among other examples. IoT devices 114 may also include, for example, any type of "smart" device or system, such as smart entertainment systems (e.g., televisions, audio systems, electronic gaming systems), smart home or office appliances (e.g., heating, ventilation, and air conditioning (HVAC) units, refrigerators, washers, dryers, coffee makers), power control systems (e.g., automatic electrical, light, and HVAC control), security systems (e.g., alarms, locks, cameras, motion detectors, fingerprint scanners, facial recognition systems), and other home automation systems, among other examples. IoT devices 114 may be statically located, such as mounted on a building, wall, floor, ground, light pole, sign, water tower, or any other fixed or static structure. IoT devices 114 may also be mobile, such as devices in vehicles or aircraft, drones, packages (e.g., for tracking shipments), mobile devices, and wearable devices, among other examples. Furthermore, IoT devices 114 may also be any type of edge device 110, including end-user devices 112 and edge gateways and routers 116.

[0023] Edge gateways and / or routers 116 can be used to facilitate communications to and from edge devices 110. For example, gateways 116 can provide communications capabilities to existing legacy devices (e.g., "brownfield" IoT devices) that were not originally developed with such capabilities. Gateways 116 can also be used to extend the geographic range of edge devices 110 with short-range, proprietary, or otherwise limited communication capabilities (such as IoT devices 114 with Bluetooth or ZigBee communication capabilities). For example, gateways 116 can act as an intermediary between IoT devices 114 and remote networks or services by using their native communication capabilities (e.g., Bluetooth, ZigBee) to provide fronthaul to IoT devices 114 and backhaul to other networks 150 and / or cloud services 120 using another wired or wireless communication medium (e.g., Ethernet, WiFi, cellular). In some embodiments, gateways 116 can be implemented as dedicated gateway devices, or as general-purpose devices (such as another IoT device 114, an end-user device 114, or another type of edge device 110).

[0024] In some instances, the gateway 116 may implement certain network management and / or application functions (e.g., IoT management and / or IoT application functions for the IoT devices 114), alone or in conjunction with other components, such as the cloud service 120 and / or other edge devices 110. For example, in some embodiments, configuration parameters and / or application logic may be pushed to or pulled from the gateway device 116, thereby allowing IoT devices 114 within range or proximity of the gateway 116 to be configured for a particular IoT application or use case.

[0025] Cloud services 120 may include services hosted remotely over network 150 or in the "cloud." For example, in some embodiments, cloud services 120 may be hosted remotely on a server (e.g., an application server or database server) in a data center. Cloud services 120 may include any service that may be utilized or available to edge devices 110, including, but not limited to, data storage, computing services (e.g., data analysis, search, diagnostics, and fault management), security services (e.g., monitoring, alerting, user authentication, etc.), mapping and navigation, geolocation services, network or infrastructure management, IoT applications and management services, payment processing, audio and video streaming, messaging, social networking, news and weather, and so forth. In some embodiments, certain cloud services 120 may include the matrix processing functionality described throughout this disclosure and, therefore, may serve as matrix processing nodes 130.

[0026] Generally speaking, edge devices 100 (and IoT devices 114 in particular) can generate extremely large volumes and varieties of data. IoT edge devices 114 typically offload this data to the cloud for processing and / or storage (e.g., by cloud services 120). However, cloud services 120 may not necessarily be well-suited to handling the rapidly increasing volume, variety, and velocity of data generated by IoT devices 114 and other edge devices 110. For example, cloud-based processing may not be ideal in certain situations, such as when processing time-sensitive or highly confidential data, or when facing network bandwidth constraints, to name a few. In some embodiments, cloud services can utilize "edge"-based processing using edge devices 110 to improve the performance of cloud services. Edge processing involves processing certain data at the edge of the network (e.g., using edge devices 110) (near the edge of the network, where the data is generated) rather than simply funneling large amounts of data to the cloud for processing and storage. Some data may still be sent to the cloud as appropriate, such as for deeper analysis and / or long-term storage. Edge processing can be used to compensate for deficiencies in cloud-based processing (e.g., when cloud-based processing is inefficient, ineffective, and / or insecure) and thereby improve the handling of the increasing volume, variety, and velocity of data generated by IoT devices 114 and / or other edge devices 110. For example, in some cases, processing data close to the source of the data (e.g., at the edge of the network) rather than in the cloud can improve performance and / or avoid system failures or disasters. Edge processing can also conserve network bandwidth, which is particularly beneficial when faced with bandwidth constraints and / or limited network connectivity.

[0027] In some embodiments, the edge devices 110 that provide edge-based processing for the cloud services 120 may be collectively referred to as "fog" because they are used to extend the "cloud" to the edge of the network, thereby forming a "fog" on the edge of the network. In some embodiments, the devices 110 in the "fog" may be connected to and / or communicate with each other using, for example, an interconnection standard or protocol. For example, in some embodiments, device interconnection may be implemented using the Open Interconnect Consortium (OIC) Standard Specification 1.0, which was developed by the Open Connectivity Foundation. TM Released on December 23, 2015, it enables devices to discover and connect to each other. Another available interconnection protocol is Thread, a networking protocol for Internet of Things (IoT) devices used in "smart" home automation and similar deployments, which has been developed by a consortium of organizations named "Thread Group." Other interconnection protocols may also be used, including, for example, the Optimal Link State Routing (OLSR) protocol or a better approach to mobile ad hoc networking (BATMAN), among other examples.

[0028] The network 150 may be used to facilitate communications between components of the computing system 100. For example, edge devices 110 (such as end-user devices 112 and IoT devices 114) may use the network 150 to communicate with each other and / or access one or more remote cloud services 120. The network 150 may include any number or types of communication networks, including, for example, a local area network, a wide area network, a public network, the Internet, a cellular network, a Wi-Fi network, a short-range network (e.g., Bluetooth or ZigBee), and / or any other wired or wireless network or communication medium.

[0029] Any, all, or some of the computing devices of system 100 may be suitable for executing any operating system, including Linux or other UNIX-based operating systems, Microsoft Windows, Windows Server, MacOS, Apple iOS, Google Android, or any custom and / or proprietary operating system, as well as virtual machines suitable for virtualizing the execution of a particular operating system.

[0030] although Figure 1 Described as comprising or relating to multiple elements, but not Figure 1 All elements shown within the system 100 of will be used in each alternative implementation of the present disclosure. In addition, one or more of the elements described in conjunction with the example of FIG. 1 may be located outside the system 100, and in other instances, certain elements may be included within or as part of one or more of the other described elements and other elements not described in the illustrated implementation. In addition, Figure 1 Some of the elements shown may be combined with other components and used for alternative or additional purposes besides those described herein.

[0031] Example Matrix Processing Architecture

[0032] Figure 2A-2C A block diagram illustrating an example embodiment of a matrix processing architecture.

[0033] In some embodiments, the matrix processing functionality described throughout this disclosure may utilize a matrix processing architecture such as Figure 2A-2C Matrix processing architecture (such as Figure 2A - Figure 2CThe Matrix Processing Architecture of FIG. 1 may be implemented and used in various systems, devices, and / or components, such as those described throughout this disclosure, including system 100 of FIG. 1 and / or any of its associated components (e.g., cloud services 120 / data center servers, edge devices 110, matrix processing nodes 130). In some embodiments, Figure 2A-2C The matrix processing architecture of NVIDIA can be used to implement artificial intelligence and machine learning in neural networks. Figure 2A-2C The matrix processing architecture shown in is only one example embodiment for performing the matrix processing functions described throughout this disclosure. Other embodiments may use different types, arrangements, and / or numbers of components. For example, other embodiments may include any number of matrix processing chips 220, matrix processing clusters 230, matrix processing units (MCUs) 234, high bandwidth memory (HBM) modules 240, and / or memory resource blocks (MRBs) 238. In addition, Figure 2A-2C All or part of any component of the Matrix Processing architecture (e.g., Matrix Processing System 200, Matrix Processing Chip 220, and / or any component of Matrix Processing Cluster 230) may be implemented as separate or standalone components or chips, or may be integrated with other components or chips, such as a System on a Chip (SoC) that integrates various computer components into a single chip.

[0034] Figure 2A A block diagram of an example embodiment of a matrix processing system 200 is shown. In the illustrated embodiment, the matrix processing system 200 includes a host processor 260, a host memory 270, matrix processing resources 210, and an interconnect bus 280.

[0035] Host processor 260 may be configured to control and / or manage matrix processing system 200. For example, in some embodiments, host processor 260 may use matrix processing resources 210 to perform complex matrix operations. Host processor 260 may be any processing resource capable of controlling and / or managing the matrix processing functions of matrix processing system 200. For example, in some embodiments, host processor 260 may use Figure 3 and Figure 4 Matrix processing system 200 is implemented as a separate or independent component communicatively coupled to matrix processing resource 210. In some embodiments, host processor 260 may be a separate or independent component communicatively coupled to matrix processing resource 210. Alternatively, in other embodiments, host processor 260 and host processing resource 210 may be integrated into the same component or chip. For example, in some embodiments, the components of matrix processing system 200 (including host processor 260 and matrix processing resource 210) may be implemented as a system on a chip (SoC).

[0036] Host memory 270 may include any type or combination of volatile and / or non-volatile memory. Examples of volatile memory include various types of random access memory (RAM), such as dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and static random access memory (SRAM), among others. Examples of non-volatile memory include disk-based storage media (e.g., magnetic and / or optical storage media), solid-state storage (e.g., any form of persistent flash memory, including planar or three-dimensional (3D) NAND flash or NOR flash), 3D cross-point memory, electrically erasable programmable read-only memory (EEPROM), and / or other types of non-volatile random access memory (RAM), among others. Host memory 270 may be used, for example, to store information (e.g., code and / or data) for use by host processor 260 during execution.

[0037] In some embodiments, an interconnect bus 280 may be used to communicatively couple the host processor 260 and the host memory 270 to the matrix processing resource 210. The interconnect bus 280 may use any interconnect protocol, such as Peripheral Component Bus Express (PCIe), Universal Serial Bus (USB), or Small Computer System Interface (SCSI), among other examples.

[0038] The matrix processing resources 210 may include any processing resources configured to perform matrix operations. For example, the matrix processing resources 210 may be configured to perform matrix multiplication operations, convolution operations, element-wise matrix operations (e.g., +, *, / , <, >, ==), dimension shuffling operations, and / or any combination of the above operations. In some embodiments, the matrix processing resources 210 may include processing resources designed and optimized to perform matrix operations. In some embodiments, the matrix processing resources 210 may also be arranged hierarchically with multiple levels of processing resources. For example, in the illustrated embodiment, the matrix processing resources 210 include multiple matrix processing chips 220, and may also include any processing resources within each matrix processing chip 220. For example, as described below in conjunction with Figure 2B and Figure 2C As discussed, each matrix processing chip 220 may include multiple high-bandwidth memory (HBM) modules 240 and multiple matrix processing clusters 230, and each matrix processing cluster 230 may include multiple matrix processing units 234. Thus, in some embodiments, the matrix processing resource 210 may include multiple matrix processing chips 220, multiple high-bandwidth memory (HBM) modules 240 and multiple matrix processing clusters 230 on each matrix processing chip 220, and / or multiple matrix processing units 234 on each matrix processing cluster 230.

[0039] The matrix processing chip 220 can be, for example, any chip or other component configured to perform matrix operations. For example, in some embodiments, the matrix processing chip 220 can be a peripheral card or chip connected to the host processor 260 using any type of interconnect interface, such as a PCIe interface. In some embodiments, the matrix processing chip 220 can be implemented using an integrated circuit, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and / or any other type of circuit. In the illustrated embodiment, the matrix processing chip 220 is configured in a circular arrangement between adjacent matrix processing chips 220 using communication channels 215. In some embodiments, the communication channels 215 can provide unidirectional communication between adjacent matrix processing chips 220. However, in other embodiments, the communication channels 215 can provide bidirectional communication between adjacent matrix processing chips 220. A circular arrangement that utilizes unidirectional communication between adjacent processing resources can be referred to as a "single-circuit" configuration, while a circular arrangement that utilizes bidirectional communication between adjacent processing resources can be referred to as a "dual-circuit" configuration.

[0040] Furthermore, although not shown, in some embodiments, the matrix processing system 200 may include a communication interface for communicating via a communication network. For example, in some embodiments, the matrix processing system 200 may communicate with one or more remote matrix processing chips via a network to perform distributed matrix operations.

[0041] Figure 2B A block diagram is shown of an example embodiment of a matrix processing chip 220. In the embodiment shown, the matrix processing chip 220 includes a controller 222, a host interface 224, inter-chip links 225, a high bandwidth memory (HBM) module 240, and a matrix processing cluster 230.

[0042] The controller 222 may be configured to control and / or manage matrix operations performed by the matrix processing chip 220. In some embodiments, the controller 222 may control and / or manage the matrix operations performed by the matrix processing chip 220. Figure 2A The host processor 260 and / or Figure 2C In some embodiments, the host processor 260, the controller 222, and / or the master control CPU (MCC) 232 may be configured to receive matrix operations or commands and distribute the matrix operations and matrix operands across the matrix processing cluster 230 and the high bandwidth memory (HBM) module 240. In some embodiments, the controller 222 may be a microprocessor, an integrated circuit, and / or any other type of circuitry and / or processing logic.

[0043] The host interface 224 may be a device that enables the matrix processing chip 220 to communicate with the host computer. Figure 2A In some embodiments, for example, the controller 222 may use the host interface 224 to communicate with the host processor 260. Figure 2A Host interface 224 may communicate with host processor 260. Host interface 224 may use any type of interconnect protocol or interface, including Peripheral Component Bus Express (PCIe), Universal Serial Bus (USB), or Small Computer System Interface (SCSI), among other examples.

[0044] Inter-chip link (ICL) 225 may enable matrix processing chip 220 to communicate with other matrix processing chips. For example, inter-chip link 225 may be used to implement Figure 2A The inter-chip links 225 may be any communication interface, for example, for enabling the matrix processing chip 220 to communicate with another matrix processing chip. In some embodiments, the matrix processing chip 220 may include multiple inter-chip links 225 (e.g., twelve inter-chip links). In some embodiments, the inter-chip links 225 may be implemented using one or more serializer / deserializer (SerDes) interfaces. A SerDes interface may be a communication interface that converts data from serial to parallel or vice versa. For example, a transmitter of a SerDes interface may include a serial-to-parallel converter, and a receiver of a SerDes interface may include a parallel-to-serial converter. In some embodiments, the matrix processing chip 220 may use multiple SerDes interfaces for each connection to another matrix processing chip (e.g., four SerDes interfaces between each pair of connected matrix processing chips).

[0045] The high-bandwidth memory (HBM) module 240 may be a memory component associated with the matrix processing chip 220 for storing matrix operands and other matrix data. In some embodiments, the high-bandwidth memory (HBM) module 240 may be designed to efficiently store and retrieve matrix data. In some embodiments, the high-bandwidth memory (HBM) module 240 may be a multidimensional memory component configured to store and retrieve data in multiple dimensions. For example, in some embodiments, the high-bandwidth memory (HBM) module 240 may be a memory component configured to store and retrieve data in two dimensions (such as rows and columns). However, other embodiments may use a memory component configured to store and retrieve data using any other number of dimensions (e.g., one, three, four, etc.). In the illustrated embodiment, the matrix processing chip 220 includes four high-bandwidth memory (HBM) modules 240a-d. In some embodiments, the high-bandwidth memory (HBM) modules 240 may be shared by the matrix processing cluster 230 of the matrix processing chip 220.

[0046] The matrix processing clusters 230 may include processing resources configured to perform matrix operations (such as matrix multiplication, convolution, and / or dimension shuffling, among other examples). In some embodiments, the matrix processing clusters 230 may be collectively used to perform specific matrix operations by performing matrix processing in parallel. In the illustrated embodiment, the matrix processing chip 220 includes twelve matrix processing clusters 230a-230l. Furthermore, in the illustrated embodiment, the matrix processing clusters 230 are configured or arranged using a two-dimensional mesh interconnect topology. The interconnect topology of the matrix processing clusters 230 may facilitate loop communication between the matrix processing clusters 230. Furthermore, other embodiments may include any number and / or arrangement of matrix processing clusters 230.

[0047] Figure 2C A block diagram is shown of an example embodiment of a matrix processing cluster 230. In the embodiment shown, the matrix processing cluster 230 includes a master control CPU (MCC) 232, a matrix processing unit (MPU) 234, a slicing engine 236, and a memory resource block (MRB) 238.

[0048] The master control CPU (MCC) 232 may be configured to control and / or manage matrix operations performed by the matrix processing cluster 230. In some embodiments, the master control CPU 232 may be a microprocessor, an integrated circuit, and / or any other type of circuit and / or processing logic. In some embodiments, the master control CPU 232 may be a processor, an integrated circuit, and / or any other type of circuit and / or processing logic. Figure 2A The host processor 260 and / or Figure 2B The controller 222 of the master CPU 232 receives instructions. Based on these instructions, the master CPU 232 can then use the matrix processing unit 234 to perform matrix operations, such as matrix multiplication, convolution and / or dimension shuffling, and so on. For example, the master CPU 232 may receive an instruction for performing a matrix multiplication operation (such as C=A*B). The instruction may include a handle or identifier for each matrix and may also indicate how the matrix should be stored in the memory resource block (MRB) 238. Subsequently, matrix A and matrix B may be decomposed into a series of smaller matrices (e.g., 32x32 matrices). Subsequently, matrix operations may be performed on the smaller matrices, and the results may be stored in the memory resource block (MRB) 238 until the output matrix C has been fully calculated.

[0049] The matrix processing unit (MPU) 234 may be configured to perform matrix operations, such as matrix multiplication, convolution, and / or dimension shuffling. In some embodiments, the matrix processing unit (MPU) 234 performs matrix operations based on commands received from the master control CPU (MCC) 232. Furthermore, in some embodiments, each matrix processing cluster 230 may include multiple matrix processing units (MPUs) 234. For example, in the illustrated embodiment, the matrix processing cluster 230 includes two matrix processing units (MPUs) 234. The matrix processing unit (MPU) 234 may be capable of performing matrix operations (such as matrix multiplication) on small matrices (e.g., 32x32 matrices). In some cases, the matrix processing unit (MPU) 234 may be designed and / or optimized to perform matrix multiplication operations. The matrix processing unit (MPU) 234 may load matrix operands from a memory resource block (MRB) 238. In some embodiments, the matrix processing unit (MPU) 234 may support the following arithmetic operations: matrix multiplication; unary matrix operations; binary matrix operations such as addition (+), subtraction (-), multiplication (*), division ( / ), bitwise XOR (exclusive OR), AND (and), OR (or), logical and arithmetic left and right shifts, comparisons (>, <, > =, ==, ! =); and column-wise, row-wise, and matrix-level operations such as summation, maximum, and minimum.

[0050] The slicing engine 236 may be configured to slice the matrix operands of a particular matrix operation into smaller partial matrices. For example, in some embodiments, the master control CPU (MCC) 232 may use the slicing engine 236 to decompose the matrix operands into smaller partial matrices for the matrix processing unit (MPU) 234. In some embodiments, the slicing engine 236 may include a convolution engine (CSE) for performing matrix slicing for convolution operations. For example, in some embodiments, the convolution engine (CSE) may slice the matrix operands in a manner that enables the convolution operation to be projected as a matrix multiplication operation, thereby enabling the same processing logic to perform both matrix multiplication operations and matrix convolution operations. In addition, in some embodiments, the slicing engine 236 and / or an associated convolution engine (CSE) may be used to perform a dimension shuffle operation to reorder the dimensions of a matrix.

[0051] Memory resource blocks (MRBs) 238 may be memory components on the matrix processing cluster 230 for storing matrix operands and other matrix data. In some embodiments, the memory resource blocks (MRBs) 238 may be designed to efficiently store and retrieve matrix data. In some embodiments, the memory resource blocks (MRBs) 238 may be multidimensional memory components configured to store and retrieve data in multiple dimensions. For example, in some embodiments, the memory resource blocks (MRBs) 238 may be memory components configured to store and retrieve data in two dimensions, such as rows and columns. In the illustrated embodiment, the matrix processing cluster 230 includes ten memory resource blocks (MRBs) 238. However, other embodiments may include a different number of memory resource blocks (MRBs) 238 on the matrix processing cluster 230. In some embodiments, each memory resource block (MRB) 238 may be capable of storing matrices of a certain size (e.g., a 256x512 matrix). In some embodiments, the memory resource blocks (MRBs) 238 may be shared by the matrix processing units (234) of a particular matrix processing cluster 230.

[0052] In some embodiments, Figure 2A-2CThe matrix processing architecture can be used to implement the matrix processing functionality described throughout this disclosure. For example, the matrix processing system 200 can be used to perform matrix operations in a distributed manner that utilizes available processing resources to achieve 100% processing efficiency. For example, in some embodiments, matrix operations can be distributed across multiple processing resources 210 optimized for matrix operations, thereby achieving full utilization of the processing resources 210 throughout the duration of the matrix operations. For example, the matrix processing system 200 can include multiple processing resources 210 designed and optimized to perform matrix operations. In some embodiments, these processing resources 210 can be configured in a single-cycle or dual-cycle arrangement. Furthermore, the processing resources 210 can be arranged hierarchically in multiple levels of processing resources. For example, in some embodiments, the processing resources 210 can include multiple matrix processing chips 220, multiple high-bandwidth memory (HBM) modules 240 and multiple matrix processing clusters 230 on each matrix processing chip 220, and / or multiple matrix processing units (MPUs) 234 on each matrix processing cluster 230. This processing architecture enables matrix operations to be distributed across multiple processing resources 210 and / or processing layers with 100% processing efficiency. Furthermore, this processing architecture enables matrix operations to be efficiently scaled across a variable number of processing resources 210 operating in parallel while still achieving 100% processing efficiency. For example, scaling can be achieved by adjusting the number of processing resources 210 used to perform a particular matrix operation, such as the number of matrix processing systems 200 or servers, the number of matrix processing chips 220 in each matrix processing system 200 or server, and so forth.

[0053] As an example, Figure 2A-2CThe matrix processing architecture can be used to implement matrix multiplication and / or convolution operations. For example, in some embodiments, the matrix multiplication operation can be distributed across multiple processing resources 210 in a manner that results in a latency for transferring matrix operands that is less than the matrix processing time, which allows the transfer of matrix operands to be completed while the matrix processing is being performed. For example, for certain matrix operations involving matrix operands having certain dimensions (e.g., matrix multiplication with "thin" matrix operands), the time required to access and transfer the matrix operands may exceed the time required to perform the actual matrix calculations, thereby resulting in idle processing time when the matrix operands are being retrieved from memory and / or being transferred to the processing resources 210. For example, a single-cycle configuration (e.g., in which each processing resource 210 only retrieves matrix operands and data from one adjacent processing resource 210 at any given time) may not be able to achieve 100% processing efficiency for these particular types of matrix operations and matrix operands. However, the dual-loop configuration of the processing resources 210 enables each processing resource to perform matrix calculations while simultaneously acquiring matrix operands and data from two adjacent processing resources 210 to the processing resource, which significantly reduces the latency of transferring matrix operands and thereby avoids any idle processing time. For example, when a dual-loop approach is used instead of a single-loop approach, the latency of transferring certain operations can be reduced by half. In this way, the latency of transferring matrix operands and matrix data can be completely masked by the matrix processing time, thereby avoiding any wasted or idle processing time and achieving 100% processing efficiency. Accordingly, even for large matrix operands and / or matrix operands with certain dimensions (such as large matrix operands that are neither square nor single vectors (e.g., "thin" matrices with a height much larger than width)), matrix operations (e.g., matrix multiplication or GEMM) can be performed efficiently. For example, even when multiplying two thin matrices, multiplying a thin matrix with a square matrix, etc., matrix multiplication can be performed efficiently. Similarly, convolution operations may be distributed across multiple processing resources 210 in a manner that results in 100% processing efficiency using available processing resources.

[0054] As an example, when a matrix operation or command is received, the convolution operation may be distributed across the processing resources 210 of the matrix processing system 200. For example, the matrix operands (or input matrices) may be partitioned based on the number of available processing resources 210. Furthermore, in some embodiments, the partitioning may span rows of the matrix operands and / or any other dimension of the matrix operands. Each partition may then be distributed to a specific processing resource 210. Each processing resource 210 may then perform multiple partial matrix operations. In some embodiments, the multiple partial matrix operations are performed in multiple stages. For example, each processing resource 210 may perform a specific stage of partial matrix operations while simultaneously sending partial matrix data to and receiving partial matrix data from its neighboring processing resources 210. For example, in a single-cycle configuration of the processing resources 210, each processing resource 210 may either send partial matrix data to each neighboring processing resource or receive partial matrix data from each neighboring processing resource. Similarly, in a dual-loop configuration of processing resources 210 , each processing resource 210 may send partial matrix data to each adjacent processing resource 210 and receive partial matrix data from each adjacent processing resource 210 .

[0055] Each processing resource 210 may then use the partial matrix data for a subsequent partial matrix operation. The result of the matrix operation may then be determined based on the partial matrix operation collectively performed by the processing resources 210.

[0056] Furthermore, if the processing resources 210 are arranged hierarchically, matrix operations can be distributed in a hierarchical manner. For example, matrix operands (or input matrices) can initially be partitioned based on the number of available matrix processing chips 220. Subsequently, each partition and the associated portion of the matrix operation can be distributed to a specific matrix processing chip 220. The partitions and the associated portion of the matrix operation distributed to a specific matrix processing chip 220 can then be similarly partitioned and distributed across the matrix processing clusters 230 and / or high-bandwidth memory (HBM) modules 240 of the specific matrix processing chip 220. For example, for certain matrix operations, a portion of the matrix operation can be distributed to each matrix processing cluster 230. Alternatively, for certain matrix operations, a portion of the matrix operation can be distributed across various "logical processing nodes" (e.g., a group of matrix processing clusters 230 associated with a high-bandwidth memory (HBM) module 240) and then distributed to each matrix processing cluster 230 of a specific logical processing node. In some embodiments, the matrix processing clusters 230 (and / or logical processing nodes) can be configured in a round-robin manner similar to the matrix processing chips 220. The partitions and partial matrix operations distributed to a particular matrix processing cluster 230 may then be partitioned and distributed across the matrix processing units (MPUs) 234 of that particular matrix processing cluster 230 in a similar manner.

[0057] Example computer processor architecture

[0058] Figure 3 and Figure 4 A block diagram illustrating an example embodiment of a computer processor that may be used in accordance with the embodiments disclosed herein. For example, Figure 3 and Figure 4 The computer processor shown in FIG can be used as a host processor associated with a matrix processing system (e.g., Figure 2A ), or as a processor associated with other components and / or devices discussed throughout this disclosure (e.g., Figure 1 Other processor and system designs known in the art for laptops, desktops, handheld PCs, personal digital assistants, engineering workstations, servers, network appliances, network hubs, switches, embedded processors, digital signal processors (DSPs), graphics devices, video game devices, set-top boxes, microcontrollers, cellular phones, portable media players, handheld devices, and various other electronic devices are also suitable. In general, a variety of systems and electronic devices that can include the processors and / or other execution logic disclosed herein are generally suitable.

[0059] Figure 3A block diagram of an example embodiment of a processor 300 is shown. Processor 300 is an example of a type of hardware device that may be used in conjunction with embodiments described throughout this disclosure. Processor 300 may be any type of processor, such as a microprocessor, an embedded processor, a digital signal processor (DSP), a network processor, a multi-core processor, a single-core processor, or other device for executing code. Although in Figure 3 Only one processor 300 is shown, but the processing element may alternatively include more than one Figure 3 The processor 300 is shown in FIG. The processor 300 may be a single-threaded core, or for at least one embodiment, the processor 300 may be multi-threaded, embodying that the processor 300 may include more than one hardware thread context (or "logical processor") per core.

[0060] Figure 3 Also shown is a memory 302, according to an embodiment, coupled to the processor 300. The memory 302 may be any of a variety of memories known or otherwise available to those skilled in the art, including various layers of a memory hierarchy. Such memory elements may include, but are not limited to, random access memory (RAM), read-only memory (ROM), logic blocks of a field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM).

[0061] The processor 300 may execute any type of instructions associated with the various algorithms, processes, or operations described in detail herein. Generally speaking, the processor 300 may transform an element or article (eg, data) from one state or thing to another.

[0062] Code 304 (which may be one or more instructions executed by processor 300) may be stored in memory 302; or code 304 may be stored in software, hardware, firmware, or any suitable combination of software, hardware, and firmware; or code may be stored in any other internal or external component, device, element, or object, as appropriate based on specific needs. In one example, processor 300 may follow a program sequence of instructions indicated by code 304. Each instruction enters front-end logic 306 and is processed by one or more decoders 308. The decoders may generate micro-operations (such as fixed-width micro-operations in a predefined format) or may generate other instructions, micro-instructions, or control signals reflecting the original code instructions as their output. Front-end logic 306 may also include register renaming logic and scheduling logic, which generally allocate resources and queue operations corresponding to instructions for execution.

[0063] Processor 300 may also include execution logic 314 having a set of execution units 316a, 316b, 316n, etc. Some embodiments may include multiple execution units dedicated to a specific function or set of functions. Other embodiments may include only one execution unit or only one execution unit that can perform a specific function. Execution logic 314 performs the operations specified by the code instructions.

[0064] After completing execution of the operations specified by the code instructions, back-end logic 318 may retire the instructions of code 304. In one embodiment, processor 300 allows out-of-order execution but requires in-order instruction retirement. Retirement logic 320 may take various known forms (e.g., a reorder buffer, etc.). In this manner, processor 300 is transformed during execution of code 304, at least with respect to the outputs generated by the decoders, the hardware registers and tables utilized by register renaming logic 310, and any registers (not shown) modified by execution logic 314.

[0065] Although Figure 3 300. Although not shown, the processing element may include other elements on the chip with processor 300. For example, the processing element may include memory control logic with processor 300. The processing element may include I / O control logic and / or may include I / O control logic integrated with memory control logic. The processing element may also include one or more caches. In some embodiments, non-volatile memory (such as flash memory or fuses) may also be included on the chip with processor 300.

[0066] Figure 4 4 shows a block diagram of an example embodiment of a multiprocessor 400. Figure 4 As shown in FIG, multiprocessor system 400 is a point-to-point interconnect system and includes a first processor 470 and a second processor 480 coupled via a point-to-point interconnect 450. In some embodiments, each of processors 470 and 480 may be a Figure 3 A version of processor 300.

[0067] Processors 470 and 480 are shown as including integrated memory controller (IMC) units 472 and 482, respectively. Processor 470 also includes point-to-point (PP) interfaces 476 and 478 as part of its bus controller unit; similarly, second processor 480 includes PP interfaces 486 and 488. Processors 470, 480 can exchange information via PP interface 450 using point-to-point (PP) interface circuits 478, 488. Figure 4As shown in FIG, IMCs 472 and 482 couple the processors to respective memories, namely, memory 432 and memory 434, which may be portions of main memory locally attached to the respective processors.

[0068] Processors 470, 480 may each exchange information with a chipset 490 via respective PP interfaces 452, 454 using point-to-point interface circuits 476, 494, 486, 498. Chipset 490 may optionally exchange information with a coprocessor 438 via a high-performance interface 439. In one embodiment, coprocessor 438 is a special-purpose processor such as, for example, a high-throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPGPU, an embedded processor, a matrix processor, or the like.

[0069] A shared cache (not shown) may be included in either processor, or external to both processors but connected to the processors via the PP interconnect, such that if the processors are placed in a low power mode, local cache information of either or both processors may be stored in the shared cache.

[0070] Chipset 490 may be coupled to first bus 416 via interface 496. In one embodiment, first bus 416 may be a Peripheral Component Interconnect (PCI) bus or a bus such as PCI Express or another third generation I / O interconnect bus, although the scope of the present disclosure is not limited in this regard.

[0071] like Figure 4 As shown in , various I / O devices 414 may be coupled to the first bus 416, along with a bus bridge 418 that couples the first bus 416 to a second bus 420. In one embodiment, one or more additional processors 415, such as a coprocessor, a high throughput MIC processor, a GPGPU processor, an accelerator (such as, for example, a graphics accelerator or a digital signal processing (DSP) unit), a matrix processor, a field programmable gate array, or any other processor, are coupled to the first bus 416. In one embodiment, the second bus 420 may be a low pin count (LPC) bus. Various devices may be coupled to the second bus 420, including, for example, a keyboard and / or mouse 422, communication devices 427, and a storage unit 428 (such as a disk drive or other mass storage device, which in one embodiment may include instructions / code and data 430). Additionally, an audio I / O 424 may be coupled to the second bus 420. Note that other architectures are possible. For example, instead of Figure 4 Instead of a point-to-point architecture, the system can implement a multi-drop bus or other such architecture.

[0072] Figure 4All or part of any component in may be implemented as separate or independent components or chips, or may be integrated with other components or chips, such as a system on a chip (SoC) that integrates various computer components into a single chip.

[0073] The various embodiments of the mechanisms disclosed herein may be implemented in hardware, software, firmware, or a combination of these implementation methods. Certain embodiments may be implemented as a computer program or program code executed on a programmable system comprising at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0074] Program code (such as Figure 4 The processing system is a computer program that processes the processing information of a processor, such as a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0075] The program code can be implemented in a high-level programming language or an object-oriented programming language to communicate with the processing system. If necessary, the program code can also be implemented in assembly language or machine language. In fact, the mechanism described herein is not limited to the scope of any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0076] One or more aspects of at least one embodiment may be implemented as representative instructions stored on a machine-readable medium that represent various logic within a processor, which, when read by a machine, causes the machine to fabricate logic for performing the techniques described herein. These representations, known as "IP cores," may be stored on a tangible, machine-readable medium and supplied to various customers or manufacturing facilities to load into fabrication machines that actually manufacture the logic or processor.

[0077] Such machine-readable storage media may include, but are not limited to, non-transitory tangible arrangements of articles manufactured or formed by a machine or apparatus, including storage media such as: a hard disk; any other type of disk, including a floppy disk, an optical disk, a compact disk read-only memory (CD-ROM), a compact disk rewritable (CD-RW), and a magneto-optical disk; semiconductor devices, such as read-only memory (ROM), random access memory (RAM) such as dynamic random access memory (DRAM) and static random access memory (SRAM), erasable programmable read-only memory (EPROM), flash memory, electrically erasable programmable read-only memory (EEPROM); phase change memory (PCM); a magnetic or optical card; or any other type of medium suitable for storing electronic instructions.

[0078] Accordingly, various embodiments of the present disclosure also include non-transitory, tangible, machine-readable media containing instructions or design data, such as hardware description language (HDL), that define the structures, circuits, devices, processors, and / or system features described herein. These embodiments are also referred to as program products.

[0079] Programmable matrix processing engine

[0080] Figure 5 An example embodiment of a programmable matrix processing engine 500 is shown. In some embodiments, the matrix processing engine 500 may be implemented by a matrix processing architecture such as, Figure 2A-2C For example, in some embodiments, the matrix processing engine 500 may be implemented by a matrix processing cluster on a matrix processing chip (e.g., from Figure 2B and Figure 2C In some embodiments, the matrix processing cluster 230 of the matrix processing chip 220 may be implemented as described above. In those embodiments, a particular matrix processing cluster may use its associated matrix processing engine 500 to perform matrix-based processing and operations, such as partial matrix operations associated with a particular matrix operation distributed across multiple matrix processing resources (e.g., as described throughout this disclosure).

[0081] In the illustrated embodiment, the matrix processing engine 500 includes a read engine 535, a slice engine 536, and an output engine 537, which are discussed further below. The illustrated embodiment also depicts various components of the underlying matrix processing architecture that may be involved when using the matrix processing engine 500 to perform matrix operations. For example, the illustrated embodiment depicts a high bandwidth memory (HBM) module 540, a master control CPU (MCC) 532, a matrix processing unit (MPU) 534, a memory resource block (MRB) 538, and a matrix routine memory 539. For example, in the illustrated embodiment, these various components are overlaid on the matrix processing engine 500 to illustrate how and when these components will be used by the matrix processing engine 500, as described further below.

[0082] The HBM module 540 may be a high bandwidth memory (HBM) module designed to efficiently store and retrieve large amounts of matrix data. For example, in some embodiments, the HBM module 540 may be a high bandwidth memory (HBM) module on a matrix processing chip (e.g., from Figure 2B HBM module 240 of the matrix processing chip 220).

[0083] MCC 532 may be a Master Control CPU (MCC) for controlling and / or managing matrix operations. For example, in some embodiments, MCC 532 may be a Master Control CPU on a particular matrix processing cluster (e.g., from Figure 2C For example, in those embodiments, the MCC 532 may be used to control and / or manage matrix operations performed on its particular cluster.

[0084] MPU 534 may be a matrix processing unit (MPU) for performing matrix operations. For example, in some embodiments, MPU 534 may be a matrix processing unit on a specific matrix processing cluster (e.g., from Figure 2C Matrix processing cluster 230 may include an MPU 234. For example, in some embodiments, the matrix processing cluster may include multiple matrix processing units (MPUs) for performing matrix operations. The illustrated embodiment depicts two matrix processing units (MPUs) 534a and 534b. In some embodiments, the MPUs 534 may perform matrix operations based on commands or instructions from the master control CPU (MCC) 532.

[0085] A memory resource block (MRB) 538 may be a memory component designed to store and retrieve matrix data. For example, in some embodiments, an MRB 538 may be a memory resource block on a particular matrix processing cluster (e.g., from Figure 2CFor example, in those embodiments, the MRB 538 may be used to store and retrieve matrix data associated with matrix operations performed on a particular cluster.

[0086] The matrix routine memory 539 may be a memory component for storing matrix routines. A matrix routine may be, for example, a programmable routine for a matrix processor that is designed to perform a specific matrix operation when executed by the matrix processor. For example, a matrix routine may include a series of instructions and / or commands that are supported by a specific matrix processor and are designed to perform a desired matrix operation when executed by the matrix processor. For example, in some embodiments, the matrix processor may be designed to support a set of instructions and / or commands for performing various basic operations. For example, in some embodiments, the matrix processor may support instructions for processing data, performing various arithmetic operations, and / or identifying matrix operands and outputs for various instructions and operations.

[0087] For example, certain instructions may be used to process data, such as reading, writing, and / or copying data (e.g., matrix data) to and from different locations, slicing matrix data, extracting matrix data, rearranging matrix data, and the like.

[0088] As another example, certain instructions may be used to perform specific arithmetic operations, including any of the following: matrix multiplication; convolution; unary matrix operations; binary matrix operations, such as addition (+), subtraction (-), multiplication (*), division ( / ), bitwise XOR, AND, OR, logical and arithmetic left and right shifts, comparisons (>, <, >=, ==, !=); and column-wise, row-wise, and matrix-level operations, such as summation, maximum, and minimum.

[0089] In addition, special "register operand" (REGOP) instructions can be used to identify matrix operands and outputs for various supported instructions and operations. Register operand instructions can be used, for example, to specify the size and location of operands and outputs of a particular instruction or operation. For example, in some embodiments, register operand instructions can be used to identify the location in a high bandwidth memory (HBM) module or memory resource block (MRB) associated with a particular operand or output. As an example, a basic matrix multiplication operation can be programmed using the following instructions: a REGOP instruction for identifying the location of each operand and the location of the output, followed by instructions for performing the matrix multiplication operation.

[0090] In this way, the basic instructions and / or commands supported by the matrix processor can be used to program matrix routines for more complex matrix operations (such as distributed matrix multiplication and / or convolution operations, dimension shuffle operations, reshape operations, etc.).

[0091] The matrix routine memory 539 may be implemented in any portion of the matrix processing architecture, such as a matrix processing chip, a matrix processing cluster, and / or a host computing system. For example, in some embodiments, a matrix processing chip (e.g., Figure 2B The matrix processing chip 220 of a matrix processing chip may include a matrix routine memory 539 that is accessible by a corresponding cluster on that matrix processing chip. As another example, in some embodiments, a matrix processing cluster (e.g., Figure 2C Matrix processing cluster 230) may include its own matrix routine memory 539. As another example, in some embodiments, a host computing system of a matrix processing architecture may include a matrix routine memory 539 accessible by its associated matrix processing resources (e.g., Figure 2A In , host memory 270 of matrix processing system 200 may include matrix routine memory accessible by matrix processing resources 210).

[0092] Furthermore, the matrix routine memory 539 can be any component or mechanism capable of storing data, including any type or combination of volatile and / or non-volatile memory, such as random access memory (RAM) (e.g., dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), static random access memory (SRAM)), flash-based memory, read-only memory (ROM), logic blocks of a field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), electrically erasable programmable ROM (EEPROM), and / or any suitable combination of the foregoing.

[0093] By way of example, in some embodiments, the matrix routines memory 539 can be implemented using random access memory (e.g., SRAM) on a matrix processing chip. For example, in these embodiments, matrix routines can be loaded onto the matrix routines memory 539 by a host computing system (e.g., the host processor 260 of FIG. 2 ). For example, the host computing system can transmit the matrix routines to the matrix processing chip via an interconnect interface (e.g., a PCIe interface), which can then store the matrix routines into its associated matrix routines memory 539. In some embodiments, a software driver of the host computing system can be used to load the matrix routines. Furthermore, in some embodiments, any existing matrix routines can be loaded onto the matrix routines memory 539 during system startup, while any additional matrix routines can be loaded later, as appropriate, after system startup.

[0094] In the illustrated example, the matrix processing engine 500 performs multiple matrix operations 501 and 502 in parallel. For example, as described above, in some embodiments, the matrix processing engine 500 may be implemented on a specific matrix processing cluster, and the specific matrix processing cluster may include multiple MPUs 534. In the illustrated example, the matrix processing engine 500 is implemented on a cluster having two MPUs 534a-b. Accordingly, the matrix processing engine 500 may perform two matrix operations 501 and 502 in parallel using the corresponding MPUs 534. The illustrated example shows the control flow of the matrix processing engine 500 for the first matrix operation 501 and the second matrix operation 502.

[0095] In the illustrated example, control flow for the first matrix operation 501 begins at the read engine 535 of the matrix processing engine 500. The read engine 535 may first retrieve matrix data (e.g., matrix data associated with the operands of the matrix operation 501) from the corresponding HBM module 540a of the matrix processing chip. The read engine 535 may then store that matrix data in certain MRBs 538a of the particular cluster associated with the read engine 535. For example, as described above, the HBM module 540a may be a high-bandwidth memory module on a particular matrix processing chip (e.g., memory shared by the matrix processing cluster of a particular matrix processing chip), and the MRBs 538a may be local memory resource blocks on the particular matrix processing cluster. Furthermore, in some embodiments, the read engine 535 may use the master control CPU (MCC) 532 of its corresponding cluster to store and retrieve data on the HBM 540 and MRBs 538.

[0096] Subsequently, slice engine 536a may "slice" the matrix data stored in MRB 538 to extract specific matrix operands associated with matrix operation 501. For example, in some cases, the associated matrix operands may only include a subset of the matrix data stored in MRB 538a, and / or the matrix operands may not be arranged contiguously in the matrix data stored in MRB 538a. Accordingly, slice engine 536a may extract specific "slices" or segments of the matrix data stored in MRB 538a and then arrange these slices to form corresponding matrix operands. For example, in the illustrated example, matrix operation 501 is associated with a convolution operation, and accordingly, slice engine 536a is used to extract the sliced ​​matrix operands and filters from the matrix data stored in MRB 538a. The sliced ​​matrix operands and filters are then stored in MRBs 538b and 538c, respectively. In some cases, the specific slicing scheme used by the slice engine 536a may depend on various factors, including the type of matrix operation 501, the number of available processing resources, the size of the operands, etc. Furthermore, in some embodiments, the specific slicing performed by the slice engine 536a for a particular operation may be programmed and / or defined using a set of instructions supported by the slice engine 536a.

[0097] The output engine 537 may then be used to calculate the result of a particular matrix operation 501. For example, the output engine 537 may use the matrix operands generated by the slice engine 536a (e.g., the matrix operands stored in the MRBs 538b and 538c) to perform the appropriate matrix operation 501. For example, in some embodiments, the output engine may first identify the associated matrix routine corresponding to the particular matrix operation 501, and then the output engine 537 may retrieve that matrix routine from the matrix routine storage 539. In some embodiments, the output engine 537 may use the master control CPU (MCC) 532 on its corresponding cluster to retrieve the matrix routine from the matrix routine storage 539.

[0098] Subsequently, if appropriate, the output engine 537 may specify or supply certain information or fields used by the matrix routine. For example, in some embodiments, certain information and / or fields for a matrix routine (such as the size and / or location of particular operands for the matrix routine) may be incomplete or unspecified. Accordingly, in some embodiments, the output engine 537 may use the MCC 532 to specify or supply any remaining information and / or fields (e.g., the size and / or location of matrix operands) for a particular matrix routine.

[0099] The output engine 537 may then execute a particular matrix routine. For example, the output engine 537 may use the MCC 532 and / or the MPU 534a to execute programmed instructions associated with the particular matrix routine. The MCC 532 may, for example, be used to perform certain tasks specified by the instruction, such as reading and writing data, communicating with other resources, and the like. The MPU 534a may, for example, be used to perform a particular arithmetic operation specified by the instruction. Furthermore, in some cases, a particular matrix routine may be repeatedly executed or looped until the particular operation has been performed or completed for all necessary data (e.g., all data for a particular matrix operand).

[0100] Subsequently, the output engine 537 may store the output or result of the matrix routine in certain MRB(s) 538d of the cluster that executes the matrix routine. In some cases, the output engine 537 may then provide the output stored in the MRB 538d to another component of the matrix processing architecture. For example, in some cases, the matrix operation 501 may be a partial matrix operation associated with a larger matrix operation distributed across multiple processing resources, and thus, the output of the matrix operation 501 may be a partial result associated with the larger distributed operation. In addition, the output of the partial matrix operation 501 may be required by (a plurality of) other processing resources involved in the distributed matrix operation. Accordingly, the output engine 537 may provide the output of the partial matrix operation 501 to appropriate resources for, for example, further processing and / or storage. In some cases, the appropriate resources may vary based on circumstances, including the type of matrix operation being performed, the implementation of (a plurality of) associated matrix routines, the number and availability of processing resources, and the like. For example, in some cases, the specific processing and / or destination of the output of the matrix operation may be programmed or defined by the associated matrix routine. Furthermore, in some embodiments, the output engine 537 may use the master control CPU (MCC) 532 on its respective cluster to provide the output of the partial matrix operation 501 to the appropriate destination.

[0101] For example, in some cases, the output engine 537 may provide the output of the partial matrix operation 501 to a specific destination for storing partial results of the distributed matrix operation. For example, for a distributed matrix operation, the corresponding partial results determined by each processing resource may be combined on a specific memory component (such as a specific HBM 540b of the matrix processing chip). For example, in some cases, the corresponding partial results determined by each cluster of the matrix processing chip may be combined on a specific HBM 540b of the matrix processing chip. Furthermore, the partial results may be stored on the HBM 540b using a specific arrangement that together form the complete result of the matrix operation.

[0102] As another example, in some cases, the output engine 537 may send the output of the partial matrix operation 501 (e.g., the output stored in the MRB 538d) back to the MPU 534a, for example, to enable the MPU 534a to use that output as an operand in a subsequent partial operation. For example, in some cases, the output of a partial operation in one stage of a distributed matrix operation may be used as an input or operand for a partial operation in another stage of the distributed matrix operation.

[0103] As another example, in some cases, the output engine 537 can provide the output of a partial matrix operation 501 (e.g., the output stored in the MRB 538d) to another matrix processing resource (e.g., another matrix processing cluster on the same matrix processing chip, or another matrix processing chip entirely). For example, in some cases, a distributed matrix operation can be distributed across multiple clusters of a matrix processing chip and / or across multiple matrix processing chips. Furthermore, in some cases, the output of a partial operation performed by a particular matrix processing resource can be used as an operand in another partial operation performed by a different processing resource.

[0104] In the example shown, the second matrix operation 502 can be performed in parallel with the first matrix operation 501. Furthermore, the control flow for the second matrix operation 502 can be similar to the control flow described above for the first matrix operation 501. However, the second matrix operation 502 can be a different matrix operation (e.g., performed using a different matrix routine), have different matrix operands and results, use different memory locations of the HBM 540 and / or MRB 538, and be performed using a different MPU 534b and associated slicing engine 536b.

[0105] Figure 6 Flowchart 600 is shown for an example embodiment of programmable matrix processing. In some embodiments, flowchart 600 may be implemented by components described throughout this disclosure (e.g., Figure 2A-2C Matrix processing architecture and / or Figure 5 programmable matrix processing engine).

[0106] The flowchart may begin at block 602 by receiving a command for performing a matrix operation. A matrix operation may include an operation on one or more matrix operands. For example, a matrix operation may include any matrix-based arithmetic operation, including element-wise matrix operations, matrix multiplication, convolution, and / or any combination of such operations.

[0107] Furthermore, in some embodiments, matrix operations can be used to implement computer-based artificial intelligence and machine learning capabilities in artificial neural networks. For example, in some embodiments, the matrix operations of block 602 can be associated with operations in an artificial neural network, such as forward propagation, backward propagation, and / or weight update operations.

[0108] Subsequently, the flowchart may proceed to block 604 to obtain matrix data from the memory. The matrix data may, for example, be associated with one or more matrix operands of a matrix operation. In some embodiments, the matrix data may be obtained from a multidimensional memory. The multidimensional memory may, for example, be a memory component designed to efficiently store and retrieve data in multiple dimensions (e.g., two dimensions). In some embodiments, the matrix data may be obtained by executing one or more instructions for obtaining matrix data from one or more memory locations of the multidimensional memory.

[0109] The flowchart may then proceed to block 606 to obtain matrix operands from the matrix data. For example, in some embodiments, the matrix operands may be obtained by slicing the matrix data, thereby extracting the matrix operands from the matrix data. Furthermore, in some embodiments, the matrix operands may be obtained by executing one or more instructions for slicing the matrix operands or extracting the matrix operands from the matrix data.

[0110] Subsequently, the flowchart may proceed to block 608 to identify matrix routines associated with matrix operations. A matrix routine may, for example, be a programmable routine for a matrix processor that is designed to perform a specific matrix operation when executed by the matrix processor. For example, a matrix routine may include a series of instructions and / or commands that are supported by a specific matrix processor and are designed to perform the desired matrix operation when executed by the matrix processor. For example, in some embodiments, the matrix processor may be designed to support a set of instructions and / or commands for performing various basic matrix operations. For example, the matrix processor may include instructions and / or commands for the following operations: identifying the memory location of matrix operands, retrieving matrix operands from memory, and / or performing specific arithmetic operations or calculations on matrix operands, among other examples. In this way, the basic instructions and / or commands supported by the matrix processor can be used to program matrix routines for more complex matrix operations.

[0111] In some embodiments, a matrix routine memory can be used to store matrix routines, and these matrix routines can be retrieved from the matrix routine memory when needed. For example, to perform a desired matrix operation, the corresponding matrix routine can first be retrieved from the matrix routine memory. However, in some cases, a particular matrix routine may not yet be stored in the matrix routine memory. Accordingly, it may be necessary to load the particular matrix routine into the matrix routine memory. Thus, in some cases, a particular matrix routine may first be retrieved from the host computing system and then stored in the matrix routine memory.

[0112] The flowchart may then proceed to block 610 to execute a matrix routine. For example, in some embodiments, the matrix routine may be executed on a matrix processor using one or more matrix operands. The flowchart may then proceed to block 612 to obtain a result of the matrix operation based on the matrix routine executed by the matrix processor. For example, in some cases, a particular matrix routine may return a result determined by a series of instructions and / or commands executed by the matrix processor.

[0113] At this point, the flowchart can be completed. However, in some embodiments, the flowchart can restart and / or repeat certain blocks. For example, in some embodiments, the flowchart can repeat at block 602 to continue receiving and processing commands for performing matrix operations.

[0114] The flowcharts and block diagrams in multiple figures illustrate the architecture, functions and operations of the possible implementations of the systems, methods and computer program products according to various aspects of the present disclosure. In this regard, each frame in the flowchart or block diagram may represent a code module, a code segment or a code portion comprising one or more executable instructions for implementing (a plurality of) specified logical functions. It should also be noted that in some alternative implementations, the multiple functions marked in the frame may not occur in the order marked in the figure. For example, depending on the functions involved, the two frames shown in succession may actually be executed substantially simultaneously, or these frames may sometimes be executed in the reverse order or in an alternative order. It will also be noted that each frame in the block diagram and / or flowchart illustration and the combination of multiple frames in the block diagram and / or flowchart illustration can be realized by a system or a variety of combinations of dedicated hardware and computer instructions based on dedicated hardware that perform specified functions or actions.

[0115] The foregoing disclosure summarizes the features of several embodiments so that those skilled in the art can better understand the various aspects of the present disclosure. Those skilled in the art will appreciate that they can easily use this disclosure as a basis for designing or modifying other processes and structures to perform the same purposes and / or achieve the same advantages of the multiple embodiments described herein. Those skilled in the art will also recognize that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that they can make various modifications, substitutions, and changes without departing from the spirit and scope of the present disclosure.

[0116] All or part of any hardware element disclosed herein can be easily provided in a system on chip (SoC) (including a central processing unit (CPU) package). SOC refers to an integrated circuit (IC) that integrates the components of a computer or other electronic system into a single chip. SoC can include digital, analog, mixed signal, and radio frequency functions, all of which can be provided on a single chip substrate. Other embodiments may include a multi-chip module (MCM), in which multiple chips are located in a single electronic package and are configured to interact closely with each other through this electronic package. In various other embodiments, the computing functions disclosed herein can be implemented in one or more silicon cores in application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and other semiconductor chips.

[0117] As used throughout this specification, the term "processor" or "microprocessor" should be understood to include not only conventional microprocessors (such as The term "industry-leading x86 and x64 architectures" also includes Matrix processors, graphics processors, and any ASIC, FPGA, microcontroller, digital signal processor (DSP), programmable logic device, programmable logic array (PLA), microcode, instruction set, emulated or virtual machine processor, or any similar "Turing-complete" device, combination of devices, or logic element (hardware or software) that enables the execution of instructions.

[0118] It should also be noted that in certain embodiments, some of the components may be omitted or combined. In a general sense, the arrangements depicted in the figures should be understood as logical partitioning, while the physical architecture may include various permutations, combinations, and / or hybrids of these elements. It is important to note that countless possible design configurations can be used to achieve the operational objectives outlined herein. Accordingly, the associated infrastructure has countless alternative arrangements, design choices, equipment possibilities, hardware configurations, software implementations, and equipment options.

[0119] In a general sense, any appropriately configured processor can execute the instruction associated with data or microcode to realize the operation described in detail herein.Any processor disclosed herein can convert element or product (e.g., data) from one state or thing to another state or thing.In another example, Fixed logic or programmable logic (e.g., software / computer instructions executed by a processor) can be utilized to realize some activities outlined herein, and the element identified herein can be a certain type of programmable processor, programmable digital logic (e.g., field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) or include digital logic, software, code, ASIC, flash memory, optical disc, CD-ROM, DVD ROM, magnetic card or optical card, other types of machine-readable media suitable for storing electronic instructions, any suitable combination of the above.

[0120] In operation, storage may store information in any suitable type of non-transitory storage medium (e.g., random access memory (RAM), read-only memory (ROM), field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or microcode), software, hardware (e.g., processor instructions or microcode), or any other suitable component, device, element, or object, where appropriate and based on specific needs. In addition, based on specific needs and implementations, information tracked, sent, received, or stored in the processor may be provided in any database, register, table, cache, queue, control list, or storage structure, all of which may be referenced within any suitable time frame. Any of the memories or storage elements disclosed herein should be interpreted as being encompassed within the broad terms "memory" and "storage," as appropriate. The non-transitory storage medium herein is expressly intended to include any non-transitory dedicated or programmable hardware configured to provide the disclosed operations or to cause the processor to perform the disclosed operations. Non-transitory storage media also expressly include a processor having stored thereon hardware-encoded instructions, and optionally including microcode instructions or sequences encoded in hardware, firmware, or software.

[0121] The computer program logic that implements all or part of the functionality described herein is embodied in various forms, including but not limited to hardware description languages, source code forms, computer executable forms, machine instructions or microcode, programmable hardware, and various intermediate forms (e.g., forms generated by HDL processors, assemblers, compilers, linkers, or locators). In an example, the source code includes a series of computer program instructions implemented in various programming languages ​​or hardware description languages, such as object code, assembly language, or high-level languages ​​(such as OpenCL, FORTRAN, C, C++, JAVA, or HTML for use with various operating systems or operating environments), hardware description languages ​​such as Spice, Verilog, and VHDL. The source code can define and use various data structures and communication messages. The source code can be in a computer executable form (e.g., via an interpreter), or the source code can be converted (e.g., via a converter, assembler, or compiler) to a computer executable form, or converted to an intermediate form (such as byte code). Any of the above may be used, where appropriate, to create or describe suitable discrete or integrated circuits, whether sequential, combinatorial, state machine, or otherwise.

[0122] In one example, any number of circuits in the accompanying drawings can be implemented on a board of an associated electronic device. The board can be a general-purpose circuit board that can fix various components of the internal electronic system of the electronic device and can further provide connectors for other peripheral devices. More specifically, the board can provide an electrical connection through which other components of the system can communicate electrically. Any appropriate processor and memory can be appropriately coupled to the board based on specific configuration needs, processing requirements, and computing design. Other components (such as external storage, additional sensors, controllers for audio / video display, and peripheral devices) can be attached to the board via a cable, as an insert card, or can be integrated into the board itself. In another example, the circuits in the accompanying drawings can be implemented as independent modules (e.g., devices with associated components and circuits configured to perform specific applications or functions) or as plug-in modules inserted into the dedicated hardware of the electronic device.

[0123] Note that for many of the examples provided herein, interactions can be described in terms of two, three, four, or more electrical components. However, this is done for purposes of clarity and example only. It should be understood that the system can also be combined and reconfigured in any suitable manner. Together with similar design alternatives, any of the components, modules, and elements shown in the drawings can be combined in a variety of possible configurations, all of which are within the broad scope of this specification. In some cases, it may be easier to describe one or more of the functions of a given set of processes by referencing only a limited number of electrical components. It should be understood that the circuits in the drawings and their teachings are easily scalable and can accommodate a large number of components and more complex / refined arrangements and configurations. Accordingly, the examples provided should not limit the scope of the circuits that are potentially applicable to countless other architectures, nor should they inhibit the broad teachings of the circuits.

[0124] Numerous other changes, substitutions, variations, alterations, and modifications may be ascertained by those skilled in the art, and this disclosure is intended to encompass all such changes, substitutions, variations, alterations, and modifications as fall within the scope of the appended claims.

[0125] Example Implementation

[0126] The following examples relate to embodiments described throughout this disclosure.

[0127] One or more embodiments may include an apparatus comprising: a multidimensional memory; a plurality of processing elements for performing a matrix operation on one or more matrix operands, wherein the plurality of processing elements comprises one or more matrix processors; wherein the plurality of processing elements is configured to: receive matrix data from the multidimensional memory, wherein the matrix data is associated with the one or more matrix operands; extract the one or more matrix operands from the matrix data; identify a matrix routine associated with the matrix operation; execute the matrix routine using the one or more matrix operands; and obtain a result of the matrix operation based on the executed matrix routine.

[0128] In an example embodiment of the apparatus, the apparatus further comprises a matrix routine memory, wherein the matrix routine memory comprises one or more matrix routines associated with one or more matrix operations.

[0129] In an example embodiment of the apparatus, the plurality of processing elements are further configured to receive the matrix routine from the matrix routine memory.

[0130] In an example embodiment of the apparatus, the plurality of processing elements are further configured to: receive the matrix routine from a host computing system; and store the matrix routine in the matrix routine memory.

[0131] In an example embodiment of the apparatus, said matrix routine comprises one or more instructions associated with said matrix operations.

[0132] In an example embodiment of the apparatus, the one or more instructions of the matrix routine comprise indications of one or more memory locations associated with the one or more matrix operands.

[0133] In an example embodiment of the apparatus, the plurality of processing elements for extracting the one or more matrix operands from the matrix data are further configured for slicing the matrix data to extract the one or more matrix operands.

[0134] In an example embodiment of the apparatus, the plurality of processing elements are further configured to execute one or more instructions for extracting the one or more matrix operands from the matrix data.

[0135] In an example embodiment of the apparatus, the plurality of processing elements are further configured to execute one or more instructions for retrieving the matrix data from one or more memory locations of the multi-dimensional memory.

[0136] In an example embodiment of the apparatus, said matrix operations comprise one or more matrix multiplication operations.

[0137] In an example embodiment of the apparatus, said matrix operations comprise one or more convolution operations.

[0138] In an example embodiment of the apparatus, said matrix operations are associated with operations in a neural network.

[0139] One or more embodiments may provide a method comprising: performing a matrix operation on one or more matrix operands, wherein performing the matrix operation comprises: receiving matrix data from a multidimensional memory, wherein the matrix data is associated with the one or more matrix operands; extracting the one or more matrix operands from the matrix data; identifying a matrix routine associated with the matrix operation; executing the matrix routine on a matrix processor using the one or more matrix operands; and obtaining a result of the matrix operation based on the matrix routine executed by the matrix processor.

[0140] In an example embodiment of the method, the method further comprises receiving the matrix routine from a matrix routine memory, wherein the matrix routine memory comprises one or more matrix routines associated with one or more matrix operations.

[0141] In an example embodiment of the method, said matrix routine comprises one or more instructions associated with said matrix operations.

[0142] In an example embodiment of the method, the one or more instructions of the matrix routine comprise indications of one or more memory locations associated with the one or more matrix operands.

[0143] One or more embodiments may provide a system comprising: a plurality of memory elements, wherein the plurality of memory elements comprise a multidimensional memory; and a plurality of processing elements for performing matrix operations on one or more matrix operands, wherein the plurality of processing elements comprise: a host processor; and one or more matrix processing chips; wherein the plurality of processing elements are configured to: receive matrix data from the multidimensional memory, wherein the matrix data is associated with the one or more matrix operands; extract the one or more matrix operands from the matrix data; identify a matrix routine associated with the matrix operation; execute the matrix routine using the one or more matrix operands; and obtain a result of the matrix operation based on the executed matrix routine.

[0144] In one example embodiment of the system, each matrix processing chip includes a plurality of matrix processing clusters.

[0145] In an example embodiment of the system, each matrix processing cluster includes a plurality of matrix processing units.

[0146] In an example embodiment of the system, each matrix processing cluster includes a plurality of memory resource blocks.

[0147] One or more embodiments may include at least one machine-accessible storage medium having instructions stored thereon, which, when executed on a machine, cause the machine to: perform a matrix operation on one or more matrix operands, wherein the instructions causing the machine to perform the matrix operation further cause the machine to: receive matrix data from a multidimensional memory, wherein the matrix data is associated with the one or more matrix operands; extract the one or more matrix operands from the matrix data; identify a matrix routine associated with the matrix operation; execute the matrix routine on a matrix processor using the one or more matrix operands; and obtain a result of the matrix operation based on the matrix routine executed by the matrix processor.

[0148] In an example embodiment of the storage medium, the instructions further cause the machine to: receive the matrix routine from a matrix routine memory, wherein the matrix routine memory includes one or more matrix routines associated with one or more matrix operations.

[0149] In an example embodiment of the storage medium, the matrix routine includes one or more instructions associated with the matrix operation.

[0150] In an example embodiment of the storage medium, the matrix routine includes an indication of one or more memory locations associated with the one or more matrix operands.

[0151] In an example embodiment of the storage medium, the instructions that cause the machine to receive the matrix data from the multi-dimensional memory further cause the machine to retrieve the matrix data from one or more memory locations of the multi-dimensional memory.

[0152] One or more embodiments may include an apparatus comprising means for performing the method of any of the foregoing examples.

[0153] One or more embodiments may include at least one machine-accessible storage medium having instructions stored thereon, which, when executed on a machine, cause the machine to: perform a method from any of the foregoing examples or implement an apparatus from any of the foregoing examples.

Claims

1. A device comprising: A plurality of matrix processing chips (220a, ..., 220d), wherein the plurality of matrix processing chips are integrated on a package, and each matrix processing chip (220a, ..., 220d) is used to process matrix instructions; an inter-chip link (ICL) (225) for connecting two or more matrix processing chips among the plurality of matrix processing chips (220a, ..., 220d), the ICL being used to enable communication between the two or more matrix processing chips (220a, ..., 220d); a plurality of high bandwidth memory (HBM) modules (240a, ..., 240d), wherein the HBM modules (240a, ..., 240d) in the plurality of HBM modules (240a, ..., 240d) are used to store matrix data for processing by the matrix processing chips (220a, ..., 220d) in the plurality of matrix processing chips (220a, ..., 220d); The matrix processing chip (220a, ..., 220d) includes: A host interface (224) for coupling the matrix processing chip (220a, ..., 220d) to a host processor (260), a controller (222) for controlling and / or managing matrix operations of the matrix routine; as well as A plurality of matrix processing units (234) are coupled to the controller (222), the plurality of matrix processing units being configured to perform a matrix multiplication operation based on a matrix multiplication instruction from the matrix routine, the matrix multiplication instruction specifying a matrix operand including a first input matrix A and a second input matrix B, the matrix processing unit (234) being configured to generate an output matrix C by multiplying the first input matrix A and the second input matrix B.

2. The device according to claim 1, wherein The matrix processing unit (234) is included in a matrix processing cluster (230), the apparatus further comprising: A plurality of cluster controllers (232), each cluster controller (232) is used to control matrix operations within a corresponding matrix processing cluster (230).

3. The device according to claim 2, wherein The plurality of cluster controllers (232) cause the matrix multiplication instruction to be executed using one or more MPUs (234) in the corresponding cluster.

4. The device according to claim 1, wherein At least one matrix processing unit (234) is configured to receive matrix data from an HBM module (240a, ..., 240d) of the plurality of HBM modules (240a, ..., 240d), wherein the matrix data is associated with the matrix operand of the matrix multiplication instruction.

5. The device according to claim 4, wherein The at least one matrix processing unit (234) is configured to execute the matrix multiplication instruction using the matrix operands.

6. The device according to claim 5, wherein The matrix multiplication instruction is one of a plurality of instructions of the matrix routine, the matrix routine being operable to be executed by at least one of the matrix processing chips (220a, . . . , 220d).

7. The device according to claim 6, wherein The matrix routines include distributed matrix multiplication routines for execution by a plurality of matrix processing chips in the matrix processing chips (220a, ..., 220d).

8. The device according to claim 6, wherein At least one of the matrix processing chips (220a, ..., 220d) is used to identify the matrix routine from a plurality of matrix routines.

9. The device according to claim 1, wherein The host interface (224) is used to couple the matrix processing chip (220a, ..., 220d) to a Peripheral Component Interconnect Express (PCIe) bus.

10. The device according to claim 6, wherein The plurality of instructions of the matrix routine are operable to be executed to perform one or more convolution operations.

11. The device according to claim 10, wherein The matrix routines are associated with operations in neural networks.

12. The device of claim 3, wherein: The controller (222) is configured to control and / or manage matrix operations in conjunction with the plurality of cluster controllers (232).

13. The device of claim 1, wherein: The multiple HBM modules (240a, ..., 240d) include four HBM modules (240a, ..., 240d) coupled to a first matrix processing chip (220a, ..., 220d) among the multiple matrix processing chips (220a, ..., 220d) and four different HBM modules (240a, ..., 240d) coupled to a second matrix processing chip (220a, ..., 220d) among the multiple matrix processing chips (220a, ..., 220d).

14. The device of claim 2, wherein: The controller (222) comprises a chip-level controller.

15. The apparatus of claim 14, wherein: The chip-level controller (222) is used to control and / or manage matrix operations in conjunction with the plurality of cluster controllers (232).

Images