Tensor processing method and apparatus therefor

Abstract

An apparatus for performing data processing according to one embodiment of this specification includes a plurality of processing elements (PEs) configured to perform tensor-based processing; and a semiconductor substrate on which the plurality of PEs are mounted, each PE including a plurality of slice circuits, each having an arithmetic pipeline; and a control circuit for controlling the plurality of slice circuits, each arithmetic pipeline on each slice circuit including i) a fetch unit for fetching at least a portion of the tensor data; ii) an operation unit for performing work on at least a portion of the fetched tensor data; and iii) a commit unit for transmitting the results of the work output from the operation unit to a destination, the control circuit being able to control the mapping between each partition of the tensor data and the plurality of arithmetic pipelines on the plurality of slice circuits.

Description

[Technical Field]

[0001] This specification relates to tensor processing, and more specifically, to a method and apparatus for processing operations on N-dimensional (N≧0) tensors. [Background technology]

[0002] Tensors are multidimensional arrays that are generalizations of vectors and matrices. In addition to scalars (N=0-dimensional tensors), vectors (N=1-dimensional tensors), and matrices (N=2-dimensional tensors), tensors can also represent higher-dimensional arrays of more generalized forms (N>3-dimensional tensors).

[0003] The rank of a tensor indicates its dimension, axis, and number of channels N. For example, a scalar can be represented with rank 0, a vector with rank 1, and a matrix with rank 2.

[0004] The shape of a tensor is a tuple structure that indicates the size of each dimension. For example, the shape of an X by Y matrix can be simply represented as (X, Y). Also, an image with X*Y pixels, each having three channels RGB (Red, Green, Blue), can be represented by a 3D tensor (X, Y, 3).

[0005] In recent years, tensors have been frequently used in artificial intelligence (AI) frameworks, and such AI frameworks provide a variety of functions and operations for generating and processing tensors. [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] The technical problem that this invention aims to solve is to provide a method and apparatus for more efficient tensor processing.

[0007] Other technical problems may be inferred from each embodiment in the detailed description. [MEANS FOR SOLVING THE PROBLEMS]

[0008] An apparatus for processing data according to one aspect of the present invention for achieving the above-described technical problems may include a plurality of PEs (processing elements) configured to perform tensor-based processing; and a semiconductor substrate on which the plurality of PEs are implemented.

[0009] Each PE may include a plurality of slice circuits each having a compute pipeline; and a control circuit that controls the plurality of slice circuits.

[0010] Each compute pipeline on each slice circuit may include i) a fetch unit that fetches at least a part of tensor data, ii) an operation unit that performs an operation on at least a part of the fetched tensor data, and iii) a commit unit that transmits the operation result output from the operation unit to a destination.

[0011] The control circuit may control mapping between each partition of the tensor data and the plurality of compute pipelines on the plurality of slice circuits.

[0012] Preferably, the plurality of slice circuits may be configured / reconfigured based on the shape of the tensor data and an instruction given to the tensor data.

[0013] Preferably, each of the plurality of slice circuits may be assigned a control register that stores configuration / reconfiguration information of the corresponding slice circuit.

[0014] Preferably, the control circuit can perform configuration / reconfiguration of each slice circuit and reading / writing of the associated control registers.

[0015] Preferably, each PE may further include a Scratch Pad Memory (SPM). The data of the plurality of control registers allocated to the plurality of slice circuits may be configured by loading respective values stored in the scratch pad memory.

[0016] Preferably, partitioning of the tensor data may be performed based on at least one of the shape of the tensor data and an instruction word given for the tensor data.

[0017] Preferably, a data memory having a bandwidth associated with the operation pipeline may be allocated for each slice circuit.

[0018] Preferably, the operation unit may include at least one of a contraction engine that performs tensor contraction on at least a part of the fetched tensor data, a vector engine that performs operations on a vector which is a one-dimensional tensor, and a transpose engine that performs data transpose.

[0019] Preferably, the contraction engine can perform the tensor contraction by reducing the rank.

[0020] Preferably, the control circuit can set a configurable fetch network through connections between different respective fetch units on different respective slice circuits.

[0021] Preferably, data fetched by the fetch network may be multicast to different respective operation units of the different respective slice circuits.

[0022] Preferably, the fetch network can perform multicast by reusing the fetched data.

[0023] Preferably, the fetch unit includes a main fetch unit and a subfetch unit, and the commit unit may include a main commit unit and a subcommit unit.

[0024] Preferably, a main data path is configured based on the main fetch unit and the main commit unit, and a sub-data path is configured based on the sub-fetch unit and the sub-commit unit, and operations on the main data path and operations on the sub-data path can be performed simultaneously within a single slice circuit.

[0025] Preferably, the control circuit can configure / reconfigure the first slice circuit and the second slice circuit so that the first operation result of the first operation unit of the first slice circuit is transmitted to the second slice circuit.

[0026] Preferably, each PE may further include a core processor. The control circuit may operate as a coprocessor for the core processor.

[0027] Preferably, the control circuit may include a command processor that executes each instruction word obtained from the core processor. Each instruction word may include at least one of the following: an instruction word for loading a control register address assigned to each slice circuit, and an instruction word for enabling each slice circuit.

[0028] Preferably, the apparatus may further include a memory controller that controls the data flow between the plurality of PEs and the memory device; and an on-chip network formed on the semiconductor substrate for the plurality of PEs and the memory controller.

[0029] Preferably, the tensor data may be related to neural network processing for artificial intelligence.

[0030] Preferably, the device may be a tensor reduction processor for accelerating neural network processing for the artificial intelligence.

[0031] A method for performing data processing according to another aspect of the present invention for achieving the technical problems described above may include: obtaining instructions for tensor data; partitioning the tensor data based on the shape of the tensor data and at least one of the instructions; assigning each partition of the tensor data to a plurality of arithmetic pipelines; and processing each partition of the tensor data through each of the plurality of arithmetic pipelines. Each arithmetic pipeline may include: i) a process of fetching each partition of the tensor data; ii) a process of performing work on the fetched partitions; and iii) a commit process of communicating the results of the work to a destination.

[0032] In another aspect of the present invention for achieving the technical problems described above, a storage medium readable by at least one processor is provided in which each program code is stored, and each operation performed by the at least one processor as the program code is executed by the at least one processor may include: obtaining instructions for tensor data; partitioning the tensor data based on the shape of the tensor data and at least one of the instructions; assigning each partition of the tensor data to a plurality of arithmetic pipelines; and processing each of each partition of the tensor data through each of the plurality of arithmetic pipelines.

[0033] Each arithmetic pipeline may include: i) a process of fetching each partition of the tensor data; ii) a process of performing operations on the fetched partitions; and iii) a commit process of communicating the results of the operations to the destination.

[0034] Preferably, the plurality of arithmetic pipelines are formed on a plurality of slice circuits, and the plurality of slice circuits can be configured / reconfigured based on the shape of the tensor data and the instruction words given to the tensor data.

[0035] Preferably, each of the plurality of slice circuits may be assigned a control register to store configuration / reconfiguration information for each slice circuit.

[0036] Preferably, each operation performed by the at least one processor may further include configuring / reconfiguring each slice circuit and reading / writing associated control registers.

[0037] Preferably, each operation performed by the at least one processor may further include determining the partitioning of the tensor data based on the shape of the tensor data and at least one of the instructions given to the tensor data.

[0038] Preferably, each operation performed by the at least one processor may further include allocating data memory having a bandwidth associated with each arithmetic pipeline.

[0039] Preferably, the operation process may include at least one of the following: a process of performing tensor contraction on at least a portion of the fetched tensor data; a process of performing operations on a vector which is a one-dimensional tensor; and a process of performing data transposition.

[0040] Preferably, at least one processor can perform the tensor reduction by reducing the rank.

[0041] Preferably, the tensor data may be related to neural network processing for artificial intelligence.

[0042] Preferably, the at least one processor may be a tensor reduction processor for accelerating neural network processing for the artificial intelligence. [Effects of the Invention]

[0043] According to one embodiment of the present invention, tensor processing can be performed more efficiently.

[0044] Other diverse technical effects can be derived from each example described in detail. [Brief explanation of the drawing]

[0045] [Figure 1] This diagram shows typical array processors and vector processors. [Figure 2] This diagram illustrates a tensor reduction scheme that performs matrix multiplication operations using fixed sizes. [Figure 3] This figure shows a data processing apparatus (1) according to one embodiment. [Figure 4] This figure shows PE(10) according to one embodiment. [Figure 5] This figure shows a more specific example of the PE(10) structure. [Figure 6] This is a diagram illustrating the partitioning and allocation of tensor data according to one embodiment. [Figure 7-9] This figure shows various examples of partitioning a 3D tensor and then assigning it to four slices. [Figure 10] This diagram illustrates the general operation of the TU(100) and an example of the configuration of the operation unit (175). [Figure 11] This figure shows a specific example of the configuration of a slice (160) according to one embodiment. [Figure 12] This figure shows a detailed example of the structure of a reduced engine (176). [Figure 13] This figure shows a detailed example of a vector engine (177) structure. [Figure 14] This is an example illustrating a fetch network according to one embodiment. [Figure 15-17] This figure illustrates various examples of processing for input tensor data in diverse embodiments. [Figure 18] This figure shows examples of data reuse and dot product in TCP(1). [Figure 19] This flowchart shows a method performed by an apparatus for data processing according to one embodiment. [Figure 20] This figure shows a case in which multiple fetch units and multiple commit units are configured within a single slice according to one embodiment. [Modes for carrying out the invention]

[0046] The examples described below are non-limiting embodiments intended to facilitate understanding of the present invention as described above, and those skilled in the art will understand that some of the embodiments can be combined, omitted, or modified.

[0047] First, as an example to which the tensor processing proposed herein may be applied, we will briefly describe artificial neural networks and their processing.

[0048] Each neuron that makes up the human brain forms a kind of signal circuit, and a data processing structure and method that mimics the signal circuit of each neuron is called an artificial neural network (ANN). In an ANN, a large number of interconnected neurons form a network, and the input / output process for individual neurons can be expressed mathematically as [Output=f(W1XInput1+W2XInput2+…+WNXInputN)]. Wi represents the weight value, and the weight value can have various values ​​depending on the type / model of the ANN, the hierarchy, each neuron, the learning result, etc. In recent years, with the development of computing technology, deep neural network (DNN) processing with many hidden layers has become possible, making training / learning and inference for AI more sophisticated and advanced. The training process in a DNN is called deep learning and includes weight value adjustment. Inference refers to the process of obtaining an output by inputting new data into a trained model. A CNN (convolutional neural network) is one of the representative DNNs and can be constructed based on a convolutional layer, a pooling layer, a fully connected layer, and / or a combination thereof. CNNs have a structure that is advantageous for learning 2D data and are typically known to perform well in image classification and detection.Beyond CNNs, a variety of DNNs are being actively researched and applied in diverse fields. These include RNNs (recurrent neural networks), which excel at processing data with time-series characteristics such as language; LSTMs (long short-term memory), which mimic forgetting of old data; GANs (generative adversarial networks), which generate fake data through competition with training data; and Large Language Models (LLMs), which can infer from user input and generate diverse words. To encompass these various forms of AI-related neural networks, including the DNNs mentioned above, we will hereafter refer to them simply as neural networks (NN).

[0049] Neural network processing can be handled by a variety of devices. While neural network processing can be handled by general-purpose CPUs, general-purpose CPUs have limitations in effectively handling the large-scale parallel processing involved in neural network processing. Therefore, processors with structures more suited to neural network processing are used. An AI accelerator generally refers to a processor used for the purpose of accelerating AI computations. Typically, GPUs (Graphics Processing Units) were initially developed for graphics processing, but due to their superior parallel processing capabilities, they are now widely used as AI accelerators for large-scale data processing, training, and inference of complex models. An NPU (Neural Processing Unit) is a processor developed to perform AI computations faster and is a processor specialized for NN / AI computations. Compared to GPUs, NPUs can implement each layer of the AI ​​model on a semiconductor chip with a physical circuit configuration, providing even higher efficiency for AI computations.

[0050] The term "processor" as used herein may encompass the concept of a stand-alone or co-processor AI accelerator configured to perform NN / AI computations faster, such as a GPU and / or NPU. An AI accelerator may be configured in a form optimized for either training or inference, or in a form for use in both training and inference. While AI accelerators can take many forms, the main performance requirements are as follows: (i) Parallel processing: AI computations require large-scale data processing, and therefore parallel processing must be supported. That is, tensor operations such as matrices and vectors must be efficiently performed. (ii) Power efficiency: Power efficiency is particularly important in environments with limited power, such as edge users / terminals, and in environments such as data centers. (iii) Low latency: Low latency is necessary in areas where real-time performance is critical, such as AI-based detection / autonomous driving, where immediate response is required. (iv) Memory performance: Sufficient memory capacity and high memory bandwidth are required to process large models and tensors. (v) Other requirements may include a high level of accuracy, precise floating-point arithmetic, and compatibility to support a variety of frameworks and libraries.

[0051] Figure 1 is a diagram illustrating the comparison between a typical array processor and a vector processor.

[0052] Array processors and vector processors share the commonality of being processors that effectively support SIMD (Single Instruction, Multiple Data) parallel processing, but they differ in the specific mechanisms of parallel processing as follows:

[0053] (a) The array processor has a large number of PEs to perform the same operation on each of the multiple elements included in the array simultaneously and in parallel.

[0054] (b) Vector processors are optimized for operations based on vectors, which are one-dimensional arrays. Instead of using a large number of PEs, vector processors execute vector instructions through parallelized pipelines. Each instruction is assigned to a different pipeline; for example, addition (ADD) and multiplication (MUL) are performed on different pipelines.

[0055] A vector processor includes vector registers for loading (LD) and storing (ST) vectors, through which data can be loaded and stored in vector units. Additionally, a separate VLEN (vector length register) is required to process vectors of different lengths. Since the elements constituting a vector can be stored apart from each other in memory, a VSTR (vector stride register) is required, where the stride value corresponds to the distance between each vector element.

[0056] Vector processors primarily support mathematical operations, especially vector and matrix operations, and can efficiently perform linear algebra, matrix multiplication, element-wise operations, and dot products. Such vector processors are also used for large-scale tensor operations that frequently occur in the field of AI, and in recent years, GPUs have sometimes included vector processors.

[0057] Tensor Contraction Approach I As used herein, tensor contraction can mean performing a reduction in the dimension of the tensor, and the term "contraction" may be replaced with expressions such as "reduction" or "collapse," or these terms may be used interchangeably.

[0058] One of the main operations of each machine learning model is tensor contraction, which can be associated with adding each element along a certain axis of the tensor. Matrix multiplication (e.g., dot product) is one example of tensor contraction; for example, the product of an m by k matrix and a k by n matrix can be understood as contracting a tensor of size (m*k, k*n) to a tensor of size m*n. Such contraction can be performed based on a common k axis between the matrices.

[0059] On the other hand, as a structure for tensor reduction, a processing technique that decomposes / reconstructs tensor operations using multiple matrix products (hereinafter referred to as "Approach I") can be considered, and for this purpose, a matrix multiplication unit that performs matrix multiplication operations of a constant / fixed size may be included in the processor.

[0060] Figure 2 is an example illustrating the matrix multiplication unit using Approach I.

[0061] First, referring to Figure 2(a), we assume that the matrix multiplication unit is composed of a systolic array with a fixed size of 128 by 128 processing elements (PEs). In this case, the matrix multiplication unit can perform up to 128 dot product operations simultaneously, so maximum utilization (maximum efficiency) can be guaranteed when the SRAM can also support 128 dot product operations, that is, when the number of data elements transmitted from the SRAM at one time is 128. In other words, if the SRAM bandwidth cannot support 128 PEs, it becomes impossible to achieve maximum utilization (maximum efficiency). Furthermore, although the data reuse rate in the matrix multiplication unit is set relatively high to a maximum of 128 times, for operations that do not require data reuse up to 128 times, there are unused PEs remaining, which has the disadvantage of wasting processing resources. In such a case as in Figure 2(a), parallelism and data locality cannot be fully utilized in tensor operations, which can lead to constraints on processing performance.

[0062] When the size of the processor's matrix multiplication unit is fixed to a small size, as shown in Figure 2(b) / (c), the data reuse range is limited, which can reduce efficiency. Furthermore, when large numbers of small PEs are connected via a NoC (network on chip), parallel tensor operations become highly complex. Additionally, programs for a large number of low-performance cores are more difficult to generate than programs for a relatively small number of high-performance cores.

[0063] Tensor data processing, such as inference, exhibits diverse tensor forms, making it important to maximize parallelism and data reuse not only through tensor form but also through batch size. However, with chips based on Approach I, it can be difficult to fully utilize the large matrix multiplication units across diverse forms and types of tensor operations.

[0064] TCP - Novel Tensor Contraction Approach The following describes a processor employing a novel tensor reduction technique that can overcome the shortcomings of Approach I, which is based on fixed-size matrix multiplication, as described above. The processor described later may be the AI ​​accelerator mentioned above, but its scope of application is not limited to AI / neural networks, and it can also be applied to processors for more general-purpose applications (e.g., CPUs) or other specialized applications (e.g., communication modems, signal processors).

[0065] The processors disclosed herein may include a tensor contraction engine, which will be referred to as a tensor contraction processor or simply TCP. Unless otherwise stated in the following description, the proposed processor may mean TCP.

[0066] In the embodiments described later, tensor contraction may be used as a basic element instead of matrix multiplication. Such novel approaches proposed herein not only enable large-scale parallel work, but also allow for pipelining on the time axis, similar to the vector processor characteristics described in Figure 1(b) above.

[0067] Figure 3 shows a data processing device 1 according to one embodiment. In Figure 3, the data processing device 1 may be TCP.

[0068] The data processing apparatus 1 may include a plurality of PEs 10, a memory controller 30, a NoC (network on chip) 20, and a silicon substrate 40.

[0069] The memory controller 30 can control the connection between the high-bandwidth memory (HBMx, not shown) and the multiple PEs 10.

[0070] The NoC 20 can provide a network for data exchange between various components 10 and 30 arranged on the silicon substrate 40. The NoC 20 supports data transmission between PEs, and PEs can transmit data to high-bandwidth memory or other PEs. As an example of data exchange between PEs, in the case of a Large Language Model (LLM) that exceeds the memory size of the data processing device 1, distributed processing is necessary, and therefore data exchange between PEs (inter-chip communication) may be required.

[0071] Each of the multiple PEs 10 can be configured to perform tensor-related data processing independently / individually. For example, the first PE 11 can perform tensor reduction independently of the second PE 12. For example, each PE can also operate as an independent device from the host. For example, each PE can be driven as an independent device by the host.

[0072] Each PE 10 operates independently, possessing its own address space. Access to unauthorized addresses can be restricted by an address translation unit (not shown). The address translation allows for the dynamic allocation of memory space and PE usage at runtime within the PE's address space. This forms the basis for communication between PEs and, combined with PCIe peer-to-peer functionality, enables PEs across multiple chips to exchange data using the same abstraction. This allows for extensible compilation and dynamic resource reservation.

[0073] Figure 4 shows PE 10 according to one embodiment.

[0074] Referring to Figure 4, PE 10 can include a core processor 110, a tensor unit (TU) 100, memory 140, and NoC 120.

[0075] The core processor 110 can control TU 100 as a coprocessor. Considering the latency involved in accessing TU 100 by the core processor 110, TU 100 can operate asynchronously.

[0076] Memory 140 can include SPM (scratch pad memory).

[0077] PE 10 may include a tensor DMA (direct memory access) engine 150. The tensor DMA engine 150 can recognize the shape of the tensor and optimize memory access while the operation is being performed. The tensor DMA engine 150 can index and transmit tensors in any dimensional order.

[0078] The core processor 110 can be booted with software stored in memory 140. The core processor 110 can load a program including tensor operations into memory 140 at the request of the host and execute the program. The core processor 110 can control the TU 100 by reading / writing each register of the TU 100's Tensor Unit Controller (TUC) 130 during the program execution process.

[0079] TU 100 contains multiple slices 160. By slicing / partitioning tensor data into smaller units in this way, a more flexible configuration is possible for various tensor forms. Depending on the settings, the entire set of slices can operate as one large PE, or each slice can operate as a small and independent parallel computing unit. In other words, unlike Approach I, which combines one large memory with a large matrix multiplication unit, the TCP structure proposed herein divides the computing pipeline, which includes each data memory (e.g., SRAM) and arithmetic unit, into many slices 160.

[0080] Each slice 160 may be allocated data memory associated with a compute pipeline. Each compute pipeline may include a fetch-operation-commit process.

[0081] TU 100 can include a large set of control registers that describe the shape and tactics of the tensor. For example, each slice 160 / each arithmetic pipeline may be allocated a control register and controlled individually, or each set of control registers may be configured and controlled identically. For example, multiple sets of control registers may be set simultaneously via broadcast / multicast.

[0082] Depending on the settings of each control register, one or a number of linked tensor operations can be performed. Each slice 160, configured through the control registers, can independently process tensor data (e.g., tensor partitions). The core processor 110 can read / write each control register linked to the arithmetic pipeline of each slice 160 via the TUC 130. Each control register contains the indexing structure for fetch operations and commits of an N-dimensional tensor, and can configure internal buffer indexing, the type and order of operations, and the topology of each slice 160. The set of control registers configured by software defines spatially / temporally parallelized tensor operations. When the activation register among the control registers is set, tensor / vector operations can be performed based on the set of control registers. Each slice 160, configured through the control registers, can independently process tensor data (e.g., tensor partitions).

[0083] The TUC 130 may include a status register that aggregates the states of all slices 160. The core processor 110 can check the state of each slice 160 through the status register.

[0084] The core processor 110 can execute a program consisting of each tensor operation by repeating a series of processes that set, activate, and then check the state of each control register. On the other hand, in some embodiments, instead of the core processor 110 directly writing to each control register, the control register set related to the tensor operation can be stored in memory 140, and the TUC 130 can be configured to load the entire control register set. In other words, instead of directly reading or writing the control registers of slice 160 in TU 100, the core processor 110 can efficiently control TU 100 through TUC 130.

[0085] The TUC 130 may include a command processor (not shown) that can sequentially execute commands given by the core processor 110. The command processor may include a command queue. The core processor 110 can push (read) commands into the command queue of the TUC 130. For example, a command may include, but is not limited to, at least one of the following:

[0086] - "dma(addr,id)": An instruction to transmit a tensor between memory (memory linked to memory controller 30) and data memory 165. For this purpose, an "id" based DMA may be used to specify source / destination tensor information (e.g., shape).

[0087] - "load(addr, csr addr, size)": An instruction that loads a control register of a given size from memory 140 to the control register address of slice 160.

[0088] - "exec(id)": Command to execute the tensor contraction operation.

[0089] - "wait" d / e (id): DMA or execution instruction wait instruction

[0090] Most instructions, including "dma" and "exec," can be executed asynchronously, with data transfer and calculations overlapping. Conversely, if synchronization is required, "wait" may be called. A common instruction sequence is: [dma(.,0)-load()-wait] d (0)-exec(0)-dma(.,1)-load()-wait e (0)-wait dIt can be configured as follows: (1)-exec(1)-..., in which case, first, a tensor is loaded using DMA with id=0 and the control register is configured (load). After DMA is complete, the (dma)id=0 operation may be performed while data for the next iteration is loaded in parallel. The core processor 110 can check the state of the command queue using polling or interrupts.

[0091] The core processor 110 can continuously push commands to the command queue of the TUC 130, preventing the command queue from becoming empty and thus enabling the TU 100 to continuously perform tensor operations. Since the command processor executes the commands in the command queue sequentially, the core processor 110 is responsible for the dynamic control flow.

[0092] For example, the core processor 110 transmits the next instruction (e.g., DMA or an instruction for a new TU operation) to the command queue while the previous tensor operation is in progress on TU 100, and then continues to execute other instructions. Operations can be performed in parallel. TU 100 begins execution after receiving a new instruction in the command queue.

[0093] Figure 5 shows a more specific example of the PE 10 structure.

[0094] Referring to Figure 5, a data memory 165 is allocated for each slice 160, and the arithmetic pipeline for each slice can include at least one of a fetch unit 170, an operation unit 175, and a commit unit 180.

[0095] The fetch unit 170 can continuously / sequentially read at least a portion of the input tensor partitioned in the data memory 165 and then provide it to the operation unit 175 in the required form.

[0096] The operation unit 175 can stream and process the fetched data. The operation unit 175 may include at least one of a reduction engine, a vector engine, and a transposition engine.

[0097] The commit unit 180 can re-store the work results of the operation unit 175 into the data memory 165.

[0098] Data memory 165 can act as a scratchpad and store tensors. It can be used to retrieve input tensors and commit output tensors after processing. Data memory 165 supports virtual addressing through a page table and translates virtual addresses to physical addresses. This feature allows for the dynamic allocation of a contiguous, large memory space, potentially simplifying tensor management.

[0099] On the other hand, while the data memory 165 has a one-dimensional address space, the data stored in the data memory 165 can be an N-dimensional tensor. For example, assuming a three-dimensional tensor with shape (X(height)=3, Y(width)=4, Z(channel)=2) as shown in Table 1, the memory address of each tensor element can change depending on the method of storage in the data memory 165. As an example, Table 2 shows storage using the [X][Y][Z]=[3][4][2](e.g., X-first, Z-last) method, and Table 3 shows storage using [Z][X][Y]=[2][3][4](e.g., Z-first, Y-last). The conversion between Table 2 and Table 3 corresponds to transposition.

[0100] JPEG2026536604000002.jpg39170

[0101] JPEG2026536604000003.jpg15170

[0102] JPEG2026536604000004.jpg14170

[0103] To explain reading / writing to data memory 165, data memory is accessed (read, write) in units of a predefined memory width, and the depth of data memory 165 can be accessed randomly. Also, within the same depth, data equal to the memory width can be read at once (parallel access). However, only one depth can be read per cycle (sequential access). Considering this, when tensor data is stored in data memory 165, the data corresponding to the last axis in the storage method can be read at once in units of the memory width, and data on axes other than the last axis can be accessed randomly, but only N (where N is an integer less than or equal to the width) pieces can be read at a time. For example, assuming a memory width of 4 in the example in Table 3, the memory access structure can be represented as shown in Table 4.

[0104] JPEG2026536604000005.jpg46170

[0105] For the Y axis, which is the last axis in the [Z][X][Y]=[2][3][4] array, four values ​​[y0=a, y1=d, y2=g, y3=j] corresponding to depth 0 can be read at once, but all of these depth 0 values ​​belong to z0. To read each value belonging to z1, random access is made from depth 0 to depth 5, and four values ​​[y0=o, y1=r, y2=v, y3=y] can be read at once, but all of these depth 5 values ​​belong to z1. Thus, in the example in Table 4, for the last axis Y, four values ​​y0 to y3 can be read at once in one width read (one cycle), but for the first axis Z, only one of z0 or z1 can be read.

[0106] In other words, the fetch unit 170 accesses the last dimension (last_dim) of the tensor data stored in the data memory 165 in memory width units, i.e., bank units (e.g., 8B, 16B, 32B / cycles), while dimensions other than the last dimension can be accessed in any dimensional order.

[0107] Commit unit 180 can operate similarly. Commit unit 180 can also perform slicing and reshaping on any dimension, as long as it is not the last dimension. The last dimension can be handled by a separate function for slicing or by transposition. For transposition, please refer to the description of the transposition engine below.

[0108] Figure 6 is a diagram illustrating the partitioning and allocation of tensor data according to one embodiment. For the sake of explanation, we assume that the input tensor to TU 100 is a three-dimensional tensor (X, Y, Z), and that TU 100 contains a total of eight slices 160.

[0109] Referring to Figure 6, the tensor data can be divided into eight data partitions, with each slice being assigned a data partition. For example, if a 3D tensor (X, Y, Z) is partitioned equally in half along each axis, the data memory 165 of each slice 160 can store the 3D tensor partitioned into [X / 2][Y / 2][Z / 2].

[0110] There are many ways to partition a tensor, and it is not limited to the examples given above. For example, any axis of the tensor can be partitioned and assigned. For instance, if Z=8, the 2D tensor (X,Y) corresponding to z0 can be assigned to the first slice, the 2D tensor (X,Y) corresponding to z1 can be assigned to the second slice, and the 2D tensor (X,Y) corresponding to z7 can be assigned to the eighth slice.

[0111] FIGS. 7 to 9 show various examples of assigning a three-dimensional tensor to four slices by partitioning the three-dimensional tensor.

[0112] In FIGS. 7 to 9, for convenience of description, considering a feed-forward layer, it is assumed that the dimension of the input tensor of the feed-forward layer is b*l*e (e.g., b=4, l=4, e=E), and the dimension of the weight value is e*f. b means the batch size, l means the sequence size, e means the input embedding size, and f means the embedding size of the feed-forward network. At this time, the contraction on the e-axis is expressed as "ble, ef→blf". The notation of "inter-slice partitioning shape|intra-slice shape" is used.

[0113] Referring to FIG. 7, as a result of 1 / 2 division on the b-axis and 1 / 2 division on the e-axis, a total of four partitions are generated, and one partition is assigned to each slice. Specifically, since the partitioning along the b-axis has a step size or a stride of 2, b 2 can be expressed as. Since the partitioning is performed over two slices, it can be expressed as [2]. This applies equally to the e-axis. Therefore, since the e-axis has a stride of E / 2 over two slices, e E / 2 can be expressed as [2]. As a result, the inter-slice partitioning shape is b 2 [2]e E / 2 [2]. Each slice receives an allocation of a shape of [2][4][E / 2] corresponding to 1 / 4 of the tensor, each of which identically has a shape of [4][4][E], and each axis has a stride of 1. As a result, the intra-slice shape is b 1 [2]l 1 [4]e 1[E / 2]. On the other hand, each partition in slice 1 and slice 3 that lacks some shading (pattern) markings corresponds to the rear region hidden by the viewpoint of the 3D tensor shown in Figure 7.

[0114] In Figure 8, the inter-slice partitioning shape is b 1 [4] The shape of the slice is l 1 [4]e 1 [E] is the case.

[0115] Referring to Figure 9, the result of dividing the e-axis into quarters is that a total of four partitions are generated, and one partition is assigned to each slice. In Figure 9, the four partitions are sequentially assigned to slices 0 through 3 along the e-axis. In Figure 9, the pattern / shading showing the mapping between the 3D tensor and slices is omitted. In Figure 9, the inter-slice partitioning shape is e E / 4 [4] The shape of the slice is b 1 [4]l 1 [4]e 1 [E / 4]

[0116] As shown in Figures 7 to 9, the shape of a tensor is lowered through tensor partitioning, and each tensor partition of the lowered shape can be assigned to each slice.

[0117] The above examples illustrate a method in which a portion of each axis of a tensor is divided, and the divided tensor is stored in the data memory of each slice with the same size; however, the present invention is not limited to this. Furthermore, in the above examples, for the sake of explanation, the input tensor was shown to be a 3-dimensional tensor, but tensors with more than 3 dimensions may also be input.

[0118] On the other hand, the order of the computing axes in the contraction operation is named the tactic. For example, suppose the expansion shape shown in Figure 9 is used for the ble and ef → blf contractions of each input tensor. The first input tensor is e E / s [s]|b 1 [B]l 1 [L]e 1 It expands as [E / s], and the second input tensor is e E / s [s]|f 1 [F]e 1 Assuming it is expanded as [E / s], B, L, E, and F represent the sizes of each dimension (b, l, e, f) of the input / weighted tensor before expansion, and s represents the number of slices (e.g., s=4 in Figure 9). The output tensor is f F / s [s]|b 1 [B]l 1 [L]f 1 Assume it is expanded as [F / s]. One possible tactic for this is e E / s [s]|b 1 [B]l 1 [L]f 1 [F]e 1 It is [E / s].

[0119] e in the tactic E / s [s] represents the outermost loop on the e-axis with a stride of E / s across s slices, which means that s outermost sub-loops are performed in parallel.

[0120] b in the tactic 1 [B]l 1 [L]f 1 [F]e 1 [E / s](ie, slice tactic) shows how each slice processes the remaining loops. Here, there are four loops on the b, l, f, and e (innermost) axes. e in slice tactic 1[E / s] represents contraction as the innermost loop. Each slice in the innermost loop undergoes contraction. For example, each slice performs the dot product of two vectors of size E / s along the e-axis. f in slice tactics 1 [F] refers to the f-axis, meaning that data fetched before the f-axis loop is reused F times (e.g., reused using the slice's internal input buffer). The f-axis loop is repeated B*L times on the two outermost loops.

[0121] e in the tactic E / s [s] means that the partial dot products with respect to the e-axis output by each of the s outermost subloops must be summed up to derive the final result. The process of calculating the final dot product from the partial dot product values ​​can be performed through the network present between each slice and is distributed according to the shape of the output tensor.

[0122] For a given input tensor and tensor operation pair, a variety of tactics are possible depending on the number of operation units, such as the expanded shapes of the input / output tensors and the number of available slices. Once each operation unit is determined, candidate expanded shapes are searched for, and a candidate that satisfies the performance and power consumption requirements is selected. If the expanded shapes do not match between adjacent operations (e.g., the expanded shape of the output of the preceding layer does not match the expanded shape of the input of the following layer), an additional bridging operation may be performed to convert the expanded shapes.

[0123] The proposed TCP supports dynamic tensor shapes through control registers that describe the tensor shape, as will be described later. For example, in the example in Figure 9 above, the b and l axes can be determined dynamically (e.g., by the token growth of each request and the number of new requests, which are dynamically determined in the LLM service).

[0124] Figure 10 is a diagram illustrating the general operation of the TU 100 and an example of the configuration of the operation unit 175.

[0125] Referring to Figure 10, the operation unit 175 may include at least one of the contraction engine 176, the vector engine 177, and the transposition engine 178.

[0126] The reduction engine 176 is for reducing (partitioned) tensor data.

[0127] In TCP 1, tensor operations are provided primitively, and TU 100 continuously accelerates each operation until the tensor operation is completed according to the length of the given tensor operation, based on the settings for that operation. For example, TU 100 processes tensors not only spatially in parallel but also continuously in the time axis. For this purpose, a fetch-operation-commit pipeline process is used, in which the pipeline can sequentially fetch the given tensor, transmit it to the operation unit, and after the given processing is completed, sequentially write the resulting tensor to data memory.

[0128] For example, as shown in Figure 10, the input tensor is divided and stored in eight slices of data memory 165. Each slice's fetch unit 170 can sequentially fetch each data from the data memory 165 based on tensor information (e.g., tensor shape / dimension information) obtained through the control register.

[0129] Each fetch unit 170 can be configured as a fetch network, as described later, and data fetched through the fetch network can be multicast to multiple operation units 175 (e.g., sequential multicast or at least partially concurrent parallel multicast).

[0130] Each operation unit 175 performs tensor contraction, vector operations, and / or tensor manipulations (e.g., transposition).

[0131] The commit unit 180 can sequentially commit the tensors, which are the work results of each operation unit 175, to the data memory 165 based on the sequence information set in the control register.

[0132] Figure 11 shows a specific example of the configuration of slice 160 according to one embodiment.

[0133] Referring to Figure 11, the fetch unit 170 may include at least one of a fetch sequencer and a fetch process unit.

[0134] The fetch sequencer can generate address sequences for accessing data memory 165. The fetch process unit can perform data preparation such as type conversion and padding.

[0135] The fetch unit 170 can configure a fetch network to transmit data not only to its own slice but also to operation units of other slices.

[0136] A fetch sequencer generates addresses based on a given tensor shape and a specified order. The fetch sequencer and fetch process unit not only generate basic N-dimensional loop-style addresses but also support indirect addressing and table lookups, providing flexibility for various types of tensor operations.

[0137] The data obtained from the data memory 165 of slice 160 can be sequentially transmitted through the fetch network to the operation unit 175, which includes a reduction engine 176, a vector engine 177, and / or a transposition engine 178.

[0138] The reduction engine 176 may include a dot product engine (DPE) 1762 that performs element-wise multiplication and generates a variety of outputs using a reduction tree. The depth of each reduction tree is configurable, and the outputs of the reduction trees can be accumulated over time through an accumulation unit 1763. Each DPE 1762 can receive two input vectors, one of which may be received by the register file 1761 of the reduction engine 176, and the other a vector sequentially transmitted by the fetch unit 170.

[0139] Figure 12 shows a detailed example of the structure of the reduction engine 176. The reduction engine 176 can perform tensor reduction through inner product operations.

[0140] Referring to Figure 12, the data path in the reduction engine 176 includes a feed unit, a register file, an inner product engine (DPE), and an accumulation unit, which can be controlled by a work sequencer.

[0141] For example, the reduction engine 176 may include eight DPEs that spatially sum the element-wise multiplication results of two input vectors through a reduction tree and perform an inner product. Each DPE can acquire two input vectors from a feed unit and a register file. For example, each input may hold 32 BF16 values, 64 FP8 values, 64 INT8 values, or 128 INT4 values. The number of each input element indicates the maximum processing capacity for a given input type.

[0142] The eight DPEs can share inputs with the feed unit (e.g., data reuse via broadcast), while each can receive inputs individually in the register file. The depth of the reduction tree can be set via a control register, which can determine the number of outputs for each DPE. In addition to add-reduction, the DPEs support max-reduction and can support various types of calculations such as max-pooling.

[0143] The feed unit processes packets transmitted through the fetch network, broadcasts the data to multiple DPEs, and reuses the fetched data. The feed unit is controlled by a work sequencer and enables additional data reuse by providing the same data multiple times to eight DPEs in a continuous cycle. The feed unit can also use shifts for data reuse in operations such as convolution and transposed tensors.

[0144] The accumulating unit receives the output of the DPE for temporal accumulation. To reuse the feed unit's inputs temporally, a large number of accumulators are required to maintain diverse partial sums. Since increasing the number of accumulators proportionally increases read / write costs, it is sometimes preferable to allocate as few accumulators as possible per unit and utilize them to the fullest extent.

[0145] As an example, an accumulating unit may contain a total of 1024 accumulators, which can be indexed differently depending on the output size of the DPE. For example, if 32 BF16 inputs (in the feed unit) are used and the reduction tree depth is 0, each DPE will produce 32 multiplication results. Thus, the total number of outputs for 8 DPEs is 256 (= 8 × 32). Given that 1024 accumulators can be used, each output of a DPE may be associated with a maximum of 4 accumulators for accumulation. However, if the reduction tree depth is 3, each DPE will produce 4 outputs (i.e., partial sums), and 8 DPEs will produce 32 outputs. In this case, each output of a DPE may be associated with 32 (= 1024 / 32) accumulators for temporal accumulation.

[0146] The register file serves as a storage location for each operand that is reused in MAC (multiplication and accumulation) operations. Since the register file provides data to each DPE, more banks may be required than the number of DPEs.

[0147] Figure 13 shows a detailed example of the vector engine 177 structure.

[0148] The vector engine 177 can perform non-linear functions, element-wise operations, reductions, and / or type conversions.

[0149] The vector engine 177 can include a cluster of numerous function units. For example, function units may be for various arithmetic and logical operations on INT32 / FP32, functions such as exp or sin, predicated operations, type conversion and / or quantization on INT4 / 8 / 16 / 32, FP8, FP16 / BF16 and FP32. Furthermore, reduction results within and between slices can be routed to a specified single slice or multicasted to multiple slices.

[0150] Returning to Figure 11, the transpose engine 178 can push data to a specified number of rows, pop data to other sets of rows, and change the last axis of the tensor to another axis in the slice. The transpose engine 178 can also use a fetch sequencer and / or commit sequencer to switch between the last axis and other axes.

[0151] The transpose engine 178 and commit unit 180, together with the vector engine, can perform tensor operations within many slices or a single slice. Tensor operations may include transpose, split, slice, concatenate, and / or reshape. Similar to the vector engine 177, such tensor operations may be performed after the inner product operation or in parallel with the inner product.

[0152] The commit unit 180 may include at least one of a commit sequencer and a commit process unit. The commit sequencer can specify the addresses of the commit data to be transmitted sequentially and determine the storage layout of the tensors. The commit process unit can support type conversion and remove padding from the data for compaction during storage.

[0153] Multiple fetch units 170 and commit units 180 can exist within a single slice 160, and each can have its own data path. Through these multiple data paths configured within a single slice, multiple independent operations can be performed simultaneously within a single slice 160.

[0154] Figure 20 shows a case in which multiple fetch units and multiple commit units are configured within a single slice according to one embodiment. In Figure 20, for ease of explanation, the number of each fetch unit and each commit unit per slice is shown as an example where each is 2, but the present invention is not limited to this, and N (where N is an integer of 1 or more) fetch units / commit units can be configured in a single slice.

[0155] When multiple fetch units are configured, one of the fetch units can operate as the main fetch unit 1701, and the remaining fetch units can operate as sub-fetch units 1702.

[0156] When multiple commit units are configured, one of the commit units can act as the main commit unit 1801, and the remaining commit units can act as sub-commit units 1802.

[0157] Multiple data paths can be configured based on the combination of each fetch unit and each commit unit, and multiple contexts (e.g., two in the case of Figure 20) can be performed simultaneously in a single slice based on these multiple data paths.

[0158] However, in some embodiments, the data path configuration may not involve fetch units / commit units. For example, it is possible to configure data paths that do not require fetch units / commit units, such as data movement between register files. When considering such data path configurations, it is possible to configure a number of data paths that exceeds the number of fetch units / commit units.

[0159] Referring to Figure 20(a), the main context performs the operation of fetching through the main fetch unit 1701, then using the reduction engine 176 and the vector engine 177, and committing through the main commit unit 1801. The subcontext performs the operation of fetching through the subfetch unit 1702 and then loading the data into the tensor register file.

[0160] Referring to Figure 20(b), this shows the operation of performing tensor contraction using the contraction engine 176 in the main context, while simultaneously performing transposition using the transposition engine 178 in the subcontext.

[0161] Figure 14 is an example illustrating a fetch network according to one embodiment.

[0162] As described above, each fetch unit 170 of each different slice 160 can constitute a fetch network, and data can be continuously fetched in a pipeline manner through the fetch network. Data fetched through the fetch network can be reused.

[0163] Referring to Figure 14, fetched tensor data can be multicast to various slices through the fetch network. Multicast data can be processed in parallel by many operation units. The fetch network can be configured / reconfigured similarly to a circuit switch (CS) network. The fetch network supports multicast, guaranteeing data order across many sources and data order within packets from a single source. A fetch network similar to a CS can improve throughput in routing, ordering, multicast, and flow control, while reducing hardware complexity.

[0164] TCP 1 can also dynamically merge PEs through the reconfiguration of the fetch network.

[0165] Figures 15 to 17 illustrate various examples of processing for input tensor data according to diverse embodiments.

[0166] The way in which the input tensor is allocated / layout in data memory 165, and the way in which the fetch unit 170, operation unit 175, and / or commit unit 180 are configured, can affect the parallelism and data locality (or data reuse) used in performing tensor operations. Since the available parallelism and data locality differ depending on the shape of the tensor and the type of tensor operation required, TCP 1 can operate flexibly, taking into account the shape of the tensor and the tensor operation.

[0167] Referring to Figure 15, a matrix multiplication is performed between two 2D tensors B01 and B02. The first tensor B01 is divided into four horizontal sections. The second tensor B02 may be stored in the register file of the reduction engine. The third tensor B03 is the result of the tensor operation and is committed to data memory 165. Each slice can be operated on independently and in parallel. In the example in Figure 15, it is assumed that no data is exchanged between slices through a fetch network or vector engine. Each slice can sequentially load each partition of the first tensor B01, provide it to the inner product engine of the operation unit, perform the inner product operation on each, commit the results, and generate an output tensor. In relation to the ble, ef→blf reduction example, Figure 15 has a tail axis that is not the e-axis, and the DPE depth is 0. Each element of the e-axis read from the register file is broadcast, multiplied, and accumulated in the accumulation unit. The partition axis of each slice is either b or l, and they are processed in parallel. The second operand is multicast to all slices and loaded into the register file.

[0168] Referring to Figure 16, a matrix multiplication is performed between two 2D tensors (A01, A02). The first tensor A01 is divided into two parts horizontally and vertically. The second tensor A02 is stored in the contraction engine's register file and can be divided into two parts along the f-axis and stored in each slice. During the fetch / transfer process of the first tensor A01, E / 2 of Slice0 and E / 2 of Slice1 are all transmitted to Slice1 and Slice0, respectively, through the fetch network, and each slice can perform the operation F / 2 along the f-axis independently and in parallel. That is, the fetch network can be configured to multicast to Slice0 and Slice1. The contraction engine (CE) of Slice0 and Slice1 uses all the data from Slice0 and Slice1 and performs an inner product using the corresponding data from the second tensor A02 according to the order of the transmitted data. The vector engine 177 does not need to reduce the data of many slices, so it performs commits in the commit unit 180 for each slice.

[0169] Referring to Figure 17, a matrix multiplication is performed between two 2D tensors 901 and 902. The first tensor 901 is divided into two parts horizontally and vertically. The second tensor 902 may be stored in the register file of the reduction engine. For the matrix multiplication, an inner product is performed on the e-axis. If divided / stored as in Figure 17, Slice0 sequentially transmits E / 2, and Slice1 sequentially transmits the remaining E / 2 to perform the inner product, and (after the necessary processing steps) the third tensor 903 may be output. Specifically, the result output from the reduction engine (CE) can be reduced (summed) across two slices, added across many slices through the reduce & root function of the vector engine (VE), and then transmitted to the target slice. The commit unit 180 can sequentially store the output tensors in the data memory 165. In Figure 17, each data partition of the first tensor 901 is used multiple times (reused) for the second tensor 902. The reduction engine (CE) can feed the transmitted input tensor to the DPE multiple times depending on the control register settings. Based on the fed data, the reduction engine can perform an inner product by sequentially transmitting the data partitions of the associated second tensor 902 to the DPE, and store each intermediate result using a number of accumulators. This allows for the reuse of inputs included in tensor operations. In relation to the ble, ef → blf reduction, in the example in Figure 17, the reduction is performed across two slices. E / 2 [2]|e 1 [E / 2] is performed, b B / 2 [2] Parallel contraction is performed along the axis.

[0170] Figure 18 shows examples of data reuse and dot product in TCP 1.

[0171] As described above, in Approach I, the data supplied to memory is transmitted downwards, and the dot product is performed while the result of the dot product operation is transmitted to the right.

[0172] In the case of the proposed TCP 1, depending on the control register settings of the computation pipeline, including the fetch network, the following actions may occur:

[0173] - Data fetched in each data memory can be multicast to the entire slice. Additionally, multiple dot product engines can be configured to perform dot product calculations within each reduction engine.

[0174] - Data fetched in each data memory is unicast only to its own reduction engine, which can then perform an inner product and reduce the result.

[0175] In this way, each operation unit can be made to its fullest potential to suit various tensor shapes depending on the settings of the control registers, and by continuously fetching, operating, and committing data on the time axis, the pipeline can be made to its fullest potential through the data path that is constructed / reconstructed based on the tensor operation.

[0176] As explained above, TCP proposed herein is a structure that can process tensor operations primitively. Unlike Approach I accelerators, which generally process unit matrix operations of a fixed size using instructions, TCP can specify the entire tensor operation of a variable size in a control register and accelerate it all at once.

[0177] Tensor operations possess very high regularity, a high level of parallelism, and data locality. TCP maximizes the parallelism of the overall tensor operations by allowing each slice to be reconfigured to suit the shape of the tensor being operated on, and ensuring that they are always processed sequentially according to a software-defined order. Through this, it is possible to maximize the use of a limited-size memory buffer and leverage the data locality inherent in the overall tensor operations.

[0178] Similar to Approach I, when processing unit matrix operations of a fixed size with instructions, the degree of data reuse is limited by the size of the matrix operation. However, TCP can improve energy efficiency by reusing more data. Furthermore, because TCP processes instructions in units of whole tensor operations, it can reduce the cost of processing instructions compared to Approach I, which processes instructions for each unit matrix operation, and thus can also reduce the size of the program.

[0179] Thus, TCP can fully utilize the parallelism and data locality in tensor contraction, thereby improving the efficiency and performance of tensor data processing used in AI and other applications. TCP can be composed of large-scale PEs, which has the advantage of simplifying related software development. Furthermore, control overhead can be minimized by performing the entire tensor contraction through a single control register setting.

[0180] To efficiently process operations in various tensor forms, a large number of PEs may be configured as a single operation unit, or as a collection of smaller, independently executable operation units. Furthermore, the fetch network may be configured / reconfigured like a circuit switch, thereby providing a structure that allows for data reuse between / within operation units.

[0181] Supporting dynamic geometry and control flow is crucial for optimizing AI accelerators, such as in LLM inference. The proposed TCP 1 can support dynamic geometry and control flow by setting / resetting control registers, enabling the generation of more efficient programs with dynamically changing computation, memory, and power consumption requirements.

[0182] As described above, the data processing device (e.g., TCP) may include a plurality of PEs 10 configured to perform tensor-based processing, and a semiconductor substrate 40 on which the plurality of PEs 10 are mounted.

[0183] Each PE 10 may include a plurality of slice circuits 160, each having its own arithmetic pipeline; and a control circuit 130 that controls the plurality of slice circuits 160.

[0184] Each arithmetic pipeline on each slice circuit 160 may include: i) a fetch unit 170 that fetches at least a portion of the tensor data; ii) an operation unit 175 that performs operations on at least a portion of the fetched tensor data; and iii) a commit unit 180 that transmits the results of the operations output from the operation unit to a destination.

[0185] The control circuit 130 can control the mapping between each partition of the tensor data and the multiple arithmetic pipelines on the multiple slice circuits 160.

[0186] As an example, the plurality of slice circuits 160 can be configured / reconfigured based on the shape of the tensor data and the instruction words given to the tensor data.

[0187] As an example, each of the multiple slice circuits 160 may be assigned a control register to store configuration / reconfiguration information for each slice circuit.

[0188] As an example, the control circuit 130 can configure / reconfigure each slice circuit 160 and read / write associated control registers.

[0189] As an example, each PE 10 may further include a scratchpad memory (SPM) 140. The set data of the multiple control registers assigned to the multiple slice circuits 160 may be constructed by loading each value stored on the scratchpad memory 140.

[0190] As an example, partitioning of the tensor data can be performed based on the shape of the tensor data and at least one of the instructions given to the tensor data.

[0191] For example, each slice circuit 160 may be allocated a data memory 165 having a bandwidth related to the arithmetic pipeline.

[0192] As an example, the operation unit 175 may include at least one of the following: a tensor reduction engine 176 that performs tensor reduction on at least a portion of the fetched tensor data; a vector engine 177 that performs operations on vectors, which are one-dimensional tensors; and a transposition engine 178 that performs data transposition.

[0193] As an example, the reduction engine 176 can perform tensor reduction by reducing the rank.

[0194] As an example, the control circuit 130 can configure a fetch network that can be formed through connections between different fetch units 170 on different slice circuits 160.

[0195] As an example, the data fetched by the fetch network can be sequentially multicast to each different operation unit 175 of each different slice circuit 160.

[0196] For example, the fetch network can perform multicast by reusing the fetched data.

[0197] For example, the fetch unit may include a main fetch unit and a subfetch unit, and the commit unit may include a main commit unit and a subcommit unit.

[0198] As an example, a main data path may be configured based on the main fetch unit and the main commit unit, and a sub-data path may be configured based on the sub-fetch unit and the sub-commit unit, allowing operations on the main data path and operations on the sub-data path to be performed simultaneously within a single slice circuit.

[0199] As an example, the control circuit 130 can configure / reconfigure the first slice circuit and the second slice circuit so that the first operation result of the first operation unit of the first slice circuit is transmitted to the second slice circuit.

[0200] For example, each PE 10 may further include a core processor 110. The control circuit 130 can operate as a coprocessor for the core processor 110.

[0201] As an example, the control circuit 130 may include a command processor that executes each instruction word obtained from the core processor 110. Each instruction word may include at least one of the following: an instruction word for loading a control register address assigned to each slice circuit 160, and an instruction word for activating each slice circuit 160.

[0202] As an example, the device may further include a memory controller 30 that controls the data flow between the plurality of PEs 10 and the storage device, and an on-chip network 20 formed on the semiconductor substrate 40 for the plurality of PEs 10 and the memory controller 30.

[0203] As an example, the aforementioned tensor data may be relevant to neural network processing for artificial intelligence.

[0204] As an example, the device may be a tensor reduction processor (TCP) 1 for accelerating neural network processing for artificial intelligence.

[0205] Figure 19 shows a flowchart of the method performed by a data processing apparatus according to one embodiment.

[0206] Referring to Figure 19, the device for data processing can obtain instructions for tensor data (1905).

[0207] A data processing device can partition the tensor data based on at least one of the shape of the tensor data and the instruction word (1910).

[0208] The data processing device can assign each partition of the tensor data to multiple arithmetic pipelines (1915).

[0209] The data processing apparatus can process each of the partitions of the tensor data through each of the plurality of arithmetic pipelines (1920). Each arithmetic pipeline may include i) a process of fetching each partition of the tensor data, ii) a process of performing work on the fetched partitions, and iii) a commit process of communicating the results of the work to a destination.

[0210] Each of the embodiments of the present invention described above can be embodied through a variety of means. For example, each embodiment of the present invention can be embodied by hardware, firmware, software, or a combination thereof.

[0211] In the case of hardware implementation, the methods according to each embodiment of the present invention can be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

[0212] In the case of implementation by firmware or software, the methods according to each embodiment of the present invention may be implemented in the form of modules, procedures, or functions that perform the functions or operations described above. The software code may be stored in a memory unit and driven by a processor. The memory unit may be located inside or outside the processor and may exchange data with the processor by various already known means.

[0213] The detailed descriptions of preferred embodiments of the present invention disclosed above are provided so that those skilled in the art can embody and implement the invention. While the above has described preferred embodiments of the invention with reference, those skilled in the art will understand that the invention can be modified and altered in various ways without departing from the scope of the invention. For example, those skilled in the art can use the configurations described in the above embodiments in combination with each other. Thus, the invention is not limited to the embodiments shown herein, but rather provides the broadest possible scope consistent with the principles and novel features disclosed herein.

[0214] The present invention may be embodied in other specific forms without departing from the spirit and essential features of the invention. Therefore, the above detailed description should not be interpreted restrictively in all respects, but should be considered illustrative. The scope of the invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention shall be included within the scope of the invention. Furthermore, embodiments may be formed by combining claims that are not explicitly referenced in the claims, or by including them as new claims through amendments after filing.

Claims

1. In a data processing device, Multiple processing elements (PEs) configured to perform tensor-based processing; and The semiconductor substrate on which the plurality of PEs are mounted includes, Each PE is Multiple slice circuits, each having its own arithmetic pipeline; and Includes a control circuit for controlling the plurality of slice circuits, Each arithmetic pipeline on each slice circuit includes i) a fetch unit that fetches at least a portion of the tensor data, ii) an operation unit that performs operations on at least a portion of the fetched tensor data, and iii) a commit unit that transmits the results of the operations output from the operation unit to the destination. The control circuit is a device that controls the mapping between each partition of the tensor data and the multiple arithmetic pipelines on the multiple slice circuits.

2. The apparatus according to claim 1, wherein the plurality of slice circuits are configured / reconfigured based on the shape of the tensor data and the instruction words given to the tensor data.

3. The apparatus according to claim 2, wherein each of the plurality of slice circuits is assigned a control register for storing configuration / reconfiguration information of each slice circuit.

4. The apparatus according to claim 3, wherein the control circuit performs configuration / reconfiguration of each slice circuit and reading / writing of associated control registers.

5. Each PE further includes a scratchpad memory (SPM), The apparatus according to claim 3, wherein the data of the multiple control registers assigned to the multiple slice circuits is configured by loading each value stored in the scratchpad memory.

6. The apparatus according to claim 1, wherein partitioning of the tensor data is performed based on the shape of the tensor data and at least one of the instructions given to the tensor data.

7. The apparatus according to claim 1, wherein each slice circuit is allocated a data memory having a bandwidth related to the arithmetic pipeline.

8. The apparatus according to claim 1, wherein the operation unit includes at least one of a tensor contraction engine that performs tensor contraction on at least a portion of the fetched tensor data, a vector engine that performs operations on a vector which is a one-dimensional tensor, and a transposition engine that performs data transposition.

9. The apparatus according to claim 8, wherein the reduction engine performs tensor reduction by reducing the rank.

10. The apparatus according to claim 1, wherein the control circuit sets up a fetch network that can be configured through connections between different fetch units on different slice circuits.

11. The apparatus according to claim 10, wherein the data fetched by the fetch network is multicast to each different operation unit of each different slice circuit.

12. The apparatus according to claim 11, wherein the fetch network performs multicast by reusing the fetched data.

13. The fetch unit includes a main fetch unit and a sub-fetch unit, The apparatus according to claim 1, wherein the commit unit includes a main commit unit and a sub-commit unit.

14. The main data path is configured based on the main fetch unit and the main commit unit. A subdata path is configured based on the subfetch unit and the subcommit unit. The apparatus according to claim 13, wherein calculations on the main data path and calculations on the sub-data path are performed simultaneously within a single slice circuit.

15. The apparatus according to claim 1, wherein the control circuit configures / reconfigures the first slice circuit and the second slice circuit so that the first operation result of the first operation unit of the first slice circuit is transmitted to the second slice circuit.

16. Each PE further includes a core processor, The apparatus according to claim 1, wherein the control circuit operates as a coprocessor for the core processor.

17. The control circuit includes a command processor that executes each instruction word obtained from the core processor, The apparatus according to claim 16, wherein each instruction word includes at least one of the following: an instruction word for loading a control register address assigned to each slice circuit, and an instruction word for activating each slice circuit.

18. A memory controller that controls the data flow between the plurality of PEs and the storage device; and The apparatus according to claim 1, further comprising an on-chip network formed on the semiconductor substrate for the plurality of PEs and the memory controller.

19. The apparatus according to claim 1, wherein the tensor data relates to neural network processing for artificial intelligence.

20. The apparatus according to claim 19, wherein the apparatus is a tensor reduction processor for accelerating neural network processing for artificial intelligence.

21. In a method for performing data processing, Obtain the command for the tensor data; Partitioning of the tensor data is performed based on the shape of the tensor data and at least one of the instructions; Assign each partition of the tensor data to multiple arithmetic pipelines; and This includes processing each of the partitions of the tensor data through each of the plurality of arithmetic pipelines, Each arithmetic pipeline is a method comprising: i) a process of fetching each partition of the tensor data; ii) a process of performing work on the fetched partitions; and iii) a commit process of communicating the results of the work to a destination.

22. Each program code is stored in a storage medium readable by at least one processor, and each operation performed by the at least one processor as a result of the execution of each program code by the processor is: Obtain the command for the tensor data; Partitioning of the tensor data is performed based on the shape of the tensor data and at least one of the instructions; Assign each partition of the tensor data to multiple arithmetic pipelines; and This includes processing each of the partitions of the tensor data through each of the plurality of arithmetic pipelines, Each arithmetic pipeline is a storage medium comprising: i) a process of fetching each partition of the tensor data; ii) a process of performing operations on the fetched partitions; and iii) a commit process of transmitting the results of the operations to a destination.

23. The plurality of arithmetic pipelines are formed on a plurality of slice circuits, The storage medium according to claim 22, wherein the plurality of slice circuits are configured / reconfigured based on the shape of the tensor data and the instruction words given to the tensor data.

24. The storage medium according to claim 23, wherein each of the plurality of slice circuits is assigned a control register for storing configuration / reconfiguration information of each slice circuit.

25. Each of the operations performed by the aforementioned at least one processor is: The storage medium according to claim 24, further comprising configuring / reconfiguring each slice circuit and reading / writing associated control registers.

26. Each of the operations performed by the aforementioned at least one processor is: The storage medium according to claim 22, further comprising determining the partitioning of the tensor data based on the shape of the tensor data and at least one of the instruction words given to the tensor data.

27. Each of the operations performed by the aforementioned at least one processor is: The storage medium according to claim 22, further comprising allocating data memory having associated bandwidth for each arithmetic pipeline.

28. The storage medium according to claim 22, wherein the operation process includes at least one of the following steps: a process of performing tensor contraction on at least a portion of the fetched tensor data; a process of performing operations on a vector which is a one-dimensional tensor; and a process of performing data transposition.

29. The storage medium according to claim 28, wherein the at least one processor performs the tensor reduction by reducing the rank.

30. The tensor data is stored in a storage medium according to claim 22, relating to neural network processing for artificial intelligence.

31. The storage medium according to claim 22, wherein the at least one processor is a tensor reduction processor for accelerating neural network processing for artificial intelligence.

Images