Tensor processing method and apparatus therefor

Abstract

An apparatus for performing data processing according to one embodiment of the present specification comprises: a plurality of processing elements (PEs) configured to perform tensor-based processing; and a semiconductor substrate on which the plurality of PEs are mounted, wherein each PE comprises: a plurality of slice circuits each having a compute pipeline; and a control circuit for controlling the plurality of slice circuits. Each compute pipeline on each slice circuit comprises: i) a fetch unit for fetching at least a part of tensor data; ii) an operation unit for performing an operation on at least a portion of the fetched tensor data, and iii) a commit unit for transmitting an operation result outputted from the operation unit to a destination, wherein the control circuit can control mapping between partitions of the tensor data and the plurality of compute pipelines on the plurality of slice circuits.

Description

Tensor processing method and device therefor

[0001] This specification relates to tensor processing, and more specifically, to a method for processing operations on N-dimensional (N≥0) tensors and a device for performing the same.

[0002] A tensor is a multidimensional array that generalizes vectors and matrices. In addition to scalars (N=0-dimensional tensors), vectors (N=1-dimensional tensors), and matrices (N=2-dimensional tensors), tensors can also represent more generalized, higher-dimensional arrays (N>3-dimensional tensors).

[0003] The rank of a tensor represents the number of dimensions / axes / channels N of the tensor. For example, a scalar can be represented as rank 0, a vector as rank 1, and a matrix as rank 2.

[0004] The shape of a tensor is a tuple structure that represents the size of each dimension. For example, the shape of an X by Y matrix can be simply expressed as (X, Y). Also, an image with X * Y pixels each having three channels RGB (Red, Green, Blue) can be expressed as a 3D tensor (X, Y, 3).

[0005] Recently, tensors have been frequently used in artificial intelligence (AI) frameworks, and these AI frameworks provide various functions and operations for creating / processing tensors.

[0006] The technical problem to be achieved by the present invention is to provide a method for performing tensor processing more efficiently and a device therefor.

[0007] Other technical challenges can be inferred from the examples in the detailed description.

[0008] A device for performing data processing according to one aspect of the present invention for achieving the above-described technical task may include a plurality of processing elements (PEs) configured to perform tensor-based processing; and a semiconductor substrate on which the plurality of PEs are mounted.

[0009] Each PE may include a plurality of slice circuits, each having a compute pipeline; and a control circuit that controls the plurality of slice circuits.

[0010] Each operation pipeline on each slice circuit may include i) a fetch unit that fetches at least a portion of tensor data, ii) an operation unit that performs an operation on at least a portion of the fetched tensor data, and iii) a commit unit that transmits the operation result output from the operation unit to a destination.

[0011] The above control circuit can control mapping between partitions of the tensor data and a plurality of operation pipelines on the plurality of slice circuits.

[0012] Preferably, the plurality of slice circuits can be configured / reconfigured based on the shape of the tensor data and a command given to the tensor data.

[0013] Preferably, each of the plurality of slice circuits may be allocated a control register that stores configuration / reconfiguration information of each slice circuit.

[0014] Preferably, the control circuit can perform configuration / reconfiguration of each slice circuit and reading / writing of associated control registers.

[0015] Preferably, each PE may further include a scratch pad memory (SPM). Data of a plurality of control registers allocated to the plurality of slice circuits may be configured by loading values ​​stored in the scratch pad memory.

[0016] Preferably, partitioning of the tensor data can be performed based on at least one of a shape of the tensor data and a command given to the tensor data.

[0017] Preferably, each slice circuit may be allocated a data memory having a bandwidth related to the computation pipeline.

[0018] Preferably, the operation unit may include at least one of a reduction engine that performs tensor contraction on at least a portion of the patched tensor data, a vector engine that performs operations on vectors that are one-dimensional tensors, and a transposition engine that performs data transposition.

[0019] Preferably, the reduction engine can perform the tensor reduction by reducing the rank.

[0020] Preferably, the control circuit can establish a configurable patch network through connections between different patch units on different slice circuits.

[0021] Preferably, data patched by the patch network can be multicast to different operation units of the different slice circuits.

[0022] Preferably, the patch network can perform the multicast by reusing the patched data.

[0023] Preferably, the patch unit may include a main patch unit and a sub patch unit, and the commit unit may include a main commit unit and a sub commit unit.

[0024] Preferably, a main data path is configured based on the main patch unit and the main commit unit, and a sub data path is configured based on the sub patch unit and the sub commit unit, and operations on the main data path and operations on the sub data path can be performed simultaneously within one slice circuit.

[0025] Preferably, the control circuit can configure / reconfigure the first slice circuit and the second slice circuit so that the first operation result of the first operation unit of the first slice circuit is transmitted to the second slice circuit.

[0026] Preferably, each PE may further include a core processor. The control circuit may operate as a co-processor for the core processor.

[0027] Preferably, the control circuit may include a command processor that executes instructions obtained from the core processor. The instructions may include at least one of an instruction for loading a control register address assigned to each slice circuit and an instruction for enabling each slice circuit.

[0028] Preferably, the device may further include a memory controller that controls data flow between the plurality of PEs and the memory device; and an on-chip network formed on the semiconductor substrate for the plurality of PEs and the memory controller.

[0029] Preferably, the tensor data may be related to neural network processing for artificial intelligence.

[0030] Preferably, the device may be a tensor reduction processor for accelerating neural network processing for the artificial intelligence.

[0031] According to another aspect of the present invention for achieving the above-described technical task, a method for performing data processing may include obtaining a command for tensor data; performing partitioning on the tensor data based on at least one of a shape of the tensor data and the command; assigning partitions of the tensor data to a plurality of compute pipelines; and processing each partition of the tensor data through each of the plurality of compute pipelines. Each compute pipeline may include: i) a process of fetching each partition of the tensor data, ii) a process of performing an operation on the fetched partition, and iii) a process of committing the operation to transmit the result of performing the operation to a destination.

[0032] According to another aspect of the present invention for achieving the above-described technical task, there may be provided a storage medium readable by at least one processor, in which program codes are stored, and operations performed by the at least one processor as the program codes are executed by the at least one processor may include: obtaining an instruction for tensor data; performing partitioning on the tensor data based on at least one of a shape of the tensor data and the instruction; assigning partitions of the tensor data to a plurality of compute pipelines; and processing each of the partitions of the tensor data through each of the plurality of compute pipelines.

[0033] Each operation pipeline may include: i) a process of patching each partition of the tensor data, ii) a process of performing an operation on the patched partition, and iii) a process of committing the result of performing the operation to a destination.

[0034] Preferably, the plurality of operation pipelines are formed on a plurality of slice circuits, and the plurality of slice circuits can be configured / reconfigured based on the shape of the tensor data and a given command for the tensor data.

[0035] Preferably, each of the plurality of slice circuits may be allocated a control register that stores configuration / reconfiguration information of each slice circuit.

[0036] Preferably, the operations performed by said at least one processor may further include performing configuration / reconfiguration of each slice circuit and reading / writing of associated control registers.

[0037] Preferably, the operations performed by the at least one processor may further include determining a partitioning of the tensor data based on at least one of a shape of the tensor data and a command given to the tensor data.

[0038] Preferably, the operations performed by said at least one processor may further include allocating data memory having an associated bandwidth for each computation pipeline.

[0039] Preferably, the operation process may include at least one of a process of performing tensor contraction on at least a portion of the patched tensor data, a process of performing an operation on a vector which is a one-dimensional tensor, and a process of performing data transpose.

[0040] Preferably, said at least one processor can perform said tensor reduction by reducing the rank.

[0041] Preferably, the tensor data may be related to neural network processing for artificial intelligence.

[0042] Preferably, said at least one processor may be a tensor reduction processor for accelerating neural network processing for said artificial intelligence.

[0043] According to one embodiment of the present invention, tensor processing can be performed more efficiently.

[0044] Various other technical effects can be derived from the embodiments described in detail.

[0045] Figure 1 illustrates a typical array processor and vector processor.

[0046] Figure 2 is a diagram for explaining a tensor reduction method that performs matrix multiplication operations according to a fixed size.

[0047] Figure 3 illustrates a device (1) for data processing according to one embodiment.

[0048] Figure 4 illustrates a PE (10) according to one embodiment.

[0049] Figure 5 shows a more specific example of the PE(10) structure.

[0050] Figure 6 is a diagram for explaining partitioning and allocation of tensor data according to one embodiment.

[0051] Figures 7 to 9 illustrate various examples of partitioning a 3D tensor and assigning it to four slices.

[0052] Figure 10 is a drawing for explaining an example of the overall operation of TU (100) and the configuration of the operation unit (175).

[0053] Fig. 11 illustrates a specific implementation example of the configuration of a slice (160) according to one embodiment.

[0054] Figure 12 shows a detailed example of the structure of a reduced engine (176).

[0055] Figure 13 shows a detailed example of the structure of a vector engine (177).

[0056] Figure 14 is an example for explaining a patch network according to one embodiment.

[0057] FIGS. 15 to 17 are diagrams illustrating examples of processing of input tensor data according to various embodiments.

[0058] Figure 18 illustrates examples of data reuse and dot product in TCP(1).

[0059] FIG. 19 illustrates a flowchart of a method performed in a device for data processing according to one embodiment.

[0060] FIG. 20 illustrates a case where multiple patch units and multiple commit units are configured within one slice according to one embodiment.

[0061] The examples described below are non-limiting examples to help understand the present invention described above, and those skilled in the art will understand that some of the examples can be combined / omitted / changed.

[0062] First, we briefly describe an artificial neural network and its processing as an example to which the tensor processing proposed in this specification can be applied.

[0063] The neurons that make up the human brain form a kind of signal circuit, and the data processing structure and method that imitates the signal circuit of neurons are called artificial neural networks (ANNs). In an ANN, multiple interconnected neurons form a network, and the input / output process for each neuron can be mathematically expressed as [Output = f(W1×Input1 + W2×Input2 + ... +WN×InputN)]. Wi represents a weight, and the weight can have various values ​​depending on the type / model of the ANN, layer, each neuron, and learning results. With the recent development of computing technology, deep neural network (DNN) processing with multiple hidden layers has become possible, and training / learning and inference for AI are becoming more sophisticated and advanced. The training process in DNNs is referred to as deep learning and includes weight adjustments, etc. Inference refers to the process of obtaining output by inputting new data into a trained model. A convolutional neural network (CNN) is a representative type of deep neural network (DNN) and can be constructed based on convolutional layers, pooling layers, fully connected layers, and / or a combination thereof. CNNs possess a structure advantageous for learning two-dimensional data and are known to exhibit excellent performance in tasks such as image classification and detection.In addition to CNNs, various types of DNNs are being actively researched and applied in various fields, including recurrent neural networks (RNNs), which excel at processing data with time-series characteristics such as language, long short-term memories (LSTMs) that mimic forgetting of old data, generative adversarial networks (GANs), which generate fake data by competing with training data, and large language models (LLMs), which can generate various words by inferring from user input. Various types of AI-related neural networks, including the DNNs described above, are collectively referred to as neural networks (NNs) hereinafter.

[0064] Neural network processing can be handled by a variety of devices. While neural network processing is not impossible with standard CPUs, they have limitations in effectively handling the large-scale parallel processing required for neural network processing. Therefore, processors with architectures more suited to neural network processing are being used. AI accelerators generally refer to processors used for the purpose of accelerating AI operations. For example, the GPU (Graphics Processing Unit), initially intended for graphics processing, has recently become widely used as an AI accelerator for large-scale data processing, complex model learning, and inference due to its superior parallel processing capabilities. The NPU (Neural Processing Unit) is a processor developed to perform AI operations more quickly and is specialized for NN / AI operations. Compared to GPUs, the NPU implements AI model layers as physical circuits on a semiconductor chip, providing greater efficiency for AI operations.

[0065] The term "processor" as used herein may encompass standalone or co-processor AI accelerators, such as GPUs and / or NPUs, that are configured to perform NN / AI operations more quickly. Depending on the implementation, AI accelerators may be configured to be optimized for either training or inference, or for use in both training and inference. Although AI accelerators come in various forms, the main performance requirements are as follows: (i) Parallel processing: AI operations require large-scale data processing, so parallel processing must be supported. That is, tensor operations such as matrices and vectors must be performed efficiently. (ii) Power efficiency: Power efficiency is particularly important in environments with limited power, such as edge users / terminals, or in environments such as data centers. (iii) Low-latency characteristics: In areas where real-time performance is important, such as AI-based detection / autonomous driving, immediate responses are required, so low-latency is necessary. (iv) Memory performance: Processing large models and tensors requires sufficient memory capacity and high memory bandwidth. (v) Additionally, high accuracy and precise floating-point operations, as well as compatibility with various frameworks and libraries, may be required.

[0066] Figure 1 is a drawing for comparing and explaining a general array processor and a vector processor.

[0067] Array processors and vector processors have in common that they effectively support parallel processing of SIMD (Single Instruction, Multiple Data), but there are differences in the specific mechanisms of parallel processing as follows.

[0068] (a) The array processor has multiple PEs to perform the same operation on each of multiple elements included in the array simultaneously and in parallel.

[0069] (b) Vector processors are optimized for processing operations based on vectors, which are one-dimensional arrays. Instead of using multiple processors, vector processors execute vector instructions through pipeline parallelism. The pipelines are divided according to each instruction; for example, addition (ADD) and multiplication (MUL) are performed in separate pipelines.

[0070] A vector processor includes vector registers for vector load (LD) / store (ST), allowing it to load and store data in vector units. Additionally, vector length registers (VLEN) are required to handle vectors of different lengths. Because each element constituting a vector can be stored separately in memory, a vector stride register (VSTR) is required, with the stride value corresponding to the spacing between vector elements.

[0071] Vector processors primarily support mathematical operations, particularly vector and matrix operations, and can efficiently perform linear algebra, matrix multiplication, element-wise operations, and inner products. These vector processors are also frequently used in AI for large-scale tensor operations, and recent GPUs also include vector processors.

[0072] Tensor Contraction Approach I

[0073] The contraction of a tensor used in this specification may mean reducing the dimension of the tensor, and the term "contraction" may be replaced with or used interchangeably with expressions such as "reduction" or shrinkage.

[0074] A key operation in machine learning models is tensor reduction, which can involve adding elements along certain axes of a given tensor. Matrix multiplication (e.g., dot product) is an example of tensor reduction. For example, the multiplication of an m-by-k matrix and a k-by-n matrix can be understood as reducing a tensor of size (m*k, k*n) to a tensor of size m*n. Such reduction can be performed based on the k-axis common to the matrices.

[0075] Meanwhile, a processing technique (hereinafter referred to as 'Approach I') that decomposes / reconstructs tensor operations into multiple matrix multiplications as a structure for tensor reduction may be considered, and for this purpose, a matrix multiplication unit that performs matrix multiplication operations of a constant / fixed size may be included in the processor.

[0076] Figure 2 is an example for explaining a matrix multiplication unit according to Approach I.

[0077] First, referring to (a) of Fig. 2, it is assumed that the matrix multiplication unit is configured as a systolic array and its size is fixed to 128 by 128 PEs (processing elements). In this case, since the matrix multiplication unit can perform up to 128 dot product operations simultaneously, the SRAM must also be able to support 128 dot product operations, that is, the number of data elements transferred from the SRAM at one time must be 128 to ensure maximum utilization (maximum efficiency). In other words, if the SRAM bandwidth cannot support 128 PEs, maximum utilization (maximum efficiency) cannot be achieved. In addition, although the maximum data reuse rate in the matrix multiplication unit is set relatively large at 128 times, there is a disadvantage in that unused PEs remain for operations that do not require data reuse up to 128 times, resulting in a waste of processing resources. In the case of (a) of Fig. 2, parallelism and data locality are not sufficiently utilized in tensor operations, which may result in limitations in processing performance.

[0078] If the size of the processor's matrix multiplication unit is fixed to a small size, as in (b) / (c) of Fig. 2, the data reuse range is limited, which can reduce efficiency. Furthermore, when a large number of small PEs are connected through a network on chip (NoC), parallel execution of tensor operations becomes highly complex. Furthermore, programs for a large number of low-performance cores are more difficult to create than programs for a relatively small number of high-performance cores.

[0079] In tensor data processing, such as inference, various tensor shapes are represented, so maximizing parallelism and data reuse based not only on tensor shape but also on batch size is crucial. However, for chips based on Approach I, it may be difficult to fully utilize the massive matrix multiplication unit across various shapes and types of tensor operations.

[0080] TCP - Novel Tensor Contraction Approach

[0081] Below, we describe a processor that employs a novel tensor reduction technique that addresses the shortcomings of Approach I, which relies on fixed-size matrix multiplication described above. While the processor described below may be an AI accelerator as mentioned above, its application is not limited to AI / neural networks, and it can also be applied to more general-purpose processors (e.g., CPUs) or other special-purpose processors (e.g., communication modems, signal processors, etc.).

[0082] The processor disclosed herein may include a tensor contraction engine, which will be referred to as a tensor contraction processor or simply TCP. Unless otherwise stated in the following description, the proposed processor may refer to a TCP.

[0083] In the embodiments described below, tensor reduction can be used as a basic element instead of matrix multiplication. This novel approach proposed in this specification not only enables large-scale parallel operations, but also enables pipelining on the time axis, similar to the vector processor characteristics discussed previously in Figure 1(b).

[0084] Fig. 3 illustrates a device (1) for data processing according to one embodiment. In Fig. 3, the device (1) for data processing may be a TCP.

[0085] A device (1) for data processing may include a plurality of PEs (10) and a memory controller (30), a network on chip (NoC) 20, and a silicon substrate (40).

[0086] The memory controller (30) can control the connection between high bandwidth memory (HBMx, not shown) and multiple PEs (10).

[0087] The NoC (20) can provide a network for data exchange between various components (10, 30, etc.) arranged on a silicon substrate (40). The NoC (20) supports data transfer between PEs, and the PEs can transfer data to high-bandwidth memory or other PEs. As an example of data exchange between PEs, a large language model (LLM) that exceeds the memory size of the device (1) for data processing may require distributed processing, and thus data exchange (inter-chip communication) between PEs may be required.

[0088] Each of the multiple PEs (10) may be configured to independently / individually perform data processing related to a tensor. For example, the first PE (11) may perform tensor reduction independently from the second PE (12). For example, each PE may operate as an independent device from the host. For example, each PE may be driven as an independent device by the host.

[0089] Each PE (10) has an independent address space and operates independently. Access to unauthorized addresses may be restricted by an address translation unit (not shown). Address translation allows dynamic allocation of memory space and PE usage at runtime within the PE's address space. This forms the basis for inter-PE communication and, combined with the PCIe peer-to-peer functionality, enables PEs across multiple chips to exchange data using the same abstraction. This enables scalable compilation and dynamic resource reservation.

[0090] Figure 4 illustrates a PE (10) according to one embodiment.

[0091] Referring to FIG. 4, PE (10) may include a core processor (110), a tensor unit (TU) (100), a memory (140), and a NoC (120).

[0092] The core processor (110) can control the TU (100) as a co-processor. Considering that there is a delay time in accessing the TU (100) from the core processor (110), the TU (100) can operate asynchronously.

[0093] The memory (140) may include a scratch pad memory (SPM).

[0094] The PE (10) may include a Tensor DMA (direct memory access) engine (150). The Tensor DMA engine (150) may recognize the shape of a tensor and optimize memory access during operations. The Tensor DMA engine (150) may index and transfer tensors in any dimension order.

[0095] The core processor (110) can be booted with software stored in the memory (140). The core processor (110) can load a program including tensor operations into the memory (140) at the request of the host and execute the program. The core processor (110) can control the TU (100) by reading / writing registers of the tensor unit controller (TUC 130) of the TU (100) during the process of executing the program.

[0096] TU (100) includes multiple slices (160). By slicing / partitioning tensor data into smaller units in this way, more flexible configurations are possible for various tensor shapes. Depending on the settings, the entire set of slices can operate as a single large PE, or each slice can operate as a small, independent parallel computing unit. That is, in the TCP structure proposed in this specification, unlike Approach I, which combines a single large memory and a large matrix multiplication unit, the computation (computation) pipeline including each data memory (e.g., SRAM) and computational unit is divided into multiple slices (160).

[0097] Each slice (160) may be allocated data memory related to a compute pipeline. Each compute pipeline may include a fetch-operation-commit process.

[0098] TU (100) may include a large set of control registers that describe the shape and tactics of the tensor. For example, each slice (160) / each computational pipeline may be individually controlled by assigning a control register, but the control register sets may also be configured identically to control them identically. For example, multiple control register sets may be configured simultaneously via broadcast / multicast.

[0099] Depending on the settings of the control registers, one or more connected tensor operations can be performed. Each slice (160) set through the control register can independently process tensor data (e.g., tensor partition). The core processor (110) can read / write the control registers linked to the operation pipeline of each slice (160) through the Tensor Unit Controller, TUC (130). The indexing structure of fetch-operation-commit of an N-dimensional tensor, the indexing of the internal buffer, the type and order of operations, the topology of the slices (160), etc. can be set in the control registers. The set of control registers set by software defines a tensor operation that is spatially / temporally parallelized. When an activation register is set among the control registers, a tensor / vector operation can be performed based on the set of the set control registers. Each slice (160) set through the control register can independently process tensor data (e.g., tensor partition).

[0100] The TUC (130) may include a status register that aggregates the status of all slices (160). The core processor (110) can check the status of each slice (160) through the status register.

[0101] The core processor (110) can execute a program consisting of tensor operations by repeating a series of processes of setting control registers, enabling them, and checking their status. Meanwhile, depending on the embodiment, instead of the core processor (110) directly writing control registers, the core processor (110) may store a set of control registers related to tensor operations in the memory (140) and have the TUC (130) load the entire set of control registers. That is, instead of directly reading / writing the control registers of the slices (160) within the TU (100), the core processor (110) can efficiently control the TU (100) through the TUC (130).

[0102] The TUC (130) may include a command processor (not shown) that can sequentially execute commands given from the core processor (110). The command processor may include a command queue. The core processor (110) may push (write) a command to the command queue of the TUC (130). For example, the command may include at least one of the following, but is not limited thereto.

[0103] - 'dma(addr, id)': A command to transfer a tensor between memory (memory connected to 30) and data memory (165). For this purpose, 'id'-based DMA that specifies source / destination tensor information (e.g., shape) can be used.

[0104] - 'load(addr, csr addr, size)': Command to load a control register of a given size from memory (140) to the control register address of a slice (160).

[0105] - 'exec(id)': Command to execute tensor reduction operation

[0106] - 'wait d / e(id)': Command to wait for DMA or execution command

[0107] Most commands, including 'dma' and 'exec', can be executed asynchronously, allowing data transfer and computation to overlap. Alternatively, 'wait' can be called if synchronization is required. An example of a typical command sequence is [dma(., 0)-load()-wait]. d (0)-exec(0)-dma(.,1)-load()-wait e (0)-wait d (1)-exec(1)-...,], in which case, the tensor is first loaded using DMA from id= 0, and the control register is configured (load). After DMA is completed, the id= 0 operation can be executed while data for the next iteration is loaded in parallel (dma). The core processor (110) can check the status of the command queue using polling or interrupt.

[0108] The core processor (110) can continuously push commands to the command queue of the TUC (130) and ensure that the command queue is not empty, thereby enabling the TU (100) to continuously perform tensor operations. Since the command processor sequentially executes commands in the command queue, the core processor (110) is responsible for dynamic control flow.

[0109] For example, the core processor (110) transmits the next command (e.g., a command for a DMA or a new TU operation) to the command queue while the previous tensor operation is in progress in the TU (100), and the core processor (110) then continues to execute other commands. The tasks can be performed in parallel. The TU (100) receives a new command from the command queue and begins execution.

[0110] Figure 5 shows a more specific example of the PE(10) structure.

[0111] Referring to FIG. 5, a data memory (165) is allocated to each slice (160), and the operation pipeline of each slice may include at least one of a fetch unit (170), an operation unit (175), and a commit unit (180).

[0112] The patch unit (170) can continuously / sequentially read at least a portion of the partitioned input tensor from the data memory (165) and provide it to the operation unit (175) in a required form.

[0113] The operation unit (175) can stream and process the patched data. The operation unit (175) can include at least one of a contraction engine, a vector engine, and a transpose engine.

[0114] The commit unit (180) can store the work result of the operation unit (175) back into the data memory (165).

[0115] Data memory (165) can serve as a scratch pad and store tensors. Data memory (165) can be used to retrieve input tensors and commit output tensors after operations. Data memory (165) supports virtual addressing via page tables, converting virtual addresses into physical addresses. This feature allows for the dynamic allocation of large, contiguous memory spaces, facilitating tensor management.

[0116] Meanwhile, the data memory (165) has a one-dimensional address space, but the data stored in the data memory (165) may be an N-dimensional tensor. For example, assuming a three-dimensional tensor having a shape (X (height) = 3, Y (width) = 4, Z (channel) = 2) as shown in Table 1, the address in the memory of each tensor element may vary depending on the storage method in the data memory (165). For example, Table 2 shows storage according to the [X][Y][Z] = [3][4][2] (e.g., X-first, Z-last) method, and Table 3 shows storage according to the [Z][X][Y] = [2][3][4] (e.g., Z-first, Y-last) method. The conversion between Tables 2 and 3 corresponds to a transpose.

[0117] [Table 1]

[0118]

[0119] [Table 2]

[0120]

[0121] [Table 3]

[0122]

[0123] Looking at the read / write of the data memory (165), the data memory is accessed (read, written) in units of a predefined memory width, and the depth of the data memory (165) can be randomly accessed. In addition, data as much as the memory width can be read at once within the same depth (parallel access). However, only 1 depth can be read per cycle (sequential access). Considering this, when tensor data is stored in the data memory (165), data corresponding to the last axis in the storage method can be read as much as the memory width at once, and data on axes other than the last axis can be randomly accessed, but can be read N at a time (where N is an integer less than or equal to the width). For example, assuming memory width = 4 in the example of Table 3, the memory access structure can be expressed as in Table 4.

[0124] [Table 4]

[0125]

[0126] [Z][X][Y] =[2][3][4] For the Y axis, which is the last axis in the array, four values ​​[y0=a, y1=d, y2=g, y3=j] corresponding to depth 0 can be read at once, but these depth 0 values ​​are all values ​​belonging to z0. To read values ​​belonging to z1, random access is performed from depth 0 to depth 5, and four values ​​[y0=o, y1=r, y2=v, y3=y] can be read at once, but these depth 5 values ​​are all values ​​belonging to z1. In this way, in the example of Table 4, four values ​​y0 to y3 can be read at once for the last axis Y per one width read (one cycle), but for the first axis Z, only either z0 or z1 can be read.

[0127] In other words, the patch unit (170) accesses the last dimension (last_dim) of the tensor data stored in the data memory (165) in units of memory width, i.e., bank (e.g., 8B, 16B, 32B / cycles), and non-last dimensions can be accessed in any dimension order.

[0128] The commit unit (180) can also operate similarly. The commit unit (180) can also perform slicing and reshaping on any dimension other than the final dimension. The final dimension may require a separate function for slicing or transposition. For transposition, please refer to the description of the transposition engine described below.

[0129] Fig. 6 is a diagram for explaining partitioning and allocation of tensor data according to one embodiment. For convenience of explanation, it is assumed that the input tensor of TU (100) is a three-dimensional tensor (X, Y, Z), and that TU (100) includes a total of eight slices (160).

[0130] Referring to FIG. 6, tensor data can be divided into eight data partitions, and each data partition can be allocated to each slice. For example, if a three-dimensional tensor (X, Y, Z) is equally partitioned into 1 / 2 for each axis, the data memory (165) of each slice (160) can store a three-dimensional tensor partitioned into [X / 2] [Y / 2] [Z / 2].

[0131] There are many ways to partition a tensor and are not limited to the example above. For example, you can partition and assign any axis of a tensor. For example, if Z = 8, you can assign a 2D tensor (X, Y) corresponding to z0 to the first slice, a 2D tensor (X, Y) corresponding to z1 to the second slice, and a 2D tensor (X, Y) corresponding to z7 to the 8th slice.

[0132] Figures 7 to 9 illustrate various examples of partitioning a 3D tensor and assigning it to four slices.

[0133] In Figs. 7 to 9, for the convenience of explanation, the feed-forward layer is considered, and the dimension of the input tensor of the feed-forward layer is b*l*e (e.g., b = 4, l = 4, e = E), and the dimension of the weight is assumed to be e*f. b represents the batch size, l represents the sequence size, e represents the embedding size of the input, and f represents the embedding size of the feed-forward network. At this time, the contraction on the e axis is expressed as "ble,ef → blf". The notation "inter-slice partitioning shape | intra-slice shape" is used.

[0134] Referring to Figure 7, a total of 4 partitions are created as a result of dividing by 1 / 2 along the b-axis and 1 / 2 along the e-axis, and 1 partition is assigned to each slice. Specifically, since partitioning along the b-axis is a step size or stride of 2, b 2 can be expressed as [2]. Since the partitioning is performed over two slices, it can be expressed as [2]. The same applies to the e-axis. Therefore, the e-axis has a stride of E / 2 over two slices, so e E / 2 [2] can be expressed as follows. As a result, the inter-slice partitioning shape is b 2 [2]e E / 2 [2]. Each slice is assigned a shape of [2][4][E / 2], which corresponds to 1 / 4 of the tensor having the same shape of [4][4][E], and each axis has a stride of 1. Consequently, the intra-slice shape is b1 [2]l 1 [4]e 1 [E / 2]. Meanwhile, partitions without some shading (pattern) markings in slices 1 and 3 correspond to the back region hidden due to the viewpoint of the 3D tensor illustrated in Fig. 7.

[0135] In Fig. 8, the inter-slice partitioning shape is b 1 [4], and the intra-slice shape is l 1 [4] e 1 [E] is.

[0136] Referring to Fig. 9, a total of four partitions are generated as a result of dividing by 1 / 4 along the e-axis, and one partition is assigned to each slice. In Fig. 9, four partitions are assigned to slices 0 to 3 in order along the e-axis. In Fig. 9, the pattern / shading indicating the mapping between the 3D tensor and the slices is omitted. In Fig. 9, the inter-slice partitioning shape is e E / 4 [4] and the intra-slice shape is b 1 [4]l 1 [4]e 1 It is [E / 4].

[0137] As illustrated in FIGS. 7 to 9, the shape of the tensor can be lowered through partitioning of the tensor, and tensor partitions of the lowered shape can be assigned to each slice.

[0138] The above example illustrates a case where a tensor is divided into equal-sized slices and stored in the data memory of each slice by dividing some of the axes of the tensor. However, the present invention is not limited to this. Furthermore, for convenience of explanation, the above examples illustrate that the input tensor is a three-dimensional tensor, but tensors with three or more dimensions may also be input.

[0139] Meanwhile, in the reduction operation, the order of the computing axes is called tactic. For example, assume that a lowered shape like that in Fig. 9 is used for the reduction of input tensors from ble, ef to blf. The first input tensor is e E / s [s] | b 1 [B]l 1 [L]e 1 [E / s] is lowered, and the second input tensor is e E / s [s] | f 1 [F]e 1 Assuming that [E / s] is lowered, B, L, E, F represent the sizes of each dimension (b, l, e, f) of the input / weight tensors before lowering, and s represents the number of slices (e.g., s=4 in Fig. 9). In addition, the output tensor is f F / s [s] | b 1 [B]l 1 [L]f 1 Assume that it is lowered like [F / s]. For this, one possible tactic is e E / s [s] | b 1 [B]l 1 [L] f 1 [F]e 1 It is [E / s].

[0140] e in that tactic E / s [s] represents an outmost loop on the e-axis with a stride of E / s over s slices, meaning that s outmost sub-loops are executed in parallel.

[0141] b in that tactic 1 [B]l 1 [L] f 1 [F]e 1[E / s] (i.e., slice tactic) indicates how each slice processes the remaining loops. Here, there are four loops on the b, l, f, and e (innermost) axes. In slice tactic, e 1 [E / s] is the innermost loop, which represents a reduction. In the innermost loop, each slice performs a reduction. For example, each slice performs a dot product between two vectors of size E / s along the e axis. In the slice tactic, f 1 [F] is for the f-axis, meaning that the data fetched before the f-axis loop is reused F times (e.g., using the internal input buffer of the slice). The f-axis loop is repeated on the two outmost loops B*L times.

[0142] e in that tactic E / s [s] indicates that the partial dot products along the e-axis output from each of the s outmost sub-loops should be combined to produce the final result. The process of computing the final dot product from the partial dot product values ​​can be performed through a network that exists between slices, and is also distributed according to the shape of the output tensor.

[0143] For a given pair of input tensors and tensor operations, various tactics may be employed depending on factors such as the number of computational units, the lowered shapes of the input / output tensors, and the number of available slices. Once the computational units are determined, candidate lowered shapes are searched, and the one that satisfies the performance and power requirements is selected. If the lowered shapes of neighboring operations do not match each other (e.g., the lowered shape of the output of the preceding layer does not match the lowered shape of the input of the subsequent layer), an additional bridge operation may be performed to transform the lowered shapes.

[0144] The proposed TCP supports dynamic tensor shapes through control registers that describe the tensor shape, as described below. For example, in the example of Figure 9 above, the b and l axes can be determined dynamically (e.g., based on the token generation length of each request and the number of new requests, which are dynamically determined in the LLM service).

[0145] Figure 10 is a drawing for explaining an example of the overall operation of TU (100) and the configuration of the operation unit (175).

[0146] Referring to FIG. 10, the operation unit (175) may include at least one of a reduction engine (176), a vector engine (177), and a transposition engine (178).

[0147] The reduction engine (176) is for reducing (partitioned) tensor data.

[0148] In TCP(1), tensor operations are provided as primitives, and TU(100) continuously accelerates operations until the tensor operation is completed according to the length of the tensor operation based on the settings for the given tensor operation. For example, TU(100) not only processes tensors spatially in parallel but also sequentially processes them along the time axis. For this purpose, the pipeline process of Fetch-Operation-Commit is used, and the pipeline can continuously fetch a given tensor, pass it to the Operation unit, and after completing the given processing, write the resulting tensor sequentially to the data memory.

[0149] For example, as shown in FIG. 10, the input tensor is divided and stored in the data memory (165) of eight slices, and the patch unit (170) of each slice can sequentially patch data from the data memory (165) based on the tensor information (e.g., tensor shape / dimension information) obtained through the control register.

[0150] Patch units (170) can form a patch network as described below, and data patched through the patch network can be multicast to a plurality of operation units (175) (e.g., sequential multicast or at least partially simultaneous parallel multicast).

[0151] Each operation unit (175) performs tensor reduction, vector operation and / or tensor manipulation (e.g., transposition, etc.).

[0152] The commit unit (180) can sequentially commit the tensor, which is the work result of each operation unit (175), to the data memory (165) based on the order information set in the control register.

[0153] Fig. 11 illustrates a specific implementation example of the configuration of a slice (160) according to one embodiment.

[0154] Referring to FIG. 11, the patch unit (170) may include at least one of a patch sequencer and a patch process unit.

[0155] The patch sequencer can generate an address sequence for accessing data memory (165). The patch processing unit can perform data preparation such as type conversion or padding.

[0156] A patch unit (170) can configure a patch network to transmit data to operation units of other slices as well as its own slice.

[0157] The patch sequencer generates addresses based on a given tensor shape and a specified order. The patch sequencer and patch processing unit not only generate basic N-dimensional loop-style addresses, but also support indirect addresses and table lookups, providing flexibility for various types of tensor operations.

[0158] Data retrieved from the data memory (165) of the slice (160) can be sequentially transmitted to an operation unit (175) including a reduction engine (176), a vector engine (177), and / or a transposition engine (178) through a patch network.

[0159] The reduction engine (176) may include dot product engines (DPEs) (1762) that perform element-wise multiplication and generate a variety of outputs using reduction trees. The depth of each reduction tree is configurable, and the outputs of the reduction trees may be accumulated temporally via an accumulation unit (1763). Each DPE (1762) may receive two input vectors, for example, one may be received from a register file (1761) of the reduction engine (176), and the other may be a vector sequentially passed from a fetch unit (170).

[0160] Fig. 12 illustrates a detailed example of the structure of a reduction engine (176). The reduction engine (176) can perform tensor reduction through a dot product operation.

[0161] Referring to FIG. 12, the data path in the reduction engine (176) includes a feed unit, a register file, a dot product engine (DPE), and an accumulation unit, which can be controlled by a task sequencer.

[0162] For example, the reduction engine (176) may include eight DPEs that perform a dot product by spatially summing the element-wise multiplication results of two input vectors through a reduction tree. Each DPE may obtain two input vectors from a feed unit and a register file. For example, each input may hold 32 BF16 values, 64 FP8 values, 64 INT8 values, or 128 INT4 values. The number of input elements represents the maximum throughput of a given input type.

[0163] The eight DPEs can share input from the feed unit (e.g., data reuse through broadcasting), and each can receive input individually from the register file. The depth of the reduction tree can be set via a control register, which determines the number of DPE outputs. Furthermore, the DPE supports max-reduction in addition to add-reduction, enabling various types of computations, such as max-pooling.

[0164] The feed unit processes packets transmitted over the fetch network and reuses fetch data by broadcasting the data to multiple DPEs. Controlled by the task sequencer, the feed unit enables additional data reuse by providing the same data to the eight DPEs multiple times in successive cycles. The feed unit can also use shifts to reuse data in operations such as convolution and transpose tensors.

[0165] The accumulation unit receives the output of the DPE for temporal accumulation. To temporally reuse the input of the feed unit, multiple accumulators are required to maintain various partial sums. Since increasing the number of accumulators proportionally increases read / write costs, it may be desirable to allocate as few accumulators as possible per unit to maximize utilization.

[0166] For example, an accumulation unit can contain a total of 1024 accumulators, which can be indexed differently depending on the output size of the DPE. For example, if 32 BF16 inputs (from the feed unit) are used and the depth of the reduction tree is 0, each DPE produces 32 multiplication results. Therefore, the total number of outputs of 8 DPEs is 256 (=8 × 32). Given that 1024 accumulators are available, each output of a DPE can be associated with at most 4 accumulators for accumulation. However, if the depth of the reduction tree is 3, each DPE produces 4 outputs (i.e., partial sums), and 8 DPEs produce 32 outputs. In this case, each output of a DPE can be associated with 32 (=1024 / 32) accumulators for temporal accumulation.

[0167] The register file serves as storage for operands reused in MAC (multiplication and accumulation) operations. Because the register file provides data to each DPE, it may require more banks than the number of DPEs.

[0168] Figure 13 shows a detailed example of the structure of a vector engine (177).

[0169] The vector engine (177) can perform non-linear functions, element-wise operations, reductions, and / or type conversions.

[0170] The vector engine (177) may include multiple clustered functional units. For example, the functional units may be for various arithmetic and logical operations on INT32 / FP32, functions such as exp or sin, predicated operations, type conversion and / or quantization on INT4 / 8 / 16 / 32, FP8, FP16 / BF16 and FP32. In addition, the results of reduction within a slice and between slices may be routed to a designated slice or multicasted to multiple slices.

[0171] Returning to Figure 11, the transpose engine (178) can push data for a specified number of rows and pop data for another set of rows to change the last axis of the tensor to another axis within the slice. The transpose engine (178) can also use a patch sequencer and / or a commit sequencer to perform the switch between the last axis and another axis.

[0172] The transpose engine (178) and the commit unit (180), together with the vector engine, can perform tensor manipulations within multiple slices or a single slice. Tensor manipulations may include transpose, split, slice, concat, and / or reshape. Similar to the vector engine (177), such tensor manipulations may be performed after the dot product operation or in parallel with the dot product.

[0173] The commit unit (180) may include at least one of a commit sequencer and a commit processing unit. The commit sequencer may determine the storage layout of a tensor by specifying the addresses of sequentially transmitted commit data. The commit processing unit may support type conversion and remove padding from data for compaction during storage.

[0174] A plurality of patch units (170) and commit units (180) may exist within a single slice (160), and each of them may have its own data path. In this way, a plurality of independent operations may be performed simultaneously within a single slice (160) through a plurality of data paths configured within a single slice.

[0175] FIG. 20 illustrates a case in which multiple patch units and multiple commit units are configured within a slice according to one embodiment. In FIG. 20, for convenience of explanation, the number of patch units and the number of commit units per slice are each 2, but the present invention is not limited thereto, and N (N is an integer greater than or equal to 1) patch units / commit units may be configured within a slice.

[0176] When multiple patch units are set, one of the patch units can operate as a Main patch unit (1701), and the remaining patch units can operate as Sub patch units (1702).

[0177] When multiple commit units are set, one of the commit units may act as the Main commit unit (1801), and the remaining commit unit(s) may act as Sub commit units (1802).

[0178] Multiple data paths can be configured based on a combination of patch units and commit units, and multiple contexts (e.g., two in the case of Fig. 20) can be executed simultaneously in one slice based on the multiple data paths.

[0179] However, depending on the embodiment, the configuration of the data path may not involve a patch unit / commit unit, and it may also be possible to configure a data path that does not require a patch unit / commit unit, such as data movement between register files. Considering a case where the data path is configured in this way, a greater number of data paths may be configured than the number of patch units / commit units.

[0180] Referring to (a) of Fig. 20, the main context performs an operation of using the post-fetch reduction engine (176) and vector engine (177) through the Main patch unit (1701) and committing through the Main commit unit (1801). The sub context represents an operation of loading data into the post-fetch tensor register file through the Sub patch unit (1702).

[0181] Referring to (b) of Fig. 20, an operation is shown in which a tensor is reduced using a reduction engine (176) in the main context, and a transposition is performed using a transposition engine (178) in the sub context.

[0182] Figure 14 is an example for explaining a patch network according to one embodiment.

[0183] As described above, fetch units (170) of different slices (160) can form a fetch network, and data can be continuously fetched in a pipelined manner through the fetch network. The fetch network can be used to reuse fetched data.

[0184] Referring to Figure 14, fetched tensor data can be multicast to multiple slices via a fetch network. The multicast data can be processed in parallel by multiple operation units. The fetch network can be configured / reconfigured similarly to a CS (circuit switch) network. The fetch network guarantees data ordering between multiple sources and data ordering within packets from a single source, and supports multicasting. A CS-like fetch network can improve throughput and reduce hardware complexity in routing, ordering, multicasting, and flow control.

[0185] TCP(1) can also dynamically merge PEs by reconfiguring the fetch network.

[0186] FIGS. 15 to 17 are diagrams illustrating examples of processing of input tensor data according to various embodiments.

[0187] Depending on how the input tensor is allocated / layed out in the data memory (165) and how the fetch unit (170), the operation unit (175), and / or the commit unit (180) are set, the parallelism and data locality (or data reuse) used in performing the tensor operation may vary. Since the available parallelism and data locality differ depending on the shape of the tensor and the type of tensor operation required, TCP (1) can operate flexibly by taking into account the shape of the tensor and the tensor operation.

[0188] Referring to Fig. 15, a matrix multiplication is performed between two two-dimensional tensors (B01, B02). The first tensor (B01) is divided into four parts in the horizontal direction. The second tensor (B02) may be stored in the register file of the reduction engine. The third tensor (B03) is committed to the data memory (165) as a result of the tensor operation. Each slice can perform operations independently and in parallel. In the example of Fig. 15, it is assumed that no data exchange between slices is performed via the fetch network or vector engine. Each slice sequentially loads partitions of the first tensor (B01), provides them to the dot product engine of the operation unit, and commits them after each dot product operation to generate an output tensor. Regarding the ble, ef → blf reduction example, Fig. 15 has a tail axis instead of the e axis, and the DPE depth is 0. Each element of the e-axis read from the register file is broadcast, multiplied, and accumulated in the accumulation unit. The partition axis of each slice is b or l, and the parallel processing is performed, and the second operand is multicast to all slices and loaded into the register file.

[0189] Referring to Fig. 16, a matrix multiplication is performed between two two-dimensional tensors (A01, A02). The first tensor (A01) is split into two in the horizontal and vertical directions. The second tensor (A02) is stored in the register file of the reduction engine, and can be split into two along the f-axis and stored in each slice. In the fetch / transfer process of the first tensor (A01), E / 2 of Slice0 and E / 2 of Slice1 are transferred to both Slice1 and Slice0 through the fetch network, and each Slice can independently and in parallel perform operations of F / 2 along the f-axis. In other words, the fetch network can be set to multicast to Slice0 and Slice1. The Contraction Engine (CE) of Slice0 and Slice1 uses both data of Slice0 and Slice1, and performs a dot product using the data of the corresponding second tensor (A02) according to the order of the transferred data. Since the vector engine (177) does not need to reduce data from multiple slices, it performs a commit in the commit unit (180) of each slice.

[0190] Referring to FIG. 17, a matrix multiplication is performed between two two-dimensional tensors (901, 902). The first tensor (901) is split into two in the horizontal and vertical directions. The second tensor (902) may be stored in the register file of the reduction engine. For the matrix multiplication, a dot product is performed along the e-axis. In the case of splitting / storage as shown in FIG. 17, Slice0 sequentially transmits E / 2, Slice1 sequentially transmits the remaining E / 2 to perform a dot product, and (after going through the necessary processing steps) the third tensor (903) can be output. Specifically, the result output from the reduction engine (CE) is reduced (summed) across two slices, and the result can be added from multiple slices and transmitted to the target slice through the reduce & route function of the vector engine (VE). The commit unit (180) can sequentially store the output tensor in the data memory (165). In Fig. 17, each data partition of the first tensor (901) is used (reused) multiple times for the second tensor (902). The reduction engine (CE) can feed the received input tensor to the DPE multiple times according to the control register settings. Depending on the data being fed, the reduction engine can sequentially transfer the data partitions of the related second tensor (902) to the DPE to perform a dot product and store the intermediate results using multiple accumulators. Accordingly, the input reuse included in the tensor operation can be utilized. Regarding the ble,ef → blf reduction, in the example of Fig. 17, the reduction is performed across two slices e E / 2 [2] | e 1 It is performed with [E / 2], and b B / 2 [2] Parallel contraction is performed along the axis.

[0191] Figure 18 illustrates examples of data reuse and dot product in TCP(1).

[0192] As described above, in Approach I, data supplied to memory is passed downward, and the dot product is performed while passing the dot product operation result to the right.

[0193] For the proposed TCP(1), the following operations can be performed depending on the control register settings of the operation pipeline including the Fetch Network.

[0194] - Data fetched from each data memory can be multicast to all Slices. Additionally, multiple Dot Product Engines can be configured within each reduction engine to compute Dot Products.

[0195] - Data fetched from each data memory is unicast only to its own reduction engine, and the reduction engine can perform a dot product and then reduce the result.

[0196] In this way, operation units can be utilized to the maximum extent possible for various tensor shapes depending on the settings of the control register, and by continuously fetching / operating / committing data along the time axis, the pipeline can be utilized to the maximum extent possible through the data path that is configured / reconfigured based on the tensor operation.

[0197] As discussed above, the TCP proposed in this specification is a structure capable of processing tensor operations as primitives. Unlike Approach I accelerators, which typically process fixed-size unit matrix operations as commands, TCP can accelerate entire tensor operations of variable size simultaneously by specifying them as control registers.

[0198] Tensor operations are highly regularized, exhibiting a high degree of parallelism and data locality. TCP maximizes the parallelism of tensor operations by reconfiguring slices to fit the shape of the tensor being operated on, ensuring they are always processed sequentially in a software-defined order. This maximizes the use of a limited-size memory buffer, leveraging the data locality inherent in all tensor operations.

[0199] While Approach I, which processes unit matrix operations of a fixed size as commands, limits data reuse by the size of the matrix operations, TCP can improve energy efficiency by reusing more data. Furthermore, because TCP processes commands in units of entire tensor operations, it can reduce the cost of command processing and program size compared to Approach I, which processes commands for each unit matrix operation.

[0200] In this way, TCP can fully leverage parallelism and data locality in tensor reduction, thereby improving the efficiency and performance of tensor data processing used in AI and other fields. TCP can be configured with large PEs, which has the advantage of simplifying related software development. Furthermore, by performing the entire tensor reduction through a single control register configuration, control overhead can be minimized.

[0201] To efficiently process various tensor-shaped operations, multiple PEs can be configured as a single computational unit or as a collection of smaller, independently executable computational units. Furthermore, the fetch network can be configured / reconfigured, similar to a circuit switch, providing a structure that enables data reuse across and within computational units.

[0202] Supporting dynamic shape and control flow is important for optimization of AI accelerators, such as LLM inference. The proposed TCP(1) supports dynamic shape and control flow by setting / resetting control registers, enabling more efficient program generation according to dynamically changing computation, memory, and power consumption requirements.

[0203] As described above, a device (e.g., TCP) for performing data processing may include a plurality of PEs (10) configured to perform tensor-based processing; and a semiconductor substrate (40) on which the plurality of PEs (10) are mounted.

[0204] Each PE (10) may include a plurality of slice circuits (160), each having a compute pipeline; and a control circuit (130) that controls the plurality of slice circuits (160).

[0205] Each operation pipeline on each slice circuit (160) may include i) a fetch unit (170) that fetches at least a portion of tensor data, ii) an operation unit (175) that performs an operation on at least a portion of the fetched tensor data, and iii) a commit unit (180) that transmits the operation result output from the operation unit to a destination.

[0206] The above control circuit (130) can control the mapping between partitions of the tensor data and a plurality of operation pipelines on the plurality of slice circuits (160).

[0207] For example, the plurality of slice circuits (160) can be configured / reconfigured based on the shape of the tensor data and a command given to the tensor data.

[0208] For example, each of the plurality of slice circuits (160) may be allocated a control register that stores configuration / reconfiguration information of each slice circuit.

[0209] For example, the control circuit (130) can perform configuration / reconfiguration of each slice circuit (160) and reading / writing of the associated control register.

[0210] For example, each PE (10) may further include a scratch pad memory (SPM) (140). The set data of the plurality of control registers allocated to the plurality of slice circuits (160) may be configured by loading values ​​stored on the scratch pad memory (140).

[0211] For example, partitioning of the tensor data may be performed based on at least one of a shape of the tensor data and a command given to the tensor data.

[0212] For example, a data memory (165) having a bandwidth related to the operation pipeline may be allocated to each slice circuit (160).

[0213] For example, the operation unit (175) may include at least one of a reduction engine (176) that performs tensor contraction on at least a portion of the patched tensor data, a vector engine (177) that performs operations on vectors that are one-dimensional tensors, and a transposition engine (178) that performs data transposition.

[0214] For example, the reduction engine (176) can perform the tensor reduction by reducing the rank.

[0215] For example, the control circuit (130) can establish a configurable patch network through connections between different patch units (170) on different slice circuits (160).

[0216] For example, data patched by the patch network can be sequentially multicast to different operation units (175) of the different slice circuits (160).

[0217] For example, the patch network can perform the multicast by reusing the patched data.

[0218] For example, the patch unit may include a main patch unit and a sub patch unit, and the commit unit may include a main commit unit and a sub commit unit.

[0219] For example, a main data path is configured based on the main patch unit and the main commit unit, a sub data path is configured based on the sub patch unit and the sub commit unit, and operations on the main data path and operations on the sub data path can be performed simultaneously within one slice circuit.

[0220] For example, the control circuit (130) can configure / reconfigure the first slice circuit and the second slice circuit so that the first operation result of the first operation unit of the first slice circuit is transmitted to the second slice circuit.

[0221] For example, each PE (10) may further include a core processor (110). The control circuit (130) may operate as a co-processor for the core processor (110).

[0222] For example, the control circuit (130) may include a command processor that executes instructions obtained from the core processor (110). The instructions may include at least one of an instruction for loading a control register address assigned to each slice circuit (160) and an instruction for enabling each slice circuit (160).

[0223] For example, the device may further include a memory controller (30) that controls data flow between the plurality of PEs (10) and the memory device; and an on-chip network (20) formed on the semiconductor substrate (40) for the plurality of PEs (10) and the memory controller (30).

[0224] For example, the tensor data may be related to neural network processing for artificial intelligence.

[0225] For example, the device may be a tensor reduction processor (TCP, 1) for accelerating neural network processing for the artificial intelligence.

[0226] FIG. 19 illustrates a flowchart of a method performed in a device for data processing according to one embodiment.

[0227] Referring to FIG. 19, a device for data processing can obtain a command for tensor data (1905).

[0228] A device for data processing can perform partitioning on the tensor data based on a shape of the tensor data and at least one of the commands (1910).

[0229] A device for data processing can assign partitions of the tensor data to multiple compute pipelines (1915).

[0230] The device for data processing can process each partition of the tensor data through each of the plurality of operation pipelines (1920). Each operation pipeline can include: i) a process of patching each partition of the tensor data, ii) a process of performing an operation on the patched partition, and iii) a process of committing the result of performing the operation to a destination.

[0231] The embodiments of the present invention described above may be implemented through various means. For example, the embodiments of the present invention may be implemented through hardware, firmware, software, or a combination thereof.

[0232] In the case of hardware implementation, the method according to embodiments of the present invention may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

[0233] When implemented using firmware or software, the methods according to embodiments of the present invention may be implemented in the form of modules, procedures, or functions that perform the functions or operations described above. The software code may be stored in a memory unit and executed by a processor. The memory unit may be located within or outside the processor and may exchange data with the processor using various known means.

[0234] The detailed description of the preferred embodiments of the present invention disclosed above has been provided to enable those skilled in the art to implement and practice the present invention. While the above description has been made with reference to preferred embodiments of the present invention, those skilled in the art will appreciate that various modifications and variations can be made to the present invention without departing from the scope of the present invention. For example, those skilled in the art can utilize the individual components described in the above-described embodiments in combination with each other. Accordingly, the present invention is not intended to be limited to the embodiments described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0235] The present invention may be embodied in other specific forms without departing from the spirit and essential characteristics thereof. Therefore, the above detailed description should not be construed as limiting in all respects but rather as illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are intended to be included therein. Furthermore, claims that are not explicitly cited in the claims may be combined to form an embodiment or incorporated into a new claim through a post-application amendment.

Claims

1. In a device that performs data processing, A plurality of processing elements (PEs) configured to perform tensor-based processing; and It includes a semiconductor substrate on which the above multiple PEs are mounted, Each PE is, a plurality of slice circuits, each having a compute pipeline; and It includes a control circuit that controls the plurality of slice circuits, Each operation pipeline on each slice circuit includes i) a fetch unit that fetches at least a portion of tensor data, ii) an operation unit that performs an operation on at least a portion of the fetched tensor data, and iii) a commit unit that transmits the operation result output from the operation unit to the destination. A device wherein the control circuit controls mapping between partitions of the tensor data and a plurality of operation pipelines on the plurality of slice circuits.

2. In paragraph 1, A device in which the plurality of slice circuits are configured / reconfigured based on the shape of the tensor data and a command given to the tensor data.

3. In paragraph 2, A device in which each of the plurality of slice circuits is allocated a control register that stores configuration / reconfiguration information of each slice circuit.

4. In paragraph 3, The above control circuit is a device that performs configuration / reconfiguration of each slice circuit and reading / writing of associated control registers.

5. In paragraph 3, Each PE further contains a Scratch Pad Memory (SPM), A device in which data of a plurality of control registers allocated to the plurality of slice circuits is configured by loading values ​​stored in the scratch pad memory.

6. In paragraph 1, A device in which partitioning of the tensor data is performed based on at least one of a shape of the tensor data and a command given to the tensor data.

7. In paragraph 1, A device in which each slice circuit is allocated data memory having a bandwidth related to the computation pipeline.

8. In paragraph 1, A device wherein the operation unit comprises at least one of a contraction engine that performs tensor contraction on at least a portion of the patched tensor data, a vector engine that performs operations on vectors that are one-dimensional tensors, and a transposition engine that performs data transposition.

9. In paragraph 8, A device wherein the above reduction engine performs the tensor reduction by reducing the rank.

10. In paragraph 1, A device wherein the above control circuit establishes a configurable patch network through connections between different patch units on different slice circuits.

11. In paragraph 10, A device in which data patched by the above patch network is multicast to different operation units of the different slice circuits.

12. In paragraph 11, The above patch network is a device that performs the multicast by reusing the patched data.

13. In paragraph 1, The above patch unit includes a main patch unit and a sub patch unit, A device wherein the above commit unit includes a main commit unit and a sub commit unit.

14. In paragraph 13, A main data path is configured based on the main patch unit and the main commit unit, A sub data path is configured based on the above sub patch unit and the above sub commit unit, A device in which operations on the main data path and operations on the sub data path are performed simultaneously within one slice circuit.

15. In paragraph 1, A device wherein the control circuit configures / reconfigures the first slice circuit and the second slice circuit so that the first operation result of the first operation unit of the first slice circuit is transmitted to the second slice circuit.

16. In paragraph 1, Each PE contains more core processors, A device wherein the above control circuit operates as a co-processor for the above core processor.

17. In paragraph 16, The above control circuit includes a command processor that executes instructions obtained from the core processor, A device wherein the above commands include at least one of a command for loading a control register address assigned to each slice circuit and a command for enabling each slice circuit.

18. In paragraph 1, a memory controller that controls data flow between the plurality of PEs and the memory device; and A device further comprising an on-chip network formed on the semiconductor substrate for the plurality of PEs and the memory controller.

19. In paragraph 1, The above tensor data is related to neural network processing for artificial intelligence, device.

20. In paragraph 19, The device is a tensor reduction processor for accelerating neural network processing for the artificial intelligence.

21. In a method of performing data processing, Obtain commands for tensor data; Perform partitioning on the tensor data based on the shape of the tensor data and at least one of the commands; Assigning partitions of the above tensor data to multiple compute pipelines; and Processing each partition of the tensor data through each of the plurality of operation pipelines, A method in which each operation pipeline includes: i) a process of patching each partition of the tensor data, ii) a process of performing an operation on the patched partition, and iii) a process of committing the result of performing the operation to a destination.

22. A storage medium readable by at least one processor, in which program codes are stored, wherein operations performed by the at least one processor as the program codes are executed by the processor are: Obtain commands for tensor data; Perform partitioning on the tensor data based on the shape of the tensor data and at least one of the commands; Assigning partitions of the above tensor data to multiple compute pipelines; and Processing each partition of the tensor data through each of the plurality of operation pipelines, A storage medium, wherein each operation pipeline includes: i) a process of patching each partition of the tensor data, ii) a process of performing an operation on the patched partition, and iii) a process of committing the result of performing the operation to a destination.

23. In paragraph 22, The above multiple operation pipelines are formed on multiple slice circuits, A storage medium in which the plurality of slice circuits are configured / reconfigured based on the shape of the tensor data and a command given to the tensor data.

24. In paragraph 23, A storage medium in which each of the plurality of slice circuits is allocated a control register that stores configuration / reconfiguration information of each slice circuit.

25. In paragraph 24, the operations performed by the at least one processor are: A storage medium further comprising: performing configuration / reconfiguration of each slice circuit and reading / writing of associated control registers.

26. In paragraph 22, the operations performed by the at least one processor are: A storage medium further comprising determining partitioning of the tensor data based on at least one of a shape of the tensor data and a command given to the tensor data.

27. In paragraph 22, the operations performed by the at least one processor are: A storage medium further comprising allocating data memory having an associated bandwidth for each operation pipeline.

28. In paragraph 22, A storage medium, wherein the above operation process includes at least one of a process of performing tensor contraction on at least a portion of the patched tensor data, a process of performing an operation on a vector which is a one-dimensional tensor, and a process of performing data transpose.

29. In paragraph 28, A storage medium wherein said at least one processor performs said tensor reduction by reducing the rank.

30. In paragraph 22, The above tensor data is a storage medium related to neural network processing for artificial intelligence.

31. In paragraph 22, A storage medium, wherein at least one processor is a tensor reduction processor for accelerating neural network processing for the artificial intelligence.

Images