Tensor processing method and apparatus therefor

Abstract

An apparatus for performing data processing according to one embodiment of the present specification includes: a plurality of processing elements (PEs) configured to perform tensor-based processing; and a semiconductor substrate on which the plurality of PEs are mounted; wherein each PE includes: a plurality of slice circuits each having a compute pipeline; and a control circuit for controlling the plurality of slice circuits. Each compute pipeline on each slice circuit includes: i) an acquisition unit for acquiring at least a portion of tensor data, ii) an operation unit for performing an operation on at least a portion of the acquired tensor data, and iii) a submission unit for transferring an operation result output from the operation unit to a destination, wherein the control circuit is capable of controlling a mapping between partitions of the tensor data and a plurality of compute pipelines on the plurality of slice circuits.

Description

Technical Field

[0001] This specification relates to tensor processing, and more specifically, to a method for performing operations on N-dimensional (N≥0) tensors and an apparatus for performing the method. Background Technology

[0002] Tensors are multidimensional arrays that generalize vectors and matrices. In addition to scalars (N=0-dimensional tensors), vectors (N=1-dimensional tensors), and matrices (N=2-dimensional tensors), tensors can represent higher-dimensional arrays (N>3-dimensional tensors) in more generalized forms.

[0003] The rank of a tensor represents the tensor's dimension / number of axes / channels, N. For example, a scalar can be represented by rank 0, a vector by rank 1, and a matrix by rank 2.

[0004] The shape of a tensor is a tuple structure representing the size of each dimension. For example, the shape of an X×Y matrix can be simply represented as (X, Y). Furthermore, an X×Y matrix with three channels RGB (red, green, blue)... An image of Y pixels can be represented as a 3D tensor (X, Y, 3).

[0005] Tensors have been frequently used in artificial intelligence (AI) frameworks recently, and these AI frameworks provide various functions and operations for generating / processing tensors. Summary of the Invention

[0006] Technical issues

[0007] The technical problem to be solved by the present invention is to provide a method and apparatus for performing tensor processing more effectively.

[0008] Other technical issues can be derived from the implementation of the specific embodiments.

[0009] Technical solution

[0010] In one aspect of the present invention for solving the above-mentioned technical problems, an apparatus for performing data processing may include: a plurality of processing elements (PEs) configured to perform tensor-based processing; and a semiconductor substrate on which the plurality of PEs are mounted.

[0011] Each PE may include: a plurality of slice circuits, each slice circuit having a computation pipeline; and control circuitry configured to control the plurality of slice circuits.

[0012] Each computation pipeline on each slice circuit may include: i) an acquisition unit configured to acquire at least a portion of tensor data, ii) a computation unit configured to perform operations on at least a portion of the acquired tensor data, and iii) a submission unit configured to deliver the computation results output from the computation unit to a destination.

[0013] The control circuit can control the mapping between the partitions of the tensor data and the multiple computation pipelines on the multiple slice circuits.

[0014] Preferably, the plurality of slice circuits can be configured / reconfigured based on the shape of the tensor data and the instructions given for the tensor data.

[0015] Preferably, a control register can be assigned to each of the plurality of slice circuits, and the control register stores the configuration / reconfiguration information of each slice circuit.

[0016] Preferably, the control circuit can perform configuration / reconfiguration of each slice circuit and read / write of the associated control registers.

[0017] Preferably, each PE may further include an on-chip sticky note memory (SPM). Data assigned to multiple control registers of the plurality of slice circuits can be configured by loading values ​​stored in the on-chip sticky note memory.

[0018] Preferably, the partitioning of the tensor data can be performed based on at least one of the shape of the tensor data and the instructions given for the tensor data.

[0019] Preferably, a data memory with bandwidth associated with the computational pipeline can be assigned to each slice circuit.

[0020] Preferably, the computation unit may include at least one of the following: a shrinking engine configured to perform tensor shrinking on at least a portion of the acquired tensor data; a vector engine configured to perform operations on vectors that are one-dimensional tensors; and a transpose engine configured to perform data transpose.

[0021] Preferably, the shrinkage engine can perform the tensor shrinkage by reducing the rank.

[0022] Preferably, the control circuit can configure a configurable acquisition network by connecting different acquisition units on different slice circuits.

[0023] Preferably, the data acquired by the acquisition network can be multicast to different processing units of the different slice circuits.

[0024] Preferably, the acquisition network can perform the multicast by reusing the acquired data.

[0025] Preferably, the acquisition unit may include a main acquisition unit and a sub-acquisition unit, and the submission unit may include a main submission unit and a sub-submission unit.

[0026] Preferably, the main data path can be configured based on the main acquisition unit and the main submission unit, and the sub-data path can be configured based on the sub-acquisition unit and the sub-submission unit, and the operations on the main data path and the operations on the sub-data path can be executed simultaneously within a slice circuit.

[0027] Preferably, the control circuit can be configured / reconfigured to include a first slice circuit and a second slice circuit for delivering a first operation result from a first arithmetic unit of the first slice circuit to the second slice circuit.

[0028] Preferably, each PE may further include a core processor, and the control circuitry may operate as a coprocessor for the core processor.

[0029] Preferably, the control circuitry may include a command processor configured to execute instructions obtained from the core processor. The instructions may include at least one of the following: instructions for loading the control register address assigned to each slice circuit; and instructions for enabling each slice circuit.

[0030] Preferably, the device may further include: a memory controller configured to control data flow between the plurality of PEs and a memory device; and an on-chip network formed on the semiconductor substrate for use with the plurality of PEs and the memory controller.

[0031] Preferably, the tensor data may be related to neural network processing used for artificial intelligence.

[0032] Preferably, the device may be a tensor shrinking processor for accelerating neural network processing for artificial intelligence.

[0033] In another aspect of the invention for solving the aforementioned technical problems, a method for performing data processing may include the following steps: obtaining instructions for tensor data; partitioning the tensor data based on at least one of the shape of the tensor data and the instructions; assigning the partitions of the tensor data to a plurality of computational pipelines; and processing each of the partitions of the tensor data through each of the plurality of computational pipelines. Each computational pipeline may include: i) a process of acquiring each partition of the tensor data, ii) a process of performing operations on the acquired partitions, and iii) a submission process of delivering the results of the operations to a destination.

[0034] In another aspect of the invention for solving the above-mentioned technical problems, a storage medium is provided, which can be read by at least one processor and store program code, which, when executed by the at least one processor, causes the at least one processor to perform an operation, the operation including: obtaining instructions for tensor data; partitioning the tensor data based on at least one of the shape of the tensor data and the instructions; assigning the partitions of the tensor data to a plurality of computational pipelines; and processing each of the partitions of the tensor data through each of the plurality of computational pipelines.

[0035] Each computation pipeline may include: i) a process of acquiring each partition of the tensor data, ii) a process of performing operations on the acquired partitions, and iii) a submission process of delivering the results of the operations to a destination.

[0036] Preferably, the plurality of computational pipelines can be formed on a plurality of slice circuits, and the plurality of slice circuits can be configured / reconfigured based on the shape of the tensor data and the instructions given for the tensor data.

[0037] Preferably, each of the plurality of slice circuits may be assigned a control register that stores the configuration / reconfiguration information of each slice circuit.

[0038] Preferably, the operations performed by the at least one processor may further include: performing configuration / reconfiguration of each slice circuit and reading / writing of the associated control registers.

[0039] Preferably, the operation performed by the at least one processor may further include: determining the partitioning of the tensor data based on at least one of the shape of the tensor data and instructions given for the tensor data.

[0040] Preferably, the operation performed by the at least one processor may further include: assigning a data memory with relevant bandwidth to each computing pipeline.

[0041] Preferably, the process of performing the operation may include at least one of the following: performing tensor shrinkage on at least a portion of the acquired tensor data; performing operations on a vector that is a one-dimensional tensor; and performing data transpose.

[0042] Preferably, the at least one processor can perform the tensor shrinkage by reducing the rank.

[0043] Preferably, the tensor data may be related to neural network processing used for artificial intelligence.

[0044] Preferably, the at least one processor may be a tensor shrinking processor for accelerating the neural network processing for artificial intelligence.

[0045] Beneficial effects

[0046] According to embodiments of the present invention, tensor processing can be performed more efficiently.

[0047] Various other technical effects can be derived from the detailed implementation methods. Attached Figure Description

[0048] Figure 1 Typical array processors and vector processors are shown.

[0049] Figure 2 This is a diagram used to explain the tensor shrinking method for performing matrix multiplication operations based on a fixed size.

[0050] Figure 3 An apparatus (1) for data processing according to one embodiment is shown.

[0051] Figure 4 The PE (10) according to the embodiment is shown.

[0052] Figure 5 A more specific example of the structure of PE(10) is shown.

[0053] Figure 6 It is a diagram used to explain the partitioning and allocation of tensor data according to the implementation method.

[0054] Figures 7 to 9 Various examples are shown in which a three-dimensional tensor is partitioned and assigned to four slices.

[0055] Figure 10 This is a diagram illustrating an example of the configuration of the overall operation and arithmetic unit (175) of the TU (100).

[0056] Figure 11 A specific implementation example of the configuration of the slice (160) according to the implementation method is shown.

[0057] Figure 12 A detailed example of the structure of the contraction engine (176) is shown.

[0058] Figure 13 A detailed example of the structure of the vector engine (177) is shown.

[0059] Figure 14 This is an example used to explain the fetch network according to the implementation method.

[0060] Figures 15 to 17 This is a diagram used to explain examples of processing input tensor data according to various implementation methods.

[0061] Figure 18 An example of data reuse and dot product in TCP(1) is shown.

[0062] Figure 19 The flow of a method performed in a data processing apparatus according to an embodiment is shown.

[0063] Figure 20 This illustrates a scenario where multiple acquisition units and multiple submission units are configured in a slice according to an embodiment. Detailed Implementation

[0064] The examples described below are non-limiting embodiments provided to facilitate understanding of the present invention described above, and those skilled in the art will understand that some embodiments can be combined, omitted, and modified.

[0065] First, as an example of tensor processing that can be applied to the present specification, artificial neural networks and their processing will be briefly described.

[0066] The neurons that make up the human brain form a signal circuit. Data processing structures and methods that mimic the signal circuits of neurons are called Artificial Neural Networks (ANNs). In an ANN, multiple interconnected neurons form a network, and the input / output process of a single neuron can be mathematically represented as [output = f(W1 × input 1 + W2 × input 2 + ... + WN × input N)]. Here, Wi represents the weight, and the weight can have various values ​​depending on the type / model of the ANN, the layers, each neuron, the learning outcome, etc. With the recent advancements in computing technology enabling the processing of deep neural networks (DNNs) with multiple hidden layers, the training / learning and inference of AI have become more complex and sophisticated. The training process in DNNs is called deep learning and can include, for example, weight adjustment. Inference refers to the process of obtaining an output by inputting new data into a trained model. Convolutional Neural Networks (CNNs) are a representative type of DNN and can be configured based on convolutional layers, pooling layers, fully connected layers, and / or combinations thereof. CNNs have structures that are advantageous for learning two-dimensional data and are known to exhibit excellent performance, for example, in image classification and detection. Besides CNNs, various DNNs (e.g., Recurrent Neural Networks (RNNs) that excel in processing data with time-series characteristics such as language, Long Short-Term Memory (LSTM) that mimics forgetting old data, Generative Adversarial Networks (GANs) that generate fake data by competing with learning data, and Large Language Models (LLMs) that can infer from user input to generate various words) have been actively researched and applied in various fields. In the following text, the various types of AI-related neural networks, including the aforementioned DNNs, are collectively referred to and simply as neural networks (NNs).

[0067] Neural network processing can be performed by a variety of devices. While it's not impossible to use a general-purpose CPU for neural network processing, general-purpose CPUs have limitations in efficiently handling the large amounts of parallel processing involved in neural network processing. Therefore, processors with architectures more suited to neural network processing are used. AI accelerators generally refer to processors designed to accelerate AI operations. For example, although graphics processing units (GPUs) were initially developed for graphics processing, they have recently been widely used as AI accelerators for large-scale data processing and for training and inference of complex models due to their excellent parallel processing capabilities. Neural processing units (NPUs) are processors developed to perform AI operations more quickly and are dedicated to NN / AI operations. Compared to GPUs, NPUs offer higher efficiency for AI operations by implementing the layers of an AI model as a physical circuit configuration on a semiconductor chip.

[0068] The term "processor" as used in this specification can refer to the concept of a standalone (independent) or auxiliary (coprocessor-based) AI accelerator (such as a GPU and / or NPU) configured to perform NN / AI operations more quickly. Depending on the implementation, an AI accelerator can be configured to be optimized for training or inference, or it can be configured to be used in both training and inference. While AI accelerators come in various forms, the main performance requirements are as follows: (i) Parallel processing: Parallel processing must be supported for this purpose, as AI operations require large-scale data processing. That is, tensor operations such as matrix and vector operations must be performed efficiently. (ii) Power efficiency: Power efficiency is important in environments with limited power (such as edge users / terminals) or in environments such as data centers. (iii) Low latency: Low latency is required in areas where real-time performance is important, such as AI-based detection / autonomous driving that requires instant response. (iv) Memory performance: Sufficient memory capacity and high memory bandwidth are required to handle large-scale models and tensors. (v) In addition, high levels of accuracy and precise floating-point operations, as well as compatibility with various frameworks and libraries, may be required.

[0069] Figure 1 This is a diagram used to compare and describe general-purpose array processors and vector processors.

[0070] Array processors and vector processors share the common feature of effectively supporting SIMD (Single Instruction, Multiple Data) parallel processing; however, they differ in the detailed mechanisms of parallel processing as follows.

[0071] (a) The array processor includes multiple processing elements (PEs) to perform the same operation in parallel on each of the multiple elements included in the array simultaneously.

[0072] (b) The vector processor is optimized for processing operations based on vectors as one-dimensional arrays and executes vector instructions in pipelined parallelism rather than using multiple PEs. The pipeline is partitioned depending on the instructions; for example, addition (ADD) and multiplication (MUL) are performed in different pipelines.

[0073] Vector processors include vector registers for loading (LD) and storing (ST) vectors, thereby enabling data to be loaded and stored in vector units. To handle vectors of varying lengths, a vector length register (VLEN) is required. Since the elements constituting a vector can be stored separately in memory, a vector stride register (VSTR) is needed, with the stride value corresponding to the distance between vector elements.

[0074] Vector processors primarily support mathematical operations, especially vector and matrix operations, and can efficiently perform linear algebra, matrix multiplication, element-wise operations, inner products, and more. Such vector processors are also used for large-scale tensor operations that frequently occur in AI applications, and recent GPUs can incorporate vector processors.

[0075] Tensor contraction method I

[0076] Tensor contraction as used in this specification may mean reducing the dimension of a tensor, and the term “contraction” may be replaced by or used interchangeably with “reduction” or other similar expressions.

[0077] One of the main operations in machine learning models is tensor shrinking, and tensor shrinking can be associated with summing elements along certain axes of the tensor. Matrix multiplication (e.g., dot product) is an example of tensor shrinking. For example, multiplication between an m×k matrix and a k×n matrix can be understood as multiplying a matrix of size (m×k) by a matrix of size k×n. k, k The tensor of n contracts to a size of m. A tensor of n. Such shrinkage can be performed based on the common k-axis between matrices.

[0078] Furthermore, as a structure for tensor shrinkage, a processing technique that decomposes / reconstructs tensor operations into multiple matrix multiplications (hereinafter referred to as "Method I") can be considered, and for this purpose, the processor may include a matrix multiplication unit that performs matrix multiplication operations of a predetermined / fixed size.

[0079] Figure 2 This is an example used to explain the matrix multiplication unit according to Method I.

[0080] First refer to Figure 2 In (a), assume the matrix multiplication unit is configured as a systolic array with a fixed size of 128 × 128 processing elements (PEs). In this case, since the matrix multiplication unit can perform up to 128 dot product operations simultaneously, the SRAM should also support 128 dot product operations; that is, the number of data elements transferred from the SRAM at one time should be 128 to ensure maximum utilization (maximum efficiency). In other words, if the SRAM bandwidth does not support 128 PEs, maximum utilization (maximum efficiency) cannot be achieved. Although the maximum data reuse rate in the matrix multiplication unit is configured to a relatively high 128x, in operations that do not require up to 128x data reuse, unused PEs may be left behind, resulting in wasted processing resources. Figure 2 In case (a), the rich parallelism and data locality in tensor operations may not be fully utilized, thus limiting processing performance.

[0081] If the size of the processor's matrix multiplication unit is fixed to be small, such as Figure 2 (b) and Figure 2 As shown in (c), the scope of data reuse is limited, thus reducing efficiency. Furthermore, if a large number of small PEs are connected via an on-chip network (NoC), high complexity is involved in parallel execution of tensor operations. Additionally, programs for a large number of low-performance cores are more difficult to generate than programs for a relatively small number of high-performance cores.

[0082] In tensor data processing, such as inference, various tensor shapes emerge, making it important to maximize parallelism and data reuse not only based on tensor shape but also on batch size. However, in Method I-based chips, it may be difficult to fully utilize the massive matrix multiplication units for tensor operations of various shapes and types.

[0083] TCP-Novel Tensor Shrinkage Method

[0084] The following describes a processor employing a novel tensor shrinkage scheme that overcomes the shortcomings of Method I, which is based on fixed-size matrix multiplication. The processor described below can be an AI accelerator, but its application is not limited to AI / neural networks; it can also be applied to more general-purpose processors (e.g., CPUs) or other specialized processors (e.g., communication modems, signal processors, etc.).

[0085] The processors disclosed in this specification may include a tensor shrinking engine and are referred to as a tensor shrinking processor or simply TCP. Unless otherwise stated, the processor mentioned in the following description may refer to TCP.

[0086] In the implementation described below, tensor shrinkage can be used as the basic element instead of matrix multiplication. Based on this novel method presented in this specification, not only can massively parallel tasks be implemented, but pipelined operations can also be performed on the time axis, similar to those described above. Figure 1 (b) describes the characteristics of the vector processor.

[0087] Figure 3 A device (1) for data processing according to an embodiment is shown. Figure 3 In this context, the device (1) used for data processing can be TCP.

[0088] The device (1) for data processing may include multiple PEs (10), a memory controller (30), an on-chip network (NoC) (20), and a silicon substrate (40).

[0089] The memory controller (30) can control the connection between the high bandwidth memory (HBMx, not shown) and multiple PEs (10).

[0090] NoC (20) can provide a network for data exchange between various components (10, 30, etc.) disposed on the silicon substrate (40). NoC (20) can support data transfer between PEs, and each PE can transfer data to a high-bandwidth memory or another PE. As an example of data exchange between PEs, large language models (LLMs) exceeding the memory size of device (1) may require distributed processing, thus necessitating data exchange (inter-chip communication) between PEs.

[0091] Each of the multiple PEs (10) can be configured to perform tensor-related data processing independently / separately. For example, the first PE (11) can perform tensor shrinking independently of the second PE (12). For example, each PE can operate as a host-independent device. For example, each PE can be driven by the host as an independent device.

[0092] Each PE (10) has an independent address space and operates independently. Access to unauthorized addresses is restricted by an address translation unit (not shown). Address translation can dynamically allocate memory space and PEs use as their address space at runtime. This forms the basis for PE-to-PE communication and, combined with PCIe peering, enables PEs on multiple chips to exchange data using the same abstraction. This allows for scalable compilation and dynamic resource reservation.

[0093] Figure 4 The PE (10) according to the embodiment is shown.

[0094] Reference Figure 4 PE (10) may include a core processor (110), a tensor unit (TU) (100), a memory (140), and a NoC (120).

[0095] The core processor (110) can control the TU (100) as a coprocessor. Considering the latency when accessing the TU (100) from the core processor (110), the TU (100) can operate asynchronously.

[0096] The memory (140) may include SPM (scratch pad memory).

[0097] PE (10) may include a Tensor Direct Memory Access (DMA) engine (150). The Tensor DMA engine (150) can optimize memory access by recognizing the shape of tensors during operation. The Tensor DMA engine (150) can index and transfer tensors in any dimensional order.

[0098] The core processor (110) can be booted using software stored in memory (140). In response to a request from the host, the core processor (110) can load a program including tensor operations into memory (140) and execute the program. While executing the program, the core processor (110) can control the TU (100) by reading / writing the registers of the Tensor Unit Controller (TUC) (130) of the TU (100).

[0099] TU (100) comprises multiple slices (160). By partitioning / dividing tensor data into smaller units in this way, more flexible configurations for various tensor shapes are possible. Depending on the configuration, the entire set of slices can operate as a large PE, or each slice can operate as a small, independent parallel computation unit. That is, unlike Method I, which combines a large memory and a large-scale matrix multiplication unit, in the TCP architecture proposed here, the computational pipeline, including each data memory (e.g., SRAM) and operation unit, is divided into multiple slices (160).

[0100] Data storage associated with the computation pipeline can be allocated to each slice (160). Each computation pipeline may include an acquire-compute-submit process.

[0101] TU (100) may include a large set of control registers describing tensor shape and tactics. For example, a control register may be assigned to each slice (160) / each computation pipeline, and thus the control register may be controlled individually; however, sets of control registers may also be configured and controlled in the same manner. For example, multiple sets of control registers may be configured at once via broadcast / multicast.

[0102] Depending on the configuration of the control registers, one or more consecutive tensor operations can be performed. Each slice (160) configured via the control registers can independently process tensor data (e.g., tensor partitions). The core processor (110) can read / write the control registers via the tensor unit controller (TUC) (130), which is linked to the computation pipeline of each slice (160). The control registers can be configured with the index structure for the get-operate-submit of N-dimensional tensors, the index of internal buffers, the type and order of operations, the topology of the slices (160), etc. A set of control registers configured by software defines the spatially / temporally parallelized tensor operations. When the enable registers in the control registers are set, tensor / vector operations can be performed based on the configured set of control registers. Each slice (160) configured via the control registers can independently process tensor data (e.g., tensor partitions).

[0103] The TUC (130) may include a status register that aggregates the status of all slices (160). The core processor (110) can check the status of each slice (160) through the status register.

[0104] The core processor (110) can execute a program consisting of tensor operations by repeating a series of processes of setting control registers, enabling control registers, and checking status. Meanwhile, according to one implementation, instead of the core processor (110) directly writing to the control registers, a set of control registers associated with tensor operations can be stored in memory (140), and the TUC (130) can load the entire set of control registers. That is, the core processor (110) can effectively control the TU (100) through the TUC (130) instead of directly reading / writing the control registers of the slice (160) in the TU (100).

[0105] The TUC (130) may include a command processor (not shown) configured to execute commands sequentially from the core processor (110). The command processor may include a command queue. The core processor (110) may push (write) commands into the command queue of the TUC (130). For example, the command may include at least one of the following, but is not limited thereto.

[0106] -'dma(addr, id)': Command for transferring tensors between memory (memory connected to memory controller (30)) and data memory (165), where source / destination tensor information (e.g., shape) can be specified using id-based DMA.

[0107] -'load(addr, csraddr, size)': A command to load a control register of a given size from memory (140) to the control register address of slice (160).

[0108] -'exec(id)': The execution command used for tensor shrinking operations.

[0109] -'wait d / e (id)': Commands used to wait for DMA or execute commands.

[0110] Most commands, including "dma" and "exec", can be executed asynchronously, and data transfer and computation can overlap. When synchronization is required, "wait" can be invoked. An example of a typical command sequence is [dma(., 0) - load() - wait]. d (0)-exec(0)-dma(.,1)-load()-wait e (0)-wait d(1)-exec(1)-...,], where the tensor is first loaded using DMA at id=0, and the control register is configured (loaded). After DMA is complete, the id=0 operation can be executed, while data (dma) for the next iteration is loaded in parallel. The core processor (110) can use polling or interrupts to check the status of the command queue.

[0111] The core processor (110) can continuously push commands to the command queue of the TUC (130) and ensure that the TU (100) continuously performs tensor operations by keeping the command queue non-empty. Since the command processor executes the commands in the command queue sequentially, the core processor (110) is responsible for dynamic control flow.

[0112] For example, while a previous tensor operation is performed in TU (100), the core processor (110) sends the next command (e.g., DMA or a command for a new TU operation) to the command queue, after which the core processor (110) continues to execute other commands. Tasks can be executed in parallel. TU (100) receives a new command from the command queue and begins execution.

[0113] Figure 5 A more specific example of the structure of PE(10) is shown.

[0114] Reference Figure 5 Each slice (160) is allocated a data storage (165), and the computation pipeline for each slice may include at least one of an acquisition unit (170), an operation unit (175), and a submission unit (180).

[0115] The acquisition unit (170) can continuously / sequentially read at least a portion of the input tensors partitioned in the data memory (165) and provide them to the operation unit (175) in the desired form.

[0116] The processing unit (175) can stream and process the acquired data. The processing unit (175) may include at least one of a shrinking engine, a vector engine, and a transpose engine.

[0117] The submission unit (180) can store the operation result of the operation unit (175) back into the data memory (165).

[0118] The data memory (165) can act as an on-chip note-taking memory and store tensors. The data memory (165) can be used to fetch input tensors and submit output tensors after computation. The data memory (165) can support virtual addressing via page tables to translate virtual addresses into physical addresses. Using this, large contiguous memory spaces can be dynamically allocated, which facilitates tensor management.

[0119] Furthermore, although the data memory (165) can have a one-dimensional address space, the data stored in the data memory (165) can be an N-dimensional tensor. For example, assuming a three-dimensional tensor has the shape shown in Table 1 (X (height) = 3, Y (width) = 4, Z (channels) = 2), the address of each tensor element in the memory can vary depending on how the tensor is stored in the data memory (165). For example, Table 2 shows the storage according to [X][Y][Z] = [3][4][2] (e.g., X-first, Z-last), and Table 3 shows the storage according to [Z][X][Y] = [2][3][4] (e.g., Z-first, Y-last). The conversion between Table 2 and Table 3 corresponds to transpose.

[0120] [Table 1]

[0121] [Table 2]

[0122] [Table 3]

[0123] Examining the read / write operations of the data memory (165), the data memory can be accessed (read, write) in units of a predefined memory width and can be accessed randomly relative to the depth of the data memory (165). Furthermore, within the same depth, data corresponding to the memory width can be read at once (parallel access). However, only one depth can be read per cycle (serial access). With this in mind, when tensor data is stored in the data memory (165), data corresponding to the last axis in the storage scheme can be read at once in units of the memory width, and data corresponding to non-last axes can be read in units of N (where N is an integer not greater than the width), although random access is possible. For example, assuming a memory width of 4 in the example of Table 3, the memory access structure can be represented as shown in Table 4.

[0124] [Table 4]

[0125] In the array [Z][X][Y]=[2][3][4], relative to the last axis Y, four values ​​corresponding to depth 0 [y0=a, y1=d, y2=g, y3=j] can be read at once, but these depth 0 values ​​all belong to z0. To read the values ​​belonging to z1, a random access from depth 0 to depth 5 can be performed, and four values ​​[y0=0, y1=r, y2=v, y3=y] can be read at once, but these depth 5 values ​​all belong to z1. As in the example in Table 4, for each width read (one cycle), for the last axis Y, four values ​​y0 to y3 can be read at once, but for the first axis Z, only one of z0 or z1 can be read.

[0126] In other words, the acquisition unit (170) can access the last dimension (last_dim) of tensor data stored in the data memory (165) in units of memory width, i.e., in library units (e.g., 8B, 16B, 32B / cycle), and can access non-last dimensions in any dimension order.

[0127] Submit unit (180) can operate similarly. Submit unit (180) can also perform slicing / reshaping relative to any dimension, except the last dimension. For slicing, the last dimension may need to be handled by a separate function or by transposition. For transposition, see the description of the transposition engine below.

[0128] Figure 6 This is a diagram used to explain the partitioning and assignment of tensor data according to the implementation method. For ease of description, it is assumed that the input tensor of TU (100) is a three-dimensional tensor (X, Y, Z), and TU (100) includes a total of 8 slices (160).

[0129] Reference Figure 6 Tensor data can be divided into 8 data partitions, and each data partition can be assigned to each slice. For example, if a three-dimensional tensor (X, Y, Z) is uniformly partitioned along each axis by 1 / 2, then the data memory (165) of each slice (160) can store the three-dimensional tensor divided into [X / 2][Y / 2][Z / 2].

[0130] The method of tensor partitioning can be changed and is not limited to the examples above. For example, tensors can be partitioned and assigned along any axis. For example, when Z=8, the two-dimensional tensor (X, Y) corresponding to z0 can be assigned to the first slice, the two-dimensional tensor (X, Y) corresponding to z1 can be assigned to the second slice, and the two-dimensional tensor (X, Y) corresponding to z7 can be assigned to the eighth slice.

[0131] Figures 7 to 9 Various examples of partitioning a three-dimensional tensor and assigning it to four slices are shown.

[0132] exist Figures 7 to 9 For ease of description, a feedforward layer is considered. Assume the dimension of the input tensor of the feedforward layer is b. 1 e (e.g., b=4, 1=4, e=E), and the dimension of the weights is e. f. Here, b represents the batch size, l represents the sequence size, e represents the input embedding size, and f represents the embedding size of the feedforward network. The contraction along the e-axis is represented as “ble,ef →blf”. The notation “inter-slice partitioning shape | intra-slice shape” is used.

[0133] Reference Figure 7 As a result of the 1 / 2 partition along the b-axis and the 1 / 2 partition along the e-axis, a total of 4 partitions are generated, and one partition is assigned to each slice. Specifically, since the partition along the b-axis has a step size or stride of 2, it can be represented as b 2 Since the partitioning is performed on two slices, it can be represented as [2]. The same applies to the e-axis. Therefore, the e-axis has a stride of E / 2 on the two slices and can be represented as e E / 2 [2]. As a result, the shape of the division between slices is b. 2 [2]e E / 2 [2]. Each slice is assigned a shape of [2][4][E / 2], which corresponds to 1 / 4 of a tensor with the same shape [4][4][E], and each axis has a stride of 1. Therefore, the shape within a slice is b. 1 [2]l 1 [4]e 1 [E / 2]. Meanwhile, the partitions without shading / pattern markers in slices 1 and 3 correspond to... Figure 7 The back side region is hidden from the viewpoint of the three-dimensional tensor shown.

[0134] exist Figure 8 In the middle, the shape between slices is b. 1 [4], the shape inside the slice is l 1 [4]e 1 [E].

[0135] Reference Figure 9 As a result of dividing the slice by 1 / 4 on the e-axis, a total of 4 partitions are generated, and one partition is assigned to each slice. Figure 9 In this context, four partitions are allocated sequentially along the e-axis, from slice 0 to slice 3. Figure 9 In this text, the pattern / shading representation showing the mapping between 3D tensors and slices is omitted. Figure 9 In the middle, the shape between slices is e E / 4 [4], the shape inside the slice is b 1 [4] / l 1 [4]e 1 [E / 4].

[0136] like Figures 7 to 9 As shown, tensor partitioning reduces the shape of a tensor, and tensor partitions with reduced shapes can be assigned to corresponding slices.

[0137] The foregoing example illustrates the case where a tensor is divided into equal-sized segments by partitioning it along some axes and stored in the data memory of each slice; however, the invention is not limited thereto. Furthermore, while the above example shows an input tensor as a three-dimensional tensor for ease of description, tensors with three or more dimensions can also be input.

[0138] Furthermore, in a contraction operation, the order in which axes are computed is called the tactic. For example, suppose as follows: Figure 9 The degradation shape shown is used to shrink the input tensor ble,ef → blf. Assume the first input tensor is reduced to e. E / s [s] | b 1 [B]l 1 [L]e 1 [E / s], the second input tensor is reduced to e. E / s [s] | f 1 [F]e 1 [E / s], where B, L, E, F represent the size of the corresponding dimensions (b, l, e, f) of the input / weight tensor before degradation, and s represents the number of slices (e.g., Figure 9 (where s=4). It is also assumed that the output tensor is reduced to f. F / s [s] | b 1 [B]l 1 [L]f 1 [F / s]. One possible strategy is e E / s [s] |b 1 [B]l 1 [L] f 1 [F]e 1 [E / s].

[0139] In this strategy, e E / s [s] represents the outermost loop on the e-axis, which has a stride of E / s across s slices, meaning that the sub-loops of the outermost loop are executed in parallel.

[0140] In this strategy, b 1 [B]l 1 [L] f1 [F]e 1 [E / s] (i.e., the slicing strategy) represents how each slice handles the remaining loops. Here, there are four loops on the b, l, f, and e (innermost) axes. In the slicing strategy, e 1 [E / s] is the innermost loop and represents contraction. Within the innermost loop, each slice performs contraction. For example, each slice performs a dot product between two vectors of size E / s along the e-axis. In the slicing strategy, f... 1 [F] corresponds to the f-axis and means that data acquired before the f-axis loop is reused F times (e.g., using the slice's internal input buffer for reuse). The f-axis loop repeats B times on the two outermost loops. L times.

[0141] In this strategy, e E / s[ [s] indicates that the partial dot products on the e-axis of the outputs of each of the s outermost sub-loops should be combined to obtain the final result. The process of calculating the final dot product from the partial dot product values ​​can be performed through a network between slices, and can also be distributed according to the shape of the output tensor.

[0142] For a given pair of input tensors and tensor operations, various strategies can be employed depending on the degradation shape of the input / output tensors, the number of available slices, and so on. Once the computational unit is determined, candidate degradation shapes are searched, and those that meet performance and power consumption requirements are selected. If the degradation shapes between adjacent operations do not match (e.g., if the degradation shape of the output of a previous layer does not match the degradation shape of the input of a subsequent layer), an additional bridge operation can be performed to transform the degradation shape.

[0143] The proposed TCP supports dynamic tensor shapes through control registers that describe the tensor shape, as described below. For example, in Figure 9 In the example, the b-axis and l-axis can be determined dynamically (e.g., depending on the token generation length for each request and the number of new requests dynamically determined in the LLM service).

[0144] Figure 10 This is a diagram illustrating the configuration example of the overall operation and arithmetic unit (175) of the TU (100).

[0145] Reference Figure 10 The operation unit (175) may include at least one of the contraction engine (176), the vector engine (177), and the transpose engine (178).

[0146] The shrink engine (176) is used to shrink (partitioned) tensor data.

[0147] In TCP(1), tensor operations are provided as primitives, and TU(100) continuously accelerates the operation according to the length of the tensor operation based on the settings of the given tensor operation until the given tensor operation is completed. For example, TU(100) processes tensors not only in parallel spatially but also performs continuous processing on the time axis. For this purpose, a fetch-operate-submit pipeline is used, which continuously fetches the given tensor and passes it to the arithmetic unit, and sequentially writes the result tensor to the data memory after the given processing is completed.

[0148] For example, such as Figure 10 As shown, when the input tensor is partitioned and stored in the data memory (165) of 8 slices, each acquisition unit (170) of each slice can sequentially acquire data from the data memory (165) based on tensor information (e.g., tensor shape / dimension information) obtained through the control register.

[0149] As described below, the acquisition unit (170) can be configured with an acquisition network and can multicast data acquired through the acquisition network to multiple processing units (175) (e.g., sequential multicast or at least partially simultaneous parallel multicast).

[0150] Each operational unit (175) performs tensor shrinking, vector operations, and / or tensor manipulation (e.g., transpose).

[0151] The submission unit (180) can sequentially submit tensors, which are the results of operations of each operation unit (175), to the data memory (165) based on the order information set in the control register.

[0152] Figure 11 A detailed implementation example of the configuration of the slice (160) according to the embodiment is shown.

[0153] Reference Figure 11 The acquisition unit (170) may include at least one of an acquisition sequencer and an extraction processing unit.

[0154] The fetch sequencer generates an address sequence for accessing the data memory (165). The fetch processing unit can perform data preparation such as type conversion and padding.

[0155] The acquisition unit (170) can be configured to acquire the network to transmit data not only to its own slice, but also to the processing units of other slices.

[0156] The acquisition sequencer generates addresses based on a given tensor shape and a specified order. The acquisition sequencer and acquisition processing unit not only generate basic N-dimensional circular type addresses, but also support indirect addressing and table lookups, thus providing flexibility for various types of tensor operations.

[0157] Data obtained from the data storage (165) of the slice (160) can be sequentially transmitted through the acquisition network to the processing unit (175) including the shrink engine (176), the vector engine (177) and / or the transpose engine (178).

[0158] The contraction engine (176) may include a dot product engine (DPE) (1762), which performs element-wise multiplication and can use a reduction tree to generate various numbers of outputs. The depth of each reduction tree is configurable, and the output of the reduction tree can be accumulated in time by an accumulation unit (1763). Each DPE (1762) may receive two input vectors, for example, one from the register file (1761) of the contraction engine (176) and the other from a vector sequentially passed from the acquisition unit (170).

[0159] Figure 12 A detailed example of the structure of the contraction engine (176) is shown. The contraction engine (176) can perform tensor contraction through dot product operations.

[0160] Reference Figure 12 The data path in the contraction engine (176) includes the feed unit, register file, dot product engine (DPE) and accumulation unit, and can be controlled by the operation sequencer.

[0161] For example, the shrink engine (176) can include eight DPEs, which perform a dot product by spatially summing the results of element-wise multiplication of two input vectors passed through a reduction tree. Each DPE can obtain two input vectors from a feed unit and a register file. For example, each input can hold 32 BF16 values, 64 FP8 values, 64 INT8 values, or 128 INT4 values. The number of input elements represents the maximum throughput for a given input type.

[0162] Eight DPEs can share input from the feed unit (e.g., via broadcast data reuse) and can receive separate input from a register file. The depth of the reduction tree, which determines the number of outputs of the DPE, can be set via a control register. Furthermore, in addition to supporting additive reduction, the DPEs also support maximum reduction, thus supporting various computations such as max pooling.

[0163] The feed unit processes packets transmitted over the network and broadcasts the data to multiple DPEs for reuse. The feed unit is controlled by an operation sequencer and achieves additional data reuse by providing the same data multiple times to eight DPEs in consecutive cycles. The feed unit can also use shifting operations such as convolution and transpose tensors for data reuse.

[0164] The accumulator unit receives the output of the DPE for time accumulation. To reuse the input of the feed unit in time, multiple accumulators are needed to maintain the various partial sums. Since increasing the number of accumulators proportionally increases read / write costs, it is preferable to allocate and utilize as few accumulators as possible per unit.

[0165] For example, an accumulation unit can include a total of 1024 accumulators, which can be indexed differently depending on the output size of the DPE. For instance, when using 32 BF16 inputs (from the feed unit) and the depth of the reduction tree is 0, each DPE produces 32 multiplication results, so the total number of outputs for eight DPEs is 256 (=8×32). Given that 1024 accumulators are available, each output of a DPE can be associated with up to four accumulators for time accumulation. However, when the depth of the reduction tree is 3, each DPE produces four outputs (i.e., partial sums), and eight DPEs produce 32 outputs. In this case, each output of a DPE can be associated with 32 (=1024 / 32) accumulators for time accumulation.

[0166] The register file is used as storage for operands that are reused in multiplication and accumulation (MAC) operations. Since the register file provides data to each DPE, it may require at least as much storage as the number of DPEs.

[0167] Figure 13 A detailed example of the structure of the vector engine (177) is shown.

[0168] The vector engine (177) can perform non-linear functions, element-wise operations, reduction and / or type conversion.

[0169] The vector engine (177) can include functional units for multiple clusters. For example, functional units can be used for various arithmetic and logical operations against INT32 / FP32, such as functions of exponentiation (exp), sine (sin), prediction operations, type conversion and / or quantization against INT4 / 8 / 16 / 32, FP8, FP16 / BF16 and FP32. The vector engine can also route reduction results to a specified slice within or between slices, or multicast them to multiple slices.

[0170] Return to Figure 11The transpose engine (178) can push data for a specified number of rows and pop data for another set of rows to change the last axis of the tensor to a different axis within that slice. The transpose engine (178) can also use a get sequencer and / or submit sequencer to perform a switch between the last axis and another axis.

[0171] The transpose engine (178) and the commit unit (180), together with the vector engine, can perform tensor manipulations on multiple slices or within a single slice. Tensor manipulations can include transpose, split, slice, reconcile, and / or reshape. Similar to the vector engine (177), such tensor manipulations can be performed after or in parallel with dot product operations.

[0172] The commit unit (180) may include at least one of a commit sequencer and a commit processing unit. The commit sequencer can determine the storage layout of the tensor by specifying the addresses of the commit data to be sent in a specific order. The commit processing unit may support type conversion and may remove padding from the data for compression during storage.

[0173] Each of the acquisition unit (170) and the submission unit (180) can be provided as multiple units within a slice (160) and can have their own data paths. By configuring multiple data paths within a slice, multiple independent operations can be performed simultaneously within a slice (160).

[0174] Figure 20 This illustrates a scenario where multiple acquisition units and multiple submission units, according to an implementation method, are configured within a single slice. Figure 20 For ease of description, an example is given where the number of acquisition units and the number of submission units for each slice are two. However, the present invention is not limited to this, and N acquisition units / submission units (N is an integer of 1 or greater) can be set in a slice.

[0175] When multiple acquisition units are set, one of the acquisition units can be operated as the main acquisition unit (1701), while the remaining acquisition units can be operated as sub-acquisition units (1702).

[0176] When multiple commit units are set, one of the commit units can be operated as the main commit unit (1801), while the remaining commit units can be operated as sub-commit units (1802).

[0177] Multiple data paths can be configured based on combinations of fetch and submit units, and based on these multiple data paths, multiple contexts can be executed simultaneously within a slice (e.g., in...). Figure 20 In the case of two).

[0178] However, according to the implementation, configuring data paths may not involve fetch / submit units. For example, data paths for moving data between register files can be configured without fetch / submit units. Considering this, a greater number of data paths than fetch / submit units can be configured.

[0179] Reference Figure 20 In (a), the main context uses the shrinking engine (176) and the vector engine (177) to perform operations acquired through the main fetch unit (1701) and operations submitted through the main submit unit (1801). The sub-context performs operations acquired through the sub-fetch unit (1702) and loads the data into the tensor register file.

[0180] Reference Figure 20 (b) illustrates an operation in which tensor shrinking is performed in the main context using a shrinking engine (176) and transposition is performed in the sub-context using a transposition engine (178).

[0181] Figure 14 This is an example used to explain the acquisition of the network according to the implementation method.

[0182] As described above, the acquisition units (170) of different slices (160) can be configured with an acquisition network, and data can be acquired continuously in a pipeline manner through the acquisition network. The acquired data can be reused through the acquisition network.

[0183] Reference Figure 14 The acquired tensor data can be multicast to multiple slices through the acquisition network. Multicast data can be processed in parallel across multiple computation units. The acquisition network can be configured / reconfigured similarly to a circuit-switched (CS) network. The acquisition network guarantees data ordering across multiple sources and within packets from a single source, and supports multicast. CS-like acquisition networks can improve throughput and reduce hardware complexity in routing, ordering, multicast, and flow control.

[0184] TCP(1) can dynamically merge PEs by reconfiguring the network.

[0185] Figures 15 to 17 This is a diagram used to explain examples of processing input tensor data according to various implementation methods.

[0186] Depending on how the input tensors are allocated / arranged in the data memory (165) and how the acquisition unit (170), operation unit (175), and / or submission unit (180) are configured, the parallelism and data locality (or data reuse) used in performing tensor operations can vary. Since the available parallelism and data locality vary depending on the shape of the tensor and the type of tensor operation required, TCP (1) can operate flexibly by taking into account the shape of the tensor and the tensor operation.

[0187] Reference Figure 15 A matrix multiplication is performed between two two-dimensional tensors (B01, B02). The first tensor (B01) is divided into four equal parts horizontally. The second tensor (B02) can be a tensor stored in the shrinking engine's register file. The third tensor (B03) is submitted to the data memory (165) as the result of the tensor operation. Each slice can be operated on independently and in parallel. Figure 15 In the example, it is assumed that data exchange between slices is not performed via a fetch network or vector engine. Each slice sequentially loads partitions of the first tensor (B01) and feeds them to the dot product engine of the computation unit, and after each dot product operation, it is submitted to produce the output tensor. Regarding the ble,ef → blf shrinking example, in Figure 15 In this process, the tail axis is not the e-axis, and the DPE depth becomes 0. Each element of the e-axis read from the register file is broadcast, multiplied, and accumulated in the accumulation unit. The partition axis of each slice is b or l, which is processed in parallel, and the second operand is multipoint-transferred to all slices and loaded into the register file.

[0188] Reference Figure 16 The matrix multiplication between two two-dimensional tensors (A01, A02) is performed. The first tensor (AO1) is divided into two equal parts in both the horizontal and vertical directions. The second tensor (A02) is stored in the register file of the shrinking engine and can be divided into groups of two along the f-axis and stored in each slice. During the acquisition / transmission of the first tensor (AO1), the E / 2 of slice 0 and the E / 2 of slice 1 are passed to slice 1 and slice 0 respectively through the acquisition network, and each slice can perform the F / 2 operation on the f-axis independently and in parallel. That is, the acquisition network can be configured to multicast to slice 0 and slice 1. The shrinking engine (CE) of slice 0 and slice 1 uses the data from both slice 0 and slice 1 and performs a dot product using the corresponding data of the second tensor (AO2) in the order of the delivered data. Since the vector engine (177) does not need to reduce the data from multiple slices, the submission unit (180) of each slice performs the submission.

[0189] Reference Figure 17Perform a matrix multiplication between two two-dimensional tensors (901, 902). The first tensor (901) is divided into two equal parts in both the horizontal and vertical directions. The second tensor (902) can be stored in the shrink engine's register file. For matrix multiplication, a dot product is performed on the e-axis. When... Figure 17 When partitioned / stored as described above, slice 0 sequentially passes E / 2, while slice 1 sequentially passes the remaining E / 2 to perform a dot product, and (after subsequent necessary processing steps) a third tensor (903) can be output. Specifically, the results output from the shrink engine (CE) are reduced (summed) on the two slices, and through the reduction and routing functions of the vector engine (VE), the results from multiple slices can be added and passed to the target slice. The submission unit (180) can sequentially store the output tensor in the data memory (165). Figure 17 In this process, each data partition of the first tensor (901) is used multiple times (reused) relative to the second tensor (902). The shrinking engine (CE) can feed the received input tensors to the DPE multiple times according to the control register configuration. Based on the fed data, the shrinking engine sequentially passes the corresponding data partitions of the second tensor (902) to the DPE to perform dot products, and multiple accumulators can be used to store intermediate results. Therefore, input reuse included in tensor operations can be utilized. Regarding ble,ef → blf shrinking, in Figure 17 In the example, on 2 slices according to e E / 2 [2] | e 1 [E / 2] contractes, and along b B / 2 [2] The axis is subjected to parallel contraction.

[0190] Figure 18 An example of data reuse and dot product in TCP(1) is shown.

[0191] As described above, in method I, the data provided to the memory is passed down, and the dot product is performed while the dot product result is passed to the right.

[0192] In the proposed TCP(1), the following operations can be performed depending on the settings of the control registers, including the computational pipeline for acquiring the network.

[0193] - Data retrieved from each data store can be multicast to all slices. Furthermore, within each shrink engine, multiple dot product engines can be configured to perform dot product operations.

[0194] - Data retrieved from each data store is unicast only to its own shrink engine, and the shrink engine can reduce the result after performing the dot product.

[0195] In this way, depending on the configuration of the control registers, the operation units suitable for various tensor shapes can be utilized to the maximum extent, and the pipeline can be fully utilized by continuously acquiring / operating / submitting data along the time axis through data paths configured / reconfigured based on tensor operations.

[0196] As described above, the TCP proposed in this specification is a structure capable of handling tensor operations as primitives. Unlike the Method I accelerator, which processes fixed-size identity matrix operations as instructions, TCP can specify variable-size tensor operations through control registers and accelerate them immediately.

[0197] Tensor operations exhibit very high regularity and a high level of parallelism and data locality. TCP allows slices to be reconfigured to fit the tensor shape for each computational objective, maximizing the parallelism of the entire tensor operation and ensuring that they are always processed sequentially in a software-defined order. This allows the full utilization of the data locality included in tensor operations by making full use of finite-sized storage buffers.

[0198] When fixed-size identity matrix operations are processed as instructions, as in Method I, the degree of data reuse is limited by the size of the matrix operation. However, TCP can improve energy efficiency by allowing more data reuse. Furthermore, since TCP processes instructions on a per-tentacle basis, it reduces the cost of processing instructions and also decreases the program size compared to Method I, which processes instructions for each identity matrix operation.

[0199] In this way, TCP can fully leverage the parallelism and data locality in tensor shrinking, thereby improving the efficiency and performance of tensor data processing used in AI and other fields. TCP can consist of large-scale PEs, which simplifies related software development. Furthermore, by configuring the entire tensor shrinking process via a single control register, control overhead can be minimized.

[0200] To efficiently handle operations of various tensor shapes, multiple PEs can be configured as a single computational unit or as a group of smaller, independently operable computational units. Furthermore, the acquisition network can be configured / reconfigured like a circuit switch, providing a structure that enables data reuse between / within computational units.

[0201] Support for dynamic shape and control flow is important for the optimization of AI accelerators (e.g., in LLM inference), and the proposed TCP (1) can support dynamic shape and control flow by setting / resetting control registers, enabling more efficient generation of programs based on dynamically changing computation, memory and power requirements.

[0202] As described above, a device for performing data processing (e.g., TCP) may include: a plurality of PEs (10) configured to perform tensor-based processing; and a semiconductor substrate on which the plurality of PEs (10) are mounted.

[0203] Each PE (10) may include: a plurality of slice circuits (160), each slice circuit having a computation pipeline; and a control circuit (130) configured to control the plurality of slice circuits (160).

[0204] Each computation pipeline on each slice circuit (160) may include: i) an acquisition unit (170) configured to acquire at least a portion of tensor data; ii) an operation unit (175) configured to perform operations on at least a portion of the acquired tensor data; and iii) a submission unit (180) configured to deliver the operation results output from the operation unit to a destination.

[0205] The control circuit (130) controls the mapping between partitions of tensor data and multiple computation pipelines on multiple slice circuits (160).

[0206] For example, multiple slice circuits (160) can be configured / reconfigured based on the shape of the tensor data and the instructions given for the tensor data.

[0207] For example, a control register can be assigned to each of the multiple slice circuits (160), which stores the configuration / reconfiguration information of each slice circuit.

[0208] For example, the control circuit (130) can perform configuration / reconfiguration of each slice circuit (160) and read / write of the associated control register.

[0209] For example, each PE (10) may also include an on-chip sticky note memory (SPM) (140). A set of data assigned to multiple control registers of multiple slice circuits (160) can be configured by loading values ​​stored in the on-chip sticky note memory (140).

[0210] For example, the partitioning of tensor data can be performed based on at least one of the shape of the tensor data and the instructions given for the tensor data.

[0211] For example, a data memory (165) with bandwidth associated with the computational pipeline can be assigned to each slice circuit (160).

[0212] For example, the operation unit (175) may include at least one of the following: a shrinkage engine (176) configured to perform tensor shrinkage on at least a portion of the acquired tensor data; a vector engine configured to perform operations on vectors as one-dimensional tensors; and a transpose engine configured to perform data transpose.

[0213] For example, the shrinking engine (176) can perform tensor shrinking by reducing the rank.

[0214] For example, the control circuit (130) can set up a configurable acquisition network through the connection between different acquisition units (170) on different slice circuits (160).

[0215] For example, data acquired by the acquisition network can be sequentially multicast to the arithmetic units (175) of different slice circuits (160).

[0216] For example, a network can perform multicast by reusing the acquired data.

[0217] For example, the acquisition unit may include a main acquisition unit and a sub-acquisition unit, and the submission unit may include a main submission unit and a sub-sub-submission unit.

[0218] For example, the main data path can be configured based on the main acquisition unit and the main submission unit, and the sub-data path can be configured based on the sub-acquisition unit and the sub-submission unit. Furthermore, operations on the main data path and operations on the sub-data path can be performed simultaneously within a slice circuit.

[0219] For example, the control circuit (130) can configure / reconfigure the first slice circuit and the second slice circuit to transmit the first operation result of the first arithmetic unit of the first slice circuit to the second slice circuit.

[0220] For example, each PE (10) may further include a core processor (110), and the control circuit (130) may operate as a coprocessor of the core processor (110).

[0221] For example, the control circuitry (130) may include an instruction processor configured to execute instructions obtained from the core processor (110). These instructions may include at least one of the following: instructions for loading the address of the control register assigned to each slice circuit; and instructions for enabling each slice circuit.

[0222] For example, the device may further include: a memory controller (30) configured to control the data flow between the plurality of PEs (10) and the memory device; and an on-chip network formed on the semiconductor substrate for use with the plurality of PEs (10) and the memory controller (30).

[0223] For example, tensor data can be associated with neural network processing used in artificial intelligence.

[0224] For example, the device could be a tensor shrinking processor (TCP, 1) to accelerate neural network processing for artificial intelligence.

[0225] Figure 19 The flow of a method performed in a data processing apparatus according to an embodiment is shown.

[0226] Reference Figure 19 Devices used for data processing can obtain instructions for tensor data (1905).

[0227] Devices for data processing can perform partitioning of tensor data based on at least one of the shape of the tensor data and instructions (1910).

[0228] Devices used for data processing can assign partitions of tensor data to multiple computation pipelines (1915).

[0229] The device for data processing can process each partition of tensor data through each of multiple computational pipelines (1920). Each computational pipeline may include: i) a process for acquiring each partition of tensor data, ii) a process for performing operations on the acquired partitions, and iii) a submission process for passing the results of the operations to a destination.

[0230] The embodiments of the present invention described above can be implemented by various means. For example, the embodiments of the present invention can be implemented by hardware, firmware, software, or a combination thereof.

[0231] In the case of hardware implementation, the method according to embodiments of the present invention can be implemented in one or more ASICs (Application-Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field-Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

[0232] When implemented via firmware or software, the method according to embodiments of the invention can be implemented in the form of modules, processes, or functions that perform the aforementioned functions or operations. Software code can be stored in memory units and driven by a processor. Memory units can be located internally or externally to the processor and can exchange data with the processor in various known ways.

[0233] A detailed description of preferred embodiments of the invention disclosed above has been provided to enable those skilled in the art to implement and practice the invention. While the invention has been described with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from its scope. For example, those skilled in the art can use each configuration described in the above embodiments in combination with each other. Therefore, the invention is not limited to the embodiments shown herein, but is accorded the widest scope consistent with the principles and novel features disclosed herein.

[0234] The invention may be embodied in other specific forms without departing from its spirit and essential characteristics. Therefore, the detailed description above should not be construed as restrictive in all respects, but rather as illustrative. The scope of the invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the scope of equivalence of the invention are included within its scope. Furthermore, claims not expressly referenced in the claims may be combined to form embodiments, or may be included as new claims after submission by amendment.

Claims

1. An apparatus for performing data processing, the apparatus comprising: Multiple processing elements PE, the multiple processing elements PE being configured to perform tensor-based processing; as well as A semiconductor substrate, wherein the plurality of PEs are mounted on the semiconductor substrate. Each PE includes: Multiple slice circuits, each slice circuit having a computational pipeline; and A control circuit, configured to control the plurality of slice circuits. Each computational pipeline on each slice circuit includes: i) an acquisition unit configured to acquire at least a portion of tensor data; ii) a computation unit configured to perform operations on at least a portion of the acquired tensor data; and iii) a submission unit configured to deliver the computation results output from the computation unit to a destination. The control circuit is configured to control the mapping between partitions of the tensor data and multiple computation pipelines on the multiple slice circuits.

2. The device according to claim 1, wherein, Configure / reconfigure the plurality of slice circuits based on the shape of the tensor data and the instructions given for the tensor data.

3. The device according to claim 2, wherein, A control register is assigned to each of the plurality of slice circuits, and the control register stores the configuration / reconfiguration information of each slice circuit.

4. The device according to claim 3, wherein, The control circuit is configured to perform configuration / reconfiguration of each slice circuit and read / write of the associated control registers.

5. The device according to claim 3, in, Each PE further includes an on-chip sticky note memory (SPM), and The data assigned to the multiple control registers of the multiple slice circuits is configured by loading values ​​stored in the on-chip sticky memory.

6. The device according to claim 1, wherein, The partitioning of the tensor data is performed based on at least one of the shape of the tensor data and the instructions given for the tensor data.

7. The device according to claim 1, wherein, Each slice circuit is assigned a data memory with bandwidth associated with the computational pipeline.

8. The device according to claim 1, wherein, The computation unit includes at least one of the following: a shrinkage engine configured to perform tensor shrinkage on at least a portion of the acquired tensor data; and a vector engine configured to perform operations on a vector that is a one-dimensional tensor. And a transpose engine configured to perform data transpose.

9. The device according to claim 8, wherein, The shrinkage engine performs the tensor shrinkage by reducing the rank.

10. The device according to claim 1, wherein, The control circuit configures the configurable acquisition network by connecting different acquisition units on different slice circuits.

11. The device according to claim 10, wherein, The data acquired by the acquisition network is multicast to different processing units of the different slice circuits.

12. The device according to claim 11, wherein, The acquisition network performs the multicast by reusing the acquired data.

13. The device according to claim 1, in, The acquisition unit includes a main acquisition unit and a sub-acquisition unit, and The submission unit includes a main submission unit and a sub-submission unit.

14. The device according to claim 13, in, The main data path is configured based on the main acquisition unit and the main submission unit. The sub-data path is configured based on the sub-acquisition unit and the sub-submission unit, and The operations on the main data path and the operations on the sub-data path are executed simultaneously within a slice circuit.

15. The device according to claim 1, wherein, The control circuit configures / reconfigures the first slice circuit and the second slice circuit to deliver the first operation result of the first operation unit of the first slice circuit to the second slice circuit.

16. The device according to claim 1, in, Each PE further includes a core processor, and The control circuit operates as a coprocessor for the core processor.

17. The device according to claim 16, in, The control circuitry includes a command processor configured to execute instructions obtained from the core processor, and The instructions include at least one of the following: instructions for loading the control register address assigned to each slice circuit; and instructions for enabling each slice circuit.

18. The device according to claim 1, wherein the device further comprises: A memory controller configured to control the data flow between the plurality of PEs and the memory device; as well as An on-chip network is formed on the semiconductor substrate for use with the plurality of PEs and the memory controller.

19. The device according to claim 1, wherein, The tensor data is related to neural network processing used in artificial intelligence.

20. The device according to claim 19, wherein, The device is a tensor shrinking processor used to accelerate neural network processing for artificial intelligence.

21. A method for performing data processing, the method comprising the steps of: Obtain instructions for tensor data; The tensor data is partitioned based on at least one of the shape of the tensor data and the instruction; The partitions of the tensor data are assigned to multiple computation pipelines; as well as Each partition of the tensor data is processed by each of the plurality of computational pipelines. Each computation pipeline includes: i) a process of acquiring each partition of the tensor data, ii) a process of performing operations on the acquired partitions, and iii) a submission process of delivering the results of the operations to a destination.

22. A storage medium capable of being read by at least one processor and storing program code, the program code causing the at least one processor to perform an operation when executed by the at least one processor, the operation comprising: Obtain instructions for tensor data; The tensor data is partitioned based on at least one of the shape of the tensor data and the instruction; The partitions of the tensor data are assigned to multiple computation pipelines; as well as Each of the partitions of the tensor data is processed by each of the plurality of computational pipelines. Each computation pipeline includes: i) a process of acquiring each partition of the tensor data, ii) a process of performing operations on the acquired partitions, and iii) a submission process of delivering the results of the operations to a destination.

23. The storage medium according to claim 22, in, The multiple computational pipelines are formed on multiple slice circuits, and The plurality of slice circuits are configured / reconfigured based on the shape of the tensor data and the instructions given for the tensor data.

24. The storage medium according to claim 23, wherein, Each of the plurality of slice circuits is assigned a control register that stores the configuration / reconfiguration information for each slice circuit.

25. The storage medium according to claim 24, wherein, The operations performed by the at least one processor further include: Perform configuration / reconfiguration and read / write of the associated control registers for each slice circuit.

26. The storage medium according to claim 22, wherein, The operations performed by the at least one processor further include: The partitioning of the tensor data is determined based on at least one of the shape of the tensor data and the instructions given for the tensor data.

27. The storage medium according to claim 22, wherein, The operations performed by the at least one processor further include: Assign a data storage device with the relevant bandwidth to each computing pipeline.

28. The storage medium according to claim 22, wherein, The process of performing the operation includes at least one of the following: performing tensor shrinkage on at least a portion of the acquired tensor data; performing operations on a vector that is a one-dimensional tensor; and performing data transpose.

29. The storage medium according to claim 28, wherein, The at least one processor performs the tensor shrinkage by reducing the rank.

30. The storage medium according to claim 22, wherein, The tensor data is related to neural network processing used in artificial intelligence.

31. The storage medium according to claim 22, wherein, The at least one processor is a tensor shrinking processor for accelerating neural network processing for artificial intelligence.

Images