Chip and data processing method

Abstract

The embodiment of the invention provides a chip and a data processing method, and relates to the field of integrated circuits. The chip is used for processing a deep learning algorithm, and the deep learning algorithm comprises a calculation-intensive operator and a memory-access-intensive operator. The chip at least comprises a calculation core particle and a bandwidth core particle, the calculation core particle and the bandwidth core particle are coupled through a core particle interconnection interface, and the bandwidth core particle is further directly connected with the high-bandwidth memory; computing intensive operators are mapped to the computing core particles, and memory access intensive operators are mapped to the bandwidth core particles; the calculation core particle is used for executing a task corresponding to the calculation-intensive operator, and the bandwidth core particle is used for executing a task corresponding to the memory access-intensive operator. According to the method, operators with different requirements on computing power specifications and bandwidth specifications in the deep learning algorithm are processed through different core grains in the same chip, and the problem of hardware resource waste caused by great difference of different parts of the deep learning type algorithm on computing and bandwidth capability requirements is effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of integrated circuits, and in particular to a chip and a data processing method. BACKGROUND

[0002] The specifications of a typical AI chip include computing performance specifications, bandwidth specifications, storage specifications, etc. The performance of an AI algorithm service is closely related to the computing performance specifications of the AI chip. However, the performance of the chip is also affected by many other factors. A typical example is that the storage I / O bandwidth may cause the computing unit performance to be unable to be exerted. In recent years, with the rise of AI large model services, the weights and intermediate cache results are huge. When a single chip is insufficient to store the weights or intermediate results, multiple chips need to be relied on to realize data sharing through networking, and therefore the storage capacity of the chip has gradually become an important performance influencing factor.

[0003] A deep learning algorithm is a computation graph composed of a series of different operators. Different operators have different computing / bandwidth / storage resource requirement characteristics. Even the same type of operator, in different configurations, also exhibits completely different resource requirement characteristics. Table 1 shows an example of the difference in storage, computing, and bandwidth characteristics required by different operators. Among them, the data volume reflects the demand for storage resources, the computing complexity reflects the demand for computing resources (Ops, i.e., the number of operations), and the computing memory ratio reflects the demand for bandwidth resources (the higher the computing amount value experienced on the chip for each unit of data loaded, the longer the time the single data stays on the chip, the slower it is consumed, and the lower the demand for bandwidth).

[0004] Table 1:

[0005]

[0006] Traditional AI chips generally adopt a single computing power design. The chip uses the same computing power to compute the computing power graph of the deep learning algorithm from beginning to end. However, in the computation graph of the deep learning algorithm, different operators have different demands for bandwidth capability, computing capability, and storage capability. Using the same chip for processing will cause different bottlenecks such as computing bottlenecks and bandwidth bottlenecks to be experienced at different time periods during the computation, resulting in a large amount of resource waste and difficulty in maximizing hardware utilization.

[0007] An improved design is to separate the tasks of a deep learning algorithm, and distribute tasks with different requirements for computing power and bandwidth capability to different chips or servers. For example, in the Transformer inference process of an LLM (Large Language Model), the Prefill and Decode stages are separated and completed on different computing nodes such as different servers or different chips. The prefill stage is to generate a cache of the user input prompt sequence and the first token, and the task characteristics of this stage are computationally intensive. The decoding stage is to calculate the next token from the latest generated token and the cache. This stage will produce a complete output through multiple iterations, and the task characteristics are bandwidth-intensive. Therefore, the prefill calculation can be performed by a high-power node, and the intermediate calculation results are transmitted to a high-bandwidth computing node, which is responsible for decoding calculation, thereby relieving the imbalance between computing and bandwidth. However, the challenge brought by this scheme is that the cache data generated in the prefill stage is huge, and needs to be transmitted at high speed over a long distance between multiple servers and multiple chips through PCIE or server network, which additionally increases the high communication cost (such as high-end switches). In addition, since the computing power of the prefill and decode stages needs to store complete weight data and calculate the complete algorithm process respectively, the internal operator calculation bottleneck and bandwidth bottleneck problem in a single algorithm inference process cannot be solved. SUMMARY

[0008] To solve the above technical problems or at least partially solve the above technical problems, embodiments of the present application provide a chip and a data processing method.

[0009] In a first aspect, embodiments of the present application provide a chip, which is used for processing a deep learning algorithm, and the deep learning algorithm at least includes a computation-intensive operator and a memory-intensive operator.

[0010] The chip at least includes a computing core and a bandwidth core, and the computing core and the bandwidth core both include a core interconnection interface, and the computing core and the bandwidth core are coupled through the core interconnection interface; and the bandwidth core is directly connected with a high-bandwidth memory.

[0011] The computation-intensive operator is mapped to the computing core, and the memory-intensive operator is mapped to the bandwidth core.

[0012] The computing core is used for executing a task corresponding to the computation-intensive operator, and the bandwidth core is used for executing a task corresponding to the memory-intensive operator.

[0013] Optionally, the computing core further includes a first computing core unit and a data moving unit.

[0014] The first computing core unit is coupled with the data movement unit; the first computing core unit, in response to receiving a task corresponding to the compute-intensive operator, performs operation on first to-be-processed data in the task corresponding to the compute-intensive operator to obtain a first operation result;

[0015] The data movement unit is coupled with the first computing core unit and the computing core particle core interconnection interface respectively; in a case where the computing core particle needs to provide the first operation result to the bandwidth core particle, the data movement unit transmits the first operation result to the bandwidth core particle through the core interconnection interface.

[0016] Optionally, the first computing core unit comprises a cache management unit;

[0017] In a case where the computing core particle does not need to provide the first operation result to the bandwidth core particle, the first computing core unit caches the first operation result to the cache management unit.

[0018] Optionally, the first computing core unit further comprises a scheduling unit, a tensor computing unit and a hybrid computing unit;

[0019] The scheduling unit is coupled with the tensor computing unit and the hybrid computing unit respectively; the scheduling unit, in response to receiving a task corresponding to the compute-intensive operator, distributes the computing task to the tensor computing unit and the hybrid computing unit;

[0020] The tensor computing unit is coupled with the scheduling unit, the cache management unit and the data movement unit respectively; the tensor computing unit, in response to receiving a first computing task distributed by the scheduling unit, executes the first computing task;

[0021] The hybrid computing unit is coupled with the scheduling unit, the cache management unit and the data movement unit respectively; the tensor computing unit, in response to receiving a second computing task distributed by the scheduling unit, executes the second computing task.

[0022] Optionally, the bandwidth core particle comprises a second computing unit, a data movement unit and a high-bandwidth memory controller;

[0023] The second computing unit is coupled with the data movement unit and the high-bandwidth memory controller; the second computing unit, in response to receiving a task corresponding to the memory-intensive operator, performs operation on second to-be-processed data in the task corresponding to the memory-intensive operator to obtain a second operation result, and provides the second operation result to the high-bandwidth memory controller;

[0024] The high-bandwidth memory controller is coupled with the second computing unit, the data movement unit and the high-bandwidth memory respectively; the high-bandwidth memory controller stores the second operation result in the high-bandwidth memory in response to receiving the second operation result; and obtains data indicated by a data read command sent by the second computing unit from the high-bandwidth memory and feeds back to the second computing unit in response to receiving the data read command;

[0025] The data movement unit is coupled with the second computing unit, the high-bandwidth memory controller, and the chiplet interconnect interface of the bandwidth chiplet; the data movement unit interacts with the computing chiplet coupled with the bandwidth chiplet through the chiplet interconnect interface.

[0026] Optionally, the second computing core unit includes a cache management unit.

[0027] In a case where the data amount of the second operation result is less than the cache capacity of the cache management unit, the second operation result is cached to the cache management unit; or,

[0028] In a case where the data amount of the second operation result is greater than or equal to the cache capacity of the cache management unit, the second operation result is provided to the high-bandwidth memory controller.

[0029] Optionally, the second computing core unit includes a scheduling unit, a tensor computing unit and a hybrid computing unit.

[0030] The scheduling unit is coupled with the tensor computing unit and the hybrid computing unit respectively; the scheduling unit distributes computing tasks to the tensor computing unit and the hybrid computing unit in response to receiving a task corresponding to the memory-intensive operator.

[0031] The tensor computing unit is coupled with the scheduling unit, the cache management unit and the data movement unit respectively; the tensor computing unit executes a third computing task in response to receiving the third computing task distributed by the scheduling unit.

[0032] The hybrid computing unit is coupled with the scheduling unit, the cache management unit and the data movement unit respectively; the tensor computing unit executes a fourth computing task in response to receiving the fourth computing task distributed by the scheduling unit.

[0033] Optionally, the computing power specification of the second computing core unit is less than the computing power specification of the first computing core unit.

[0034] Optionally, the chip further comprises a storage chip, the storage chip comprises a chip interconnection interface, the storage chip is coupled with the computing chip through the chip interconnection interface; and the storage chip is further coupled with a dynamic random access memory.

[0035] The storage chip stores data received from the computing chip in the dynamic random access memory in response to the data.

[0036] The storage chip reads data indicated by a data read command received from the computing chip from the dynamic random access memory in response to the data read command, and feeds back the data to the computing chip.

[0037] Optionally, the storage chip comprises a data transfer unit and a memory controller.

[0038] The memory controller is coupled with the data transfer unit and the dynamic random access memory respectively, and is configured to write data into the dynamic random access memory or read data from the dynamic random access memory.

[0039] The data transfer unit is coupled with the chip interconnection interface of the storage chip and the memory controller, and is configured to transfer data to the memory controller.

[0040] Optionally, the number of the computing chips is the same as the number of the storage chips, each computing chip is coupled with a different storage chip; or the number of the computing chips is more than the number of the storage chips, part of the computing chips share one storage chip, and the remaining computing chips exclusively use one storage chip; or the number of the computing chips is less than the number of the storage chips, each computing chip is coupled with one or more storage chips.

[0041] Optionally, the storage capacity of the bandwidth chip is less than the storage capacity of the storage chip.

[0042] In a second aspect, an embodiment of the present application provides a data processing method, the method comprising:

[0043] Traversing a computation graph of a to-be-processed deep learning model, the computation graph comprising a plurality of operators;

[0044] Identifying the type of a current operator;

[0045] In a case where the type of the current operator is computation-intensive, assigning the current operator to a computing chip of a chip provided by any embodiment of the present application.

[0046] When the type of the current operator is memory-intensive, the current operator is assigned to the bandwidth chip of the chip provided in any embodiment of the present invention.

[0047] Optionally, the method includes: calculating the storage capacity requirement corresponding to the current operator and calculating the memory access ratio; if the storage capacity requirement is greater than a first threshold, allocating the current operator to a computing chip; or, if the storage capacity requirement is less than or equal to the first threshold and the memory access ratio is greater than a second threshold, allocating the current operator to a computing chip; if the storage capacity requirement is less than or equal to the first threshold and the memory access ratio is less than or equal to the second threshold, allocating the current operator to a bandwidth chip.

[0048] The technical solutions provided by the embodiments of the present invention bring at least the following beneficial effects:

[0049] This invention provides a chip comprising computing cores and bandwidth cores, coupled via a core interconnect interface. The bandwidth core is also directly connected to high-bandwidth memory based on 3D stacking technology. This chip is used to process deep learning algorithms, which include at least computationally intensive operators and memory-intensive operators. Computationally intensive operators are mapped to computing cores, and memory-intensive operators are mapped to bandwidth cores. The computing core executes the task corresponding to the computationally intensive operator, and the bandwidth core executes the task corresponding to the memory-intensive operator. This effectively distributes operators with different computational and bandwidth requirements in deep learning algorithms to different cores within the same chip, thus solving the problem of wasted hardware resources caused by the significant differences in computational and bandwidth requirements of different parts of deep learning algorithms. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0051] Figure 1 A block diagram of a chip according to an embodiment of the present invention is shown;

[0052] Figure 2 A block diagram of a chip provided according to another embodiment of the present invention is shown;

[0053] Figure 3A and Figure 3B The following are schematic diagrams of the chip packaging provided in this embodiment;

[0054] Figure 3C A 3D perspective schematic diagram illustrating the chip packaging method provided in an embodiment of the present invention;

[0055] Figure 4A structural schematic diagram of a computing core provided by an embodiment of the present application is shown;

[0056] Figure 5 A structural schematic diagram of a bandwidth core provided by an embodiment of the present application is shown;

[0057] Figure 6 A structural schematic diagram of a storage core provided by an embodiment of the present application is shown;

[0058] Figure 7 An example of mapping of an LLM Transformer on the chip architecture of the present application is shown;

[0059] Figure 8 A flowchart of a data processing method provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0060] Exemplary embodiments of the present application are described herein with reference to the drawings, in which various details of the embodiments of the present application are set forth in order to provide a thorough understanding of the present application. However, persons of ordinary skill in the art will readily appreciate that the embodiments of the present application can be practiced without such details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0061] The terms "first", "second", and the like in the description and claims of the present application are used for distinguishing between similar objects and are not necessarily used to describe a sequence or order, unless specified otherwise. It is to be understood that the data used with the first, second, etc. can be interchanged, under appropriate circumstances, so that the embodiments of the present application are capable of functioning with either data arrangement, unless specified otherwise. The terms "first", "second", and the like are used to distinguish between objects created by transformation of a single object, and are not necessarily used to describe a sequence or order, unless specified otherwise.

[0062] Figure 1A block diagram of a chip provided by an embodiment of the present application is shown. The chip of the embodiment of the present application is used to process deep learning algorithms. Among deep learning and other machine learning algorithms, operators are the basis for various calculations in model training and inference execution, responsible for performing specific mathematical or logical operation operations. For example, in a neural network, operators can perform convolution, pooling, activation function, etc. operations, which are necessary for building and training an effective model. Specifically, in a neural network, a convolution operator is used to extract features of input data, such as edge and texture information in an image; a pooling operator is used to reduce the dimensionality of data while preserving important features; and an activation function operator is used to introduce non-linear characteristics, so that the neural network can learn and approximate complex non-linear functions. These operators work together to enable the neural network to handle complex tasks such as image recognition, speech recognition, and natural language processing. In addition to operators in neural networks, the field of artificial intelligence involves many other types of operators. For example, in a decision tree algorithm, operators may involve calculating information gain or Gini impurity to determine how to split nodes in the tree. In a support vector machine, operators may involve calculating distances between data points and decision boundaries.

[0063] In an embodiment of the present application, operators in a deep learning algorithm can be classified into, for example, compute-intensive operators and compute-memory-intensive operators (hereinafter referred to as memory-intensive operators) according to their requirements for computing power specifications and bandwidth specifications. Optionally, operators can be identified according to their corresponding storage capacity requirements and compute-memory ratios. If the storage capacity requirement of the current operator is greater than a first threshold, the type of the operator is determined to be compute-intensive. If the storage capacity requirement of the current operator is less than or equal to the first threshold, and the compute-memory ratio is greater than a second threshold, the type of the operator is determined to be compute-intensive. If the storage capacity requirement of the current operator is less than or equal to the first threshold, and the compute-memory ratio is less than or equal to the second threshold, the type of the operator is determined to be memory-intensive.

[0064] Still optionally, referring to Table 1, compute-intensive operators or memory-intensive operators can be identified according to the data volume, computational complexity, and compute-memory ratio of the operators.

[0065] As Figure 1As shown, the chip includes a compute die and a bandwidth die. The compute die and the bandwidth die each include a die-to-die (D2D) interface, and the compute die and the bandwidth die are coupled via the die-to-die interface. Alternatively, the compute die and the bandwidth die each include one or more die-to-die interfaces, and the number of die-to-die interfaces included in the compute die and the bandwidth die can be the same or different, which is not limited in the present application. The bandwidth die is also directly connected to a high-bandwidth memory, for example, the bandwidth die is directly stacked with a 3D-DRAM, or the bandwidth die is directly connected to a high-bandwidth memory (HBM) based on a 3D stacking process, and the high-bandwidth memory is used to achieve high-bandwidth capability.

[0066] In the process of processing deep learning algorithms by the chip, the compute-intensive operators are mapped to the compute die, and the memory-intensive operators are mapped to the bandwidth die. The compute die processes tasks corresponding to the compute-intensive operators, and the bandwidth die processes tasks corresponding to the memory-intensive operators, thereby microscopically splitting the deep learning algorithm from the operator level, and solving the problem of waste of hardware resources caused by different requirements of compute-intensive operators and memory-intensive operators for computing and bandwidth capability.

[0067] Alternatively, the number of compute dies included in the chip can be one or more, and the number of bandwidth dies can also be one or more, and the number of compute dies and the number of bandwidth dies can be flexibly configured according to application requirements. The number of compute dies and the number of bandwidth dies can be the same or different, which is not limited in the present application.

[0068] Figure 2 A block diagram of a chip provided by another embodiment of the present application is shown. In the shown chip, Figure 1 Based on the shown chip, Figure 2 The shown chip further includes a storage die coupled to a dynamic random access memory (DRAM). The storage die is responsible for large-capacity data storage. The storage die is coupled to the compute die via a die-to-die interface, and stores data sent by the compute die. Alternatively, the storage die stores the data sent by the compute die in the dynamic random access memory in response to receiving the data. Alternatively, the storage die reads data indicated by a data read command from the dynamic random access memory in response to receiving the data read command sent by the compute die, and feeds back the data to the compute die.

[0069] In an optional embodiment, the number of computing corelets is the same as the number of storage corelets, and each computing corelet is coupled with a different storage corelet. Optionally, the number of computing corelets is greater than the number of storage corelets, and some of the computing corelets share the same storage corelet, and the remaining computing corelets exclusively use one storage corelet. Optionally, the number of computing corelets is less than the number of storage corelets, and each computing corelet is coupled with one or more storage corelets.

[0070] Figure 3A and Figure 3B respectively show the packaging schematic diagram of the chip provided by the embodiment. Figure 3C The 3D perspective schematic diagram of the packaging mode of the chip provided by the embodiment is shown. In the embodiment, the chip can include one or more computing corelets, one or more bandwidth corelets, and one or more storage corelets. One computing corelet is matched with one or more storage corelets, and the number of computing corelets and the number of bandwidth corelets can be combined according to the sequence, for example, according to the actual business computing and bandwidth demand ratio.

[0071] As shown in Figure 3A , the chip includes one computing corelet, one bandwidth corelet, and two storage corelets (denoted as storage corelet 1 and storage corelet 2, respectively). The computing corelet is coupled with the bandwidth corelet, the storage corelet 1, and the storage corelet 2, respectively. When processing the task corresponding to the computation-intensive operator, the computing corelet can store the computing result or intermediate data into the storage corelet 1 and / or the storage corelet 2.

[0072] As shown in Figure 3B , the chip includes two computing corelets (denoted as computing corelet 1 and computing corelet 2), two bandwidth corelets (denoted as bandwidth corelet 1 and bandwidth corelet 2), and two storage corelets (denoted as storage corelet 3 and storage corelet 4). The computing corelet 1 is coupled with the bandwidth corelet 1, the bandwidth corelet 2, and the storage corelet 3 through different corelet interconnection interfaces, respectively. The computing corelet 2 is coupled with the bandwidth corelet 1, the bandwidth corelet 2, and the storage corelet 4 through different corelet interconnection interfaces, respectively. The bandwidth corelet 1 is coupled with the computing corelet 1 and the computing corelet 2 through different corelet interconnection interfaces, respectively. The bandwidth corelet 2 is coupled with the computing corelet 1 and the computing corelet 2 through different corelet interconnection interfaces, respectively. The storage corelet 3 is coupled with the computing corelet 1, and the storage corelet 4 is coupled with the computing corelet 2.

[0073] In an optional embodiment, if the chip provided by the embodiment of the application is integrated with N computing cores, M storage cores and P bandwidth cores, the M storage cores should be as uniformly matched with the N computing cores as possible, for example, each computing core is loaded with M / N storage cores, and loading the same number of storage cores helps to simplify the task and data partitioning complexity. For the operators allocated to the bandwidth cores, they are further partitioned into P sub-tasks and completed by the P bandwidth cores. Similarly, for the operators allocated to the computing cores, they are further partitioned into N sub-tasks and completed by the N computing cores (and M storage cores). Taking matrix multiplication as an example, a matrix with a dimension of (m, k) is multiplied by a matrix with a dimension of (k, n), and then the task can be partitioned in different dimensions, for example, it can be allocated to three parts (m / 3, k) x (k, n) or (m, k) x (k, n / 3). If the dimension partitioning cannot be divided, one part of the task is allowed to be different from the others. For example, in the case of m = 11, the m dimension is divided into three parts, which can be divided into m = 4 + 4 + 3.

[0074] The embodiment of the application provides a chip using heterogeneous hybrid cores (such as computing cores, bandwidth cores and storage cores), which can effectively solve the problem of waste of hardware resources caused by the large difference in the demand for computing, bandwidth and storage capacity of different parts of deep learning type algorithms. Compared with the prior art which only splits the computing bottleneck and the bandwidth bottleneck from the macro business level of AI, the embodiment of the application can split from the operator level at the micro level of deep learning algorithms, and support balanced and flexible matching of computing, bandwidth and storage capacity according to the actual business characteristics.

[0075] The embodiment of the application proposes three types of core design and flexible combination based thereon, and the interconnection topology of the three types of cores is not limited to the grid form, but can also be ring, star, full connection and the like. In addition, the different types of cores of the application support different number combinations, respectively expand the storage / computing / bandwidth capacity, to meet the differentiated needs of different types of AI models.

[0076] Figure 4 A structure diagram of a computing core provided by an embodiment of the application is shown. As shown in Figure 4 The computing core includes a first computing core unit, a data moving unit and a plurality of core interconnection interfaces.

[0077] The first computing core unit is coupled with the data moving unit. The first computing core unit performs operation on first to-be-processed data in the task corresponding to the compute-intensive operator to obtain a first operation result in response to receiving the task corresponding to the compute-intensive operator.

[0078] The data migration unit is coupled with the first computing core unit and the corelet interconnection interface of the corelet respectively. In the case that the corelet needs to provide the first operation result to the bandwidth corelet, the data migration unit transmits the first operation result to the bandwidth corelet through the corelet interconnection interface.

[0079] Further, as shown in Figure 4 The first computing core unit includes a cache management unit, a scheduling unit, a tensor computing unit and a hybrid computing unit.

[0080] In the case that the corelet does not need to provide the first operation result to the bandwidth corelet, the first computing core unit caches the first operation result to the cache management unit. Since the capacity of the cache management unit is limited, when the data volume exceeds the capacity of the cache management unit, the first computing core unit will transmit the data to the storage corelet through the data migration unit, so that the large-size cache data required by the first computing core unit is accessed to the storage corelet through the corelet interconnection interface by the data migration unit, and the small-size data (such as small-size temporary data) is accessed to the local cache management unit.

[0081] The scheduling unit is coupled with the tensor computing unit and the hybrid computing unit respectively. The scheduling unit distributes the computing task to the tensor computing unit and the hybrid computing unit in response to receiving the task corresponding to the compute-intensive operator.

[0082] The tensor computing unit is coupled with the scheduling unit, the cache management unit and the data migration unit respectively. The tensor computing unit executes the first computing task in response to receiving the first computing task distributed by the scheduling unit. Optionally, the tensor computing unit can be responsible for matrix multiplication, convolution operation and the like.

[0083] The tensor is a mathematical object that can represent data in multiple dimensions. In deep learning, the tensor can be used to represent data and parameters.

[0084] The hybrid computing unit is coupled with the scheduling unit, the cache management unit and the data migration unit respectively. The hybrid computing unit executes the second computing task in response to receiving the second computing task distributed by the scheduling unit. The hybrid computing unit can be responsible for the operation of scalar, vector or special function.

[0085] In the embodiment, the scheduling unit is responsible for the task distribution of various computing units, so as to complete different computing functions such as scalar, vector, tensor and special function.

[0086] Figure 5 The structure of the bandwidth corelet of the embodiment is shown. As shown in Figure 5 The bandwidth corelet includes a second computing core unit, a data migration unit, a high-bandwidth memory controller and one or more corelet interconnection interfaces.

[0087] The second computing unit is coupled with the data moving unit, the high-bandwidth memory controller. The second computing unit performs operation on the second to-be-processed data in the task corresponding to the memory-intensive operator to obtain a second operation result, and provides the second operation result to the high-bandwidth memory controller in response to receiving the task corresponding to the memory-intensive operator.

[0088] The high-bandwidth memory controller is coupled with the second computing unit, the data moving unit and the high-bandwidth memory respectively. The high-bandwidth memory controller stores the second operation result in the high-bandwidth memory in response to receiving the second operation result, and acquires data indicated by a data reading command sent by the second computing unit from the high-bandwidth memory and feeds back to the second computing unit in response to receiving the data reading command.

[0089] The data moving unit is coupled with the second computing unit, the high-bandwidth memory controller and the chiplet interconnection interface of the bandwidth chiplet. The data moving unit interacts with the computing chiplet of the bandwidth chiplet through the chiplet interconnection interface.

[0090] Further, as shown in Figure 4 The second computing core unit includes a cache management unit, a scheduling unit, a tensor computing unit and a hybrid computing unit.

[0091] In a case where the data amount of the second operation result is less than the cache capacity of the cache management unit, the second operation result is cached to the cache management unit; or in a case where the data amount of the second operation result is greater than or equal to the cache capacity of the cache management unit, the second operation result is provided to the high-bandwidth memory controller.

[0092] The scheduling unit is coupled with the tensor computing unit and the hybrid computing unit respectively. The scheduling unit distributes computing tasks to the tensor computing unit and the hybrid computing unit in response to receiving the task corresponding to the memory-intensive operator.

[0093] The tensor computing unit is coupled with the scheduling unit, the cache management unit and the data moving unit respectively. The tensor computing unit executes a third computing task in response to receiving the third computing task distributed by the scheduling unit.

[0094] The hybrid computing unit is coupled with the scheduling unit, the cache management unit and the data moving unit respectively. The tensor computing unit executes a fourth computing task in response to receiving the fourth computing task distributed by the scheduling unit.

[0095] Figure 6 A structure diagram of a storage chiplet is shown. As shown in Figure 6 The storage chiplet includes a data moving unit, a memory controller and one or more chiplet interconnection interfaces.

[0096] The memory controller is coupled with the data movement unit and the dynamic random access memory, respectively. The memory controller is configured to write data to the dynamic random access memory or read data from the dynamic random access memory.

[0097] The data movement unit is coupled with the chiplet interconnect interface of the storage chiplet and the memory controller. The data movement unit is configured to move data to the memory controller.

[0098] In an optional embodiment, the storage capacity of the bandwidth chiplet is less than the storage capacity of the storage chiplet, and the computing power specification of the bandwidth chiplet is less than the computing power specification of the computing chiplet, i.e., the computing power specification of the second computing core unit is less than the computing power specification of the first computing core unit.

[0099] Figure 7 An example of mapping of an LLM Transformer on the chip architecture of the present application is shown. Different algorithm parts (single operator, or a computing subgraph composed of multiple operators) have different requirements for computing power, storage capacity, and bandwidth capacity. Through the compiler software, different algorithm parts can be assigned and distributed to different chiplet computations. Transformer is a kind of deep learning algorithm, and is the main basic algorithm structure of large model services. A typical Transformer consists of two parts, an encoder and a decoder. The encoder and the decoder are respectively composed of multiple repeated computing subgraph blocks cascaded, and the computing subgraph block includes a multi-head attention unit (Multi-head Attention), a normalization unit, and a feedforward network layer (FeedForward). The function of the encoder is to convert the input (text, speech, etc. sequence) into an intermediate representation (hidden space), and the function of the decoder is to generate the output result from the intermediate representation. According to different algorithm characteristics, only the encoder (such as BERT, used for text classification, semantic analysis, information extraction, etc.) can be used, or only the decoder (such as GPT-2 / GPT-3 / Llama, autoregressive generation) can be used, or both the encoder and the decoder (such as T5, translation task) can be used. From the perspective of computation, the encoder and the decoder are built by the same type of operators, and the main operators include matrix multiplication (GEMM), tensor concatenation (Concat), normalization exponential function (Softmax), normalization (Norm), tensor addition (Elementwise Add), and nonlinear (Non-linear).

[0100] In the example shown in FIG. 6, the bandwidth chiplet 600 is used to provide the data transfer between the storage chiplet 500 and the memory controller 400. The bandwidth chiplet 600 is configured to provide the data transfer between the storage chiplet 500 and the memory controller 400. Figure 7For example, firstly, in the pre-filling stage, the multi-head attention unit (MOF) has high requirements for computational power and intermediate result storage capacity, so it is allocated to computational kernels (with storage kernels serving as storage extensions of the MOF). The feedforward network layer and related element operation layers have high bandwidth requirements, so they are allocated to bandwidth kernels. Computational kernels and bandwidth kernels interact through a kernel-to-kernel (D2D) interface, with their respective data transfer units responsible for coordinated scheduling. Secondly, in the decoding stage, the storage requirements of the MOF increase rapidly with the context, and the entire process is almost entirely a bandwidth bottleneck. To reduce the additional bandwidth overhead caused by intermediate caching, the MOF can still be allocated to computational kernels (with storage kernels serving as storage extensions of the MOF), while the remaining parts can still be allocated to bandwidth kernels.

[0101] Figure 7 This invention demonstrates mapping methods for different operators using the LLM Transformer as an example, but it is also applicable to deploying other deep learning algorithms. Compared to traditional methods, this invention offers higher hardware utilization and lower hardware costs.

[0102] by Figure 7For example, first, compared with the traditional method of only separating at the macro level according to the calculation and memory access characteristics of the pre-filling stage and the decoding stage, and assigning the two stages to different computing devices, the present application supports separating the micro components within the two stages to achieve the best match of different local business characteristics and hardware capabilities. Second, the pre-filling and decoding stages are actually the same algorithm model and the same set of weight data, but in the traditional separation method, the pre-filling computing device and the decoding computing device each need to store a set of weight data locally. Considering that the weight size of a large model is huge, a large storage waste will be formed here. Third, the pre-filling stage will generate a large intermediate cache result, which is one of the necessary inputs for the decoding stage calculation and increases linearly with the length of the context sequence, and even much higher than the weight size of a large model. Therefore, in the traditional two-stage separation method, a large network transmission cost is required to transfer this part of data from the pre-filling computing device to the decoding computing device. In the present application, since the calculations of the multi-head attention units in the two stages can be mapped to the same computing core for completion, there is no transmission of this part, and the relevant results are directly stored locally on the chip. Fourth, the actual business is generally more complex than the standard process described above, such as fusion models of non-Transformer structure and Transformer structure, mixed models of multiple AI models, and more complex resource differences of different operators. Since the present application maps and matches the operators (or calculation subgraphs) as the basic unit to different resource type cores, compared with the traditional single chip or algorithm node network division method, the hardware utilization rate can be significantly improved, and it is also easier to customize the core combination chip for the resource characteristics of a specific business.

[0103] Figure 8 A flowchart of a data processing method provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the method comprises the following steps. Figure 8

[0104] Step S801: traversing a calculation graph of a to-be-processed deep learning model, the calculation graph comprising a plurality of operators.

[0105] Step S802: identifying the type of a current operator. For example, calculating the storage capacity requirement and the calculation memory access ratio of the current operator. Wherein, referring to Table 1, the storage capacity requirement and the calculation memory access ratio of the current operator can be determined according to the type and configuration of the current operator.

[0106] Step S803: determining whether the storage capacity requirement is greater than a first threshold. If yes, executing step S804, and if no, executing step S805.

[0107] Step S804: assigning the current operator to a computing core.

[0108] ​Step S805: judging whether the calculated access ratio is greater than a second threshold. If yes, step S804 is executed, and if no, step S806 is executed.

[0109] Step S806: assigning the current operator to the bandwidth corelet.

[0110] Step S807: judging whether the current operator is the last operator. If yes, the flow ends, and if no, the flow returns to step S801.

[0111] In the chip provided by the embodiment of the present application, the computing corelet executes the task corresponding to the compute-intensive operator, and the bandwidth corelet executes the task corresponding to the memory access-intensive operator, so that the operators with different requirements for computing power specifications and bandwidth specifications in the deep learning algorithm are distributed to different corelets in the same chip, and the problem of waste of hardware resources caused by the great difference in requirements for computing and bandwidth capabilities of different parts of the deep learning type algorithm is effectively solved.

[0112] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0113] Each embodiment in the specification is described in a related manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0114] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application is included in the protection scope of the present application.

Claims

1. A chip, characterized in that, The chip is used to process deep learning algorithms, which include at least computationally intensive operators and memory-intensive operators; The chip includes at least a computing chip and a bandwidth chip, both of which include a chip interconnect interface. The computing chip and the bandwidth chip are coupled through the chip interconnect interface. The bandwidth chip is also directly connected to a high-bandwidth memory. The compute-intensive operator is mapped to the compute core, and the memory-intensive operator is mapped to the bandwidth core; The computing kernel is used to execute the task corresponding to the computing-intensive operator, and the bandwidth kernel is used to execute the task corresponding to the memory-intensive operator.

2. The chip according to claim 1, characterized in that, The computing chip also includes a first computing core unit and a data transfer unit; The first computing core unit is coupled to the data transfer unit; in response to receiving the task corresponding to the computing-intensive operator, the first computing core unit performs calculations on the first data to be processed in the task corresponding to the computing-intensive operator to obtain a first calculation result. The data transfer unit is coupled to the first computing core unit and the core interconnect interface of the computing core respectively; when the computing core needs to provide the first calculation result to the bandwidth core, the data transfer unit transmits the first calculation result to the bandwidth core through the core interconnect interface.

3. The chip according to claim 2, characterized in that, The first computing core unit includes a cache management unit; If the computing core does not need to provide the first calculation result to the bandwidth core, the first computing core unit caches the first calculation result to the cache management unit.

4. The chip according to claim 3, characterized in that, The first computing core unit also includes: a scheduling unit, a tensor computing unit, and a hybrid computing unit; The scheduling unit is coupled to the tensor computation unit and the hybrid computation unit respectively; in response to receiving the task corresponding to the computationally intensive operator, the scheduling unit distributes the computation task to the tensor computation unit and the hybrid computation unit. The tensor computation unit is coupled to the scheduling unit, the cache management unit, and the data transfer unit respectively; the tensor computation unit executes the first computation task in response to receiving the first computation task distributed by the scheduling unit. The hybrid computing unit is coupled to the scheduling unit, the cache management unit, and the data transfer unit respectively; the tensor computing unit executes the second computing task in response to receiving the second computing task distributed by the scheduling unit.

5. The chip according to any one of claims 1-4, characterized in that, The bandwidth core includes: a second computing unit, a data transfer unit, and a high-bandwidth memory controller; The second computing unit is coupled to the data transfer unit and the high-bandwidth memory controller. In response to receiving the task corresponding to the memory-intensive operator, the second computing unit performs calculations on the second data to be processed in the task corresponding to the memory-intensive operator to obtain a second calculation result, and provides the second calculation result to the high-bandwidth memory controller. The high-bandwidth memory controller is coupled to the second computing unit, the data transfer unit, and the high-bandwidth memory respectively; in response to receiving the second calculation result, the high-bandwidth memory controller stores the second calculation result in the high-bandwidth memory; and in response to receiving a data read command sent by the second computing unit, the high-bandwidth memory controller retrieves the data indicated by the data read command from the high-bandwidth memory and feeds it back to the second computing unit. The data transfer unit is coupled to the second computing unit, the high-bandwidth memory controller, and the core interconnect interface of the bandwidth core; the data transfer unit interacts with the computing core coupled to the bandwidth core through the core interconnect interface.

6. The chip according to claim 5, characterized in that, The second computing core unit includes a cache management unit; If the amount of data in the second calculation result is less than the cache capacity of the cache management unit, the second calculation result is cached in the cache management unit. or, If the amount of data in the second calculation result is greater than or equal to the cache capacity of the cache management unit, the second calculation result is provided to the high-bandwidth memory controller.

7. The chip according to claim 6, characterized in that, The second core computing unit includes: a scheduling unit, a tensor computing unit, and a hybrid computing unit; The scheduling unit is coupled to the tensor computing unit and the hybrid computing unit respectively; in response to receiving the task corresponding to the memory-intensive operator, the scheduling unit distributes the computing task to the tensor computing unit and the hybrid computing unit. The tensor computation unit is coupled to the scheduling unit, the cache management unit, and the data transfer unit respectively; the tensor computation unit executes the third computation task in response to receiving the third computation task distributed by the scheduling unit. The hybrid computing unit is coupled to the scheduling unit, the cache management unit, and the data transfer unit respectively; the tensor computing unit executes the fourth computing task in response to receiving the fourth computing task distributed by the scheduling unit.

8. The chip according to any one of claims 5-7, characterized in that, The computing power of the second computing core unit is less than that of the first computing core unit.

9. The chip according to any one of claims 1-8, characterized in that, The chip also includes a storage chip, which includes a chip interconnect interface and is coupled to the computing chip through the chip interconnect interface; the storage chip is also coupled to a dynamic random access memory. In response to receiving data sent by the computing core, the storage core stores the data in the dynamic random access memory; or, In response to receiving a data read command from the computing chip, the storage chip reads the data indicated by the data read command from the dynamic random access memory and feeds it back to the computing chip.

10. The chip according to claim 9, characterized in that, The storage chip includes a data transfer unit and a memory controller; The memory controller is coupled to the data transfer unit and the dynamic random access memory respectively. The memory controller is used to write data to the dynamic random access memory or read data from the dynamic random access memory. The data transfer unit is coupled to the chip interconnect interface of the storage chip and the memory controller, and the data transfer unit is used to transfer data to the memory controller.

11. The chip according to claim 9, characterized in that, The number of computing cores is the same as the number of storage cores, and each computing core is coupled to a different storage core. or, The number of computing cores exceeds the number of storage cores. Some computing cores share the same storage core, while the remaining computing cores occupy a storage core exclusively. or, The number of compute cores is less than the number of storage cores, and each compute core is coupled to one or more storage cores.

12. The chip according to claim 9, characterized in that, The storage capacity of the bandwidth core is less than the storage capacity of the storage core.

13. A data processing method, characterized in that, include: Traverse the computation graph of the deep learning model to be processed, wherein the computation graph includes multiple operators; For the current operator, identify the type of the current operator; If the type of the current operator is computationally intensive, the current operator is assigned to the computing chip of the chip as described in any one of claims 1-12; If the current operator is of the memory-intensive type, the current operator is assigned to the bandwidth chip of the chip as described in any one of claims 1-12.

14. The method according to claim 13, characterized in that, The method includes: Calculate the storage capacity requirement corresponding to the current operator and calculate the memory access ratio; If the storage capacity requirement exceeds a first threshold, the current operator is allocated to a computing chip; or, If the storage capacity requirement is less than or equal to a first threshold and the computation-to-memory ratio is greater than a second threshold, the current operator is assigned to a computational chip. If the storage capacity requirement is less than or equal to a first threshold and the calculated memory access ratio is less than or equal to a second threshold, the current operator is assigned to a bandwidth core.

Images