Computing devices and computing methods

Abstract

The application provides a computing device and a computing method. The computing device comprises a plurality of cores and a central unit, the plurality of cores and the central unit are connected in a star architecture, the central unit is located at the center of the star architecture and is connected in communication with the plurality of cores, and the plurality of cores interact through the central unit; when performing a computing task, the computing task is split into a plurality of subtasks according to a preset strategy, the plurality of cores are respectively used for executing the plurality of subtasks and generating corresponding intermediate operation results; and the plurality of cores respectively send the generated intermediate operation results to the central unit, so that the central unit obtains an operation result of the computing task according to the intermediate operation results of the plurality of cores. The computing device and the computing method provided by the application can improve the operation speed and reduce the operation delay.

Description

Technical Field

[0001] This application belongs to the field of chip design technology and relates to a computing device and computing method. Background Technology

[0002] As the demand for computing power and memory bandwidth from AI (Artificial Intelligence) applications continues to grow rapidly, AI accelerator designs are gradually evolving towards multi-die architectures. In a multi-die architecture, dies can be interconnected via high-speed bandwidth. However, as the number of dies increases, the interconnect bandwidth between them also needs to increase. This means that in existing multi-die AI accelerators, the interconnect bandwidth becomes a bottleneck restricting computational performance. Summary of the Invention

[0003] This application provides a computing device and computing method to improve the computing speed of multi-chip architecture and reduce computing latency.

[0004] In a first aspect, embodiments of this application provide a computing device. The computing device includes multiple chips and a central unit. The multiple chips and the central unit are connected in a star topology. The central unit is located at the center of the star topology and is communicatively connected to the multiple chips. The multiple chips interact with each other through the central unit. When executing a computing task, the computing task is divided into multiple subtasks according to a preset strategy. The multiple chips are respectively used to execute the multiple subtasks and generate corresponding intermediate computation results. The multiple chips respectively send the generated intermediate computation results to the central unit, so that the central unit can obtain the computation result of the computing task based on the intermediate computation results of the multiple chips.

[0005] In some implementations, obtaining the computation result of the computing task by the central unit based on the intermediate computation results of the plurality of dies includes: the central unit merging the intermediate computation results of the plurality of dies to obtain the computation result of the computing task.

[0006] In some implementations, obtaining the computation result of the computing task by the central unit based on the intermediate computation results of the plurality of dies includes: corresponding to each die, the central unit sends the intermediate computation result of the die to its corresponding target die, the target die completes the merging of some intermediate computation results, and sends the merged result to the remaining dies through the central unit, so that all dies obtain the computation result of the computing task based on the received merged result.

[0007] In some implementations, the central unit is used to coordinate data transmission between the plurality of dies.

[0008] In some implementations, the plurality of dies send the intermediate computation results to the central unit in parallel.

[0009] In some implementations, the computational task includes computational tasks of AI operators, which include embedding operators and / or matrix multiplication operators.

[0010] In some implementations, the method of splitting the computation task of the embedding operator includes: splitting the embedding table of the embedding operator into multiple sub-embedded tables by row, storing the multiple sub-embedded tables by the multiple chips respectively, and each sub-embedded table retaining the complete index of the corresponding embedding table; the multiple chips respectively execute the multiple sub-tasks and generate corresponding intermediate computation results, including: when performing a query operation, the multiple chips respectively query the locally stored sub-embedded tables according to the complete index and generate query results.

[0011] In some implementations, the method of splitting the computation task of the matrix multiplication operator includes: splitting the weight matrix in the matrix multiplication operator into multiple weight parts along the column direction; or, splitting the weight matrix in the matrix multiplication operator into multiple weight parts along the row direction, and correspondingly splitting the input matrix in the matrix multiplication operator into multiple input parts.

[0012] Secondly, embodiments of this application provide a calculation method. The calculation method is applied to a computing device, which includes multiple dies and a central unit. The multiple dies and the central unit are connected in a star topology. The central unit is located at the center of the star topology and is communicatively connected to the multiple dies. The multiple dies interact with each other through the central unit. The calculation method includes: when executing a calculation task, splitting the calculation task into multiple subtasks according to a preset strategy; executing the multiple subtasks through the multiple dies respectively, and generating corresponding intermediate calculation results; and obtaining the calculation result of the calculation task through the central unit based on the intermediate calculation results of the multiple dies.

[0013] As described above, embodiments of this application provide a computing device and a computing method. The computing device includes multiple chips and a central unit, with the chips and central unit connected in a star topology. When performing computational tasks, the computational task can be divided into multiple subtasks, with each chip executing one or more subtasks and generating corresponding intermediate computation results. The intermediate computation results from each chip are then combined to obtain the final computation result. This approach helps improve computational speed and reduce computational latency.

[0014] Furthermore, in this embodiment, the central unit can be a device with computing capabilities, used to merge the intermediate computation results of each die to obtain the final computation result. This method eliminates the need for transmitting intermediate computation results between dies; instead, the merging is performed by the central unit located at the topology center, which helps to further reduce latency. Attached Figure Description

[0015] Figure 1 The diagram shown is a schematic representation of a computing device provided in an embodiment of this application.

[0016] Figure 2A and Figure 2B This is a schematic diagram illustrating the merging of intermediate calculation results in some implementation methods.

[0017] Figure 3 This is a schematic diagram illustrating the merging of intermediate calculation results in some implementation methods.

[0018] Figures 4 to 7 The diagram shown is a schematic representation of operator splitting in an embodiment of this application. Detailed Implementation

[0019] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0020] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the illustrations only show the components related to this application and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0021] To meet the ever-increasing demands for computing power and memory bandwidth, chip design is increasingly moving towards multi-die packaging architectures. For example, some solutions package two dies together, interconnected via high-speed bandwidth. Dies can communicate via NV-HBI (NV-High Bandwidth Interface), which provides 10TB / s of bidirectional bandwidth, allowing the two dies to operate as a unified CUDA (Compute Unified Device Architecture) GPU (Graphics Processing Unit). However, as the memory bandwidth of a single die increases, this approach requires a corresponding increase in the interconnect bandwidth between dies, resulting in higher cost and power consumption.

[0022] As more and more AI accelerators adopt multi-chip architecture, the interconnect bandwidth between chips is gradually becoming one of the important factors restricting the development of AI accelerators.

[0023] In response to at least the above-mentioned problems, embodiments of this application provide a computing device and a computing method. Figure 1 The diagram shown is an architectural diagram of a computing device provided in an embodiment of this application. Figure 1 As shown, the computing device includes N dies (Die0 to DieN-1) and a central unit, where N is a positive integer greater than 1. The N dies and the central unit are connected in a star topology, with the central unit located at the center of the star topology and communicating with each die. The dies can interact with each other through the central unit.

[0024] For example, data can be transmitted between the die and the central unit via a high-speed interconnect to ensure efficient collaborative computing under low latency and high bandwidth conditions.

[0025] In this embodiment, the die can serve as a computing unit, on which a computing core, on-chip cache, and high-speed interface can be integrated. Different dies do not communicate directly; instead, data exchange is achieved through a central unit. This approach avoids complex interconnections between dies and simplifies data scheduling. During computation, the computational task to be performed can be divided into multiple subtasks according to a preset strategy. These subtasks can be assigned to different dies for processing to generate corresponding intermediate computation results. Each die can execute its corresponding subtask in parallel and send its generated intermediate computation results to the central unit, which then obtains the computation result of the computational task based on the intermediate computation results from multiple dies.

[0026] In some implementations, the central unit is used to coordinate data transmission between the dies. For example, corresponding to each die, the central unit can send the intermediate calculation results of the die to its corresponding target die, which then merges some of the intermediate calculation results and sends the merged result to the remaining dies through the central unit, so that all dies can obtain the calculation result of the computation task based on the received merged result.

[0027] Taking N=4 as an example, Figure 2A and Figure 2B This diagram illustrates the merging of intermediate calculation results from Die0 to Die3. (Example) Figure 2A As shown, each die undergoes a merging of intermediate calculation results. Specifically, the intermediate results from the Die0 operation are a0 to d0, the intermediate results from the Die1 operation are a1 to d1, and so on. After the calculation is complete, several rounds of merging are required to obtain the final result. The specific process is as follows:

[0028] In the first round, Die0 transmits a0 to Die1 via the central unit (Die1 is the target die corresponding to Die0 at this time), and the merging of a0 and a1 is completed at Die1. Die1 transmits b1 to Die2 via the central unit (Die2 is the target die corresponding to Die1 at this time), and the merging of b1 and b2 is completed at Die2. Die2 transmits c2 to Die3 via the central unit (Die3 is the target die corresponding to Die2 at this time), and the merging of c2 and c3 is completed at Die3. Die3 transmits d3 to Die0 via the central unit (Die0 is the target die corresponding to Die3 at this time), and the merging of d0 and d3 is completed at Die0. The merging process in subsequent rounds is similar to that in the first round; see details below. Figure 2A As shown, it will not be repeated here. After the last round of merging, Die0 can obtain the merged results of b0 to b3, Die1 can obtain the merged results of c0 to c3, Die2 can obtain the merged results of d0 to d3, and Die3 can obtain the merged results of a0 to a3.

[0029] In the above process, the amount of data sent and received by each die is 3 / 4*data, where data is the total length of the intermediate calculation results obtained by each die.

[0030] like Figure 2B As shown, after the merging is complete, each die sends the merged result to the other dies through the central unit to ensure that all dies can obtain the final calculation result. During this process, the amount of data sent and received by each die is 3 / 4 * data.

[0031] In summary, when adopting Figure 2A and Figure 2B In the method shown, the amount of data sent and received by each die is 3 / 4*data + 3 / 4*data = 3 / 2data.

[0032] In some implementations, the central unit can merge the intermediate calculation results of multiple dies to obtain the final calculation result.

[0033] Taking N=4 as an example, Figure 3 This diagram illustrates how intermediate computation results from multiple dies are merged via a central unit in these embodiments. For example... Figure 3 As shown, each die sends its intermediate computation results to the central unit. The central unit merges the intermediate computation results from all dies and then sends the merged result back to each die. During this process, each die sends and receives 1 data unit. Compared to... Figure 2A and Figure 2B As shown, this method reduces the amount of data sent and received by the chip, and the corresponding latency is also shortened.

[0034] In some implementations, the host computer breaks down the computation task into multiple subtasks according to a preset strategy, sends each subtask to a separate die, and sends a merging task to the central unit. After each die completes its corresponding subtask, it sends the intermediate computation results to the central unit, which then executes the merging task to combine the intermediate computation results to obtain the final computation result.

[0035] In some implementations, the host sends computation tasks and merging tasks to the central unit. The central unit breaks down the computation tasks into multiple subtasks according to a preset strategy and sends each subtask to its corresponding die. After each die completes its subtask, it sends the intermediate computation results to the central unit, which then performs a merging task to combine the intermediate computation results to obtain the final computation result.

[0036] In some implementations, each die can send intermediate computation results to the central unit in parallel.

[0037] In some implementations, each subtask includes a task ID (Identification Number). Subtasks corresponding to the same computation task have the same task ID. The central unit can distinguish the computation task corresponding to the intermediate computation results sent by each die based on the task ID, thereby ensuring the accuracy of the merged result.

[0038] Tables 1 and 2 show the following: Figure 2A and Figure 2B The merging methods shown (method 1 in the table) and Figure 3The table shows a performance comparison of the merging methods (method 2 is shown in the table), where bw_per_line is the interconnect bandwidth between the central unit and the die.

[0039] Table 1. Performance Comparison of Different Merging Methods

[0040]

[0041] Table 2. Performance Comparison of Different Merging Methods

[0042]

[0043]

[0044] According to Tables 1 and 2, compared to Figure 2A and Figure 2B Regarding the merging method shown, Figure 3 The merging method shown can reduce latency by nearly half.

[0045] In some implementations, the computational task includes the computational task of AI operators, which include embedding operators and / or MatMul operators.

[0046] Figure 4 This is an example diagram showing the breakdown of the computational task for the Embedding operator. For example... Figure 4 As shown, the embedding table can be split into multiple sub-embedded tables by row, and each die stores these sub-embedded tables (e.g., embedding table 0, embedding table 1), with each sub-embedded table retaining a complete index of its corresponding embedding table. When performing a query, each die only needs to query the locally stored sub-embedded table using the complete index to obtain the query result; cross-die queries are not involved. For indexes that are not found during the query, the result is set to zero. Finally, the query results from all dies are summed to obtain the final calculation result.

[0047] Figure 5 This is an example diagram showing the breakdown of the computational task for the MatMul operator. (Example:) Figure 5 As shown, there are two ways to decompose the computation task of the MatMul operator. One is to split the weight matrix of the MatMul operator along the column direction, with each die calculating a portion of the weights. After computation, the intermediate results from each die are concatenated to obtain the final result. The second method is to split the weight matrix of the MatMul operator along the row direction, and simultaneously split the input matrix accordingly. The intermediate result of each die is a partial sum of the final result; these partial sums are then accumulated to obtain the final result.

[0048] Figure 6 and Figure 7 This is shown as a case where the two splitting methods corresponding to the MatMul operator are combined in an LLM (Large Language Model) based on Transformer. The Transformer layer includes Self-Attention (...). Figure 6 As shown) and MLP (Multilayer Perceptron) Figure 7 As shown, there are two parts. In the first matrix multiplication operation, the weight matrix (W) is multiplied... Q W K W V The input matrix X is split along the column direction, while f remains unchanged. Here, f is the identity operator. In the next matrix multiplication operation, the weight matrix B is split along the row direction, and correspondingly, the input matrix Y is also split accordingly. Here, g is the AllReduce operation. After computation, a single inter-die AllReduce operation is introduced. AllReduce is a communication operation in distributed computing used to aggregate (reduce) data across multiple dies and broadcast the results. In this application, the AllReduce operation is used to aggregate the partial sums of all dies to obtain the complete output matrix Z (i.e., the final computation result). In this case, the central unit can execute the AllReduce operation, completing data synchronization through efficient low-latency communication mechanisms (such as NVLink of the GPU or high-speed interconnect of the CPU). Since the AllReduce operation only needs to be executed once after computation, it has low complexity.

[0049] In summary, this application provides a computing device and a computing method. The computing device includes multiple chips and a central unit, with the chips and central unit connected in a star topology. When performing computational tasks, the task can be divided into multiple subtasks, with each chip executing one or more subtasks and generating corresponding intermediate computation results. The intermediate computation results from each chip are merged to obtain the final computation result. This approach improves computational speed and reduces computational latency. Furthermore, in this embodiment, the central unit can be a device with computational capabilities, used to merge the intermediate computation results from each chip to obtain the final computation result. This method eliminates the need for inter-chip transmission of intermediate computation results for merging; instead, the merging is performed by the central unit located at the center of the topology, further reducing latency.

[0050] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A computing device, characterized in that, The computing device includes multiple chips and a central unit. The multiple chips and the central unit are connected in a star topology. The central unit is located at the center of the star topology and is communicatively connected to the multiple chips. The multiple chips interact with each other through the central unit. When performing a computation task, the computation task is divided into multiple sub-tasks according to a preset strategy. The multiple chips are used to execute the multiple sub-tasks respectively and generate corresponding intermediate calculation results. The multiple dies each send the generated intermediate calculation results to the central unit, so that the central unit can obtain the calculation result of the calculation task based on the intermediate calculation results of the multiple dies.

2. The computing device according to claim 1, characterized in that, The calculation results of the computational task obtained by the central unit based on the intermediate calculation results of the multiple dies include: The central unit merges the intermediate calculation results of the multiple dies to obtain the calculation result of the calculation task.

3. The computing device according to claim 1, characterized in that, The calculation results of the computational task obtained by the central unit based on the intermediate calculation results of the multiple dies include: For each die, the central unit sends the intermediate calculation results of the die to its corresponding target die, which then merges some of the intermediate calculation results and sends the merged results to the remaining dies through the central unit, so that all dies can obtain the calculation results of the computing task based on the received merged results.

4. The computing device according to claim 3, characterized in that, The central unit is used to coordinate data transmission between the multiple dies.

5. The computing device according to claim 2, characterized in that, The multiple dies send the intermediate calculation results to the central unit in parallel.

6. The computing device according to claim 1, characterized in that, The computational task includes computational tasks for AI operators, which include embedding operators and / or matrix multiplication operators.

7. The computing device according to claim 6, characterized in that, The method for splitting the computation task of the embedding operator includes: splitting the embedding table of the embedding operator into multiple sub-embedded tables by row, storing the multiple sub-embedded tables by the multiple chips respectively, and each sub-embedded table retaining the complete index of the corresponding embedding table; The multiple chips execute the multiple subtasks respectively and generate corresponding intermediate calculation results, including: when performing a query operation, the multiple chips query the locally stored sub-embedded tables according to the complete index and generate query results.

8. The computing device according to claim 6, characterized in that, The method for splitting the computation task of the matrix multiplication operator includes: splitting the weight matrix in the matrix multiplication operator into multiple weight parts along the column direction; or, splitting the weight matrix in the matrix multiplication operator into multiple weight parts along the row direction, and correspondingly splitting the input matrix in the matrix multiplication operator into multiple input parts.

9. A calculation method, characterized in that, The invention is applied to a computing device, which includes multiple dies and a central unit. The multiple dies and the central unit are connected in a star topology. The central unit is located at the center of the star topology and is communicatively connected to the multiple dies. The multiple dies interact with each other through the central unit. The calculation method includes: When performing a computation task, the computation task is divided into multiple sub-tasks according to a preset strategy; The multiple sub-tasks are executed by the multiple dies respectively, and corresponding intermediate calculation results are generated; The central unit obtains the computation result of the computation task based on the intermediate computation results of the multiple dies.

Images