A SoC cluster and distributed computing method
Through SoC clustering and distributed computing methods, using Ethernet and controllers to optimize communication between SoCs, the problem of low data communication between GPUs is solved, efficient distributed computing and task decomposition is realized, and large-scale neural network models are improved. Operational efficiency on multiple SoCs.
Patent Information
- Application Number
- CN202310770466.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-27
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-06-27
AI Technical Summary
In the prior art, data communication between GPUs is inefficient, which cannot effectively solve the data communication requirements of large neural network models when they are distributed on multiple GPUs, resulting in limited computing efficiency.
Using SoC cluster and distributed computing methods, point-to-point and collective communication is realized through Ethernet and controllers in SoC clusters, shared memory and network flow algorithms are used to optimize task decomposition and traffic control, and communication efficiency between SoCs is improved.
It realizes efficient distributed computing, solves the problem that a single SoC cannot run large-scale computing tasks, simplifies the structure of distributed computing systems, and improves the efficiency and communication speed of computing tasks.
Smart Images

Figure CN116795769B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of distributed computing technology, and in particular to a SoC cluster and a distributed computing method. Background Art
[0002] In recent years, neural networks, especially LLM (large language model), have developed rapidly. The parameters of the model have exceeded 100 billion and will continue to grow, which will bring huge demand for computing power. At present, the training and reasoning of neural networks mainly rely on GPU (Graphics Processing Unit) or NPU (Neural network Processing Unit). The current maximum computing power of a single GPU is about 100TFLOPS and the memory is about 100G. LLM often requires EFLOPS-level computing power and 1T-level memory space during training. A single GPU cannot complete such a huge computing task alone, so it is necessary to adopt parallel computing methods and use multiple GPUs for calculation, but this requires massive data communication between GPUs. For example, when training a neural network, during the forward propagation process, it is necessary to synchronize the intermediate calculation results between GPUs, and during the reverse propagation process, it is even more necessary to synchronize all model parameters between GPUs. When reasoning about a neural network, it is also necessary to synchronize the calculation results between GPUs. Because the amount of data that needs to be transmitted is very large, the communication bandwidth becomes a bottleneck, affecting the efficiency of parallel computing. This requires solving two problems: how to improve the efficiency of data communication between GPUs and how to split the neural network model so that it can be distributed to multiple GPUs for running, minimizing the data communication requirements between them and maximizing the computing speed.
[0003] Existing AI servers are usually based on traditional PC architecture, where the CPU communicates with GPU, NIC (Network Interface Card) and other devices through the PCIE (Peripheral component Interconnect Express) bus. The CPU and GPU each have their own memory. When a single GPU cannot handle large computing tasks, multiple GPUs are installed on the server, and a large amount of data needs to be transmitted between GPUs. In order to accelerate GPU-GPU communication, NVIDIA uses NVLINK and NVSWITCH technologies to establish direct channels between GPUs.
[0004] The disadvantage of this method is that the collective communication between GPUs is inefficient. For example, after a GPU updates its weight parameters, it needs to synchronize them to all other GPUs. Since NVSWITCH is a point-to-point connection technology, it needs to send these parameters to all other GPUs separately, repeating several times. If optimized through ring network technology, that is, the GPU only sends one copy of the data to the next GPU on the ring network, and this GPU then relays it to the next GPU, the communication delay will increase several times. In addition, the data communication efficiency between the CPU and the GPU is low. When the CPU sends data to the GPU, the CPU needs to read the data from the memory and send it to the GPU through the PCIE bus. The GPU writes the data to the GPU memory, which requires two memory reads and writes and one PCIE communication. Vice versa.
[0005] Existing supercomputing centers mostly use heterogeneous computing architecture, that is, the computing units are divided into CPU and GPU, the CPU is responsible for conventional numerical calculations, and the GPU is responsible for graphics and AI calculations, etc. They are interconnected through high-speed networks, such as InfiniBand or converged Ethernet, and the NIC is generally attached to the CPU.
[0006] The disadvantage of this method is that the data communication efficiency between the CPU and the GPU is low. When the CPU sends data to the GPU, the CPU reads the data from the memory and sends it to the GPU through the PCIE bus. The GPU writes the data to the GPU memory, which requires two memory reads and writes and one PCIE communication. Vice versa. Similarly, the data communication efficiency between GPUs is also low. When the GPU sends data to the GPU, it needs to send the data to the NIC through the PCIE bus with the assistance of the CPU. The NIC sends the data to the receiving NIC through the high-speed network. After the receiving NIC receives the data, it sends it to the receiving GPU through the PCIE bus. During this period, two PCIE communications and one network communication are required.
[0007] In order to distribute the neural network model on multiple GPUs, the neural network model needs to be split. Currently, the main neural network model splitting technologies are tensor parallelism and pipeline parallelism.
[0008] Tensor parallelism uses the decomposable nature of matrix multiplication to decompose the parameter matrix into several slices (such as taking each row vector of the matrix as a slice), and performing matrix multiplication of different slices on different GPUs. The calculation results are then sent to all other GPUs. The receiving GPU merges all the calculation results of the sending end for the next calculation. This method decomposes the computing tasks to multiple GPUs for simultaneous execution, which can increase the calculation speed of a single batch of input data exponentially. However, the disadvantage is that whenever non-matrix operations, such as non-linear activation functions, are encountered, the calculation results need to be merged once before the non-matrix operations can be performed, so the amount of data synchronized between GPUs is large. Another disadvantage is that it can only be used on some specific neural network models, such as Transformer, but cannot be used on CNN models.
[0009] Pipeline parallelism uses the sequential structure of most neural networks to split the neural network into several layers, which are taken over by different GPUs and executed in layers in series. After the previous GPU completes the calculation, the intermediate results are sent to the next GPU until the last GPU outputs the results. Obviously, this method cannot improve the speed of a single training or reasoning, but when processing batch data, after a GPU completes its work steps, it can process the next batch of data without waiting for the entire task to be completed. Multiple GPUs form a pipeline, and the calculation tasks are decomposed into multiple GPUs for execution on a macro level, which can also multiply the calculation speed of the entire system. Its disadvantage is that it relies on the sequential structure of the neural network and cannot improve the calculation speed of a single batch of data. Summary of the invention
[0010] The embodiments of the present application provide a SoC cluster and a distributed computing method. The present application solves the problem that a single SoC cannot run large computing tasks through the SoC cluster. Furthermore, it solves the problem of how to split the neural network model onto multiple SoCs and run them efficiently.
[0011] In a first aspect, an embodiment of the present application provides a SoC cluster, comprising a plurality of SoCs and an Ethernet and a controller connected thereto. The SoC is used for distributed computing and end-to-end communication. The Ethernet is used to carry point-to-point and / or collective communications between SoCs, and the SoC is directly connected to the Ethernet via a built-in network communication unit. The controller is used for at least one of the following functions: sensing the remaining computing power and memory of the SoC, and the remaining bandwidth of the Ethernet. The computing tasks are decomposed according to the computing power and memory that each SoC can provide, the bandwidth of the Ethernet connection, and the computing task's requirements for computing power, memory, and bandwidth. Traffic control is performed on the SoC and Ethernet during distributed computing.
[0012] Furthermore, the SoC includes a computing unit, a network communication unit and a shared memory. The computing unit and the network communication unit share the data generated by the distributed computing process through the shared memory. Furthermore, each SoC is connected to a read-write memory dedicated to the SoC, and the read-write memory is used to receive and store programs and data related to the distributed computing steps.
[0013] Furthermore, the Ethernet has the following characteristics: comprising one or more Ethernet switching chips, or one or more Ethernet switches. The multiple Ethernet switching chips or Ethernet switches adopt the CLOS architecture. A dynamic routing mechanism is adopted to select one or more paths with sufficient bandwidth between SoCs so that communication between SoCs is unblocked. This dynamic routing mechanism supports point-to-point communication (i.e., unicast) and / or collective communication (i.e., multicast) between SoCs.
[0014] Furthermore, in response to collective communication between N SoCs, the transmitting SoC sends only one copy to Ethernet, which completes the replication to multiple receiving SoCs; the controller adjusts the sending rate of the SoC so that the sending traffic of multiple SoCs aggregated on at least one link of the Ethernet does not exceed the remaining bandwidth of this link, or is not greater than the inlet bandwidth of any receiving SoC.
[0015] Further, in response to point-to-point communication between SoCs, the controller adjusts the SoC sending rate so that it is no greater than the remaining bandwidth on at least one Ethernet link through which the traffic is sent or the ingress bandwidth of the receiving SoC.
[0016] In a second aspect, an embodiment of the present application further provides a method for distributed computing of a SoC cluster, using the SoC cluster described in any one of the embodiments of the first aspect, comprising the following steps:
[0017] Abstract computing tasks into computing graphs;
[0018] Generate a task decomposition diagram and decompose the computing task based on the computing power and memory provided by each SoC, the bandwidth of the Ethernet connection, and the computing power, memory, and bandwidth requirements of the computing task;
[0019] Send the decomposed computing tasks and data to SoC;
[0020] A SoC runs several computational steps assigned to it, and after the computation is completed, it sends the computational results to several SoCs that undertake downstream computational steps;
[0021] Before sending the results to the downstream SoC, adjust the SoC's sending rate and the flow control mechanism on the corresponding Ethernet link to ensure that there is no packet loss in the communication between the upstream and downstream SoCs;
[0022] In response to the SoC receiving the calculation results of several upstream calculation tasks, the calculation within the SoC is started.
[0023] Preferably, generating a task decomposition diagram according to the computational graph corresponding to the computational task includes the following steps:
[0024] Each computation step is mapped to two nodes, input and output, with a weighted edge between them, whose weight is the cost required for this computation step, and the memory required for this computation is used as a constraint. Also, each network communication step is mapped to a weighted edge, whose weight is the cost required for this network communication, and the network bandwidth is used as a constraint. This edge connects the output of the upstream computation step and the input of the downstream computation step. The cost is the time, cost or other expenses spent on network communication.
[0025] Preferably, at least one of the following responses is completed:
[0026] In response to a computational step becoming a bottleneck, multiple SoCs share this step;
[0027] In response to a certain SoC taking on multiple computing steps, they are merged into the same SoC for computing;
[0028] In response to a communication step becoming a bottleneck, increase its network bandwidth;
[0029] In response to a SoC undertaking multiple computation steps, the weight of the communication step between two computation steps is set to 0;
[0030] In response to the downstream computing steps being shared by N SoCs, the computing results are broadcast to these SoCs.
[0031] Preferably, when mapping the computing task graph to the SoC cluster, the optimization is performed with the goal of minimizing the computing task completion time. The weight of the edge corresponding to the computing step is the computing time, and the weight of the edge corresponding to the communication step is the communication time. The network flow algorithm is used to find a decomposition solution so that the time consumed on the longest path in the entire computing task is minimized.
[0032] Preferably, when mapping the computing task graph to the SoC cluster, the optimization is performed with the goal of minimizing the total cost of the computing task. The weight of the edge corresponding to the computing step is a fixed value x, representing the cost of an SoC, and the weight of the edge corresponding to the communication step is a fixed value y, representing the cost of an Ethernet link. The network flow algorithm is used to find a decomposition solution to minimize the total cost of the entire computing task.
[0033] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:
[0034] This application solves the problem that a single SoC cannot run large computing tasks through a SoC cluster; it simplifies the architecture of the distributed computing system, which only consists of SoC, Ethernet and controller. By mapping the calculation graph corresponding to the computing task to the SoC cluster, distributed computing of any computing task can be achieved. The task allocation in the SoC cluster is optimized through the task decomposition graph and the network flow algorithm to improve the efficiency of the distributed computing system. The shared memory technology within the SoC avoids data copying between the computing unit and the network communication unit within the SoC, further improves the speed of communication between SoCs, and ultimately improves the efficiency of the distributed computing system. By carrying collective communication through the Ethernet broadcast network and implementing flow control, the efficiency of collective communication between multiple SoCs is improved, and ultimately the efficiency of the distributed computing system is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0036] Figure 1 This is a SoC cluster structure diagram of an embodiment of the present application;
[0037] Figure 2 This is a diagram showing the internal composition of the SoC in the embodiment of the present application;
[0038] Figure 3 It is a schematic diagram of point-to-point (P2P) communication between SoCs;
[0039] Figure 4 A schematic diagram of collective communication between SoCs in an embodiment of the present application;
[0040] Figure 5 This is a controller structure diagram of an embodiment of the present application;
[0041] Figure 6 This is a flow chart of a method for controlling point-to-point communication traffic of a SoC cluster according to an embodiment of the present application;
[0042] Figure 7 This is a diagram of a SoC cluster collective communication traffic control system according to an embodiment of the present application;
[0043] Figure 8 This is a flow chart of a method for distributed computing of a SoC cluster according to an embodiment of the present application;
[0044] Fig. 9 A schematic diagram of distributed computing for realizing the computing graph of the SoC cluster in the embodiment of the present application;
[0045] Fig.10A schematic diagram of the task decomposition of the computation graph of an embodiment of the present application. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0047] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.
[0048] Figure 1 This is a structural diagram of a SoC cluster according to an embodiment of the present application. An embodiment of the present application provides a SoC cluster, comprising a plurality of (for example, at least three) SoCs 1 and Ethernets 2 and controllers 3 connected thereto.
[0049] The SoC is used for distributed computing and end-to-end communication. The SoC, namely System on Chip,
[0050] The Ethernet is used to carry point-to-point and collective communications between SoCs, and the SoC is directly connected to the Ethernet through a built-in network communication unit.
[0051] The controller is used for at least one of the following functions:
[0052] Sense the remaining computing power and memory of the SoC, as well as the remaining Ethernet bandwidth.
[0053] The computing tasks are decomposed according to the computing power and memory that each SoC can provide, the bandwidth of the Ethernet connection, and the computing power, memory, and bandwidth requirements of the computing tasks.
[0054] Flow control between SoC and Ethernet in distributed computing.
[0055] The SoC cluster completes the entire computing task and outputs the results according to the topological order defined by the task decomposition graph.
[0056] It should be noted that the computing power of SoC in this application refers to the maximum number of calculations that the SoC can complete per unit time, and the remaining computing power of SoC refers to the number of calculations that can be completed per unit time assuming that the SoC has already assumed the computing tasks.
[0057] The Ethernet remaining bandwidth refers to how much bandwidth is left to carry new network traffic, assuming that a certain Ethernet link has already carried network traffic.
[0058] As the best embodiment of the present application, multiple SoC (System on Chip) chips are connected through lossless and non-blocking Converged Ethernet to form a computing cluster, and distributed computing is achieved under the coordination of a controller.
[0059] SoC is used for distributed computing and end-to-end communication.
[0060] Ethernet carries point-to-point and collective communications between SoCs, achieving lossless transmission without packet loss during transmission. For example, the SoC cluster does not have separate computing and storage units, which are all distributed inside the SoC. Alternatively, each SoC is connected to a read-write memory dedicated to this SoC, which is used to receive and store programs and data related to the distributed computing steps.
[0061] Furthermore, the Ethernet has the following characteristics: it includes one Ethernet switching chip, or multiple Ethernet switching chips, or multiple Ethernet switches.
[0062] The plurality of Ethernet switching chips or Ethernet switches adopt CLOS architecture. For example, the converged Ethernet adopts CLOS architecture to achieve non-blocking.
[0063] A dynamic routing mechanism is used to select one or more paths with sufficient bandwidth between SoCs, so that communication between SoCs is unblocked. This dynamic routing mechanism supports both point-to-point communication (i.e. unicast) and collective communication (i.e. multicast) between SoCs.
[0064] Figure 2 This is a diagram of the internal composition of the SoC of the embodiment of the present application. Further, the SoC includes a computing unit, a network communication unit and a shared memory, and is connected to a readable memory dedicated to the SoC. The data generated by the computing unit is shared with the network communication unit through the shared memory without the need for copying.
[0065] Each SoC is a Turing-complete computing unit, such as Figure 5 As shown in the figure, the SoC has computing units such as CPU / GPU and NPU, storage units such as RAM, and network communication units such as Ethernet NIC. RAM is shared between CPU / GPU, NPU and NIC, that is, shared memory. CPU / GPU is used for general computing, NPU is used for neural network computing, and NIC is used for network communication. SoC is directly connected to Ethernet through NIC.
[0066] Figure 3The figure is a schematic diagram of point-to-point (P2P) communication between SoCs. Furthermore, to avoid packet loss due to congestion in data communication between SoCs, the controller calculates the communication bandwidth requirements between each SoC in advance according to the requirements of parallel computing tasks, and controls the transmission rate of the SoC according to the bandwidth of the Ethernet between SoCs, thereby avoiding congestion.
[0067] In response to the point-to-point communication between SoCs, the controller adjusts the SoC sending rate so that it is not greater than the remaining bandwidth on at least one Ethernet link through which the traffic is sent or the ingress bandwidth of the receiving SoC. That is, the sending rate of the sending SoC is not greater than the bandwidth between the sending SoC and the receiving SoC.
[0068] like Figure 3 As shown in the figure, with the help of shared RAM in SoC and RDMA protocol between SoCs, efficient P2P communication is achieved between SoCs. Assume that communication is required between the NPUs of two SoCs. The P2P communication message is generated by the NPU and handed over to the NIC through the shared RAM. The NIC sends the message to the converged Ethernet network through the RDMA protocol and reaches the receiving NIC. The receiving NIC hands the message to the NPU through the shared RAM. During this period, the NIC and NPU only pass memory pointers through shared memory technology, avoiding data copying and improving communication efficiency. At this time, flow control needs to calculate the bandwidth between the two SoCs, and the SoC sending rate is not greater than the above bandwidth.
[0069] Figure 4 This is a schematic diagram of the Collective communication between SoCs in an embodiment of the present application. Furthermore, in response to collective communication between N SoCs, the transmitting SoC only sends one copy to Ethernet, and Ethernet completes the replication to multiple receiving SoCs. The controller adjusts the sending rate of the SoC so that the sending traffic of multiple SoCs aggregated on at least one link of the Ethernet does not exceed the remaining bandwidth of the link, or is not greater than the inlet bandwidth of any receiving SoC. A simple scenario is that N SoCs communicate with each other, and Ethernet receives data from the transmitting SoC and broadcasts it to multiple receiving SoCs. The sending rate of a single transmitting SoC is no greater than 1 / (N-1) of the inlet bandwidth of the receiving SoC.
[0070] For example, in the same physical server, SoCs are interconnected through non-blocking switching boards. Multiple physical servers are interconnected through the converged Ethernet switching network of the CLOS architecture, which can span multiple subnets to achieve large-scale networking.
[0071] For example, in order to efficiently implement collective communication between SoCs, such as All-Reduce, All-Gather and other All-to-All operations, this system uses the broadcast characteristics of Ethernet to improve the efficiency of RDMA in converged Ethernet. Assume that on the SoC, the collective communication message is generated by the NPU and handed over to the NIC through the shared RAM. The NIC only sends the message once to the switching network board, and the switching network board broadcasts it to all related SoCs on the physical server. If necessary, the switching network board further broadcasts the message to the Ethernet switching network between physical servers, and finally reaches all SoCs located in different physical servers. The receiving NIC hands the message to the NPU through the shared RAM. In order to cross the subnet (broadcast domain) and avoid irrelevant SoCs from participating in the communication, multicast technology can be used to reduce the broadcast range. At this time, the flow control needs to consider the simultaneous transmission and reception between N SoCs, and adjust the transmission rate of the SoC so that the transmission traffic of multiple SoCs aggregated on at least one link of the Ethernet does not exceed the remaining bandwidth of this link, or is not greater than the inlet bandwidth of any receiving SoC.
[0072] Figure 5 This is a controller structure diagram of an embodiment of the present application. The controller is used for at least one of the following functions: receiving computing tasks and abstracting them into computing graphs. According to the computing power and memory that each SoC can provide and the bandwidth of the Ethernet connection, a task decomposition graph is generated to decompose the computing tasks. The decomposed computing tasks and data are sent to the SoC.
[0073] In a cluster, each SoC can perform a part of distributed computing, including several computing steps. The controller performs task allocation, and the controller can be implemented as a specific SoC. The controller is used to manage the entire computing cluster and realize unified scheduling of computing and network resources.
[0074] To implement the above technical solution, the present application proposes a controller comprising a resource sensing module 301, a resource allocation module 302, and a resource control module 303 which are interconnected.
[0075] The resource perception module is used to perceive computing and network resources and receive computing tasks, such as steps 110, 130, and 140 of the embodiment.
[0076] The resource allocation module is used to generate a task decomposition diagram based on the computing power and memory that each SoC can provide and the bandwidth of the Ethernet connection, and decompose the computing tasks, such as step 220 of the embodiment.
[0077] The resource control module is used to send the decomposed computing tasks and data to the SoC and control the traffic between the SoCs, such as steps 230 to 280 of the embodiment.
[0078] The specific methods for implementing the functions of the resource perception module, resource allocation module, and resource control module are as described in the various method embodiments of the present application and will not be repeated here.
[0079] For example, the controller described in the present application, in order to implement the method for implementing a computation graph by a SoC cluster described in the above embodiment, is also used for at least one of the following functions:
[0080] Determine the number of SoCs participating in the communication; determine the inlet bandwidth of the receiving SoC; and adjust the sending rate of the sending SoC.
[0081] Furthermore, the resource sensing module is also used to sense the number of SoCs participating in the communication and the inlet bandwidth of the receiving end SoC.
[0082] The resource control module is also used to adjust the sending rate of the sending end SoC.
[0083] The controller is connected to the Ethernet. To avoid packet loss due to congestion in data communication between SoCs, the controller calculates the communication bandwidth requirements between each SoC in advance according to the requirements of parallel computing tasks, and controls the transmission rate of the SoC according to the bandwidth of the Ethernet between SoCs, thereby avoiding congestion.
[0084] It should be noted that the controller may be undertaken by a SoC or an independent processor, which is not further limited here.
[0085] Figure 6 This is a flow chart of a SoC cluster point-to-point control method according to an embodiment of the present application.
[0086] The embodiment of the present application further provides a SoC cluster control method, which is used for the point-to-point communication of the SoC cluster according to any embodiment of the first aspect of the present application, and comprises the following steps:
[0087] Step 110: Sense the port bandwidths of the transmitting end SoC and the receiving end SoC, and the bandwidth of the Ethernet link between the ports;
[0088] Step 120: Adjust the sending rate of the transmitting end SoC to be smaller than the bandwidth of the transmitting end SoC and the receiving end SoC.
[0089] For example, during point-to-point transmission, the sending end computing unit generates data and sends the data to the sending end shared memory to obtain the corresponding memory pointer, and sends the memory pointer to the sending end network communication unit. The sending end network communication unit receives the memory pointer sent by the sending end computing unit, obtains the data in the sending end shared memory according to the memory pointer, and sends it to the Ethernet.
[0090] The receiving end network communication unit is used to receive data and store it in the receiving end shared memory, and send the memory pointer to the receiving end calculation unit. The receiving end calculation unit receives the memory pointer and obtains the data in the receiving end shared memory.
[0091] Figure 7 This is a diagram of a SoC cluster collective communication control system according to an embodiment of the present application. In the illustrated system, the embodiment of the present application also provides a SoC cluster control method for SoC collective communication, using the SoC cluster described in any one of the embodiments of the first aspect of the present application to sense the remaining computing power, memory, and Ethernet remaining bandwidth of the SoC, specifically including the following steps 130 to 140; performing flow control, specifically including the following steps 150:
[0092] Step 130: determining a number of SoCs participating in the communication;
[0093] Step 140: Determine the Ethernet link and remaining bandwidth through which the collective communication between the SoCs passes, including the receiving end SoC inlet bandwidth;
[0094] Step 150: Adjust the sending rate of the sending SoC so that the sending traffic of multiple SoCs aggregated on at least one link of the Ethernet does not exceed the remaining bandwidth of the link, or is not greater than the inlet bandwidth of any receiving SoC.
[0095] Figure 8 This is a flow chart of a SoC cluster distributed computing method according to an embodiment of the present application.
[0096] The present application also provides a SoC cluster distributed computing method, using the SoC cluster described in any one of the above embodiments, comprising the steps of:
[0097] Step 210: abstract the computing task into a computing graph;
[0098] Step 220: Generate a task decomposition diagram based on the computing power and memory that each SoC can provide and the bandwidth of the Ethernet connection to decompose the computing task;
[0099] Step 230: Send the decomposed computing tasks and data to the SoC.
[0100] Step 240: A SoC runs several computing steps assigned to it, and after the computing is completed, sends the computing results to all SoCs that undertake downstream computing steps;
[0101] Step 250, in response to the downstream computing step being shared by N SoCs, broadcasting the computing result to these SoCs;
[0102] Step 260: Before sending the result to the downstream SoC, adjust the sending rate of the SoC and the flow control mechanism on the corresponding Ethernet link so that there is no packet loss in the communication between the upstream and downstream SoCs;
[0103] Step 270, in response to the downstream computing step being shared by N SoCs, broadcasting the computing results to these SoCs;
[0104] Step 280 , in response to the SoC receiving the calculation results of all upstream calculation tasks, starting the calculation within the SoC.
[0105] For further explanation of the task decomposition of step 220, see Figures 9-10 .
[0106] Fig. 9 This is a schematic diagram of distributed computing of the SoC cluster implementation calculation graph of the embodiment of this application. Based on high-efficiency P2P and collective communication, this cluster implements high-efficiency distributed computing, such as training and reasoning of large-scale neural networks. Due to the huge model parameters, a single SoC cannot accommodate all the parameters and must be distributed on different SoCs. For simpler neural networks, the tensor parallel and pipeline parallel technologies introduced earlier can be used for distributed computing.
[0107] However, a complex neural network does not only have a sequential structure, but is a DAG (directed acyclic graph) composed of multiple computing steps, called a computational graph. This directed acyclic graph can also be mapped to a SoC cluster, with different SoCs taking on different computing steps. If a SoC requires multiple inputs, it will wait for the calculation results of all upstream SoCs to be delivered through the network before starting the calculation; after a SoC completes the calculation, it will send the result to other downstream SoCs through the network; repeat the above process until all SoCs complete the calculation. Among them, a calculation step in the computational graph can be undertaken by multiple SoCs, at which point they can share the computing task in a tensor parallel manner, thereby speeding up this calculation step. An SoC can also undertake multiple computing steps, so that the computing load of each SoC in the system is more balanced. Such as Figure 7 shown.
[0108] Fig.10 This is a task decomposition diagram of the computation graph of the embodiment of the present application. Further, according to the computation graph corresponding to the computation task, a task decomposition diagram is generated, which includes the following steps:
[0109] Each computation step is mapped to two nodes, input and output, with a weighted edge between them. The weight is the cost of this computation step, and the memory required for this computation is used as a constraint.
[0110] Each network communication step is mapped to a weighted edge, whose weight is the cost required for this network communication and is constrained by the network bandwidth. This edge connects the output of the upstream computing step and the input of the downstream computing step.
[0111] Each computation step is mapped to two nodes, input and output, with a weighted edge between them, whose weight is the time required for this computation step, and the memory required for this computation is used as a constraint. Each network communication step is mapped to a weighted edge, whose weight is the time required for this network communication, and the network bandwidth is used as a constraint. This edge connects the output of the upstream computation step and the input of the downstream computation step. In response to the fact that a SoC can undertake multiple computation steps, the weight of the communication step between two computation steps is set to 0.
[0112] For example, in order to realize the mapping of the above computational graph to the SoC cluster topology, the controller is used to decompose the computational tasks, comprehensively consider the computational overhead and network overhead, find the optimal allocation method, and maximize the execution efficiency of the entire computational graph. One implementation of computational task decomposition is the network flow algorithm. At this time, the computational and network overheads should be represented in the same graph, which is called the task decomposition graph. In the task decomposition graph, each computational step is mapped to two nodes, input and output, with a weighted edge between them, whose weight is the time required for this computational step, and the memory required for this computation is used as a constraint; each network communication step is mapped to a weighted edge, whose weight is the time required for this network communication, and the network bandwidth is used as a constraint. This edge connects the output of the upstream computational step and the input of the downstream computational step. If an SoC can undertake multiple computational steps, the weight of the communication step between the two computational steps is set to 0. With this task decomposition graph, algorithms such as minimum cost maximum flow can be used to calculate the optimal computational graph decomposition solution. When a computing step becomes a bottleneck, multiple SoCs can share this step; when a SoC can undertake multiple computing steps, they can be merged into the same SoC for computing; when a communication step becomes a bottleneck, you can consider increasing its network bandwidth. This optimizes the efficiency of the entire distributed computing system. Optimization strategies include time priority and cost priority. Time priority refers to optimization with the goal of minimizing the time it takes to complete a computing task, and cost priority refers to optimization with the goal of minimizing the total cost of a computing task (such as the number of SoCs).
[0113] Furthermore, after the task decomposition diagram is generated, at least one of the following responses is completed to map the computation steps to the SoC cluster:
[0114] In response to a computational step becoming a bottleneck, multiple SoCs share this step;
[0115] In response to a certain SoC taking on multiple computing steps, they are merged into the same SoC for computing;
[0116] In response to a communication step becoming a bottleneck, increase its network bandwidth;
[0117] In response to one SoC undertaking multiple computation steps, the weight of the communication step between two computation steps is set to zero.
[0118] Once the task decomposition graph is obtained, the optimal computational graph decomposition solution can be calculated using algorithms such as minimum cost maximum flow.
[0119] When a computing step becomes a bottleneck, multiple SoCs can share this step; when a SoC can undertake multiple computing steps, they can be merged into the same SoC for computing; when a communication step becomes a bottleneck, you can consider increasing its network bandwidth. This optimizes the efficiency of the entire distributed computing system. Optimization strategies include time priority and cost priority. Time priority refers to optimization with the goal of minimizing the time it takes to complete a computing task, and cost priority refers to optimization with the goal of minimizing the total cost of a computing task (such as the number of SoCs).
[0120] Furthermore, when mapping the computing task graph to the SoC cluster, the optimization is performed with the goal of minimizing the computing task completion time. At this time, the weight of the edge corresponding to the computing step is the computing time, and the weight of the edge corresponding to the communication step is the communication time. The network flow algorithm is used to find a decomposition solution so that the time consumed on the longest path in the entire computing task is minimized.
[0121] Furthermore, when mapping the computing task graph to the SoC cluster, the optimization is performed with the goal of minimizing the total cost of the computing task. At this time, the weight of the edge corresponding to the computing step is a fixed value x, that is, the cost of one SoC, and the weight of the edge corresponding to the communication step is a fixed value y, that is, the cost of one Ethernet link. The network flow is used to find a decomposition solution to minimize the total cost of the entire computing task.
[0122] In summary, the workflow of the entire SoC cluster system to implement distributed computing includes the following steps:
[0123] Submit the computation task to the controller.
[0124] The controller abstracts the computing task into a computing graph, generates a task decomposition graph based on the computing power and memory provided by each SoC and the bandwidth of the Ethernet connection, and decomposes the computing task.
[0125] The controller sends the decomposed computing tasks and data to the SoC.
[0126] The controller initiates the entire computation task.
[0127] A SoC runs several computing steps assigned to it. After the calculation is completed, the intermediate results are sent to all SoCs that undertake downstream computing steps. In particular, if a downstream computing step is shared by multiple SoCs, the intermediate results need to be copied to these SoCs. The controller controls the flow of the sending SoC based on the data communication volume required to transmit the intermediate results and the bandwidth between SoCs.
[0128] A SoC receives the intermediate calculation results of all upstream computing tasks and starts the calculation within this SoC.
[0129] Repeat the above two steps until the final calculation step is completed and the final result is output.
[0130] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0131] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0132] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A SoC cluster, characterized in that: Includes multiple SoCs and the Ethernet and controllers that connect them; The SoC is used for distributed computing and end-to-end communication. In the distributed computing workflow, the SoC runs the computing steps assigned to it and sends intermediate results to multiple SoCs that undertake downstream computing steps; The Ethernet is used to carry collective communication between SoCs, and the SoC is directly connected to the Ethernet through a built-in network communication unit; The controller is used to: Sense the remaining computing power and memory of the SoC, as well as the remaining Ethernet bandwidth; The computing tasks are decomposed according to the computing power and memory that each SoC can provide, the bandwidth of the Ethernet connection, and the computing power, memory, and bandwidth requirements of the computing tasks. When generating the task decomposition graph, each computing step is mapped to two input and output nodes, with a weighted edge between them, whose weight is the cost required for this computing step, and the memory required for this calculation is used as a constraint; each network communication step is mapped to a weighted edge, whose weight is the cost required for this network communication, and the network bandwidth is used as a constraint. This edge connects the output of the upstream computing step and the input of the downstream computing step; In the distributed computing process, the flow of SoC and Ethernet is controlled according to the data communication volume required for delivering intermediate results and the bandwidth between SoCs; in response to the collective communication between N SoCs, the controller adjusts the sending rate of SoC so that the sending flow of multiple SoCs aggregated on at least one link of Ethernet does not exceed the remaining bandwidth of this link, or is not greater than the inlet bandwidth of any receiving SoC; When mapping the computing task graph to the Soc cluster, optimization is performed with the goal of minimizing the computing task completion time or the total cost of the computing task.
2. The SoC cluster according to claim 1, characterized in that: The SoC includes a computing unit, a network communication unit and a shared memory; The computing unit and the network communication unit share data generated by the distributed computing process through the shared memory.
3. The SoC cluster according to claim 1, characterized in that: The Ethernet has the following characteristics: Contains one or more Ethernet switching chips, or one or more Ethernet switches; Multiple Ethernet switching chips or Ethernet switches adopt CLOS architecture; A dynamic routing mechanism is used to select one or more paths with sufficient bandwidth between SoCs, so that communication between SoCs is unblocked; The dynamic routing mechanism supports point-to-point communication and / or collective communication between SoCs.
4. The SoC cluster according to claim 1, characterized in that: The transmitting SoC sends only one copy to Ethernet, which then replicates it to multiple receiving SoCs.
5. The SoC cluster according to claim 1, characterized in that: In response to the point-to-point communication between SoCs, the controller adjusts the SoC sending rate so that it is not greater than the remaining bandwidth of at least one Ethernet link through which the traffic is sent or the ingress bandwidth of the receiving SoC.
6. A method for distributed computing of a SoC cluster, characterized in that: Using the SoC cluster according to any one of claims 1 to 5 comprises the following steps: Abstract computing tasks into computing graphs; Generate a task decomposition diagram and decompose the computing task based on the computing power and memory provided by each SoC, the bandwidth of the Ethernet connection, and the computing power, memory, and bandwidth requirements of the computing task; Send the decomposed computing tasks and data to SoC; A SoC runs several computational steps assigned to it, and after the computation is completed, it sends the computational results to several SoCs that undertake downstream computational steps; Before sending the results to the downstream SoC, adjust the SoC's sending rate and the flow control mechanism on the corresponding Ethernet link to ensure that there is no packet loss in the communication between the upstream and downstream SoCs; In response to the SoC receiving the calculation results of several upstream calculation tasks, the calculation within the SoC is started.
7. The method for distributed computing of SoC cluster according to claim 6, characterized in that: According to the calculation graph corresponding to the calculation task, generate a task decomposition graph, which includes the following steps: Each computation step is mapped to two nodes, input and output, with a weighted edge between them. The weight is the cost of this computation step, and the memory required for this computation is used as a constraint. Each network communication step is mapped to a weighted edge, which connects the output of the upstream computing step and the input of the downstream computing step. Its weight is the cost required for this network communication, and the network bandwidth is used as a constraint.
8. The method for distributed computing of SoC cluster according to claim 6, characterized in that: Complete at least one of the following responses: In response to a computational step becoming a bottleneck, multiple SoCs share this step; In response to a certain SoC taking on multiple computing steps, they are merged into the same SoC for computing; In response to a communication step becoming a bottleneck, increase its network bandwidth; In response to a SoC undertaking multiple computation steps, the weight of the communication step between two computation steps is set to 0; In response to the downstream computing steps being shared by N SoCs, the computing results are broadcast to these SoCs.
9. The method for distributed computing of SoC cluster according to claim 6, characterized in that: When mapping the computing task graph to the SoC cluster, the optimization is performed with the goal of minimizing the computing task completion time; The weight of the edge corresponding to the computation step is the computation time, and the weight of the edge corresponding to the communication step is the communication time; The network flow algorithm is used to find a decomposition solution so that the time consumed on the longest path in the entire computing task is minimized.
10. The method for distributed computing of SoC cluster according to claim 6, characterized in that: When mapping the computing task graph to the SoC cluster, the optimization is performed with the goal of minimizing the total cost of the computing tasks; The weight of the edge corresponding to the computation step is a fixed value x, representing the cost of one SoC, and the weight of the edge corresponding to the communication step is a fixed value y, representing the cost of one Ethernet link; The network flow algorithm is used to find a decomposition solution to minimize the total cost of the entire computing task.
