Distributed computing system and distributed computing method

Abstract

This invention proposes a distributed computing system and a distributed computing method. The distributed computing system includes a first computing device and a second computing device. The second computing device is connected to the first computing device via a cable. The distributed computing module calculates the buffer size required by the model to determine a memory usage strategy. The distributed computing module splits the model to generate multiple sub-models. The distributed computing module allocates the multiple sub-models to multiple display memories of the first and second computing devices according to the memory usage strategy, and selectively uses at least one of the multiple dynamic random access memories of the first and second computing devices to distribute the training of the model.

Description

Technical Field

[0001] This invention relates to a system, and more particularly to a distributed computing system and a distributed computing method. Prior Technology

[0002] In the training of large-scale models, a significant amount of display memory (VRAM) is often required. However, insufficient computational resources and buffers are common problems. Therefore, how to integrate the memory resources of multiple computing devices and optimize training efficiency is an important issue in this field. Summary of the Invention

[0003] This invention provides a distributed computing system and a distributed computing method that can efficiently train large models through multiple computing devices.

[0004] The distributed computing system of the present invention includes a first computing device, a second computing device, and a distributed computing module. The second computing device is connected to the first computing device via a cable. The distributed computing module is coupled to the first computing device and the second computing device and is disposed within either the first or second computing device. The distributed computing module calculates the buffer size required for the calculation model to determine a memory usage strategy. The distributed computing module splits the model to generate multiple sub-models. The distributed computing module allocates the multiple sub-models to multiple display memories of the first and second computing devices according to the memory usage strategy, and selectively uses at least one of the multiple dynamic random access memories of the first and second computing devices to distribute the training of the model.

[0005] The distributed computing method of the present invention is applicable to a distributed computing system. The distributed computing system includes a first computing device and a second computing device. The second computing device is connected to the first computing device via a cable. The distributed computing method includes the following steps: calculating the buffer size required by the model to determine a memory usage strategy; splitting the model to generate multiple sub-models; and allocating the multiple sub-models to multiple display memories of the first and second computing devices according to the memory usage strategy, and selectively using at least one of the multiple dynamic random access memories of the first and second computing devices to distribute the training of the model.

[0006] Based on the above, the distributed computing system and distributed computing method of the present invention can distribute the training of large-scale models through multiple display memories of the first computing device and the second computing device.

[0007] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings for detailed explanation. Simple Explanation of the Diagram

[0008] Figure 1 is a schematic diagram of a distributed computing system according to an embodiment of the present invention. Figure 2 is a schematic diagram of the software architecture of an embodiment of the present invention. Figure 3 is a flowchart of a distributed computing method according to an embodiment of the present invention. Figure 4 is a flowchart of a distributed computing method according to another embodiment of the present invention. Figure 5 is a schematic diagram of model splitting and allocation according to an embodiment of the present invention. Figure 6 is a schematic diagram of the forward transmission pipeline according to an embodiment of the present invention. Figure 7 is a schematic diagram of micro-batch data processing according to an embodiment of the present invention. Implementation

[0009] To make the contents of this invention more readily apparent, the following specific embodiments are provided as examples on which this invention can indeed be implemented. Furthermore, wherever possible, elements / components / steps using the same reference numerals in the drawings and embodiments represent the same or similar parts.

[0010] Figure 1 is a schematic diagram of a distributed computing system according to an embodiment of the present invention. Referring to Figure 1, the distributed computing system 100 includes a first computing device 110 and a second computing device 120. The first computing device 110 includes a processor 111, a connection interface 112, a display memory 113, and a dynamic random-access memory (DRAM) 114. The processor 111 is coupled to the connection interface 112, the display memory (VRAM), and the DRAM 114. The second computing device 120 includes a processor 121, a connection interface 122, a display memory 123, and a DRAM 124. The processor 121 is coupled to the connection interface 122, the display memory 123, and the DRAM 124.

[0011] It is worth noting that the number of display memories 113 and 123 and dynamic random access memories 114 and 124 in this invention can be one or more, and is not limited to those shown in FIG1. ​​Furthermore, the first computing device 110 and the second computing device 120 may also include other functional devices, circuits, and other types of memory for model training or model computation, and are not limited to those shown in FIG1. ​​In one embodiment, the first computing device 110 and the second computing device 120 may also each include a non-volatile memory express solid-state drive (NVMe SSD), and be allocated for model training or model computation. Additionally, in one embodiment, the distributed computing system 100 may also include multiple computing devices connected in series via multiple cables, and is not limited to those shown in FIG1.

[0012] In this embodiment, the first computing device 110 and the second computing device 120 can each be a computer host, and the connection interface 112 of the first computing device 110 is connected to the connection interface 122 of the second computing device 120 via a cable 130. In this embodiment, processors 111 and 121 can each be a central processing unit (CPU). In this embodiment, the cable 130 can be a Thunderbolt cable (e.g., Thunderbolt 4, Thunderbolt 5, or higher).

[0013] In this embodiment, the first computing device 110 and the second computing device 120 can transmit and communicate data via cable 130 to achieve model training or model computation. In this embodiment, the first computing device 110 and the second computing device 120 can be equipped with SSH programs (such as the openssh-server suite or sshpass suite) so that the first computing device 110 and the second computing device 120 can be assigned static IP addresses respectively to achieve stable communication. Furthermore, the first computing device 110 and the second computing device 120 can synchronize training data and assign computation tasks based on the SSH program.

[0014] Figure 2 is a schematic diagram of the software architecture according to an embodiment of the present invention. Referring to Figures 1 and 2, in this embodiment, one of the first computing device 110 and the second computing device 120 may store the distributed computing module 210, the model 220, and the user interface 230. In this embodiment, the model 220 may be a large model, such as a large language model (LM) or a large multimodal model (LMM). In this embodiment, one of the processors 111 and 121 may execute the distributed computing module 210 to perform distributed training on the model 220. Furthermore, one of the processors 111 and 121 may execute the user interface 230 to provide related functions through the user interface 230.

[0015] Specifically, the user interface 230 may, for example, display multiple selection fields. These selection fields may be used for selecting a model, setting the dataset (the dataset to be used for training), fine-tuning strategies and settings for model training, training settings, learning rate settings, setting the batch size (affecting the amount of display memory used), setting the number of training cycles, hardware settings, memory usage strategies (i.e., memory offloading strategies), and related interface settings, etc., and the present invention is not limited thereto. In this embodiment, the user interface 230 may also present the relevant model training content to the user in a visual manner, thereby enabling convenient relevant model training operations.

[0016] Figure 3 is a flowchart of a distributed computing method according to an embodiment of the present invention. Referring to Figures 1 to 3, the distributed computing system 100 can execute the following steps S310 to S330. In step S310, the distributed computing module 210 can calculate the buffer size required for model 220 to determine the memory usage strategy. In step S320, the distributed computing module 210 can split model 220 to generate multiple sub-models. In step S330, the distributed computing module 210 can allocate the multiple sub-models to multiple display memories of the first computing device 110 and the second computing device 120 according to the memory usage strategy, and selectively use at least one of the multiple dynamic random access memories of the first computing device 110 and the second computing device 120 to distribute the training of model 220.

[0017] In this embodiment, when the distributed computing module 210 determines that the display memory 113 and 123 of the first computing device 110 and the second computing device 120 are sufficient to accommodate the buffer size required by the model 220, the memory usage strategy is to use at least one of the display memory 113 and 123 of the first computing device 110 and the second computing device 120 to train the plurality of sub-models.

[0018] In this embodiment, when the distributed computing module 210 determines that the display memory 113 and 123 of the first computing device 110 and the second computing device 120 cannot accommodate the buffer size required by the model 220, the memory usage strategy includes allocating at least a portion of the optimizer state data of the multiple sub-models to at least one of the dynamic random access memory 114 and 124 of the first computing device 110 and the second computing device 120. The optimizer state data may include, for example, learning rate, gradient, checkpoint, and / or training logs.

[0019] In this embodiment, when the distributed computing module 210 determines that the display memory 113 and 123 of the first computing device 110 and the second computing device 120 cannot accommodate the buffer size required by the model 220, the memory usage strategy includes allocating at least a portion of the optimizer state data of the multiple sub-models and at least a portion of the multiple model parameters to at least one of the dynamic random access memory 114 and 124 of the first computing device 110 and the second computing device 120. Model parameters may include, for example, weights or other learnable variables that need to be optimized during model training.

[0020] In this embodiment, the display memory 113 and 123 of the first computing device 110 and the second computing device 120 can be used for high-speed calculations, and the dynamic random access memory 114 and 124 of the first computing device 110 and the second computing device 120 can be used to store relevant data required or generated during model training. Therefore, the distributed computing system 100 of this embodiment can dynamically adjust the memory usage state according to the memory usage strategies of different models, thereby effectively improving the overall resource utilization and model training efficiency.

[0021] Figure 4 is a flowchart of a distributed computing method according to another embodiment of the present invention. Figure 4 provides a further detailed implementation flow of the embodiment shown in Figure 3. Referring to Figures 1, 2, and 4, in this embodiment, the distributed computing system 100 can execute the following steps S410 to S470. In step S410, the distributed computing module 210 can split the model 220 to generate multiple sub-models. The distributed computing module 210 can allocate the multiple sub-models to multiple memories of the first computing device 110 and the second computing device 120.

[0022] For example, refer to Figures 5 and 6. Figure 5 is a schematic diagram of model splitting and allocation according to an embodiment of the present invention. Figure 6 is a schematic diagram of the forward pass pipeline according to an embodiment of the present invention. As shown in Figures 5 and 6, model 220 can be split into multiple sub-models F0 to F3. Sub-models F0 to F3 can be allocated to different display memories of the first computing device 110 and the second computing device 120 for execution. Sub-models F0 to F3 can sequentially input the calculated loss function into the next sub-model and summarize it. The output data B0 to B3 (e.g., gradients) calculated sequentially by sub-models F0 to F3 can be summarized and input back into sub-models F0 to F3 as the basis for updating model parameters in the next step.

[0023] In step S420, the distributed computing module 210 can divide the input data into multiple micro-batch data, and process the multiple micro-batch data simultaneously through different sub-models of pipeline parallel processing, wherein the number of the multiple micro-batch data matches the capacity of parallel processing.

[0024] For example, refer to Figure 7. Figure 7 is a schematic diagram of micro-batch data processing according to an embodiment of the present invention. Input data can be divided into multiple smaller micro-batch data, and input sequentially into sub-models F(0,0) to F(0,3). Sub-model F(0,0) can first process one micro-batch data to output data to sub-model F(1,0) at the next time point. Simultaneously, during the processing of sub-model F(1,0), sub-model F(0,1) can process the next micro-batch data. And so on, sub-models F(0,0) to F(3,3) can perform parallel processing to generate output data B(0,0) to B(3,3).

[0025] In step S430, the distributed computing module 210 can perform a forward pass pipeline operation to process the multiple micro-batch data sequentially. In step S440, the distributed computing module 210 can calculate gradients in reverse. In step S450, the distributed computing module 210 can synchronize parameters. The distributed computing module 210 can use a distributed communication library (e.g., PyTorch, TensorFlow) to perform data transmission between the first computing device 110 and the second computing device 120 to reduce latency and optimize data transmission. Furthermore, the distributed computing module 210 can use mechanisms such as synchronization barriers or explicit waiting to ensure that the sub-models executed by all memories are correctly aligned during the forward and backward passes in parallel processing during model training or model computation.

[0026] In step S460, the distributed computing module 210 can perform pipeline parallel scheduling operations. In this embodiment, the distributed computing module 210 trains the multiple sub-models using the display memories 113 and 123 of the first computing device 110 and the second computing device 120 through micro-batch interleaving and micro-batch overlapping. Furthermore, the distributed computing module 210 can adjust the memory usage of the first computing device 110 and the second computing device 120 through gradient checkpointing. Specifically, the distributed computing module 210 can adjust model partitioning through balanced computation to reduce the idle time of the computing devices, and can use gradient checkpointing to reduce memory usage during forward pass computation.

[0027] In step S470, the distributed computing module 210 can debug and test the pipeline. In this embodiment, the distributed computing module 210 can verify the data flow to ensure that the output of one stage is correctly received as input by the next stage. In this embodiment, the distributed computing module 210 can monitor performance to identify computing bottlenecks by measuring latency and throughput. The distributed computing module 210 can use analysis tools to analyze device utilization and perform corresponding optimizations. The distributed computing module 210 can test gradients to verify the correctness of gradients during backpropagation, thereby ensuring correct model training. Furthermore, when the distributed computing system 100 adds computing devices, the distributed computing module 210 can also reallocate the multiple sub-models to balance the workload of the additional devices. Accordingly, the distributed computing method of this embodiment can train large models in a distributed computing manner using multiple computing devices.

[0028] In summary, the distributed computing system and method of the present invention can train large-scale models by combining the memory resources of multiple computing devices. Furthermore, the distributed computing system and method of the present invention can generate corresponding memory usage strategies based on different model types, thereby enabling efficient use of memory resources to train large-scale models.

[0029] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

[0030] 100: Distributed computing system 110: First arithmetic unit 111, 121: Processor 112, 122: Connection Interface 113, 123: Display memory 114, 124: Dynamic Random Access Memory 120: Second arithmetic unit 130: Cable 210: Distributed computing module 220: Model 230: User Interface S310~S330, S410~S470: Steps F0~F3, F(0,0)~F(3,3): Sub-models B0~B3, B(0,0)~B(3,3): Output data

Claims

1. A distributed computing system, comprising: A first arithmetic unit; A second computing device is connected to the first computing device via a cable; A distributed computing module is coupled to the first computing device and the second computing device and disposed within the first computing device or the second computing device. The distributed computing module calculates a buffer size required for a model to determine a memory usage strategy. The distributed computing module splits the model to generate multiple sub-models. The distributed computing module allocates these sub-models to multiple display memories of the first computing device and the second computing device according to the memory usage strategy, and selectively uses at least one of the multiple dynamic random access memories of the first computing device and the second computing device to distribute the training of the model.

2. The distributed computing system as described in claim 1, wherein the distributed computing module divides an input data into multiple micro-batch data and processes the micro-batch data simultaneously through different sub-models.

3. The distributed computing system as described in claim 1, wherein the distributed computing module performs data transmission between the first computing device and the second computing device through a distributed communication library.

4. The distributed computing system as claimed in claim 1, wherein the distributed computing module trains the sub-models through the display memory of the first computing device and the second computing device in a micro-batch interleaving and micro-batch overlapping manner.

5. The distributed computing system as claimed in claim 1, wherein the distributed computing module adjusts the memory usage of the first computing device and the second computing device through a gradient checkpoint operation.

6. The distributed computing system as described in claim 1, wherein the model is a large language model or a large multimodal model.

7. The distributed computing system as claimed in claim 1, wherein when the distributed computing module determines that the display memory of the first computing device and the second computing device is sufficient to accommodate the buffer size required by the model, the memory usage strategy is to use only at least one of the display memory of the first computing device and the second computing device to train the sub-models.

8. The distributed computing system of claim 1, wherein when the distributed computing module determines that the display memory of the first computing device and the second computing device cannot accommodate the buffer size required by the model, the memory usage strategy includes allocating at least a portion of the multiple optimizer state data of the sub-models to at least one of the dynamic random access memories of the first computing device and the second computing device.

9. The distributed computing system of claim 1, wherein when the distributed computing module determines that the display memory of the first computing device and the second computing device cannot accommodate the buffer size required by the model, the memory usage strategy includes allocating at least a portion of the multiple optimizer state data of the sub-models and at least a portion of the multiple model parameters to at least one of the dynamic random access memories of the first computing device and the second computing device.

10. The distributed computing system as described in claim 1, wherein the cable is a Thunderbolt cable.

11. A distributed computing method applicable to a distributed computing system, the distributed computing system including a first computing device and a second computing device, the second computing device being connected to the first computing device via a cable, wherein the distributed computing method includes: Calculate the buffer size required for a model to determine a memory usage strategy; The model is split into multiple sub-models; And according to the memory usage strategy, the sub-models are allocated to multiple display memories of the first computing device and the second computing device, and at least one of the multiple dynamic random access memories of the first computing device and the second computing device is selectively used to train the model in a distributed manner.

12. The distributed computing method as described in claim 11 further includes: The input data is divided into multiple micro-batch data, and these micro-batch data are processed simultaneously through different sub-models.

13. The distributed computing method as described in claim 11 further includes: Data transmission between the first computing device and the second computing device is carried out through a distributed communication library.

14. The distributed computing method as described in claim 11 further includes: The sub-models are trained using the display memory of the first and second computing devices in a micro-batch interleaving and micro-batch overlapping manner.

15. The distributed computing method as described in claim 11 further includes: The memory usage of the first and second computing devices is adjusted through a gradient checkpoint operation.

16. The distributed computing method as described in claim 11, wherein the model is a large language model or a large multimodal model.

17. The distributed computing method as claimed in claim 11, wherein when the display memories of the first computing device and the second computing device are large enough to accommodate the buffer required by the model, the memory usage strategy is to use only at least one of the display memories of the first computing device and the second computing device to train the sub-models.

18. The distributed computing method of claim 11, wherein when the display memory of the first computing device and the second computing device cannot accommodate the buffer size required by the model, the memory usage strategy includes allocating at least a portion of the multiple optimizer state data of the sub-models to at least one of the dynamic random access memories of the first computing device and the second computing device.

19. The distributed computing method of claim 11, wherein when the display memory of the first computing device and the second computing device cannot accommodate the buffer size required by the model, the memory usage strategy includes allocating at least a portion of the multiple optimizer state data of the sub-models and at least a portion of the multiple model parameters to at least one of the dynamic random access memories of the first computing device and the second computing device.

20. The distributed computing method as described in claim 11, wherein the cable is a Thunderbolt cable.

Images