Deep learning training system based on gpu cluster and related methods
By using a GPU cluster-based deep learning training system with topological ring structure and Horovod framework, we have achieved efficient training of complex models in the field of petroleum geophysical exploration. This solves the problems of low training efficiency and communication bottlenecks, and improves the scalability and efficiency of model training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA NAT PETROLEUM CORP
- Filing Date
- 2024-11-27
- Publication Date
- 2026-05-29
AI Technical Summary
In the field of petroleum geophysical exploration, existing technologies struggle to efficiently train complex deep learning models, especially when faced with large amounts of data, numerous model parameters, tight time constraints, and the need for real-time adjustments. Existing methods suffer from low training efficiency and communication bottlenecks.
A deep learning training system based on GPU clusters is adopted, which distributes training batches to multiple GPU nodes through the host and executes model training in parallel. The topology ring structure is used to share gradient data and update parameters, avoiding the need to aggregate gradient data to the host and then broadcast it. The Horovod deep learning framework is used to simplify the configuration.
It improves model training efficiency, reduces communication latency, enhances system scalability and GPU utilization, and significantly shortens training time.
Smart Images
Figure CN122114222A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a deep learning training system and related methods based on GPU clusters. Background Technology
[0002] With the widespread application of artificial intelligence technologies, such as deep learning, in engineering, accelerating the training process of deep learning models to improve training efficiency has become crucial in the face of increasingly massive amounts of data. This is particularly true in the field of machine learning applications in petroleum geophysical exploration. Firstly, seismic data is vast, and with the application of efficient geophysical exploration technologies such as "broadband and high-resolution" geophysical systems, the daily volume of construction data is measured in gigabytes (Gb) and terabytes (Tb). Secondly, the introduction of multi-dimensional data, including geological, seismic, and well logging data, has led to increasingly complex network models with more and more parameters and feature dimensions. Thirdly, many business operations require training a high-quality deep learning model within a short period. Fourthly, in scenarios such as underground drilling and real-time pressure prediction, it is often necessary to adjust the trained deep learning model promptly based on real-time results from the field. These real-world complexities place higher demands on the efficient training of deep learning models. Summary of the Invention
[0003] In view of the above problems, the purpose of this invention is to provide a deep learning training method and apparatus based on GPU clusters.
[0004] In a first aspect, embodiments of the present invention provide a deep learning training method based on a GPU cluster, comprising:
[0005] A GPU cluster consisting of a host and multiple GPU nodes; the multiple GPU nodes form a topology ring;
[0006] The host is used to allocate multiple training batches of the same pre-trained deep learning model to multiple corresponding GPU nodes, and broadcast the initial parameter values of the model to each GPU node so that the multiple GPU nodes can execute the training process of the model in parallel; and to save the parameter values of the model determined after training.
[0007] The GPU node is used to receive a single training batch allocated by the host and the initial parameter values of the model; execute the training process of the model; send the corresponding gradient data to the right neighbor and receive the corresponding gradient data from the left neighbor in the topology ring according to the calculated gradient data, and realize gradient data sharing on multiple GPU nodes through multiple loops, and determine the update parameters of the model; and send the parameter values of the model determined after training to the host.
[0008] In one embodiment, the GPU node is further configured to divide the calculated gradient data into multiple blocks, such that the total number of GPU nodes is the same as the total number of blocks on each GPU node; and to number the GPU nodes and the blocks on the GPU nodes, such that blocks with the same number on the multiple GPU nodes correspond to each other.
[0009] In one embodiment, the multiple cyclic transmissions include a first communication phase and a second communication phase;
[0010] The GPU node number and the block number are both j, where j is a natural number from 1 to N.
[0011] The GPU node is configured to perform a first communication phase and a second communication phase based on the GPU node number and the block number on the GPU node.
[0012] In one embodiment, during the first communication phase, N-1 operations are performed; the GPU node performs gradient data communication according to the i-th execution process it is in.
[0013] In one embodiment, when i=1, the j-th GPU node is used to send a copy of the j-th block to the right neighbor block with the corresponding number for accumulation;
[0014] When i≠1, the j-th GPU node is used to send the replica on the ji-th block to the block with the corresponding neighbor number for accumulation.
[0015] In one embodiment, during the second communication phase, N-1 operations are performed; the GPU node performs gradient data communication according to the i-th execution process it is in.
[0016] In one embodiment, when i=1, the j-th GPU node is used to send a copy of the (j+1)-th block to the right neighbor block with the corresponding number to overwrite it;
[0017] When i≠1, the j-th GPU node is used to send a copy of the j-th block to the right neighbor block with the corresponding number to overwrite it.
[0018] In one embodiment, after completing the second communication phase, the GPU node is further configured to average the gradient data across its different blocks and update the parameters of the model based on the average value.
[0019] In one embodiment, the host is further configured to schedule the training process of the same pre-trained deep learning model to various GPU nodes.
[0020] Secondly, embodiments of the present invention provide a method for training a deep learning model using the above-described system, comprising:
[0021] The host will allocate multiple training batches of the same pre-trained deep learning model to multiple corresponding GPU nodes, and broadcast the initial values of the model's parameters to each GPU node, so that the multiple GPU nodes can execute the training process of the model in parallel.
[0022] The GPU node receives a single training batch allocated by the host and the initial parameter values of the model; executes the training process of the model; based on the calculated gradient data, it sends the corresponding gradient data to its right neighbor and receives it from its left neighbor in the topology ring, and achieves gradient data sharing on multiple GPU nodes through multiple loops, and determines the updated parameters of the model; and sends the parameter values of the model determined after training to the host.
[0023] The host receives and saves the parameter values of the model sent by the GPU node.
[0024] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:
[0025] This invention provides a deep learning training system based on a GPU cluster, comprising a host and a GPU cluster consisting of multiple GPU nodes forming a topological ring. Addressing the characteristics and requirements of exploration and development data acquired in the petroleum physical exploration field, such as geological, seismic, and well logging data, which are characterized by large volumes, diverse features, heterogeneous data from multiple sources, high-quality pre-trained models, and the need for real-time deployment in certain scenarios, this invention distributes multiple training batches of the same pre-trained deep learning model to multiple corresponding GPU nodes. This allows multiple GPU nodes to execute the model's training process in parallel, providing good scalability and operability. By simply increasing the number of GPU nodes, model performance can be significantly improved and the model training process accelerated. In this embodiment, each GPU node executes the model's training process, and through multiple loops, gradient data sharing and model update parameters are determined across multiple GPU nodes, greatly improving the model's training efficiency.
[0026] Furthermore, through the first and second communication phases, the GPU nodes achieve gradient data transfer and sharing among multiple GPU nodes. This avoids the data congestion that occurs when the gradient data from multiple GPU nodes is first aggregated and sent to the host, and then the host broadcasts the model parameters determined by the host to each GPU node. This increases the smoothness of data communication and transmission, and improves the training efficiency of the model.
[0027] Furthermore, the host schedules the training process of the same pre-trained deep learning model to various GPU nodes, which meet the fine-grained resource scheduling strategy, thereby improving the utilization of the GPU during the model training process.
[0028] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0029] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0030] Figure 1 This is a schematic diagram of the structure of a deep learning training system based on a GPU cluster in an embodiment of the present invention;
[0031] Figure 2 This is a schematic diagram of the structure of a GPU cluster based on the K8s orchestration platform in an embodiment of the present invention;
[0032] Figure 3 This is a technical flowchart of the filtering node in the GPU cluster in an embodiment of the present invention;
[0033] Figure 4 This is a flowchart illustrating the technical process of scheduling the training process of a pre-trained deep learning model to various GPU nodes in an embodiment of the present invention.
[0034] Figure 5 This is a schematic diagram of the first communication stage in an embodiment of the present invention;
[0035] Figure 6 This is a schematic diagram of the second communication stage in an embodiment of the present invention;
[0036] Figure 7 This is a schematic diagram of a conventional communication method in an embodiment of the present invention;
[0037] Figure 8a This is a schematic diagram of the layer-by-layer amplitude slice results of the seismic data volume in an embodiment of the present invention;
[0038] Figure 8b This is a schematic diagram of the river channel data volume slice results predicted by the deep learning model in an embodiment of the present invention;
[0039] Figure 8c This is a schematic diagram of the similarity slicing results of the seismic data volume in an embodiment of the present invention.
[0040] Explanation of reference numerals in the attached figures:
[0041] 1. Host; 2. GPU node; 3. Topology ring. Detailed Implementation
[0042] This invention provides a deep learning training method and apparatus based on a GPU cluster. While exemplary embodiments of this disclosure are shown in the accompanying drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this disclosure and to fully convey the scope of this disclosure to those skilled in the art.
[0043] This invention provides a deep learning training system based on a GPU cluster, comprising:
[0044] A GPU cluster consisting of a host and multiple GPU nodes; the multiple GPU nodes form a topology ring;
[0045] The host is used to allocate multiple training batches of the same pre-trained deep learning model to multiple corresponding GPU nodes, and broadcast the initial parameter values of the model to each GPU node so that the multiple GPU nodes can execute the training process of the model in parallel; and to save the parameter values of the model determined after training.
[0046] The GPU node is used to receive a single training batch allocated by the host and the initial parameter values of the model; execute the training process of the model; send the corresponding gradient data to the right neighbor and receive the corresponding gradient data from the left neighbor in the topology ring according to the calculated gradient data, and realize gradient data sharing on multiple GPU nodes through multiple loops, and determine the update parameters of the model; and send the parameter values of the model determined after training to the host.
[0047] To address the drawback of long training times caused by the large amount of training data and numerous feature dimensions in the intersection of petroleum geophysical exploration and machine learning, this invention provides a deep learning training system based on a GPU cluster, which greatly improves model training efficiency.
[0048] Reference Figure 1 For example, a GPU cluster consists of a host 1 and three GPU nodes 2, forming a topology ring 3. Each GPU node 2 has a left neighbor and a right neighbor, and can only receive data from its left neighbor and send data only to its right neighbor. In summary, the host 1 of the GPU cluster is used to manage the entire GPU cluster, while the multiple GPU nodes 2 are used to run the training process of pre-trained deep learning models.
[0049] In one embodiment, the GPU cluster is built on a cloud-native workload-based Kubernetes container orchestration platform.
[0050] This Kubernetes container orchestration platform can be used for container scheduling, automated deployment, and container scaling. (See reference...) Figure 2 A GPU cluster based on the Kubernetes container orchestration platform consists of a host and multiple nodes. The host contains an API server, scheduler, controller, and Etcd. Each node contains a Kubelet, a Kube agent, and multiple Pods, each containing containers such as Container 1, Container 2, etc. On the host, the API server serves as the sole entry point for resource operations; other components on the host need to call the interfaces provided by the API server to implement their respective functions. The scheduler, acting as the scheduler for each node, binds an input Pod to a node according to user requirements. On each node, the Kubelet acts as the node's monitor and communicator with the host. The Pod is the smallest unit managed by the Kubernetes container orchestration platform.
[0051] In one embodiment, the host in the GPU cluster is also used to schedule the training process of the same pre-trained deep learning model to various GPU nodes.
[0052] For example, in a GPU cluster, different types of applications have different requirements for computing resources, and different nodes provide different types of computing resources. Each node has multiple GPUs. Through fine-grained resource scheduling strategies, the GPU node that meets the needs of the model training process is searched, and the model training process is bound to the process of this GPU node. First, refer to... Figure 3 The system categorizes multiple nodes that provide computing resources and Pods that request computing resources by type. Based on whether GPUs are provided, multiple nodes are divided into CPU nodes and GPU nodes. Based on whether GPUs are requested, Pods are divided into CPU-type Pods and GPU-type Pods.
[0053] Next, two new extended resources are defined: the first is GPU memory (gpu-mem), and the second is the number of GPUs (gpu-count). These two scalar resources are used to describe the vector resources, and combined with this resource, a working mechanism supporting fine-grained GPU sharing is established. The specific workflow is as follows:
[0054] 1) Register resource information. The GPU Share Device Plugin uses the nvml library to query the number of GPUs and the video memory of each GPU. It then uses ListAndWatch() to report the total video memory of the node's GPUs (number of GPUs × video memory) as an extended resource to the node's Kubelet. The Kubelet further reports this to the host's API server. For example, refer to... Figure 4If each node contains two GPUs, and each GPU contains 16276 MiB of video memory, from the user's perspective, the total GPU resources of the node are 16276 × 2 = 32552 MiB of video memory; therefore, these two extended resources on the node are reported.
[0055] 2) Optimize the scheduling process. After completing all filtering actions, the Kubernetes scheduler calls the `filter` method in the newly written GPU extended scheduling module. This is because the default Kubernetes scheduler, when calculating extended resources, can only determine whether the total amount of resources meets the user's demand for idle resources, but cannot specifically determine whether the demand is met on a single GPU. Therefore, the GPU extended scheduling module is needed to check whether a single GPU contains available resources. When a node that meets the conditions is found, the `bind` method of the GPU extended scheduling module is delegated to bind the node to the Pod; however, if it is found that none of the GPUs on the node meet the conditions, binding will not occur. For example, refer to... Figure 4 The GPU cluster contains three nodes: N1, N2, and N3. Each node contains two GPUs. When a user requests 8138 MiB of GPU memory, the default scheduler scans all nodes. The scan finds that node N1 has 16276 × 2 - 16276 - 12207 = 4069 MiB of remaining resources, which does not meet the resource requirement. Nodes N2 and N3 both have 8138 MiB of remaining resources, which, according to traditional scheduling methods, meets the default scheduler's conditions. However, according to the secondary filtering search results from the GPU extended scheduling module, although node N2 has 8138 MiB of available resources, its two GPUs, GPU0 and GPU1, each only have 4069 MiB of available resources, which is insufficient to meet the requirement of 8138 MiB of memory for a single GPU. Node N3, on the other hand, has all its available resources belonging to GPU0, thus meeting the scheduling requirement for a single GPU. Therefore, node N3 is the GPU node selected by the GPU extended scheduling module that meets the user's needs, such as supporting the deep learning model training process.
[0056] The system provided in this invention enables multiple GPU nodes to execute the training process of a deep learning model in parallel, with each GPU node allocated only one training batch of the training sample set. During the training process, a loss function is determined after forward propagation, gradient data is calculated based on the loss function, the model parameters are updated, and backpropagation is performed to complete the training process. Specifically, the updated model parameters are calculated based on the average value of the gradient data shared across all GPU nodes. The communication strategy between the GPU nodes is described below.
[0057] In one embodiment, the GPU node is further configured to divide the calculated gradient data into multiple blocks such that the total number of GPU nodes is the same as the total number of blocks on each GPU node; and to number the GPU nodes and the blocks on the GPU nodes such that blocks with the same number on multiple GPU nodes correspond to each other.
[0058] In one embodiment, the multiple cyclic transmissions include a first communication phase and a second communication phase; the GPU node number and the block number are j, where j is a natural number from 1 to N; the GPU node executes the first communication phase and the second communication phase according to the GPU node number and the block number on the GPU node.
[0059] In one embodiment, during the first communication phase, N-1 operations are performed; the GPU node performs gradient data communication based on the i-th execution process it is in.
[0060] In one embodiment, when i = 1, the j-th GPU node is used to send a copy of the j-th block to the right neighbor block with the corresponding number for accumulation; when i ≠ 1, the j-th GPU node is used to send a copy of the ji-th block to the neighbor block with the corresponding number for accumulation.
[0061] In one embodiment, during the second communication phase, N-1 operations are performed; the GPU node performs gradient data communication based on the i-th execution process it is in.
[0062] In one embodiment, when i = 1, the j-th GPU node is used to send a copy of the (j+1)-th block to the right neighbor's corresponding numbered block for overwriting; when i ≠ 1, the j-th GPU node is used to send a copy of the j-th block to the right neighbor's corresponding numbered block for overwriting.
[0063] Reference Figure 1 , Figure 5 and Figure 6 As shown, for example, a GPU cluster contains 3 GPU nodes, with GPUs GPU1, GPU2, and GPU3 on each node. The communication strategy is that any GPU can only receive data from its left neighbor and send data to its right neighbor, and all GPUs participate in computation. The communication process includes a first communication phase ( Figure 5 ), and the second communication phase ( Figure 6 ).
[0064] In the first communication phase, refer to Figure 5Taking GPU1 as an example, the gradient data calculated after backpropagation is divided into blocks a1, b1, and c1; for example, the corresponding blocks b2 and b3 in GPU2 and GPU3 are respectively. According to the communication strategy in the above embodiment, a total of two operations are performed. After the first communication phase, each GPU node accumulates the gradient data of one block from the other GPU nodes. That is, each GPU node has a block containing the local final shared result. These blocks are the sum of the corresponding blocks in all GPU nodes, such as... Figure 5 The depth block is shown in c.
[0065] In the second communication phase, refer to Figure 6 According to the communication strategy in the above embodiment, the same two operations are performed. After the second communication stage, each GPU node contains the gradient data of all blocks of the final shared result in each GPU node.
[0066] In one embodiment, after completing the second communication phase, the GPU node is further configured to average the gradient data across its different blocks and update the parameters of the model based on the average value.
[0067] This invention employs a data-parallel approach to model training, making the exchange of gradient data between GPU nodes unavoidable. When the amount of data in the training dataset is relatively small, traditional communication methods are used, referring to... Figure 7 After the gradient data is determined on each GPU node, the gradient data from each GPU node is aggregated and sent to the host. The host then calculates the average of these gradient data and the model parameter values before sending them back to the GPU nodes. However, for the massive training samples in real-world engineering projects, the bandwidth limitations of the communication channels between GPUs will prolong the gradient data transmission time. Therefore, in this embodiment of the invention, multiple GPU nodes form a topological ring, referring to... Figure 1 , Figure 5 and Figure 6 In this model training, gradient data is circulated among the individual GPU nodes and is not aggregated on the host machine. This means the GPUs on the host machine do not participate in the transfer and computation of gradient data. Since this communication is not limited by the maximum bandwidth of a single GPU, communication efficiency is significantly improved, thereby increasing the model's training efficiency.
[0068] The communication efficiency of the topology ring proposed in this embodiment is briefly analyzed below. In both the first and second communication stages, each GPU node simultaneously sends and receives gradient data. Assuming the total amount of gradient data calculated on each GPU is k, the corresponding gradient data amount for each block after average division is k / N. After N-1 cycles of sending and receiving gradient data, there is a communication capacity of (N-1)×(K / N). Overall, after two communication stages, the total amount of data transmitted by each GPU node is:
[0069] V c =2k(N-1) / N;
[0070] Where k is the size of the gradient data calculated by the GPU node, and N is the number of GPUs (or GPU nodes).
[0071] As can be seen from the above formula, as the number of GPU nodes gradually increases, the amount of communication data for a single GPU is approximately equal to 2k, and the relationship with the number of GPU nodes N becomes increasingly smaller. Compared with traditional communication methods, the communication strategy provided in this embodiment of the invention not only avoids congestion during gradient data transmission between GPU nodes, improving communication efficiency, but also enhances scalability by increasing the number of GPU nodes.
[0072] This invention provides a method for training a deep learning model using the above-described system, comprising:
[0073] The host will allocate multiple training batches of the same pre-trained deep learning model to multiple corresponding GPU nodes, and broadcast the initial values of the model's parameters to each GPU node, so that the multiple GPU nodes can execute the training process of the model in parallel.
[0074] The GPU node receives a single training batch allocated by the host and the initial parameter values of the model; executes the training process of the model; based on the calculated gradient data, it sends the corresponding gradient data to its right neighbor and receives it from its left neighbor in the topology ring, and achieves gradient data sharing on multiple GPU nodes through multiple loops, and determines the updated parameters of the model; and sends the parameter values of the model determined after training to the host.
[0075] The host receives and saves the parameter values of the model sent by the GPU node.
[0076] Before training a deep learning model using the training method improved in this embodiment of the invention, a suitable deep learning framework for the GPU cluster needs to be configured. For example, when building a Kubernetes container orchestration platform, the Horovod deep learning framework can be used. Using the Horovod deep learning framework requires only simple and unified configuration, which can solve the problems of large code modifications and error-prone debugging when integrating different types of deep learning architectures onto a distributed parallel training platform. The specific configuration process is as follows:
[0077] 1) Initialize Horovod using hvd.init();
[0078] 2) Bind a GPU node to a deep learning model training process. The code is: config = tf.ConfigProto(), config.gpu_options.visible_device_list = str(hvd.local_rank());
[0079] 3) The optimizer is encapsulated to support the Horovod deep learning framework. The implementation code is opt = hvd.DistributedOptimizer(opt);
[0080] 4) The initial values of the parameters of the deep learning model are broadcast to each GPU node through the host. The code is hooks = [hvd.BroadcastGlobalVariablesHook(0)];
[0081] 5) Save the checkpoints in the host's database Etcd. The code is: checkpoint_dir = ' / tmp / train_logs' if hvd.rank() == 0 else None. Checkpoints include model parameters (weights and biases), optimizer status, and other relevant training information.
[0082] The Horovod deep learning framework requires only minor modifications and does not involve other cumbersome operations in the GPU cluster, thus improving the convenience of deep model training.
[0083] Finally, taking the training system and training method provided in this embodiment of the invention as an example, we will illustrate their application in a real oil exploration project. The exploration area of a certain work area in a basin is 546 km². 2The spatial sampling interval was 25m, the temporal sampling interval was 4ms, and the post-stack 3D seismic data volume was approximately 3300M. A deep learning model was used to predict river channel development. The total sample dataset contained 11G of labeled data, and the model had 45 million parameters. First, the model was trained on an NVIDIA A100 machine for 50 epochs, taking approximately 140 hours to complete. Then, the system provided in this embodiment of the invention was used to train the model. The GPU cluster included three A100 GPU nodes, resulting in an overall training time of approximately 48 hours, significantly improving training efficiency.
[0084] Reference Figure 8a , Figure 8b and Figure 8c The figure shows a comparison of different properties along the slice of the target layer. Figure 8a This displays slices of seismic data volume along the layers; Figure 8b This shows slices of the predicted channel data volume along the layers; Figure 8c The similarity slices of the seismic data volume are displayed. Because both channels and faults exhibit discontinuities in the similarity attribute, they are difficult to clearly separate. Furthermore, the similarity attribute only reflects the channel location, not the amplitude variations within the channel. However, the embodiments of the present invention not only accurately depict the channel location (as shown by the green ellipse) but also clearly reflect the amplitude variations within the channel (see...). Figure 8b (As shown).
[0085] Obviously, those skilled in the art can make various modifications to this invention without departing from its spirit and scope. Therefore, if these modifications fall within the scope of the claims and their equivalents, this invention is also intended to include these modifications.
Claims
1. A deep learning training system based on a GPU cluster, characterized in that, include: A GPU cluster consisting of a host and multiple GPU nodes; the multiple GPU nodes form a topology ring; The host is used to allocate multiple training batches of the same pre-trained deep learning model to multiple corresponding GPU nodes, and broadcast the initial values of the model's parameters to each GPU node, so that the multiple GPU nodes can execute the training process of the model in parallel. And save the parameter values of the model determined after training; The GPU node is used to receive a single training batch allocated by the host and the initial parameter values of the model; Execute the training process of the model; Based on the calculated gradient data, the corresponding gradient data is sent to the right neighbor and received from the left neighbor in the topological ring. After multiple loops, gradient data sharing is achieved on the multiple GPU nodes, and the update parameters of the model are determined. And send the parameter values of the model, determined after training, to the host.
2. The system as described in claim 1, characterized in that, The GPU node is further configured to divide the calculated gradient data into multiple blocks, such that the total number of GPU nodes is the same as the total number of blocks on each GPU node; and to number the GPU nodes and the blocks on the GPU nodes, such that blocks with the same number on the multiple GPU nodes correspond to each other.
3. The system as described in claim 2, characterized in that, The multiple cyclic transmissions include a first communication phase and a second communication phase. The GPU node number and the block number are both j, where j is a natural number from 1 to N. The GPU node is configured to perform a first communication phase and a second communication phase based on the GPU node number and the block number on the GPU node.
4. The system as described in claim 3, characterized in that, In the first communication phase, N-1 operations are performed; the GPU node performs gradient data communication according to the i-th execution process it is in.
5. The system as described in claim 4, characterized in that, When i=1, the j-th GPU node is used to send a copy of the j-th block to the right neighbor block with the corresponding number for accumulation; When i≠1, the j-th GPU node is used to send the replica on the ji-th block to the block with the corresponding neighbor number for accumulation.
6. The system as described in claim 5, characterized in that, In the second communication phase, N-1 operations are performed; the GPU node performs gradient data communication according to the i-th execution process it is in.
7. The system as described in claim 6, characterized in that, When i=1, the j-th GPU node is used to send a copy of the (j+1)-th block to the right neighbor block with the corresponding number to overwrite it; When i≠1, the j-th GPU node is used to send a copy of the j-th block to the right neighbor block with the corresponding number to overwrite it.
8. The system as described in claim 7, characterized in that, After completing the second communication phase, the GPU node is also used to average the gradient data on its different blocks and update the parameters of the model based on the average value.
9. The system according to any one of claims 1-8, characterized in that, The host is also used to schedule the training process of the same pre-trained deep learning model to various GPU nodes.
10. A method for training a deep learning model using the system according to any one of claims 1-9, characterized in that, include: The host will allocate multiple training batches of the same pre-trained deep learning model to multiple corresponding GPU nodes, and broadcast the initial values of the model's parameters to each GPU node, so that the multiple GPU nodes can execute the training process of the model in parallel. The GPU node receives a single training batch allocated by the host and the initial parameter values of the model; Execute the training process of the model; Based on the calculated gradient data, the corresponding gradient data is sent to the right neighbor and received from the left neighbor in the topological ring. After multiple loops, gradient data sharing is achieved on the multiple GPU nodes, and the update parameters of the model are determined. And send the parameter values of the model, determined after training, to the host; The host receives and saves the parameter values of the model sent by the GPU node.