Method for resetting data transmission network in distributed training task training process

Abstract

The invention provides a method for resetting a data transmission network in a distributed training task training process, which comprises the following steps of: after a distributed training task isscheduled to a training cluster with an RDMA (Remote Direct Memory Access) network, respectively acquiring an RDMA network IP (Internet Protocol) of each sub-task computing node, and determining a master node (the rest is slave nodes) from each sub-task computing node; wherein the collection module is used for collecting training cluster RDMA network information; and after the collection is completed, updating the environment configuration parameters of each subtask according to the cluster RDMA network information so as to realize communication according to the updated environment configuration parameters in the distributed training process and achieve the purpose of resetting the data transmission network to the RDMA network.

Description

technical field [0001] The present invention relates to the field of distributed machine learning and the field of container cloud technology; in particular, it relates to a method for resetting a data transmission network during the training process of a distributed training task. Background technique [0002] Machine learning, especially deep learning, has seen widespread success in AI-driven services. As models become more complex, their training becomes increasingly computationally expensive. To achieve efficient and timely training, it is necessary to explore the advantages of parallel computing in distributed systems. Industry leaders such as Microsoft, Facebook, and Google have begun to try to run distributed machine learning training tasks on production clusters consisting of hundreds or thousands of servers. [0003] However, a physical cluster for distributed training with practical significance, from construction and deployment to operation and maintenance, is e...

AI Technical Summary

Problems solved by technology

[0010] To sum up, since the RDMA network information (such as RDMA network IP) cannot be obtained in advance, even if the current container training cluster has an RDMA network, the training application running on each computing node (that is, the container / container group used for training) will not Inability to discover and efficiently use RDMA networks

[0011] In addition, although some methods mentioned above can also implement training tasks deployed in RDMA network physical clusters, manual configuration (the RDMA network IP is the environment configuration parameter of the network connection parameters), it is inevitable that there will be no mistakes; even if there is only a little error in the configuration, The entire training cluster is also unable to provide an effective RDMA network for data transmission in the training task, which leads to the failure of the entire deployment

Images