Distributed machine learning system and method of adaptive RDMA network

Abstract

Disclosed are a distributed machine learning system and method of an adaptive RDMA network. After a distributed training task is scheduled to a training cluster, the training cluster network environment is detected, and the training cluster network is adaptively selected according to the detection to be used for distributed training task communication; the optimal network environment in the training process is selected for the distributed task as much as possible, so that the efficient RDMA network is fully used for communication in the distributed training process, the communication bottleneck problem existing in distributed training task deployment in the prior art is solved, and then distributed training efficiency is improved.

Description

technical field [0001] The present invention relates to the field of distributed machine learning; in particular, it relates to a distributed machine learning system and method for adaptive RDMA network. Background technique [0002] Machine learning, especially deep learning, has seen widespread success in AI-driven services. As models become more complex, their training becomes increasingly computationally expensive. To achieve efficient and timely training, it is necessary to explore the advantages of parallel computing in distributed systems. Industry leaders such as Microsoft, Facebook, and Google have begun to try to run distributed machine learning training tasks on production clusters consisting of hundreds or thousands of servers. [0003] However, a physical cluster for distributed training with practical significance, from construction and deployment to operation and maintenance, is extremely professional, complex and even cumbersome. Applying container cloud t...

AI Technical Summary

Problems solved by technology

[0010] Therefore, even when distributed training tasks are scheduled to a dual-network container training cluster with an RDMA network, after the distributed training starts, the training applications running on each computing node of the cluster (here refers to the container / container group used for training) The program is also unable to discover and effectively use the RDMA network, and thus still unable to break through the communication bottleneck and achieve efficient training

[0011] In addition, even if distributed training tasks are deployed in a multi-network physical cluster with an RDMA network, special environment configuration parameters (environment configuration parameters with the RDMA network IP as the network connection parameter) need to be generated manually / using scripts; and manual configuration, Inevitably error-free; and not suitable for large-scale cluster deployments

[0012] It should be pointed out that it is precisely because most distributed machine learning frameworks rely too much on the network environment set and provided during deployment scheduling during the training process, and are not aware of the network type of the training cluster to be scheduled, which leads to distributed training. After the task is scheduled to the training cluster, it can only choose to use the default network (normal) or the RDMA network (customized) during the training process, instead of adaptively selecting according to the current network actual situation of the scheduled training cluster The network is used for data transmission of the training application during the training process

Images