Distributed machine learning system acceleration method based on network reconfiguration

Abstract

The invention relates to a distributed machine learning system acceleration method based on network reconfiguration. The method comprises the following steps that 1, a scheduler obtains a certain number of to-be-trained models from a database; 2, each training task is divided into a plurality of sub-tasks according to the position preference of each training task, and the sub-tasks are deployed toeach server; 3, the scheduling strategy manager continuously allocates GPU resources according to the difficulty degree of reaching the time fairness of each task distance; 4, the size of the TCP buffer area is dynamically adjusted according to the current network condition; step 5, the scheduling strategy manager records a corresponding result into a database according to a load operation condition; and step 6, whether to continue scheduling or not is determined according to the condition of the to-be-trained model in the database. According to the task scheduling strategy of time fairness and position sensitivity, the problems of long-tail delay and the like in the training process of the machine learning model can be avoided, and the service quality of the cluster is improved.

Description

technical field [0001] The invention relates to a method for realizing communication and load scheduling of a machine learning system, especially a method for reducing model training time by rationally optimizing scheduling in a large-scale distributed machine learning system. Background technique [0002] With the continuous development of artificial intelligence technology, massive data training and the emergence of large-scale models make single-machine model training increasingly unable to meet the performance requirements of artificial intelligence applications. Therefore, some scholars have proposed distributed machines such as data parallelism and model parallelism. Learning techniques to improve the training speed of the model. Data parallelism has been extensively researched and optimized for performance. However, as the scale of the model increases, for large models that cannot be accommodated in the memory of a single machine, model parallelism is the only way to ...

AI Technical Summary

Problems solved by technology

[0009] Since the schedulers used in the current distributed machine learning system are designed for big data processing tasks, these schedulers do not fully consider the characteristics of the machine learning task load when scheduling, which often causes problems such as long tail delays in model training.

Images