Distributed machine learning-oriented data transmission method and system

Abstract

The invention discloses a distributed machine learning-oriented data transmission method and system. The method comprises the steps that to-be-transmitted data is acquired, at least two original datastreams are formed based on the to-be-transmitted data, wherein each original data stream comprises at least two original data packets, a target transmission rate corresponding to each original data stream is acquired based on a data center network topology structure, based on the target transmission rate corresponding to each original data stream, original data packets are synchronously sent in the at least two original data streams to a corresponding receiver server through a switch, it is monitored in real time whether a congestion notification packet sent by the receiver server through theswitch can be received within a preset time period so that a congestion monitoring result is obtained, a transmission rate is adjusted based on the congestion monitoring result, and the target transmission rate corresponding to each original data stream is updated. The method can effectively balance the transmission time difference in the data transmission process caused by network congestion ina data center network topology structure.

Description

technical field [0001] The present invention relates to the technical field of distributed computing, in particular to a data transmission method and system for distributed machine learning. Background technique [0002] With the emergence of big data and efficient computing resources, machine learning, one of the rapidly developing fields of artificial intelligence (AI), is used in many application scenarios (such as natural language processing, image recognition, speech recognition and machine translation, etc.) have achieved major breakthroughs. However, as the complexity of tasks increases, the size of data and deep learning models becomes increasingly large. The simple way of using a single machine plus GPU computing can no longer meet the computing power and storage space required by large-scale deep learning models and large-scale training data sets. In order to improve the training efficiency of deep learning models, researchers use distributed technology to perfor...

AI Technical Summary

Problems solved by technology

[0005] The present invention provides a data transmission method and system oriented to distributed machine learning, so as to solve the problem of large differences in data stream transmission time due to network congestion in the current data transmission process oriented to distributed machine learning

Images