Automated task parallel method suitable for distributed machine learning and system thereof

Abstract

The invention provides an automated task parallel method suitable for distributed machine learning and a system thereof. The method and the system solve defects of a programming interface in existing distributed machine learning, and tight coupling of system data access behaviors and application logic caused by just providing a reading-writing interface of key value pairs. The defect intensifies network bandwidth resource competition in distributed cluster, and causes that programming personnel is difficult to perform parallelization on a task. The system comprises a work node module, a service node module, a host node module, a tensor module, a scheduling module, a message tracking module, a stage module, a stage group module, and an executing engine module. Through providing higher-level programming abstraction, the system decouples logic of reading-writing access behaviors and an application program. In operation, the system firstly dynamically partitions tasks according to the load condition of a service node, and then machine learning tasks are automatically executed in a parallel manner, so as to greatly reduce burden of programming personnel to compile high-concurrency machine learning applications.

Description

technical field [0001] The invention belongs to the cross-technical field of distributed computing and machine learning, and in particular relates to a parallel method and system for automatic tasks suitable for distributed machine learning. Background technique [0002] As a traditional method of mining data value, machine learning algorithms are widely used in fields such as natural language processing, text analysis, speech recognition, automatic driving of motor vehicles, and bioinformatics. With the advent of the era of big data, the value of data is becoming more and more prominent, especially the commercial value contained in it, so machine learning has been valued. However, as the scale of data and the corresponding model parameters that need to be learned become larger and larger, a single computing node cannot meet the needs of large-scale machine learning due to its limited memory resources, computing resources, and memory access bandwidth resources. Distributing...

AI Technical Summary

Problems solved by technology

The above method and system can effectively solve the network transmission bottleneck problem in the existing distributed machine learning system and improve the system concurrency, thereby improving the overall performance of the system

Images