Distributed data processing system, distributed computing task deployment system and method

Abstract

The present disclosure provides a distributed data processing system, a distributed computing task deployment system and method. The distributed data processing system is used for parallel processing of data on the plurality of computing devices, each computing device includes a forward data processing component and a backward data processing component, wherein at least one computing device includes other computing devices A model parameter component that does not have and a model parameter update component corresponding to one of the model parameter components, and the model parameter component uses a set of data to be processed in parallel through its corresponding broadcast component. Input to the broadcast component of other computing devices, and the model parameter update component obtains the corresponding global gradient value from the corresponding gradient convergence component to perform update processing.

Description

technical field [0001] The present disclosure relates to a data processing technology, and more specifically, the present disclosure relates to a distributed data processing system, a distributed computing task deployment system and a method thereof. Background technique [0002] With the development of machine learning and the gradual deepening of artificial neural network research, the concept of deep learning has been widely concerned and applied. Deep learning is a special kind of machine learning. It uses a network hierarchical structure to express the learning objects, combines simple concepts into abstract concepts, and realizes abstract concept expression through simple concept calculations. At present, deep learning has made great progress in the fields of image recognition, speech recognition and natural language processing. Deep learning involves a large number of model parameters, resulting in a huge amount of calculation, and the scale of training data is large...

AI Technical Summary

Problems solved by technology

[0006] The computing devices commonly used in deep learning usually include CPU, GPU, FPGA, and ASIC (application-specific integrated circuit), etc. GPU, FPGA, and ASIC can be collectively referred to as data acceleration processing devices. High memory bandwidth, but its memory capacity often has problems such as limited memory, high unit price, and difficulty in expansion. If the memory capacity of data acceleration processing equipment is blindly increased to meet the training of large-scale models, it will bring high costs to enterprises.

Obviously, the repeated calculations and memory in the existing data parallelism come from distributed training parameters and the Optimizer are also distributed (Distributed) placed on each computing device, which will undoubtedly improve the memory resources of computing devices. On the other hand, it will inevitably lead to the transmission overhead of parameter synchronization between devices

It is a huge waste for current computing devices, especially GPUs with expensive memory resources.

Images