A distributed depth learning method and system based on a data parallel strategy

Abstract

The invention discloses a distributed depth learning method and a system based on a data parallel strategy. The system of the invention comprises a distributed computing framework Spark, a PyTorch depth learning framework, a lightweight Web application framework Flask, a pickle, a urllib2 and other related components. The Spark framework provides the functions of cluster resource management, datadistribution and distributed computing. The PyTorch Deep Learning Framework provides the interface defined by neural network and the function of upper layer training and computation of neural network;the flask framework provides parameter server functionality; the urllib2 module is responsible for providing the network communication function between the working node and the parameter server node;Pickle is responsible for serializing and deserializing the parameters in the neural network model for transmission over the network. The invention effectively combines PyTorch and Spark, decouples PyTorch and bottom distributed cluster through Spark, absorbs respective advantages, provides convenient training interface, and efficiently realizes distributed training process based on data parallelism.

Description

Technical field [0001] The invention relates to a deep learning training system, in particular to a distributed deep learning method and system based on a data parallel strategy. Background technique [0002] In recent years, with the advent of big data, and the rapid development of artificial intelligence, especially deep learning, deep neural network models trained on big data sets have achieved breakthrough improvements and extensive applications in many fields, including speech recognition and Image recognition to natural language processing, etc. Deep learning improves its ability by continuously seeking derivation and iterative updating of the model. It requires a lot of calculation and is a typical computationally intensive task. Therefore, the training process of these neural networks is very time-consuming. Although GPU (graphics processing unit) hardware technology, network model structure and training methods have made some progress in recent years, the fact that sing...

AI Technical Summary

Problems solved by technology

Although PyTorch has an interface for distributed training, its programming needs to configure node clusters, which is relatively complicated and has no cluster resource management function. When adding new nodes or adding new computing resources to a distributed cluster, the program needs to be rewritten; In terms of updating, it cannot effectively reduce the communication overhead in the network, making distributed training less effective in terms of scalability and flexibility

Images