Distributed training framework construction method based on NFDT and distributed training method and system

Abstract

The invention discloses a distributed training framework construction method based on NFDT and a distributed training method and a system. The distributed training method comprises the following stepsthat a single-machine user stores a data set and codes in a cluster network and generates a unique hash value; a distributed training module receives a hash value corresponding to a task demand and downloads a data set and a standalone version code from a cluster network; the distributed training module generates a distributed training framework by utilizing the standalone version code; and the distributed training framework calls the computing resources to perform distributed training. According to the method, a user only needs to upload a data set and a stand-alone training code of the user, the training task is compressed into the cluster in a decentralized storage mode, the corresponding training equipment in the cluster decompresses the training task according to the received hash value, and training safety and data reliability are guaranteed. The NFDT framework performs model interpretation and analysis on a model file generated in a standalone version training process, and a training function from a standalone version code structure to a distributed mode is achieved.

Description

technical field [0001] The present invention relates to the field of Internet technology, in particular to a distributed training framework construction method based on NFDT (New Framwork for Distributed Tensorflow, distributed Tensorflow new framework), a distributed training method and a system. Background technique [0002] When using the distributed Tensorflow (tensor flow) framework to build a distributed training platform system, it is found that there are two main problems: [0003] First, when performing distributed Tensorflow training, users need to modify the stand-alone code to distributed code, which increases the difficulty for users. Although it is the Horvovd framework launched by Uber, or the estimator API or Tensorflow2.0 launched by Google , are all underlying packages of Tensorflow, which greatly reduces the difference between the distributed framework and the stand-alone version. However, for many beginners, they still need to be familiar with new APIs o...

AI Technical Summary

Problems solved by technology

However, for many beginners, they still need to be familiar with new APIs or frameworks before they can be deployed into a code structure that can perform distributed training, and the effect is still not ideal.

From the perspective of the overall implementation process, implementing distributed training on the stand-alone framework requires a lot of framework rewriting and cluster definition. Compared with the stand-alone version, such a distributed framework is more complex and difficult to define

[0004] The second is that when a centralized distributed storage dataset is used, if a server is down, the data will be lost, which will affect the overall training efficiency and data security.

Therefore, if the data is placed in a centralized place, the user's data security will not be guaranteed

In addition, centralized storage leads to very high bandwidth transmission costs. For example, in the field of computer vision, image data sets are often hundreds of gigabytes or thousands of gigabytes. Whether uploading data or downloading data, bandwidth will be consumed. Bandwidth cannot be ignored and compared with storage costs It is expensive, and centralized storage services, in order to ensure that users in various places have a good experience, they will deploy data centers on the backbone network, and the backbone network is very expensive in any country

Images