Distributed parallel deep neural network performance evaluation method for super computer

Abstract

The invention discloses a distributed parallel deep neural network performance evaluation method for a super computer, and belongs to the field of high-performance calculation and deep neural networks. The method sequentially comprises the steps of distributed parallel granularity design, deep neural network training framework platform configuration, distributed parallel communication protocol deployment, deep neural network optimization, training test data set slicing and test mode application, and a universal test method is provided for developers. The design of the distributed parallel granularity ensures the comprehensive test of a single node and multiple nodes; the reliability of application layer deployment and bottom layer communication is ensured through coupling of a deep neuralnetwork training framework pytorch and a bottom layer distributed communication framework MPI; the single-node test of different processors and the multi-node test of different processors are helpfulfor realizing the full utilization of super computer computing resources, improving the program computing performance, shortening the neural network training time and improving the neural network training convergence rate.

Description

technical field [0001] The invention belongs to the field of high-performance computing and deep neural network, in particular to a supercomputer-oriented distributed parallel deep neural network performance evaluation method. Background technique [0002] The processors used by the Tianhe-3 prototype include FT-2000+ (FTP) and MT-2000+ (MTP). FTP includes 64 FTC662 processor cores with armv8 architecture, and the main frequency is 2.2-2.4GHZ. The 32MB secondary cache can provide 204.8GB / s memory access bandwidth, and the typical working energy consumption is about 100W; while the MTP processor, which contains a total of 128 armv8 cores, is organized into 4 super nodes, and the main frequency can reach up to 2.0GHZ, the consumption of the whole processor is 240W. The prototype machine does not support development frameworks for deep neural networks such as caffe, pytorch, tensorflow, etc., nor does it provide evaluation methods for distributed parallel deep neural networks ...

AI Technical Summary

Problems solved by technology

The prototype machine does not support development frameworks for deep neural networks such as caffe, pytorch, tensorflow, etc., nor does it provide evaluation methods for distributed parallel deep neural networks on the platform, so it is impossible to directly carry out the evaluation of corresponding distributed parallel deep neural networks, and then It is impossible to fully evaluate and effectively utilize the powerful computing power of the Tianhe-3 supercomputer

Images