Method and system for optimizing parallel I/O (input/output) by reducing inter-progress communication expense

Abstract

The invention discloses a method for improving parallel I / O (input / output) efficiency by reducing inter-progress communication based on progress affinity. The method comprises the following steps of firstly, analyzing a parallel I / O program using Collective I / O as mainstream, and collecting and calculating the node information of a cluster machine and the configuration information of an MPI (message passing interface) program; enabling the system to calculate the matching result of a possible machine node and an agent progress through pre-processing, and determining an optimum matching strategy through a property predicting module; finally, writing the pre-processed matching strategy into a configuration file, namely a machine file. The method has the advantages that after indicating by the experiment result, the configuration is simple, and the optimum progress distributing scheme is determined for the running of the program through the simple and quick pre-processing under the condition of not modifying the original program code, so the inter-progress communication expense is reduced, and the parallel I / O property is improved; the problem of lower parallel I / O efficiency in the existing high-property calculating field is solved, and the property is obviously improved by reducing the inter-progress data communication expense under the condition of no influence on the other deployment of the existing system.

Description

technical field [0001] The invention belongs to the field of I / O subsystems in high-performance computing, and more specifically relates to a method and system for improving parallel I / O efficiency based on reducing inter-process communication overhead. Background technique [0002] In recent years, big data processing has continued to heat up in the field of scientific computing. On the one hand, it reflects the growing demand for big data processing in the field of scientific computing. At the same time, it can also be seen that in the field of traditional high-performance , many scientific computing applications are changing from computation-intensive to data-intensive, and high-performance computing is facing new challenges brought by big data. [0003] The Data Transmission Parallel I / O Interface (Message-Passing Interface, MPI-IO) standard formulates a parallel I / O interface—aggregated I / O (Collective I / O) interface, which provides scientific computing in the era of bi...

AI Technical Summary

Problems solved by technology

In this way, the data request in the file domain of the proxy process running on a certain machine node may come from other machine nodes. When the proxy process exchanges data with other processes, some cross-machine node data communication overhead will be generated.

Images