Storage and optimization method of MPI (Message Passing Interface) parallel data

Abstract

The invention relates to a storage and optimization method of the parallel data of an MPI (Message Passing Interface). The method comprises the following steps of: (1) calling an MPI function relative to a target filet to be written, i.e. opening the file, writing the file, closing the file and encapsulating in a same-parameter and different-name type; (2) acquiring an integral multiple distance X from a current operation data offset to the next fsBlkSize as well as an integral multiple distance Y from the tail offset formed by offset+count to the last fsBlkSize according to a data block fsBlkSize of the current file system, the offset of the current operation data and a bit number (count) to be written in the file writing function encapsulated in the step (1), and acquiring different processing modes by the encapsulated file writing function according to the acquired X and Y values; and (3) substituting the encapsulated function in the step (1) for all MPI functions to be called by all target files to be written in an application program. Without any edition to the file system, the invention can ensure the result accuracy of the write operation at receivable efficiency by little edition of an application layer.

Description

technical field [0001] The invention relates to a storage optimization method, in particular to an MPI parallel data storage optimization method, belonging to the field of computer parallel processing. Background technique [0002] The parallel processing capability of computers has important application value in some fields; MPI (Message Passing Interface, Message Passing Interface) provides programmers with the most popular parallel programming environment in the world at present. Computers, workstation networks, and cluster systems are well supported. However, in practical applications, it is found that when some parallel file systems execute the MPI_File_write_at call to perform parallel write operations between multiple nodes and multiple processes on the same target file, because the first and last offsets of some process operation data are not within the size of the file system data block Integer multiples of the position will lead to inconsistency in the final write...

AI Technical Summary

Problems solved by technology

However, in practical applications, it is found that when some parallel file systems execute MPI_File_write_at calls to perform parallel write operations between multiple nodes and multiple processes on the same target file, because the head and tail offsets of some process operation data are not within the size of the file system data block Integer multiples of the position will lead to inconsistency in the final write result

[0003] To solve this problem from the parallel file system itself, it is necessary to start from the overall structure of the file system, invest a lot of manpower and material resources, and consume high costs; before each write operation at the application layer, the first and last offsets of the written data are guaranteed in advance They are all located at integer multiples of the file system data block size, and parallel write operations can be completed more efficiently. However, programmers need to pay attention to this problem at all times during their work, which virtually increases the amount of programming tasks and weakens the flexibility of the program.

Images