A non-volatile memory management method and system based on mpi-io middleware

Abstract

The invention discloses a non-volatile memory management method and system based on MPI-IO middleware. The non-volatile memory management method of the invention includes that the MPI process applies to the operating system or restores the non-volatile memory in the non-volatile memory device during initialization. Volatile memory space, the MPI process releases the corresponding non-volatile memory space before the end of the process, and can use the non-volatile memory device to cache IO data, the MPI process completes the target file according to the cache metadata cached in the non-volatile memory space read or write. The invention can uniformly manage multiple non-volatile memory devices in a high-performance computing environment; cache IO data, reduce the IO overhead of programs; perform efficient temporary storage and recovery for files in large-scale computing; non-exclusive use of non-volatile memory Memory device, make full use of hardware resources; do not modify the API provided to users, easy to use.

Description

technical field [0001] The invention relates to the application technology of non-volatile memory in the field of large-scale computing, in particular to a non-volatile memory management method and system based on MPI-IO middleware. Background technique [0002] In recent years, high-performance computing has played a significant role in scientific research and engineering: large-scale scientific and engineering computing problems require the use of high-performance computer clusters for modeling and calculation. High-performance computing needs to process a large amount of data, including reading models, reading and writing checkpoint files (checkpoint files), etc. Model files and checkpoint files have a large amount of data, and the program needs to generate a large number of IO requests and serious IO loads when reading from or writing to the external memory. Due to the physical limitations of external memory devices and complex IO patterns in high-performance computing ...

AI Technical Summary

Problems solved by technology

This type of technology realizes the function of the file system and uses non-volatile memory as a high-performance persistent storage device; however, the technology is oriented to all files, lacks pertinence, and runs exclusively on non-volatile memory devices

In addition, the above two technical solutions only manage the non-volatile memory of a single node, and cannot uniformly manage the non-volatile memory devices in multiple nodes

Images