Multilevel nested data transmission method and system matched with high-performance computer structure

Abstract

The invention discloses a multilevel nested data transmission method and a multilevel nested data transmission system matched with a high-performance computer structure. The system comprises a plurality of node communication components running in parallel at a plurality of computing nodes and a plurality of data transmission scheduling modules running in parallel on a plurality of CPUs (central processing unit) of the computing nodes, wherein each node communication component runs at one computing node, and comprises a communication description management module for storing and managing description about data transmission operation to be performed by the corresponding computing node and an inter-node data transmission execution module for managing and executing a data transmission request between the corresponding computing node and the other computing nodes; each data transmission scheduling module runs on one CPU, and is used for creating, scheduling and executing a communication-related transaction in the corresponding computing node. The method and the system can be matched with a universal processor multilevel nested system structure, and the data transmission performance can be improved.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a multi-level nested data transmission method and system matching the structure of a high-performance computer. Background technique [0002] Numerical simulation parallel applications usually require data communication between different parallel computing units: send some data in this computing unit to other computing units, and receive data sent by other computing units. Such data communication is an important part of parallel applications in numerical simulation, and even becomes a major time overhead in some applications. At present, the parallel application of numerical simulation is becoming more and more refined: such as using three-dimensional real physical models, simulating long-term scales, complex coupling of multi-physics processes, etc. The refinement of numerical simulation makes simulation with tens of thousands or even hundreds of thousands of cores an urgent ...

AI Technical Summary

Problems solved by technology

[0005] However, the "MPI process parallel data transmission method" does not distinguish between intra-node and inter-node data transmission, and does not match the "general-purpose processor multi-level nesting" architecture

This makes the application using this scheme have to perform additional packaging on the data to be transmitted even if it is data transmission within the computing node, introducing additional overhead such as data packaging and library function calls, resulting in loss of data transmission performance

Moreover, this data transmission method ignores the performance non-uniformity of data transmission between nodes, data transmission between CPUs in a node and data transmission between CPU cores, making it difficult to schedule data transmission for the "general-purpose processor multi-level nesting" architecture. optimization

Due to the fact that many current MPI systems cannot effectively implement asynchronous data communication, the "MPI process parallel data transmission method" cannot effectively use the parallel characteristics of data transmission within nodes and between nodes to improve data transmission throughput.

In addition, when tens of thousands or even hundreds of thousands of core data are transmitted, the number of MPI processes in this scheme will reach tens of thousands to hundreds of thousands, and its startup time is long (several minutes to tens of minutes), and the trouble-free time is short (usually less less than 1 hour), unable to meet the long-time simulation requirements of numerical simulation programs

Images