A parallel I/O optimization method and system based on reducing inter-process communication overhead

A technology for inter-process communication and overhead, applied in the direction of inter-program communication, concurrent instruction execution, multi-program device, etc., to achieve the effect of improving efficiency, obvious improvement, and convenient deployment

Active Publication Date: 2018-03-27
HUAZHONG UNIV OF SCI & TECH
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

In this way, the data request in the file domain of the proxy process running on a certain machine node may come from other machine nodes. When the proxy process exchanges data with other processes, some cross-machine node data communication overhead will be generated.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A parallel I/O optimization method and system based on reducing inter-process communication overhead
  • A parallel I/O optimization method and system based on reducing inter-process communication overhead
  • A parallel I/O optimization method and system based on reducing inter-process communication overhead

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0041] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

[0042] The overall idea of ​​the present invention is to provide a matching strategy between an agent process and a machine node based on various program operating parameters provided by users in advance and configuration information of the current cluster environment. Through this strategy, configure the deployment of the agent process on the machine node, so that as many processes as possible ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method for reducing inter-process communication based on process affinity so as to improve the efficiency of parallel I / O, including: firstly analyzing the parallel I / O program with Collective I / O as the mainstream, collecting and counting the cluster Machine node information and MPI program configuration information; then, the system calculates the matching results of various possible machine nodes and agent processes through preprocessing operations, and then determines the best matching strategy through the performance prediction module; finally, the preprocessing is obtained The matching strategy is written in the configuration file. Experimental results show that the system is simple to configure, and can determine the best process allocation scheme for program operation through simple express preprocessing without modifying the original program code, thereby reducing inter-process communication overhead and improving parallel I / O performance purposes.

Description

technical field [0001] The invention belongs to the field of I / O subsystems in high-performance computing, and more specifically relates to a method and system for improving parallel I / O efficiency based on reducing inter-process communication overhead. Background technique [0002] In recent years, big data processing has continued to heat up in the field of scientific computing. On the one hand, it reflects the growing demand for big data processing in the field of scientific computing. At the same time, it can also be seen that in the field of traditional high-performance , many scientific computing applications are changing from computation-intensive to data-intensive, and high-performance computing is facing new challenges brought by big data. [0003] The Data Transmission Parallel I / O Interface (Message-Passing Interface, MPI-IO) standard formulates a parallel I / O interface—aggregated I / O (Collective I / O) interface, which provides scientific computing in the era of bi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/54G06F9/38
Inventor 石宣化金海王志翔黎明
Owner HUAZHONG UNIV OF SCI & TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap