A parallel IO system-level parameter tuning method and system

Abstract

The application discloses a parallel IO system-level parameter tuning method and system, uses a unified representation of IO system related performance of measurement data shielding different processors and system architecture influence, determines system parameters; based on the determined system parameters, measurement is carried out on a given platform, and measurement results are obtained; according to the program IO characteristics, the obtained measurement results are used to screen the MPIIO parameters affecting the parallel IO performance, and parameter tuning is realized. The parameters are screened through standard measurement and theoretical calculation, multiple running is not needed, and the cost is low; the method is suitable for different platforms and scale programs, and has strong generalization; after tuning, the IO performance is significantly improved, and the parallel IO tuning problem is effectively solved.

Description

Technical Field

[0001] This invention belongs to the field of parallel I / O technology, specifically relating to a method and system for optimizing parallel I / O system-level parameters. Background Technology

[0002] High-performance computing (HPC) is a powerful processor cluster technology that enables processors to work in parallel to process massive multidimensional datasets and solve complex problems at extremely high speeds. Over the past decade, the computing power of HPCs has grown exponentially. With the rapid development of computing power, I / O has gradually become a key factor constraining the performance of parallel programs. Parallel I / O stacks offer many adjustable I / O parameters; tuning these parameters can improve program I / O performance, thereby reducing the mismatch between I / O and computational performance.

[0003] The MPI-IO layer of the Message Passing Interface (MPI) and the parallel file system contribute to improving the performance of parallel I / O to some extent. They offer a large number of configuration parameters that, if set properly, can significantly increase I / O bandwidth. These parameters include the number of aggregators, the collection buffer size, and the stripe size and number in the Lustre file system. The values ​​of these different parameters affect the overall I / O time, and thus the I / O bandwidth. Fewer aggregators and larger stripes may result in fewer iterations required to perform POSIX read / write operations, but excessively large stripes can lead to reduced I / O parallel efficiency. Larger stripe sizes and numbers in the Lustre file system are beneficial; however, the challenge lies in determining the optimal values ​​for stripe size and number, which depend on data size, the number of nodes, and many other factors. Determining these parameters is extremely complex due to the complex and unpredictable interactions of the I / O stack, network, application I / O patterns, and other file system-related middleware.

[0004] Currently, existing methods for finding a set of parallel I / O parameters that perform well for a program are all based on heuristic search. These methods require running the program multiple times, resulting in huge data collection costs and making them difficult to apply to large-scale program I / O tuning. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for optimizing parallel I / O system-level parameters, which addresses the shortcomings of the prior art and solves the technical problems of high system stack complexity, high optimization cost and difficulty in large-scale parallel programming encountered in parallel I / O optimization.

[0006] The present invention adopts the following technical solution:

[0007] A method for optimizing parallel I / O system-level parameters includes the following steps:

[0008] Use standardized measurement data that shields the effects of different processors and system architectures to uniformly characterize the performance of the I / O system and determine system parameters;

[0009] Measurements are performed on a given platform based on defined system parameters to obtain measurement results.

[0010] Based on the program's I / O characteristics, the obtained measurement results are used to screen the MPIIO parameters that affect parallel I / O performance, thereby achieving parameter tuning.

[0011] Preferably, the system parameters include: performance parameters of a single object storage target (OST), read / write bandwidth parameters of a single process under exclusive single core, maximum read / write bandwidth parameters of a single node, and parallelism parameters of the amount of data read / written in a single operation; the performance parameters of a single OST are used to characterize the maximum IO capability of a single storage unit, the read / write bandwidth parameters of a single process under exclusive single core are used to characterize the basic IO performance of a single process, the maximum read / write bandwidth parameters of a single node are used to characterize the overall IO capability of a single node, and the parallelism parameters of the amount of data read / written in a single operation are used to characterize the performance variation characteristics when multiple processes cooperate in IO.

[0012] Preferably, the measurement results include: the maximum read / write bandwidth of a single OST, the read / write bandwidth of a single process under exclusive single core, the maximum read / write bandwidth provided by a single node, and the threshold for the amount of data in a single IO operation whose read / write bandwidth increases with the number of participating processes; the maximum read / write bandwidth of a single OST is used to reflect the upper limit of IO of the storage unit, the read / write bandwidth of a single process under exclusive single core is used to reflect the process-level IO performance, the maximum read / write bandwidth of a single node is used to reflect the upper limit of node-level IO performance, and the threshold for the amount of data in a single IO operation is used to determine the effectiveness of multi-process parallel IO.

[0013] Preferably, the measurement is performed on a given platform based on determined system parameters, as follows:

[0014] For the Lustre parallel file system, the maximum read / write bandwidth of a single OST is measured by increasing the number of processes involved in reading and writing in the measurement data until the bandwidth stabilizes.

[0015] For the maximum read / write bandwidth of a single process on a single core, when measuring the read / write bandwidth of a single process on a single core, the number of OSTs is increased. If the measured data does not increase with the increase of the number of OSTs, the current value is taken as the read / write bandwidth of a single process on a single core.

[0016] For the maximum read / write bandwidth of a single node, the number of running processes on a single node is increased, and the read / write bandwidth is measured. The bottleneck reached by the read / write bandwidth as the number of processes increases is taken as the read / write bandwidth that a single node can provide.

[0017] For a set amount of data, the threshold for the amount of IO data in a single read / write operation is split into two processes for reading and writing. The threshold for the amount of IO data in a single read / write operation is measured. When the IO data amount exceeds the threshold, the read / write bandwidth increases with the increase in the number of participating processes. When the IO data amount is below the threshold, the bandwidth does not increase.

[0018] Preferably, the obtained measurement results are used to screen MPIIO parameters that affect parallel I / O performance, specifically as follows:

[0019] Suppose a program has Total nodes, One process, with a single IO data volume of [number]. Data; the maximum bandwidth of a single process in the system is The maximum bandwidth of a single node is The maximum bandwidth of a single OST is The maximum number of processes on a single node that do not generate I / O contention is t, and the I / O data threshold is... Ceil(x) and factor(x) refer to rounding up x and factoring x, respectively.

[0020] First, calculate the total amount of aggregated data in a single run. Select the number of aggregators Number of stripes Band size and aggregator allocation This yields the MPIIO parameters that affect parallel I / O performance.

[0021] Preferably, the number of aggregators :

[0022]

[0023] in, It is a rounding function. The amount of data in a single I / O operation. This is the threshold for IO data.

[0024] Preferably, the number of strips :

[0025]

[0026]

[0027] in, It is a rounding function. The amount of data in a single I / O operation. For IO data threshold, Maximum bandwidth for a single process This represents the maximum bandwidth for a single OST.

[0028] Preferably, the strip size :

[0029]

[0030] make

[0031]

[0032]

[0033]

[0034] in, For the number of stripes, The number of aggregators, For intermediate parameters, This refers to the amount of data processed in a single I / O operation.

[0035] Preferably, the aggregator is distributed. :

[0036]

[0037]

[0038]

[0039] in, The number of aggregators, This represents the number of nodes in the program. This represents the maximum number of processes on a single node that do not generate I / O contention. This represents the number of aggregators.

[0040] Secondly, embodiments of the present invention provide a parallel I / O system-level parameter tuning system, comprising:

[0041] The parameter module uses standardized measurement data that shields the effects of different processors and system architectures to uniformly characterize the performance of the I / O system and determine system parameters.

[0042] The strategy module performs measurements on a given platform based on defined system parameters and obtains the measurement results.

[0043] The tuning module uses measurement results to filter MPIIO parameters that affect parallel I / O performance based on the program's I / O characteristics, thereby achieving parameter tuning.

[0044] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described parallel I / O system-level parameter tuning method.

[0045] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described parallel I / O system-level parameter tuning method.

[0046] Fifthly, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described parallel I / O system-level parameter tuning method.

[0047] In a sixth aspect, embodiments of the present invention provide an electronic device, including a computer program, which, when executed by the electronic device, implements the steps of the above-described parallel I / O system-level parameter tuning method.

[0048] Compared with the prior art, the present invention has at least the following beneficial effects:

[0049] A parallel I / O system-level parameter tuning method is proposed. By standardizing measurement data to mask processor and architecture differences, it uniformly characterizes I / O system performance, solving the problems of existing heuristic methods requiring multiple runs and incurring high costs. Its three-step process (determining parameters → measurement → parameter selection) is logically coherent, directly optimizing based on system performance parameters and theoretical analysis, without requiring extensive data modeling, and is applicable to programs of different scales. In principle, it obtains program I / O characteristics through a single run or analysis, and combines this with selected key MPI / IO parameters to achieve performance optimization, especially suitable for large-scale parallel programs.

[0050] Furthermore, the system parameters comprehensively reflect the system's fundamental capabilities: OST performance reflects the storage unit's limits, single-process bandwidth reflects process-level fundamental performance, node bandwidth reflects the overall node capability, and parallelism parameters reflect the multi-process collaborative effect. This setting avoids parameter omissions, provides accurate basis for subsequent measurement and optimization, ensures that optimization is based on the system's true performance, reduces optimization deviations caused by incomplete parameters, and improves optimization accuracy.

[0051] Furthermore, the measurement results quantify key system performance indicators. The maximum bandwidth of OST clarifies the storage limit, single-process bandwidth reflects the process's I / O potential, the maximum node bandwidth defines node-level performance bottlenecks, and the single I / O data volume threshold judges the effectiveness of multi-process parallelism. This setting provides concrete data support for subsequent parameter selection, avoiding a disconnect between theoretical assumptions and reality, ensuring that the selected MPIIO parameters are adapted to the system's actual capabilities, and improving the rationality and reliability of tuning.

[0052] Furthermore, by gradually adjusting variables to eliminate interference (such as avoiding insufficient load due to insufficient processes / nodes in OST performance measurement, and eliminating OST bottlenecks in single-process bandwidth measurement), the data is ensured to accurately reflect the system's limits. This method reduces measurement errors, makes the obtained system parameters more accurate, provides a reliable basis for subsequent parameter calculations, avoids tuning failures due to data distortion, and improves the stability of the method.

[0053] Furthermore, by combining factors such as the amount of data processed in a single I / O operation and system bandwidth thresholds, abstract optimization is transformed into calculable parameter selection, avoiding subjective judgment based on experience. This process links system performance with program requirements; for example, the number of stripes must match the OST bandwidth and process capacity to ensure that parameters are adapted to the program's I / O mode and the system's carrying capacity, effectively improving I / O bandwidth.

[0054] Furthermore, calculating the number of aggregators based on the critical value that multi-process parallelism is superior to single-process parallelism ensures that the amount of data processed by each aggregator is not less than the effective parallel threshold, balancing the aggregation granularity: avoiding both excessive communication costs and bandwidth bottlenecks caused by too few aggregators, and inefficient aggregation of small data caused by too many aggregators. This setting optimizes aggregation efficiency and improves data aggregation and transmission performance.

[0055] Furthermore, the formula for the number of stripes takes into account both bandwidth requirements and allocation efficiency, avoiding bandwidth waste caused by insufficient number of stripes or low parallel efficiency caused by excessive number of stripes, thereby improving storage system utilization and I / O parallelism.

[0056] Furthermore, stripe size selection matches the stripe size with the number of stripes and aggregators, reducing file system lock overhead and node communication costs, optimizing data distribution, and improving IO efficiency.

[0057] Furthermore, to prevent node network contention caused by aggregator concentration, load balancing of each node is achieved, node resources are fully utilized, and situations where some nodes are overloaded and others are idle are avoided, thereby improving overall parallel I / O efficiency. In the experiment, significant performance improvements were observed under different numbers of nodes, verifying its load balancing effect.

[0058] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0059] In summary, this invention uses standardized measurement and theoretical calculation to select parameters, eliminating the need for multiple runs and reducing costs; it is applicable to different platforms and program sizes, demonstrating strong generalization; and after optimization, IO performance is significantly improved, effectively solving the problem of parallel IO optimization.

[0060] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0061] Figure 1 A comparison chart of bandwidth before and after optimization for different numbers of LAMMPS nodes;

[0062] Figure 2 This is a flowchart of the method of the present invention;

[0063] Figure 3 A schematic diagram of a computer device provided in an embodiment of the present invention;

[0064] Figure 4 This is a block diagram of a chip provided according to an embodiment of the present invention.

[0065] Among them, 60. Computer equipment; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0068] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0069] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0070] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0071] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0072] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0073] This invention provides a method for optimizing parallel I / O system-level parameters. It finds a method based on system performance parameters and theoretical analysis to optimize parallel I / O stack parameters with only one program run. This method is applicable to I / O optimization for various scales, even large-scale parallel programs. Based on parallel I / O theoretical analysis, it directly obtains a set of high-performance I / O parameters. Experiments show that I / O performance is improved by more than 3 times in a 10-node environment on the Tianhe platform. Compared to machine learning methods, it has lower measurement costs as it does not require measuring large amounts of data for modeling and optimization. In terms of application generalization, it can be used on different applications without repeatedly collecting data for modeling and optimization, demonstrating strong generalization ability. Regarding platform versatility, the method uses several architecture-independent and easily measurable parameters, making it suitable for various platforms. In terms of scalability, it can optimize I / O for any scale, even large-scale parallel programs, ensuring superior performance while achieving optimization costs independent of scale.

[0074] Taking the Tianhe platform as an example, it is configured as a thmt1 computing partition with 10 computing nodes, each node with 16 CPU cores + 64GB of memory, for a total of 160 cores.

[0075] Please see Figure 2 This invention discloses a parallel I / O system-level parameter tuning method. By measuring certain performance parameters of a computer system and based on theoretical analysis, it obtains the program's I / O characteristics after running the program once or under conditions provided by the programmer. Based on the principles of parallel I / O and key MPI I / O parameters selected from experimental data, it finds suitable I / O stack parameters for the program, optimizing its I / O performance. This method is scalable and can be used for I / O tuning of parallel programs of various scales, even large-scale programs. Specifically, it includes the following steps:

[0076] S1. Use a series of standardized measurement data that shield the effects of different processors and system architectures to uniformly characterize the performance of the I / O system and determine the system parameters that need to be measured.

[0077] Based on the underlying I / O mechanism, the impact of different processors and system architectures is shielded, and a series of standardized measurement data are used to uniformly characterize the relevant system performance, determining the system parameters that need to be measured; the performance of a single OST is denoted as... The read / write bandwidth of a single process on a single core is denoted as... The read / write bandwidth that a single node can provide is denoted as . The parallelism of measuring the amount of data read and written in a single operation is denoted as... ,in The total amount of data read and written in a single I / O operation. This represents the number of processes participating in this I / O operation. represent The process is processing a total of data. I / O bandwidth under operation.

[0078] The measurement tools chosen were IOR and Darshan. IOR is a widely used parallel I / O performance benchmark tool designed to simulate I / O patterns in scientific applications. It can perform large-scale data read and write operations and can be configured to use different I / O methods, including POSIX and MPI-IO. Through these operations, IOR helps users evaluate the performance of storage systems, such as bandwidth, latency, and I / O operation efficiency. Furthermore, IOR supports various file access modes, such as sequential access, random access, or interleaved access, making it well-suited for simulating I / O behavior in various real-world applications. Darshan is a lightweight I / O profiling library specifically designed to understand the I / O behavior of parallel applications. It can automatically collect detailed I / O operation statistics during application runtime without significantly impacting the application's performance. Darshan's main function is to help users identify I / O bottlenecks in applications, thereby guiding optimization efforts. It can provide detailed information on file access patterns, I / O request size, access frequency, and more.

[0079] S2. Measure the required parameters on a given platform: For parallel file systems such as Lustre, measure the maximum read / write bandwidth of a single OST; measure the read / write bandwidth of a single process under exclusive single-core operation; measure the maximum read / write bandwidth that a single node can provide; measure the threshold of single IO data volume that can be increased by increasing the number of participating processes.

[0080] When measuring the performance of a single OST, a sufficiently large load needs to be applied to it, which may be unattainable by a single process or node. Therefore, the number of processes and nodes involved in read and write operations needs to be gradually increased until the bandwidth stabilizes. It is worth noting that to measure the maximum performance of an OST, the size of the data written by a single process needs to be aligned with the stripe size. This can reduce potential file system locking overhead and yield a more reasonable and stable performance measurement. The final measurement data is recorded as follows: .

[0081] The maximum bandwidth of a single process was measured to be =180MB / s, maximum bandwidth of a single node is =550MB / s, the maximum bandwidth of a single OST is =250MB / s, with a single large IO write operation of 200MB.

[0082] When measuring the read / write bandwidth of a single process on a single core, to ensure the measurement is not affected by the OST performance bottleneck, the number of OSTs is gradually increased during the measurement. When the measurement data does not increase with the increase of OST, the performance tends to stabilize, and the current value is taken as the read / write bandwidth of a single process on a single core. The final measurement data is recorded as follows. .

[0083] When measuring the read / write bandwidth provided by a single node, assuming the disk array does not become a performance bottleneck, the number of processes on the single node is increased sequentially while measuring the read / write bandwidth. The read / write bandwidth will gradually reach a bottleneck as the number of processes increases; this bottleneck represents the total read / write bandwidth provided by the single node. The final measurement data is counted as... .

[0084] In measuring the parallelism of single read / write data volume, a certain amount of single IO data is specified. Number of changing processes , The size of the data is determined by The processes are evenly distributed for I / O operations, and the changes... and get .

[0085] A single process has an upper limit to its read / write speed. For a certain amount of data, parallel writing using multiple processes can be achieved. However, this results in a continuous dataset being split into several non-contiguous segments, incurring additional overhead. Therefore, the performance improvement from multiple processes may not be guaranteed. We need to find a data volume where splitting this volume into two processes for reading and writing will be faster than using a single process. To address this, we measure a threshold for the amount of data to be read / written in a single I / O operation. For a given amount of data, the read / write speed can be improved by increasing the number of participating processes. This threshold is denoted as […]. That is to say, With a large amount of data, the read / write speed of two processes is 50% faster than that of a single process.

[0086] S3. Filter out the MPIIO parameters that mainly affect parallel I / O performance, and select high-performance MPIIO parameters for the program based on the program I / O characteristics and the system parameters calculated in step S2.

[0087] Two-phase I / O, also known as set buffering, is an optimization applicable only to set I / O operations. In two-phase I / O, the set of independent I / O operations that make up the set operation are analyzed to determine which data regions must be transferred (read or written). If an application's data access patterns exhibit obvious collective characteristics, with multiple processes simultaneously reading or writing data blocks, performing set I / O can effectively reduce the number of communications and optimize data layout, thereby improving performance. The I / O parameter `cb_nodes` determines the number of aggregators, which affects the granularity of data aggregation. For example, using fewer aggregators results in larger-granularity aggregation, but this leads to increased communication costs and a bottleneck in the total bandwidth of the aggregators. Conversely, using more aggregators results in smaller-granularity aggregation, but the aggregated data still consists of a large number of small data points, leading to inefficiency and minimal performance improvement. Therefore, choosing an appropriate number of aggregators is crucial.

[0088] Another parameter related to aggregators is cb_config_list, which provides explicit control over the distribution of aggregators. By changing it, you can achieve a reasonable way to organize the allocation of aggregators. For example, if its value is x, it means that the maximum number of aggregators per node does not exceed x. During the tuning process, too many aggregators should not be allocated to the same node, as this will lead to excessive network load on the node and large network contention between different aggregators.

[0089] There are two parameters related to parallel file systems: the number of stripes and the stripe size. The number of stripes determines the maximum bandwidth that the current storage system can provide. For example, a larger number of stripes means more OSTs can be utilized for this I / O, and thus a larger maximum bandwidth. However, a larger number of stripes does not necessarily mean faster I / O, as its I / O parallel efficiency may be lower, resulting in lower overall storage utilization. For instance, larger stripes can cause the load to concentrate on a single stripe. If a large number of small I / Os occur, they will be distributed across a single stripe (because the stripe is large, these I / Os cannot fill a single stripe), and different processes may access the same stripe simultaneously, incurring locking protocol overhead for the file system and resulting in lower bandwidth. Smaller stripes cause a single compute node to communicate with multiple OSTs, resulting in higher communication costs. In ROM I / O, OST allocation has its own mechanism, which by default is:

[0090] 1) If the number of OSTs is less than the maximum number of aggregators, then the actual number of aggregators is equal to the number of OSTs, and each aggregator corresponds to one OST;

[0091] 2) If the number of OSTs is greater than the set number of aggregators, then the actual number of aggregators is equal to the greatest common divisor n of the number of OSTs and the set number of aggregators. The number of OSTs corresponding to each aggregator is divided by n OSTs.

[0092] For example, if the maximum number of aggregators is set to 2 and the OST is set to 5, then there is actually 1 aggregator, and one aggregator is responsible for 5 OSTs. If the maximum number of aggregators is set to 2 and the OST is set to 4, then there are actually 2 aggregators, and each aggregator is responsible for 4 / 2 OSTs.

[0093] Suppose a program has Total nodes, One process, with a single IO data volume of [number]. Data. The maximum bandwidth for a single process in the system is... The maximum bandwidth of a single node is The maximum bandwidth of a single OST is The maximum number of processes on a single node that do not generate I / O contention is t, and the I / O data threshold in S2 is: Ceil(x) and factor(x) respectively refer to rounding up x and factoring x.

[0094] Step 1: Calculate the total amount of aggregated data in a single step. ;

[0095] This amount of data can be obtained through a single run or from the program's source code.

[0096] Step 2: Select the number of aggregators ;

[0097]

[0098] Step 3: Select the number of strips ;

[0099]

[0100]

[0101] Step 4: Select the strip size ;

[0102]

[0103]

[0104]

[0105] Step 5: Select Aggregator Assignment .

[0106]

[0107]

[0108]

[0109] The final MPIIO configuration parameters are obtained by following the above five steps.

[0110] In another embodiment of the present invention, a parallel I / O system-level parameter tuning system is provided. This system can be used to implement the above-mentioned parallel I / O system-level parameter tuning method. Specifically, the parallel I / O system-level parameter tuning system includes a parameter module, a strategy module, and a tuning module.

[0111] The parameter module uses standardized measurement data that shields the effects of different processors and system architectures to uniformly characterize the performance of the I / O system and determine the system parameters.

[0112] The strategy module performs measurements on a given platform based on defined system parameters and obtains the measurement results.

[0113] The tuning module uses measurement results to filter MPIIO parameters that affect parallel I / O performance based on the program's I / O characteristics, thereby achieving parameter tuning.

[0114] This invention provides a terminal device comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement corresponding method flows or corresponding functions. The processor described in this embodiment can be used for parallel I / O system-level parameter tuning methods, including:

[0115] The performance of the I / O system is uniformly characterized using standardized measurement data that shields the influence of different processors and system architectures, and system parameters are determined. Measurements are then performed on a given platform based on the determined system parameters to obtain measurement results. Based on the program's I / O characteristics, the obtained measurement results are used to filter MPIIO parameters that affect parallel I / O performance, thereby achieving parameter tuning.

[0116] Please see Figure 3 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the method for estimating the concentration of radioactive iodine species in the containment vessel after an accident, as described in this embodiment. To avoid repetition, these details are not elaborated here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the parallel I / O system-level parameter tuning system of this embodiment. To avoid repetition, these details are not elaborated here.

[0117] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 3 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0118] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0119] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device 60.

[0120] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0121] Please see Figure 4 The terminal device is an electronic device 600, which is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0122] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 2 The steps are shown in the figure.

[0123] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0124] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0125] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0126] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem). This communication can be performed via input / output interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network, wide area network, and / or public network, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0127] Example 4

[0128] This invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the terminal device and extended storage media supported by the terminal device; it can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). More specific examples of the computer-readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical fiber, portable compact disk read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0129] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0130] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0131] One or more instructions stored in a computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the parallel I / O system-level parameter tuning method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps:

[0132] The performance of the I / O system is uniformly characterized using standardized measurement data that shields the influence of different processors and system architectures, and system parameters are determined. Measurements are then performed on a given platform based on the determined system parameters to obtain measurement results. Based on the program's I / O characteristics, the obtained measurement results are used to filter MPIIO parameters that affect parallel I / O performance, thereby achieving parameter tuning.

[0133] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0134] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0135] The tuning program is a LAMMPS program with 50 million atoms, 100 nodes, 1600 processes, 1000 simulation steps, and output every 50 steps, for a total of 20 writes. These 20 writes mitigate the random fluctuations of the system to some extent.

[0136] Single IO data volume: If a node runs 5000 atoms, the length of an atom record is about 28 bytes. With 50 million atoms, a single record is about 1335MB of data.

[0137] The formula yields the following values: striping_unit140509184, striping_factor10, cb_nodes10, and cb_config_list*:4.

[0138] Before program optimization: I / O time was 124.186s, and I / O bandwidth was 159.558MB / s.

[0139] After program optimization: IO time is 26.950s, IO bandwidth is 735.250MB / s, and overall IO bandwidth is improved by 360.8%.

[0140] This method was used to optimize programs of different sizes, and the results are shown below:

[0141] Table 1

[0142]

[0143] Figure 1 The chart shows the comparison of IO bandwidth before and after optimization with different numbers of nodes. Table 1 shows the improvement ratio. It can be seen that this method has a good optimization effect under different scales.

[0144] In summary, this invention provides a parallel I / O system-level parameter tuning method and system. By measuring certain performance parameters of the system, and based on theoretical analysis, the program's I / O characteristics are obtained by running the program at most once or by the programmer providing them. Combined with important MPI I / O parameters selected through analysis of parallel I / O principles and experimental data, suitable I / O parameters are found for the program, optimizing its I / O performance. This method is scalable, does not require multiple runs, has low tuning costs, and can be used for large-scale parallel program I / O tuning.

[0145] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0146] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0147] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0148] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random-access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0152] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A method for optimizing parallel I / O system-level parameters, characterized in that, Includes the following steps: Use standardized measurement data that shields the effects of different processors and system architectures to uniformly characterize the performance of the I / O system and determine system parameters; Measurements are performed on a given platform based on defined system parameters to obtain measurement results. Based on the program's I / O characteristics, the obtained measurement results are used to screen MPIIO parameters that affect parallel I / O performance, and parameter tuning is achieved, specifically as follows: Suppose a program has Total nodes, One process, with a single IO data volume of [number]. Data; the maximum bandwidth of a single process in the system is The maximum bandwidth of a single node is The maximum bandwidth of a single OST is The maximum number of processes on a single node that do not generate I / O contention is t, and the I / O data threshold is... Ceil(x) and factor(x) refer to rounding up x and factoring x, respectively. First, calculate the total amount of aggregated data in a single transaction. Select the number of aggregators Number of stripes Band size and aggregator allocation This yields the MPIIO parameters that affect parallel I / O performance; Number of aggregators : Number of stripes : Band size : make in, For intermediate parameters; Aggregator allocation : 。 2. The parallel I / O system-level parameter tuning method according to claim 1, characterized in that, System parameters include: performance parameters of a single object storage target (OST), read / write bandwidth parameters of a single process under dedicated single core, maximum read / write bandwidth parameters of a single node, and parallelism parameters of the amount of data read / written in a single operation. The performance parameters of a single OST are used to characterize the maximum I / O capability of a single storage unit, the read / write bandwidth parameters of a single process under dedicated single core are used to characterize the basic I / O performance of a single process, the maximum read / write bandwidth parameters of a single node are used to characterize the overall I / O capability of a single node, and the parallelism parameters of the amount of data read / written in a single operation are used to characterize the performance variation characteristics when multiple processes cooperate in I / O.

3. The parallel I / O system-level parameter tuning method according to claim 1, characterized in that, The measurement results include: the maximum read / write bandwidth of a single OST, the read / write bandwidth of a single process under dedicated single core, the maximum read / write bandwidth provided by a single node, and the threshold for the amount of data per IO operation that increases with the number of participating processes. The maximum read / write bandwidth of a single OST is used to reflect the IO limit of the storage unit, the read / write bandwidth of a single process under dedicated single core is used to reflect process-level IO performance, the maximum read / write bandwidth of a single node is used to reflect the node-level IO performance limit, and the threshold for the amount of data per IO operation is used to determine the effectiveness of multi-process parallel IO.

4. The parallel I / O system-level parameter tuning method according to claim 3, characterized in that, Measurements are performed on a given platform based on defined system parameters, as follows: For the Lustre parallel file system, the maximum read / write bandwidth of a single OST is measured by increasing the number of processes involved in reading and writing in the measurement data until the bandwidth stabilizes. For the maximum read / write bandwidth of a single process on a single core, when measuring the read / write bandwidth of a single process on a single core, the number of OSTs is increased. If the measured data does not increase with the increase of the number of OSTs, the current value is taken as the read / write bandwidth of a single process on a single core. For the maximum read / write bandwidth of a single node, the number of running processes on a single node is increased, and the read / write bandwidth is measured. The bottleneck reached by the increase in the number of processes is taken as the read / write bandwidth that a single node can provide. For a set amount of data, the threshold for the amount of IO data in a single read / write operation is split into two processes for reading and writing. The threshold for the amount of IO data in a single read / write operation is measured. When the threshold for the amount of IO data is exceeded, the read / write bandwidth increases with the increase in the number of participating processes. When the threshold for the amount of IO data is below the threshold for the amount of IO data, the bandwidth does not increase.

5. A parallel I / O system-level parameter tuning system, characterized in that, include: The parameter module uses standardized measurement data that shields the effects of different processors and system architectures to uniformly characterize the performance of the I / O system and determine system parameters. The strategy module performs measurements on a given platform based on defined system parameters and obtains the measurement results. The tuning module, based on the program's I / O characteristics, uses the obtained measurement results to filter MPIIO parameters that affect parallel I / O performance, and achieves parameter tuning, specifically as follows: Suppose a program has Total nodes, One process, with a single IO data volume of [number]. Data; the maximum bandwidth of a single process in the system is The maximum bandwidth of a single node is The maximum bandwidth of a single OST is The maximum number of processes on a single node that do not generate I / O contention is t, and the I / O data threshold is... Ceil(x) and factor(x) refer to rounding up x and factoring x, respectively. First, calculate the total amount of aggregated data in a single transaction. Select the number of aggregators Number of stripes Band size and aggregator allocation This yields the MPIIO parameters that affect parallel I / O performance; Number of aggregators : Number of stripes : Band size : make in, For intermediate parameters; Aggregator allocation : 。 6. The parallel I / O system-level parameter tuning system according to claim 5, characterized in that, System parameters include: performance parameters of a single object storage target (OST), read / write bandwidth parameters of a single process under dedicated single core, maximum read / write bandwidth parameters of a single node, and parallelism parameters of the amount of data read / written in a single operation. The performance parameters of a single OST are used to characterize the maximum I / O capability of a single storage unit, the read / write bandwidth parameters of a single process under dedicated single core are used to characterize the basic I / O performance of a single process, the maximum read / write bandwidth parameters of a single node are used to characterize the overall I / O capability of a single node, and the parallelism parameters of the amount of data read / written in a single operation are used to characterize the performance variation characteristics when multiple processes cooperate in I / O.

7. The parallel I / O system-level parameter tuning system according to claim 5, characterized in that, The measurement results include: the maximum read / write bandwidth of a single OST, the read / write bandwidth of a single process under dedicated single core, the maximum read / write bandwidth provided by a single node, and the threshold for the amount of data per IO operation that increases with the number of participating processes. The maximum read / write bandwidth of a single OST is used to reflect the IO limit of the storage unit, the read / write bandwidth of a single process under dedicated single core is used to reflect process-level IO performance, the maximum read / write bandwidth of a single node is used to reflect the node-level IO performance limit, and the threshold for the amount of data per IO operation is used to determine the effectiveness of multi-process parallel IO.

8. The parallel I / O system-level parameter tuning system according to claim 7, characterized in that, Measurements are performed on a given platform based on defined system parameters, as follows: For the Lustre parallel file system, the maximum read / write bandwidth of a single OST is measured by increasing the number of processes involved in reading and writing in the measurement data until the bandwidth stabilizes. For the maximum read / write bandwidth of a single process on a single core, when measuring the read / write bandwidth of a single process on a single core, the number of OSTs is increased. If the measured data does not increase with the increase of the number of OSTs, the current value is taken as the read / write bandwidth of a single process on a single core. For the maximum read / write bandwidth of a single node, the number of running processes on a single node is increased, and the read / write bandwidth is measured. The bottleneck reached by the increase in the number of processes is taken as the read / write bandwidth that a single node can provide. For a set amount of data, the threshold for the amount of IO data in a single read / write operation is split into two processes for reading and writing. The threshold for the amount of IO data in a single read / write operation is measured. When the threshold for the amount of IO data is exceeded, the read / write bandwidth increases with the increase in the number of participating processes. When the threshold for the amount of IO data is below the threshold for the amount of IO data, the bandwidth does not increase.

Images