Asynchronous checkpoint cache control method, device and medium for high-performance computing system

Abstract

This application discloses an asynchronous checkpoint caching control method, device, and medium for high-performance computing systems. The method includes the following steps: dividing the computing nodes in the system into SBB groups and PFS groups, and ensuring that the SBB and PFS groups complete checkpoint writing with minimal time difference; starting a server on each SBB node, embedding a client in each process of each computing node in the SBB group, and having each client communicate with an SBB server; during the checkpoint caching phase, using different modes to write checkpoints from the SBB and PFS groups to the corresponding SBB and PFS groups, respectively, where the SBB group uses a client-server mode and the PFS group uses a file I / O mode; during the checkpoint refresh phase, checkpoints in the SBB are refreshed using the client-server mode. This application can effectively alleviate performance bottlenecks during large-scale checkpoint applications in HPC systems, improve checkpoint efficiency, and reduce the impact on computing tasks.

Description

Technical Field

[0001] This application relates to the field of high performance computing (HPC) system technology, and in particular to an asynchronous checkpoint cache control method, device and medium for high performance computing systems. Background Technology

[0002] As HPC systems continue to scale, the number of integrated processor cores and accelerators increases, correspondingly increasing the probability of hardware failures. The mean time between failures (MTBF) is showing a significant downward trend, and in the future, the MTBF of exascale systems may be measured in minutes. Scientific computing applications in HPC systems typically require large-scale numerical simulations or complex modeling problems, with single task execution times potentially ranging from tens to hundreds of hours. During task execution, there is a risk of job interruption due to system failures. To address the risk of job interruption due to system failures, checkpointing technology is widely used as a fault-tolerance mechanism for HPC systems.

[0003] Checkpointing typically involves directly writing checkpoints to a Parallel File System (PFS) for storage. However, for large-scale applications, checkpoints can reach hundreds of gigabytes or even tens of terabytes in size. In this scenario, the limited bandwidth of PFS becomes a performance bottleneck, making it difficult for PFS to complete the checkpoint writing within an acceptable timeframe, thus significantly impacting the application's computational efficiency. Asynchronous checkpointing can alleviate this problem. Asynchronous checkpointing first caches the checkpoints in local storage, and then a background process flushes the cached data to PFS, thereby improving the reliability of the checkpoints. Because asynchronous checkpointing only blocks the application when writing the checkpoints to the cache, the background data flushing operation is transparent to the application and therefore does not block it. Furthermore, since cache write speeds are usually quite fast, applications using asynchronous checkpointing can quickly resolve checkpoint-related blocking.

[0004] However, as the computing power of HPC systems continues to grow, the computational scale of applications allowed by the system also increases, and the checkpoint size increases accordingly. However, the capacity of local storage on the compute nodes in HPC systems has not increased accordingly, resulting in certain limitations when applying asynchronous checkpointing technology.

[0005] 1. Insufficient cache space. Since asynchronous checkpointing technology typically uses local memory as a cache, large-scale applications usually consume a significant amount of memory during runtime, leaving limited memory available for checkpoint caching. This can cause asynchronous checkpointing technology to fail to complete the checkpoint caching process due to insufficient cache space. In this case, asynchronous checkpointing technology can only degenerate into synchronous checkpointing technology, which directly writes the checkpoint to PFS. However, due to the limited bandwidth of PFS, synchronous checkpointing technology usually requires blocking the application for a considerable period of time.

[0006] 2. Low utilization efficiency of new storage technologies. Current HPC systems are gradually introducing a new type of storage called Shared Burst Buffer (SBB). This type of storage often uses high-performance NVMe SSDs as storage devices. Compared to local memory, these devices usually have a larger capacity, and compared to PFS, they have a higher bandwidth advantage. However, compared to PFS, SBB has lower reliability due to factors such as the limited lifespan of the SSDs. To maximize the reliability of checkpoints, asynchronous checkpointing technology needs to flush the checkpoints in the SBB to the PFS if it needs to use the SBB cache. However, current mainstream implementations of asynchronous checkpointing rely on distributed file systems (SLS) to write and read data from the SBB. The SLS essentially provides a unified organization and abstraction of the SBB, and the underlying details of the SBB are beyond the control of the file system interface caller. When reading and writing to the SBB through the SLS, for checkpoints cached in the SBB, current mainstream asynchronous checkpointing implementations cannot allow the SBB to directly write the cached checkpoints to the PFS. Instead, the compute nodes need to read the checkpoints into their local buffers via the SLS, and then write the checkpoints from their local buffers to the PFS. In this process, checkpoints cached in the SBB need to be relayed to the PFS by the compute nodes. In large-scale applications, checkpoints are usually large, and the amount of data that needs to be relayed by the compute nodes is significant, leading to inefficiency in the process of refreshing cached checkpoints to the PFS. Summary of the Invention

[0007] The technical problem to be solved by this application is: In view of the above-mentioned problems existing in the prior art, this application provides an asynchronous checkpoint cache control method, device and medium for high-performance computing systems that is simple to implement, low in cost, highly efficient in storage utilization, and efficient in caching and refreshing. It can effectively alleviate the performance bottleneck in the process of large-scale application of checkpoints in HPC systems, improve checkpoint efficiency and reduce the impact on computing tasks.

[0008] To solve the above-mentioned technical problems, the technical solution proposed in this application is as follows:

[0009] An asynchronous checkpoint cache control method for a high-performance computing system includes the following steps:

[0010] The compute nodes in the HPC system are divided into SBB group and PFS group. The SBB group uses SBB as checkpoint cache and the PFS group uses PFS as checkpoint cache. The optimal compute node partitioning method for the SBB group and PFS group is determined so that the SBB group and PFS group complete the checkpoint write with the minimum time difference.

[0011] A server is started on each SBB node, and a client is embedded in each process on each compute node of the SBB group. Each client communicates with an SBB server.

[0012] During the checkpoint caching phase, checkpoints for the SBB group and PFS group are written to the SBB and PFS respectively using different modes. For the SBB group, the client-server mode is used to write checkpoints to the SBB, and for the PFS group, the file I / O mode is used to write checkpoints to the PFS.

[0013] During the checkpoint refresh phase, checkpoints in the cache are refreshed to PFS. Checkpoints already cached in PFS remain unchanged, while checkpoints cached in SBB are refreshed to PFS using a client-server model.

[0014] Furthermore, the step of determining the optimal computation node partitioning method for the SBB group and the PFS group so that the SBB group and the PFS group complete the checkpoint write with the minimum time difference includes:

[0015] Construct the following computation node partitioning model:

[0016] F( = | |

[0017] in, The number of nodes that concurrently write to PFS. The number of nodes that concurrently write to the SBB. The total amount of data written to the SBB. The total amount of data written to PFS. , They are respectively at a concurrency level of The bandwidth of SBB at a concurrency level of The bandwidth of PFS at that time This indicates the total write time for group SBB. F( represents the total write time of the PFS group) (Regarding) and The function represents the absolute difference in write time between the SBB group and the PFS group;

[0018] By finding F( The solution that yields the minimum value is obtained, which leads to the optimal computation node partitioning method that enables the SBB group and PFS group to complete the checkpoint writing with the minimum time difference.

[0019] Furthermore, it also includes setting the following constraints:

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026] in, D This represents the total number of checkpoint data points for the operation. N The number of nodes used for job execution. It is about The function, the functional relationship is used express, It is about The function, the functional relationship is used express.

[0027] In searching for F( In the process of finding the minimum solution, the bandwidth of SBB and PFS under different concurrency levels is obtained to form a concurrency-SBB bandwidth table and a concurrency-PFS bandwidth table. These tables are then traversed to find all possible bandwidth values ​​that satisfy the constraints and the corresponding F(t) is calculated. The multiple F( values) obtained from the search will be used to find the F( values). In the value, F( The value corresponding to the minimum value and The values ​​are used as the number of computing nodes for the SBB group and the PFS group, respectively.

[0028] Furthermore, in the checkpoint caching phase, writing checkpoints to the SBB using a client-server model includes:

[0029] The client first checks if there are any incomplete checkpoint refresh operations. If so, it waits for the checkpoint refresh operation to be completed; otherwise, it starts the current checkpoint caching process.

[0030] After starting the current checkpoint caching process, the client registers each state that its process needs to cache as an RDMA memory region;

[0031] For each state to be cached, the client sends a checkpoint cache RPC request to the corresponding server, which carries the starting address and length information of the current state data in the process memory.

[0032] The client is blocked to prevent the current cached state from being modified until a positive response to the RPC request is received from the server.

[0033] After receiving a positive response regarding all states, the client determines that the SBB checkpoint caching is complete, unblocks the client, and continues executing the computation task.

[0034] Furthermore, it also includes the server providing an affirmative response to the RPC requests sent by the client, including:

[0035] In response to RPC requests from clients regarding checkpoint caching, the server pre-creates a fixed-size buffer and registers it as an RDMA memory region.

[0036] The server initializes the RDMA operation counter cnt to represent the number of operations required to complete one state buffer for the current client;

[0037] The server uses RDMA to bypass the operating system kernel, reads the state from the memory of the corresponding client process according to the starting address and length of the current state, writes the state in the RDMA memory region into the local storage device, and updates the value of the RDMA operation counter cnt.

[0038] Determine whether the value of the RDMA operation counter has reached the preset value. If not, otherwise re-execute the process whereby the server uses RDMA to bypass the operating system kernel, reads the state from the memory of the corresponding client process according to the starting address and length of the current state in the RDMA memory region, writes the state in the RDMA memory region into the local storage device, and updates the value of the RDMA operation counter cnt. If yes, it is determined that all state data of the current RPC request has been read and written to the local storage device, and the server gives an affirmative response to the corresponding client RPC request.

[0039] Furthermore, during the checkpoint refresh phase, the process of using a client-server model to refresh the data cached in the SBB to the PFS includes:

[0040] The client corresponding to each process on each compute node of the SBB group sends an RPC request for checkpoint refresh to the corresponding server;

[0041] After the client sends an RPC request for checkpoint refresh to the server and returns, it continues to execute the computation task.

[0042] After receiving a checkpoint refresh request, the server creates a buffer and reads the checkpoint cache corresponding to the client from the local storage device into the buffer. Then, it uses the file system interface provided by PFS to refresh the data in the buffer to PFS.

[0043] After the server flushes all checkpoint caches of a client to PFS, it responds positively to the corresponding client RPC requests.

[0044] When the client caches the next checkpoint, it checks whether the server has responded positively to the RPC request to determine whether the most recent checkpoint refresh has been completed.

[0045] Furthermore, the checkpoint refresh phase also includes maintaining the connection established between each client and the SBB server during the checkpoint caching phase, so that the client can use services from the same server in both the checkpoint caching and refresh phases.

[0046] During the checkpoint refresh phase, the compute nodes of the PFS group and the SBB group also maintain synchronization while performing computation tasks. That is, when the SBB group compute node sends a checkpoint refresh RPC request to the server, the PFS group compute node blocks computation. When the SBB group compute node continues to perform computation tasks, the PFS group compute node unblocks to continue computation.

[0047] Furthermore, during the restart phase, it also includes reading checkpoints from PFS for all application processes to the corresponding regions in the address space of the process in the specified state, in order to restore the computation of the application processes before the failure.

[0048] An electronic device includes a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to perform the method described above.

[0049] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0050] Compared with the prior art, the beneficial effects of this application are as follows:

[0051] 1. This application integrates PFS and SBB, two heterogeneous storage systems in HPC systems, as checkpoint caches. By fully utilizing the heterogeneous storage resources of SBB and PFS in HPC systems to cache checkpoints, the I / O pressure can be distributed, avoiding the performance bottleneck that may exist in a single storage system and shortening the time spent on checkpoint caching.

[0052] 2. This application optimizes the refresh of the SBB cache by adopting a "client-server" model to interact with the remote heterogeneous storage SBB, instead of accessing the SBB through the distributed file system. When it is necessary to refresh the checkpoint cached in the SBB to the PFS, the compute node sends the request as a client, the SBB receives the request as a server, and directly refreshes the data to the PFS. This reduces the participation of the compute node in the checkpoint refresh process, eliminates the need for compute nodes to transfer data, significantly improves the checkpoint refresh efficiency, and reduces the impact on computing tasks. Attached Figure Description

[0053] Figure 1 This is a schematic diagram illustrating the implementation process of the asynchronous checkpoint caching control method in the high-performance computing system of this embodiment. Detailed Implementation

[0054] The present application will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present application.

[0055] To facilitate understanding, the relevant technical background involved in this application will be introduced first by way of example.

[0056] In the HPC field, applications typically perform large-scale numerical simulations or complex modeling tasks, with single runs lasting tens or even hundreds of hours. In recent years, as HPC systems have grown in scale, their Mean Time Between Failures (MTBF) have been shortening. Especially in future exascale systems, the MTBF may be measured in minutes. Checkpointing is a fault-tolerance mechanism to address the risk of job interruption caused by such system-level failures. The core mechanism of checkpointing lies in persistently saving the critical state of the application process at a specific execution moment. Application processes typically perform checkpointing operations periodically. To ensure the consistency of the saved critical state and prevent changes in the critical state due to the continued execution of the application process during the saving process, the application process is usually required to be in a blocked state during checkpointing.

[0057] When a system failure occurs, the application process loads the most recent available checkpoint version from persistent storage to restore its critical state. For applications using checkpointing, when a system failure occurs, instead of restarting from scratch, recovery is made from the checkpoint, greatly minimizing the loss of computational progress. Checkpointing technology can be divided into traditional checkpointing and asynchronous checkpointing based on its I / O characteristics. Traditional checkpointing, due to its synchronous write-to-PFS feature, is also called synchronous checkpointing. Traditional checkpointing, the earlier version, typically writes checkpoints directly to PFS for storage. However, PFS bandwidth is often limited. When large-scale application processes generate terabytes of checkpoint data, synchronous writes to PFS can lead to significant I / O bottlenecks, forcing application processes to be blocked for extended periods, severely impacting computational efficiency.

[0058] Asynchronous checkpointing technology can alleviate the problem of excessive blocking time for application processes caused by traditional or synchronous checkpointing technologies. It first caches checkpoints in local storage, and then a background process flushes the cached data to PFS, thereby improving checkpoint reliability. PFS is a file system specifically designed for HPC systems, shared among multiple users, jobs, and nodes. It needs to provide not only high-concurrency data access capabilities but also high-capacity persistent data storage capabilities, typically reaching petabyte (PB) levels. NVMe SSDs, as electronic storage devices without mechanical parts, use flash memory chips to store data. Data access does not involve the movement of mechanical parts, thus enabling fast data access and offering high bandwidth advantages. However, NVMe SSDs are manufactured using advanced semiconductor processes, resulting in high costs and making large-scale deployment difficult to achieve the petabyte-level storage capacity required by PFS. HDDs, traditional mechanical storage devices, access data through a moving read / write head and one or more rotating magnetic platters. This mechanical seek process limits the bandwidth of HDD data transfer, making its bandwidth lower than that of NVMe SSDs. However, HDDs are significantly cheaper to manufacture and use than NVMe SSDs. Therefore, considering cost, HDDs are often deployed on a large scale as the underlying storage device for PFS in HPC systems to economically achieve the petabyte-level storage capacity required by PFS. Because PFS uses HDDs as its storage device, its bandwidth is limited, which means that when application processes write data to PFS in a large-scale, high-concurrency manner, such as writing terabyte-level checkpoint data, it often takes a considerable amount of time.

[0059] SBB (Shared Block Buffer) is a new type of storage. Conceptually, an SBB is a high-speed shared storage system, typically composed of multiple high-performance NVMe SSDs connected to compute nodes via high-speed networks such as InfiniBand or Omni-Path. Conceptually, the capacity of an SBB is often smaller than that of a PFS (Programmable Memory System) but larger than the compute node's local memory, while its theoretical peak bandwidth is often lower than that of the compute node's local memory but higher than that of the PFS. The main design goal of SBB is to alleviate the I / O bottleneck of PFS by providing an I / O buffer layer between the high-speed compute node's local memory and the low-speed PFS. Application processes can cache data in the SBB and then flush it to the PFS in the background. SBBs typically consist of a large number of nodes and storage devices, and their underlying physical structure can be very complex. Current asynchronous checkpointing technologies can only rely on the unified abstraction of SBBs by distributed file systems for data access.

[0060] SBB and PFS, as shared storage across multiple applications and nodes in HPC systems, typically have sufficient capacity. Their usable bandwidth, however, is related to concurrency, which is defined as the number of compute nodes concurrently writing to shared storage such as SBB or PFS. The following example illustrates the relationship between shared storage bandwidth and concurrency: when two compute nodes simultaneously write to PFS, the bandwidth utilized by PFS is x GB / s. Increasing the concurrency to four compute nodes results in a bandwidth of y GB / s. Since the concurrency in both scenarios is relatively low, insufficient to fully utilize PFS bandwidth, x and y are not equal (x and y only equal when both scenarios fully utilize PFS bandwidth, reaching its peak bandwidth). Furthermore, because the concurrency in the second scenario is higher than in the first, y > x. The bandwidth utilized by SBB exhibits the same pattern as PFS, also being related to concurrency.

[0061] Although SBB uses NVMe SSDs as storage devices, its theoretical peak bandwidth can exceed that of PFS, which uses HDDs. However, when SBB concurrency is insufficient while PFS concurrency is high, the actual bandwidth delivered by SBB may be lower than that of PFS. Furthermore, while ideally, SBB within an HPC system should have a peak bandwidth far exceeding that of PFS to effectively alleviate I / O bottlenecks between compute nodes and the persistent storage layer, significant differences exist in the construction and maintenance budgets of HPC systems in actual deployments. For systems with limited budgets, the number of NVMe SSDs that can be deployed may be insufficient to form a large-scale, high-bandwidth SBB cluster. In such cases, when facing high-concurrency, high-throughput data write demands from large-scale parallel applications, such as checkpointing requirements, a limited-scale SBB may not provide sufficient aggregate write bandwidth, thus becoming a bottleneck in system performance and failing to fully meet the application's needs for high-speed data caching and exchange.

[0062] While asynchronous checkpointing can reduce application process blocking time, current HPC systems face several challenges. Firstly, limited local memory capacity prevents the full caching of checkpoints for large-scale applications. As HPC systems grow, so does the computational scale of applications. During runtime, large-scale applications consume significant amounts of local memory, leaving insufficient available memory to cache complete checkpoint data. This forces asynchronous checkpointing to degenerate into synchronous checkpointing, writing all checkpoint data directly to the slow PFS without any caching, resulting in prolonged application process blocking and hindering the execution of computational tasks.

[0063] On the other hand, when using the new, high-capacity Storage Block (SBB) as a cache, relying on a distributed file system to refresh the SBB cache is inefficient. Although HPC systems are increasingly deploying SBBs, which often use NVMe SSDs as storage devices, their bandwidth is typically higher than that of the Processing File System (PFS) in their design concept. Furthermore, as a storage resource that can be shared among multiple jobs, the capacity of an SBB is usually sufficient for checkpoint storage. While SBBs are suitable as checkpoint caches due to their high bandwidth and large capacity, current asynchronous checkpointing technologies rely on a distributed file system to access the SBB. Therefore, when refreshing the SBB cache to the final storage (PFS), the data must first be read locally by the compute nodes and then written back to the PFS. This checkpoint cache refresh process requires compute nodes to relay data. For large-scale applications with massive checkpoint data volumes, this relay process significantly reduces refresh efficiency and also greatly interferes with the application processes on the compute nodes.

[0064] To address the aforementioned issues, this application constructs a novel framework, HeteroCheck, based on asynchronous checkpointing technology. This framework effectively alleviates potential performance bottlenecks during large-scale checkpointing applications in HPC systems, while simultaneously improving checkpointing efficiency and reducing the impact on computational tasks. Furthermore, this application replaces the distributed file system with a client-server model in its asynchronous checkpointing technology. The compute node starts the client, and the SBB node starts the server. When the SBB cache is refreshed, the client sends a refresh request to the server. Upon receiving the request, the server directly writes the data stored in the SBB to the PFS, eliminating the need for data relay from the compute node. This simplifies the data flow during SBB cache checkpoint refresh from "SBB → compute node → PFS" to "SBB → PFS," significantly reducing network overhead caused by data movement in high-performance computing systems. It also offloads the SBB refresh task from the compute node, freeing up CPU and network resources and minimizing the impact on computational tasks on the compute node.

[0065] This application integrates heterogeneous storage (including SBB and PFS) in HPC systems as its available cache. Compared with the traditional strategy of storing checkpoints in a single storage medium, the heterogeneous storage strategy of this application can fully integrate the diverse storage resources in HPC systems, avoid the performance bottlenecks that may be caused by a single medium, and thus significantly shorten the time of checkpoint operations. At the same time, by using a "client-server" approach to interact with remote storage resources of SBB, when it is necessary to refresh the checkpoints in SBB to PFS, the compute node will act as the client and SBB will act as the server. The compute node only needs to send a checkpoint refresh request to the server on SBB, and the data cached in the checkpoints will be directly refreshed to PFS by the server. Compared with the traditional method of reading and writing to SBB through a distributed file system, since the checkpoint refresh process no longer requires the compute node to transfer data, the efficiency of checkpoint refresh can be effectively improved.

[0066] For large-scale applications, the local memory of compute nodes may be insufficient to fully cache checkpoints. In this case, current asynchronous checkpointing techniques can only degenerate into synchronous checkpointing, directly writing all checkpoints to the slow PFS, significantly increasing the blocking time of application processes. This application's HeteroCheck framework can use both PFS and SBB as caches simultaneously, with SBB sharing the load on PFS and alleviating potential performance bottlenecks on PFS. Furthermore, for large-scale applications, SBB alone may not be sufficient to meet the high aggregate bandwidth demands of application processes. This application's dual-storage caching strategy can fully utilize PFS to alleviate the load on SBB and avoid the performance bottlenecks that may exist with a single storage system.

[0067] For large-scale applications, checkpoints can typically reach petabyte (PB) levels. When using a Service Block Buffer (SBB) as a cache, current asynchronous checkpointing technologies rely on a distributed file system (SFS) to access data within the SBB. During checkpoint refresh, the SFS transports data to compute nodes, which then flush the data into the PFS. This checkpoint refresh process requires significant involvement from compute nodes as intermediaries, which is not only inefficient but can also interfere with application processes on those nodes. This application's HeteroCheck framework offloads the SBB checkpoint cache refresh task directly to the SBB server, eliminating the need for compute nodes. This reduces compute node involvement during the checkpoint refresh process and eliminates the need for compute nodes to relay data, thus effectively improving refresh efficiency.

[0068] The present application will be further described below with reference to specific embodiments.

[0069] like Figure 1 As shown, the steps of the asynchronous checkpoint cache control method of the high-performance computing system in this embodiment include:

[0070] Step S01. Divide the compute nodes in the HPC system into two groups, namely the SBB group and the PFS group. The SBB group uses SBB as the checkpoint cache and the PFS group uses PFS as the checkpoint cache. Determine the optimal compute node partitioning method for the SBB group and the PFS group so that the SBB group and the PFS group can complete the checkpoint write with the minimum time difference.

[0071] Step S02. Start a server on each SBB node, embed a client in each process on each compute node in the SBB group, and each client communicates with an SBB server so that the SBB can provide direct services to the compute nodes.

[0072] Step S03. During the checkpoint caching stage, checkpoints of the SBB group and PFS group are written to the SBB and PFS respectively using different modes. For the SBB group, the client-server mode is used to write checkpoints to the SBB, and for the PFS group, the file I / O mode is used to write checkpoints to the PFS.

[0073] Step S04. In the checkpoint refresh phase, refresh the cached checkpoints to PFS. Checkpoints already cached in PFS remain unchanged, while checkpoints cached in SBB are refreshed to PFS using a client-server model.

[0074] In this embodiment, an asynchronous checkpointing framework, HeteroCheck, is constructed. This framework integrates two heterogeneous storage systems, SBB and PFS, in the HPC system as checkpoint caches, avoiding the performance bottleneck of checkpoint caching that may exist with a single storage system. At the same time, a "client-server" model is adopted to interact with the remote heterogeneous storage SBB. When it is necessary to refresh the checkpoints cached in the SBB to the PFS, the compute node acts as a client to send a request, and the SBB acts as a server to receive the request and directly refresh the data to the PFS. This reduces the involvement of compute nodes in the checkpoint refresh process, eliminates the need for compute nodes to relay data, and improves refresh efficiency.

[0075] In this embodiment, multiple storage methods are used for checkpoint caching to avoid the problem that a single storage might be unable to fully cache checkpoints due to limited capacity, and also to avoid the problem that a single storage might have insufficient bandwidth, leading to excessively long checkpoint times and application blocking times. Specifically, the HeteroCheck framework in this embodiment uses both SBB and PFS storage to cache checkpoints, with the cached data for one checkpoint distributed across the two storage methods. This embodiment divides the compute nodes into two groups: one group caches checkpoints in the SBB (SBB group), and the other group caches checkpoints in the PFS (PFS group). This embodiment uses an algorithm based on minimizing tail latency to determine the optimal number of nodes for the SBB and PFS groups, allowing both groups to complete checkpoint writing as simultaneously as possible. A sequential selection method is used, adding the compute node with the smallest number to the SBB group until the required number of nodes for the SBB group is met, with the remaining compute nodes added to the PFS group.

[0076] Applications in HPC systems are mostly parallel computing programs. A large parallel computing program typically distributes computational tasks evenly across multiple processes during runtime. These processes usually run in parallel on different computing nodes. At checkpoints, each process needs to save its computational state. Therefore, a checkpoint in the entire application is actually an independent checkpoint for multiple processes. Parallel applications are typically load-balanced, ensuring that the amount of checkpoint data for each process remains consistent. This embodiment specifically addresses the case where the amount of checkpoint data is the same for each process.

[0077] In this embodiment, HeteroCheck uses SBB and PFS for checkpoint caching. SBB and PFS are shared storage across multiple applications and nodes in an HPC system, and their capacity is usually sufficient. However, their available bandwidth depends on the concurrency of the applications during runtime. After dividing compute nodes into SBB and PFS groups, the HeteroCheck framework in this embodiment needs to ensure that both SBB and PFS groups complete checkpoint writing simultaneously to minimize checkpoint caching time. Otherwise, "tail latency" may occur, where one group has completed writing while the other has not. In this case, the storage corresponding to the group that has completed writing is idle and unused, and the checkpoint caching time is determined by the other group that has not yet completed writing. This embodiment, based on dividing compute nodes into SBB and PFS groups, further controls the number of nodes concurrently writing to the PFS group and the number of nodes concurrently writing to the SBB group, minimizing the time difference between checkpoint writing completion times for the SBB and PFS groups.

[0078] In this embodiment, by determining the optimal computing node partitioning method for the SBB group and the PFS group, so that the SBB group and the PFS group complete the checkpoint write with the minimum time difference, the following is included:

[0079] Construct the following computation node partitioning model:

[0080] F( = | | (1)

[0081] in, The number of nodes that concurrently write to PFS. The number of nodes that concurrently write to the SBB. The total amount of data written to the SBB. The total amount of data written to PFS. , They are respectively at a concurrency level of The bandwidth of SBB at a concurrency level of The bandwidth of PFS at that time This indicates the total write time for group SBB. F( represents the total write time of the PFS group) (Regarding) and The function represents the absolute difference in write time between the SBB group and the PFS group;

[0082] By finding F( The solution that minimizes the value yields the optimal computation node partitioning method that allows the SBB group and PFS group to complete the checkpoint writing with the minimum time difference. That is, the optimal solution of equation (1) is the one that minimizes F( The solution that obtains the minimum value is obtained by minimizing the difference F( ), when the difference F( When the minimum value is obtained, a node allocation scheme that makes the write time of the SBB group and the PFS group closest can be found, so that the SBB group and the PFS group can complete the write of the checkpoint in the closest time, thereby minimizing the time for caching the checkpoint and avoiding the "tail latency" effect.

[0083] As a preferred embodiment, the following constraints may be further set:

[0084] (2)

[0085] (3)

[0086] (4)

[0087] (5)

[0088] (6)

[0089] (7)

[0090] in, D This represents the total number of checkpoint data points for the operation. N The number of nodes used for job execution. It is about The function, the functional relationship is used express, It is about The function, the functional relationship is used express.

[0091] In searching for F( In the process of finding the minimum solution, the bandwidth of SBB and PFS under different concurrency levels is obtained to form a concurrency-SBB bandwidth table and a concurrency-PFS bandwidth table. These tables are then traversed to find all possible bandwidth values ​​that satisfy the constraints and the corresponding F(t) is calculated. The multiple F( values) obtained from the search will be used to find the F( values). In the value, F( The value corresponding to the minimum value and The values ​​are used as the number of computing nodes for the SBB group and the PFS group, respectively.

[0092] This embodiment uses the above method to determine the optimal number of nodes for the SBB group and the PFS group. The computing node with the smallest number is added to the SBB group in sequence by sequential selection until the number of nodes required by the SBB group is met. The remaining computing nodes are added to the PFS group, so that the two groups of nodes can complete the checkpoint write as simultaneously as possible, thereby minimizing the tail delay effect.

[0093] Specifically, and This is a function of concurrency. HPC systems can use I / O benchmark programs such as IOR to measure the bandwidth of SBB and PFS at different concurrency levels, thus creating concurrency-SBB bandwidth tables and concurrency-PFS bandwidth tables respectively. This embodiment solves for F( In the process of finding the minimum value, firstly, the bandwidths of SBB and PFS under different concurrency levels are obtained, forming a concurrency-SBB bandwidth table and a concurrency-PFS bandwidth table that correspond to the concurrency level and SBB bandwidth. Then, under the condition of satisfying the constraints, the two tables are traversed to determine all possible bandwidth values. Substitute all possible bandwidth values ​​into equation (1) to calculate the corresponding F( ) values, resulting in multiple F( ) value, in multiple F( Find the minimum value among the values, and at this time the corresponding and The value represents the number of computing nodes for the SBB group and the PFS group, respectively. Furthermore, if multiple equal minimum values ​​exist, the first found minimum value can be taken as the number of nodes for concurrent writing in the SBB group and the PFS group.

[0094] In this embodiment, during the checkpoint caching phase, writing checkpoints to the SBB using a client-server model includes:

[0095] The client first checks if there are any incomplete checkpoint refresh operations. If so, it waits for the checkpoint refresh operation to be completed; otherwise, it starts the current checkpoint caching process.

[0096] After starting the current checkpoint caching process, the client registers each state that its process needs to cache as an RDMA memory region;

[0097] For each state to be cached, the client sends a checkpoint cache RPC request to the corresponding server, which carries the starting address and length information of the current state data in the process memory.

[0098] The client is blocked to prevent the current cached state from being modified until a positive response to the RPC request is received from the server.

[0099] After receiving a positive response regarding all states, the client determines that the SBB checkpoint caching is complete, unblocks the client, and continues executing the computation task.

[0100] Furthermore, it also includes the server responding positively to the RPC requests sent by the client, the steps of which include:

[0101] ① For each received RPC request from a client regarding the checkpoint cache, the server pre-creates a fixed-size buffer and registers it as an RDMA memory region.

[0102] If the allocation is dynamically based on the actual length of each client's state data, then processing significantly longer states would require requesting an extremely large buffer, which could easily exhaust the available memory of the SBB server node, leading to an out-of-memory (OOM) risk. This embodiment solves the resource management problem when multiple clients make concurrent requests by adopting a fixed-size strategy.

[0103] ② The server initializes the RDMA operation counter cnt, which represents the number of operations required to complete one state buffer for the current client.

[0104] Specifically, the formula for calculating the number of operations required for a client-side state cache can be expressed as: , where state_length is the length of the state data and buf_size is the fixed buffer size of the server.

[0105] In a specific application embodiment, the RDMA operation counter cnt can be initialized to the number of operations required for one state cache of the client as calculated in the above manner. Then, after each cache operation is completed, the RDMA operation counter cnt is decremented by 1. When the RDMA operation counter cnt is 0, it is determined that the current state data cache is complete.

[0106] ③ The server uses RDMA to bypass the operating system kernel and reads the state from the memory of the corresponding client process into the server's RDMA memory area (a fixed-size buffer) according to the starting address and length of the state. RDMA can improve network throughput and data transmission efficiency.

[0107] ④ The server writes the state in the buffer to the local storage device (such as NVMe SSD) and updates the value of the RDMA operation counter cnt.

[0108] ⑤ Determine whether the value of the RDMA operation counter has reached the preset value. If not, otherwise re-execute the process whereby the server uses RDMA to bypass the operating system kernel, reads the state from the memory of the corresponding client process according to the starting address and length of the current state in the RDMA memory region, writes the state in the RDMA memory region into the local storage device, and updates the value of the RDMA operation counter cnt. If yes, it is determined that all state data of the current RPC request has been read and written to the local storage device, and the server gives an affirmative response to the corresponding client RPC request.

[0109] Specifically, the server checks whether the RDMA operation counter cnt has reached a preset value, such as whether it has decremented to 0. If so, steps ③ and ④ are repeated; otherwise, it determines that all data has been read and written to the local NVMe SSD, and the server responds positively to the corresponding client RPC request. Once the client receives a positive response regarding all statuses, it considers the SBB checkpoint caching complete, unblocks, and continues executing the computation task.

[0110] In this embodiment, caching the checkpoints of the PFS group during the checkpoint caching phase includes:

[0111] PFS group compute nodes write checkpoints directly to PFS through the I / O interface of the parallel file system.

[0112] In this embodiment, during the checkpoint refresh phase, the SBB group uses a client-server model to refresh the data cached in the SBB to the PFS, including:

[0113] The client corresponding to each process on each compute node of the SBB group sends an RPC request for checkpoint refresh to the corresponding server;

[0114] The client sends an RPC request for checkpoint refresh to the server and returns immediately to continue executing the computation task;

[0115] After receiving the checkpoint refresh request, the server creates a buffer and reads the checkpoint cache corresponding to the client in the local NVMe SSD into the buffer, and then uses the file system interface provided by PFS to refresh the data in the buffer to PFS.

[0116] After the server flushes all checkpoint caches of a client to PFS, it responds positively to the corresponding client RPC requests.

[0117] When the client caches the next checkpoint, it checks whether the server has responded positively to the RPC request to determine whether the most recent checkpoint refresh has been completed.

[0118] More preferably, during the checkpoint refresh phase, the compute nodes of the PFS group and the SBB group maintain synchronization while executing computation tasks. That is, when an SBB group compute node sends a checkpoint refresh RPC request to the server, the PFS group compute node blocks its computation; when the SBB group compute node resumes its computation task, the PFS group compute node unblocks and continues its computation. This synchronization mechanism ensures the consistency of the computation state of each process in the application.

[0119] More preferably, during the checkpoint refresh phase, the system also maintains the connection established between each client and the SBB server during the checkpoint caching phase, to ensure that the client uses services from the same server in both the checkpoint caching and refresh phases, and to ensure that the server finds the correct cache during the refresh phase.

[0120] In this embodiment, during the restart phase, checkpoints for all application processes are read from the PFS to the corresponding regions in the address space of the processes with critical states, in order to restore the computations of the application processes before the failure. Specifically, the checkpoint cache of the "SBB group" is flushed to the PFS, while the checkpoint cache of the "PFS group" does not need to be flushed because it is directly stored in the PFS. As long as the cache in the SBB is flushed, the checkpoints of all application processes can be found in the PFS. When an application process encounters a system failure and needs to restart, in this embodiment, HeteroCheck reads checkpoints from the PFS to the corresponding regions in the address space of the processes with critical states for all application processes, thereby restoring the computations of the application processes before the failure.

[0121] In summary, this application integrates PFS and SBB, two heterogeneous storage systems in an HPC system, as checkpoint caches. This fully utilizes the heterogeneous storage resources of SBB and PFS in the HPC system to cache checkpoints, distributing I / O pressure, avoiding performance bottlenecks that may exist with a single storage system, and shortening the time spent caching checkpoints. Simultaneously, by optimizing SBB cache refresh, a "client-server" model is adopted to interact with the remote heterogeneous storage SBB, rather than accessing the SBB through a distributed file system. When checkpoints cached in the SBB need to be refreshed to the PFS, the compute node acts as the client, sending a request, and the SBB acts as the server, receiving the request and directly refreshing the data to the PFS. This reduces the involvement of compute nodes in the checkpoint refresh process, eliminates the need for compute nodes to relay data, and significantly improves checkpoint refresh efficiency.

[0122] This embodiment further provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program and the processor is used to execute the computer program to perform the method as described above.

[0123] It is understood that the method described in this embodiment can be executed by a single device, such as a computer or server, or it can be applied to a distributed scenario where multiple devices cooperate to complete the task. In a distributed scenario, one of the multiple devices may execute only one or more steps of the method described in this embodiment, and the multiple devices interact to complete the method. The processor can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the method described in this embodiment. The memory can be implemented using read-only memory (ROM), random access memory (RAM), static storage devices, and dynamic storage devices. The memory can store the operating system and other applications. When the method described in this embodiment is implemented through software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0124] This embodiment further provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0125] Those skilled in the art will understand that the above embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0126] The above description is merely a preferred embodiment of this application and is not intended to limit the application in any way. Although this application has disclosed preferred embodiments above, it is not intended to limit the application. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application without departing from the content of the technical solution of this application should fall within the protection scope of the technical solution of this application.

Claims

1. An asynchronous checkpoint cache control method for a high-performance computing system, characterized in that the steps include... The application comprises the following steps: Dividing the computing nodes in the HPC system into SBB groups and PFS groups, wherein the SBB groups use SBB as checkpoint cache, the PFS groups use PFS as checkpoint cache, and the optimal computing node division mode of the SBB groups and the PFS groups is determined to make the SBB groups and the PFS groups complete checkpoint writing with minimum time difference; Starting a server on each SBB node, and embedding a client in each process on each computing node of the SBB group, wherein each client communicates with a SBB server; In the checkpoint cache stage, the checkpoints of the SBB groups and the PFS groups are written into SBB and PFS respectively using different modes, wherein the checkpoints of the SBB groups are written into SBB using the client-server mode, and the checkpoints of the PFS groups are written into PFS using the file I / O mode; In the checkpoint flush stage, the checkpoints in the cache are flushed to PFS, wherein the checkpoints already cached in PFS remain unchanged, and the checkpoints cached in SBB are flushed to PFS using the client-server mode; The optimal computing node division mode of the SBB groups and the PFS groups is determined to make the SBB groups and the PFS groups complete checkpoint writing with minimum time difference, which comprises the following computing node division model: The following computing node division model is constructed: F( )=∣ ∣ wherein, is the number of nodes writing concurrently to the PFS, is the number of nodes writing concurrently to the SBB, is the total amount of data written to the SBB, is the total amount of data written to the PFS, , are the bandwidths of the SBB and PFS, respectively, at a concurrency of denotes the total write time of the SBB group, denotes the total write time of the PFS group, F( ) is a function of and denotes the absolute difference between the write times of the SBB and PFS groups.​​ By finding the solution that makes F( ) minimum, the optimal way of dividing the computing nodes that makes the SBB group and the PFS group complete the checkpoint write with the minimum time difference is obtained.

2. The method of Claim 1, wherein, The following constraint conditions are set: wherein, D is the total amount of checkpoint data for the job, N is the number of nodes used for the job run, is a function of , the functional relationship being represented by , is a function of , the functional relationship being represented by ; In the process of finding the solution that makes F( ) minimum, by obtaining the bandwidths of SBB and PFS under different concurrences, a concurrence-SBB bandwidth table and a concurrence-PFS bandwidth table are formed, the concurrence-SBB bandwidth table and the concurrence-PFS bandwidth table are traversed, all possible bandwidth values satisfying the constraint condition are found out and the corresponding F( ) values are calculated, and the values of and corresponding to the time when F( ) among the multiple F( ) values found out takes the minimum value are taken as the numbers of computing nodes of the SBB group and the PFS group respectively.

3. The method of Claim 1, wherein, In the checkpoint cache stage, the checkpoints are written into SBB using the client-server mode, which comprises the following steps: The client first checks whether there is an unfinished checkpoint flush operation, and waits for the completion of the checkpoint flush operation if there is, otherwise starts the current checkpoint cache process; After starting the current checkpoint cache process, the client registers each state of the process that needs to be cached as an RDMA memory region; The client sends a checkpoint cache RPC request to the corresponding server for each state to be cached, and the request carries the start address and length information of the current state data in the process memory; The client is blocked so that the current state to be cached is not modified until a positive response to the RPC request is received from the server; After receiving the positive response about all states, the client determines that the SBB checkpoint cache is complete, unblocks the client, and continues to execute the computing task.

4. The method of Claim 3, wherein, The server positively responds to the RPC request sent by the client, which comprises the following steps: The server pre-creates a fixed-size buffer and registers it as an RDMA memory region for the RPC request sent by the client about checkpoint cache; The server initializes the RDMA operation counter cnt to represent the number of operations required to complete the cache of one state of the current client; The server uses RDMA to bypass the operating system kernel to read the state from the memory of the corresponding client process into the RDMA memory region according to the start address and length of the current state, and writes the state in the RDMA memory region into the local storage device, and updates the value of the RDMA operation counter cnt. determining whether the value of the RDMA operation counter reaches a preset value, if not, re-executing the server reading the state from the memory of the corresponding client process according to the start address and length of the current state using RDMA bypassing the operating system kernel from the RDMA memory region, writing the state in the RDMA memory region into the local storage device, and updating the value of the RDMA operation counter cnt; if yes, determining that all state data of the current RPC request is read and written into the local storage device, and the server positively replies to the corresponding client RPC request.

5. The method of claim 1-4, wherein, In the checkpoint flush phase, the data buffered in the SBB is flushed to the PFS by the client-server mode, comprising: each process on each computing node of the SBB group sends a checkpoint flush RPC request to the corresponding server; the client returns after sending the checkpoint flush RPC request to the server, and continues to execute the computing task; the server receives the checkpoint flush request, creates a buffer, reads the checkpoint of the client in the local storage device into the buffer, and uses the file system interface provided by the PFS to flush the data in the buffer to the PFS; the server flushes all checkpoints of a client to the PFS, and positively replies to the corresponding client RPC request; the client checks whether the server positively replies to the RPC request to determine whether the last checkpoint flush is completed when the next checkpoint is buffered.

6. The method of Claim 5, wherein, In the checkpoint flush phase, it also includes maintaining the connection established by each client in the checkpoint buffering phase and the SBB server, so that the client uses the service from the same server in the two phases of checkpoint buffering and flushing; In the checkpoint flush phase, it also includes synchronizing the computing nodes of the PFS group and the SBB group when executing the computing task, that is, when the SBB group computing node sends a checkpoint flush RPC request to the server, the PFS group computing node blocks the computing, and when the SBB group computing node continues to execute the computing task, the PFS group computing node unblocks to continue the computing.

7. The method of claim 1-4, wherein, It also includes reading the checkpoint from the PFS to the corresponding area in the process address space of the specified state in the restart phase, to recover the computing of the application process before the failure.

8. An electronic device comprising a processor and a memory for storing a computer program, characterized in that The processor is configured to execute the computer program to execute the method of any one of claims 1-7.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-7.

Images