A parallel IO write consistency guarantee method based on state preheating

Abstract

This invention provides a parallel I / O write consistency guarantee method based on state preheating, belonging to the field of computer technology. This invention collects the I / O operation timestamp sequence of MPI processes and the inter-process communication topology, and uses a risk assessment model fused with temporal causal convolution and graph attention to output risk scores for each variable's write operation. It then uses an adaptive preheating timing estimation algorithm based on a queuing network to calculate the expected completion time variance of the I / O path, determining whether to trigger the preheating process. Finally, it uses a preheating intensity adjustment function to fuse the risk scores, MPI process size, and historical write success rate, triggering policy upgrades and retrying when deviations are detected. The execution results are updated with reinforcement learning reward signals to update the parameters of the artificial intelligence model and the weights of the policy decision model, achieving closed-loop adaptive adjustment. This solves the technical problem that parallel I / O write consistency guarantee mechanisms cannot adaptively adjust the synchronization strategy strength according to the real-time system load status and the risk of abnormal propagation between processes.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and more specifically, relates to a method for ensuring consistency of parallel I / O writes based on state preheating. Background Technology

[0002] Parallel I / O systems are widely used in large-scale scientific simulations in high-performance computing. Typical scenarios include checkpoint writing and data archiving in multiphysics coupled computations such as climate and ocean models. In these scenarios, thousands or even tens of thousands of MPI processes need to collaboratively write variable data to distributed file systems such as Lustre and GPFS through parallel I / O libraries such as PNetCDF. Existing technologies typically use fixed global barrier synchronization strategies or static threshold triggering mechanisms to ensure write consistency.

[0003] However, the fixed synchronization strategy cannot detect the real-time load fluctuations of OST service nodes, MDS service nodes and network link nodes. When the system is stable, it introduces unnecessary synchronization overhead, and when the load is abnormal, it may cause write failure due to insufficient strategy strength. The static threshold mechanism cannot dynamically capture the abnormal propagation path between processes, and it is difficult to distinguish between the first write and repeated write scenarios. In scenarios such as delayed allocation in Lustre file system, it is easy to cause misjudgment due to the lag in metadata update.

[0004] In other words, existing technologies have a technical problem where parallel I / O write consistency guarantee mechanisms cannot adaptively adjust the strength of synchronization strategies based on the real-time system load status and the risk of abnormal propagation between processes. Summary of the Invention

[0005] In view of this, the present invention provides a parallel IO write consistency guarantee method based on state preheating, which can solve the technical problem that the existing parallel IO write consistency guarantee mechanism cannot adaptively adjust the strength of the synchronization strategy according to the real-time load state of the system and the risk of abnormal propagation between processes.

[0006] This invention is implemented as follows: This invention provides a parallel I / O write consistency guarantee method based on state preheating, including the following steps:

[0007] The system collects the file inode generation time, PNetCDF internal file mode flag, and process local execution counter of the parallel IO system. The first write judgment result is obtained through a multi-level cross-validation mechanism. The system collects the IO operation timestamp sequence, inter-process communication topology snapshot, and write success rate of each MPI process. The data is then input into a risk assessment model that combines temporal causal convolution and graph attention to obtain the risk score of write operations for each variable.

[0008] Collect the real-time load status of each OST service node, MDS service node, and network link node, input the data into an adaptive preheating timing estimation algorithm based on a queuing network, and calculate the variance of the expected completion time of the IO path. When the variance of the expected completion time of the IO path exceeds the stability threshold, the preheating process is triggered; otherwise, the preheating process is skipped.

[0009] The system takes the risk score, the current number of MPI processes, and the historical write success rate as inputs, and calculates the comprehensive preheating intensity value through the preheating intensity adjustment function. Based on the range to which the comprehensive preheating intensity value belongs, it selects a strong synchronization strategy, a lightweight verification strategy, or a skip strategy to perform the state preheating operation.

[0010] Based on the selected strategy, the pre-synchronization phase, atomic write phase, and post-synchronization verification phase are executed. During the pre-synchronization phase, the pio_syncfile interface is called to enable all MPI processes to complete the synchronization of the preceding operations. During the atomic write phase, the pio_put_var interface is called in a globally consistent order to complete the data writing. During the post-synchronization verification phase, the root process reads the values ​​of key variables and compares them element by element with the expected values. If there is a discrepancy in the comparison, the strategy upgrade and retry process is triggered.

[0011] The policy execution time, write success rate, and expected completion time variance of the IO path for this IO process are recorded in the historical log. The parameters of the temporal causal convolution and graph attention fusion risk assessment model are updated with reinforcement learning reward signals that record successful writes as 1 and failed writes as -1, and the weights of the policy decision model are also updated.

[0012] Exceptions captured during the writing process are logged, and a retry process is automatically triggered based on a preset failure count threshold. The writing process is then switched to a strong synchronization strategy and re-executed. Failure events are recorded in the historical log for use in the next round of strategy decision model weight updates.

[0013] The temporal causal convolution and graph attention fusion risk assessment model consists of four parts: a single-process temporal feature extraction branch, an inter-process graph state aggregation branch, an anomaly subnet branch, and a risk score output layer. The outputs of the two branches are concatenated and then fused with the anomaly subnet branch through an adaptive jump connection module to output the risk score of each variable write operation.

[0014] The single-process temporal feature extraction branch is implemented by stacking causal dilated convolutional layers. Causal padding ensures that the prediction does not depend on future information, and extracts the IO operation temporal pattern within a single process.

[0015] The inter-process graph state aggregation branch models the inter-process communication topology as a dynamic graph, with nodes representing each MPI process. Edge weights are updated in real time by the inter-process message passing frequency, and a graph attention layer is introduced at each time step to aggregate the hidden state information of neighboring processes.

[0016] Specifically, the adaptive jump connection module jumps to the abnormal subnet branch for fine-grained analysis when the L2 norm of the output of the single-process temporal feature extraction branch exceeds the dynamic threshold. The abnormal subnet branch is composed of fully connected layers with ReLU activation function.

[0017] In the temporal causal convolution and graph attention fusion risk assessment model, regarding GPU memory allocation, the edge weight matrix of the graph state aggregation branch is allocated in GPU memory; regarding memory allocation, the convolution parameters of the single-process temporal feature extraction branch are allocated in CPU memory; the two branches achieve pipelined computation through asynchronous transmission of CUDA streams, and the number of CUDA streams... With the current number of MPI processes The relationship is .

[0018] The adaptive warm-up timing estimation algorithm based on the queuing network models the parallel IO system as an open Jackson queuing network. It uses the real-time load status of each service node to solve the flow balance equation and obtains the sum of the average dwell time variances of each service node as the expected completion time variance of the IO path. After the warm-up is completed, the load status is collected again and recalculated to form a closed-loop feedback.

[0019] Specifically, the stability threshold is obtained by running IO benchmark tests covering different MPI process sizes and file system types on the target platform. The variance of the expected completion time of the IO path when the write success rate first falls below the success rate threshold is used as a candidate stability threshold. The final stability threshold is determined after iterative verification.

[0020] The preheating intensity adjustment function is based on risk scoring. Current number of MPI processes and historical write success rate As input, calculate the comprehensive preheating intensity value. The formula is ,in , , The weighting coefficients and , This represents the maximum number of MPI processes currently supported by the platform.

[0021] In the strong synchronization strategy, a layered barrier strategy is used to perform the pre-synchronization phase. First, intra-node barriers based on shared memory are executed within the computing nodes, and then network-level barriers are executed between nodes. In addition, non-blocking collective operations are combined to pipeline the synchronization phase and the computing phase.

[0022] The multi-level cross-validation mechanism combines the file inode generation time, PNetCDF internal file mode flag, and process local runtime counter for comprehensive judgment. When all three pieces of information indicate that it is the first write, the first write judgment result is output as true; otherwise, it is false.

[0023] Specifically, the PNetCDF internal file mode flag intercepts PNetCDF key function calls through the LD_PRELOAD mechanism, records the call sequence and return values ​​of the ncmpi_create and ncmpi_def_var functions, establishes an externally visible state image, and provides the PNetCDF internal file mode flag in the state image to the multi-level cross-validation mechanism.

[0024] The training dataset for the temporal causal convolution and graph attention fusion risk assessment model is specifically obtained by running the CESM sea ice coupling mode on the Sunway high-performance computing platform, collecting IO operation timestamp sequences, write success rate labels, and inter-process communication topology snapshots under different MPI process sizes and different file system types. The ratio of abnormal samples to normal samples in the first write is 1:3, and the timestamp sequences are normalized based on the maximum timestamp of each process's IO operation.

[0025] Specifically, the training of the temporal causal convolution and graph attention fusion risk assessment model uses a weighted sum of mean squared error loss and reinforcement learning policy gradient loss as the total loss function, employs the Adam optimizer, and triggers early stopping if the loss does not decrease in multiple consecutive rounds of validation.

[0026] Wherein, the reference value for the preset failure count threshold is the 75th percentile of the number of retries in the write failure scenario IO test; the reference value for the stability threshold is the variance of the expected completion time of the IO path when the write success rate first falls below 99% in 100 to 200 benchmark tests; the weighting coefficient , , The reference value was determined by traversing the range of 0.1 to 0.9 with a step size of 0.1 using a grid search method and after 5 rounds of cross-validation.

[0027] This invention proposes a parallel I / O write consistency guarantee method based on state preheating. By constructing a risk assessment model that combines temporal causal convolution and graph attention, and combining an adaptive preheating timing estimation algorithm based on queuing network with a multi-parameter preheating intensity adjustment function, the real-time adaptive adjustment of the synchronization strategy strength is achieved.

[0028] This invention utilizes causal convolution to extract the timing pattern of single-process I / O operations and employs a graph attention mechanism to dynamically capture inter-process state dependencies, enabling risk scoring to perceive abnormal propagation paths in large-scale parallel scenarios. It also utilizes an open Jackson queuing network to quantitatively model multi-level queuing behavior of I / O paths, transforming the judgment of preheating trigger timing from empirical rules to quantitative inference based on real-time load status. Furthermore, it employs a multi-parameter preheating intensity adjustment function to uniformly map risk scoring, MPI process size, and historical write success rate into a comprehensive preheating intensity value, providing a quantifiable basis for strategy selection and avoiding the instability of single threshold judgments in boundary scenarios.

[0029] In summary, this invention solves the technical problem mentioned in the background art that the parallel IO write consistency guarantee mechanism cannot adaptively adjust the strength of the synchronization strategy according to the real-time load status of the system and the risk of abnormal propagation between processes. Attached Figure Description

[0030] Figure 1 This is a flowchart of the method of the present invention.

[0031] Figure 2 This is a risk score distribution diagram for the write operation of each variable in the risk assessment model of temporal causal convolution and graph attention fusion.

[0032] Figure 3 This is a distribution diagram of the average queue length for each OST service node.

[0033] Figure 4 This is a diagram showing the error distribution of each key variable in the post-synchronization verification stage.

[0034] Figure 5 A comparison chart of risk score distribution before and after the reinforcement learning reward signal update. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below.

[0036] like Figure 1 The diagram shown is a flowchart of a parallel I / O write consistency guarantee method based on state preheating provided by the present invention. This method includes the following steps:

[0037] S01. Collect the file inode generation time, PNetCDF internal file mode flag and process local execution counter of the parallel IO system, and obtain the first write judgment result through a multi-level cross-validation mechanism; collect the IO operation timestamp sequence, inter-process communication topology snapshot and write success rate of each MPI process, input the temporal causal convolution and graph attention fusion risk assessment model, and obtain the risk score of write operation for each variable.

[0038] S02. Collect the real-time load status of each OST service node, MDS service node and network link node, input the adaptive preheating timing estimation algorithm based on queuing network, and calculate the expected completion time variance of the IO path; when the expected completion time variance of the IO path exceeds the stability threshold, trigger the preheating process; when the expected completion time variance of the IO path does not exceed the stability threshold, skip the preheating process.

[0039] S03. Using the risk score, the current number of MPI processes, and the historical write success rate as inputs, calculate the comprehensive preheating intensity value through the preheating intensity adjustment function; based on the range to which the comprehensive preheating intensity value belongs, select a strong synchronization strategy, a lightweight verification strategy, or a skip strategy, and execute the state preheating operation.

[0040] S04. Based on the strategy selected in S03, execute the pre-synchronization phase, atomic write phase, and post-synchronization verification phase. In the pre-synchronization phase, the pio_syncfile interface is called to enable all MPI processes to complete the synchronization of the preceding operations. In the atomic write phase, the pio_put_var interface is called in a globally consistent order to complete the data writing. In the post-synchronization verification phase, the root process reads the values ​​of key variables and compares them element by element with the expected values. If there is a discrepancy in the comparison, the strategy upgrade and retry process is triggered.

[0041] S05. Record the policy execution time, write success rate and expected completion time variance of the IO path for this IO process to the historical log. Update the parameters of the temporal causal convolution and graph attention fusion risk assessment model with the reinforcement learning reward signal of writing success as 1 and writing failure as -1, and update the weights of the policy decision model.

[0042] S06. Log the exceptions captured during the writing process, automatically trigger the retry process according to the preset failure count threshold, switch to the strong synchronization strategy to re-execute the writing, and record the failure events to the historical log for the next round of strategy decision model weight update.

[0043] The specific structure of the temporal causal convolution and graph attention fusion risk assessment model is as follows: the model is divided into four parts: a single-process temporal feature extraction branch, an inter-process graph state aggregation branch, an anomaly subnetwork branch, and a risk score output layer. The single-process temporal feature extraction branch takes the IO operation timestamp sequence of each MPI process as input and is implemented using stacked causal dilated convolutional layers, with the dilation rate according to... , , , The algorithm progressively increases in size, with a total of four layers. Each layer has a convolutional kernel size of 3 and 64 channels. Causal padding ensures that predictions do not rely on future information. It extracts the temporal patterns of I / O operations within a single process, with an output dimension of 64. The inter-process graph state aggregation branch models the inter-process communication topology as a dynamic graph, with nodes representing each MPI process. Edge weights are updated in real-time based on the inter-process message passing frequency. A graph attention layer is introduced at each time step, with four attention heads, each with a dimension of 16, aggregating the hidden state information of neighboring processes, resulting in an output dimension of 64. The outputs of the two branches are concatenated along the feature dimension and then fed into an adaptive skip connection module. When the L2 norm of the single-process temporal feature extraction branch exceeds a dynamic threshold, the algorithm jumps to the anomaly subnet branch for fine-grained analysis. The anomaly subnet branch consists of two fully connected layers with a hidden layer dimension of 128 and a ReLU activation function. The outputs of the normal path and the anomaly subnet branch are then gated and weighted and fed into a risk scoring output layer, which outputs the risk score for each variable's write operation, with values ​​ranging from 0 to 1. Regarding GPU memory allocation, the edge weight matrix of the graph state aggregation branch is allocated in GPU memory; regarding memory allocation, the convolution parameters of the single-process temporal feature extraction branch are allocated in CPU memory; the number of CUDA streams... With the current number of MPI processes The relationship is ,in The unit is "piece". The unit is 1, and the two branches implement pipelined computation through asynchronous transmission of CUDA streams. The model uses a learning rate. A reward discount factor of 0.95 is used to fuse reinforcement learning reward signals for online parameter updates. The temporal causal convolution and graph attention fusion risk assessment model ensures the causality of temporal modeling through causal convolution, avoiding information leakage; it dynamically captures inter-process state dependencies through graph attention, enabling the model to perceive abnormal propagation paths between processes in large-scale parallel scenarios; and it dynamically switches between normal paths and abnormal subnet branches through adaptive skip connections, making the risk score highly sensitive to real abnormal events. This reduces missed and false judgments in preheating trigger decisions, improving the reliability and adaptability of overall IO write consistency assurance.

[0044] The steps for establishing the training dataset for the temporal causal convolution and graph attention fusion risk assessment model specifically include: running the CESM sea ice coupling mode on the Sunway high-performance computing platform, collecting IO operation timestamp sequences, write success rate labels, and inter-process communication topology snapshots under different MPI process sizes (128 to 16384 processes) and different file system types (Lustre file system, GPFS file system, BeeGFS file system), collecting no less than 10,000 samples in total, with the ratio of abnormal first write samples to normal samples being 1:3; normalizing the collected IO operation timestamp sequences based on the maximum timestamp of each process's IO operation; and dividing the training set, validation set, and test set into an 8:1:1 ratio.

[0045] The specific steps for training the temporal causal convolution and graph attention fusion risk assessment model include: using the weighted sum of mean squared error loss and reinforcement learning policy gradient loss as the total loss function, with a weight ratio of 0.7:0.3; employing the Adam optimizer with an initial learning rate of... The batch size is 32; the training rounds are 100, and the accuracy of the risk score is evaluated on the validation set every 10 rounds. If the loss does not decrease for 3 consecutive rounds of validation, early stopping is triggered; after training, the model weights are saved and deployed in the IO write consistency guarantee method.

[0046] The adaptive warm-up timing estimation algorithm based on queuing networks models the parallel I / O system as an open Jackson queuing network. Each OST service node, MDS service node, and network link node corresponds to a service node. The service rate of each service node is estimated in real time using an exponentially weighted moving average based on historical monitoring data. Before each write operation, the flow balance equation is solved using the real-time load status of each OST service node, MDS service node, and network link node to obtain the expected average dwell time and queue length of each service node. The variance of the average dwell time of each service node is summed to obtain the expected completion time variance of the I / O path. After warm-up, the real-time load status of each OST service node, MDS service node, and network link node is collected again, and the expected completion time variance of the I / O path is recalculated, forming a closed-loop feedback. The adaptive warm-up timing estimation algorithm based on queuing networks models the multi-level queuing behavior of the I / O system with a mathematical network, transforming the judgment of warm-up triggering timing from empirical rules to quantitative inference based on real-time load status. This enables timely warm-up triggering when the system load fluctuates greatly and reduces unnecessary synchronization overhead when the system is stable, thereby achieving an adaptive balance between reliability and performance.

[0047] The stability threshold is obtained by running 100 to 200 IO benchmark tests covering different MPI process sizes and file system types on the target platform, recording the expected completion time variance of the IO path and the corresponding write success rate for each test, and using the expected completion time variance of the IO path when the write success rate first falls below 99% as a candidate stability threshold. The final stability threshold is determined after 3 to 5 rounds of iterative verification.

[0048] The preheating intensity adjustment function is based on risk scoring. Current number of MPI processes and historical write success rate As input, calculate the comprehensive preheating intensity value. The formula is as follows: ;in , , These are the weighting coefficients. , The maximum number of MPI processes currently supported by the platform. , , All are dimensionless quantities. Dimensionless quantity; weighting coefficient , , The specific value was determined by running 50 to 100 comparative experiments on the target platform, aiming to maximize the write success rate. A grid search method was used to traverse the range of 0.1 to 0.9 with a step size of 0.1, and the result was determined after 5 rounds of cross-validation. When the comprehensive preheating intensity value meets... When the overall preheating intensity value meets the requirements, the skip strategy is selected, and no additional synchronization operation is performed; When a lightweight verification strategy is selected, the root process performs data verification after the write operation is completed; when the overall preheating intensity value meets the requirements... When a strong synchronization strategy is selected, the pio_syncfile interface is called to perform full MPI process synchronization before writing. The preheating intensity adjustment function uses multi-parameter weighted fusion to uniformly map the risk score, MPI process size, and historical write success rate into a comprehensive preheating intensity value, making the strategy selection have a quantifiable basis and avoiding the instability of a single threshold judgment in boundary scenarios.

[0049] The multi-level cross-validation mechanism combines the file inode generation time, PNetCDF internal file mode flag, and process local runtime counter for comprehensive judgment. When all three pieces of information indicate that it is the first write, the first write judgment result is output as true; otherwise, the first write judgment result is output as false, in order to avoid misjudgment caused by the lag in metadata updates in scenarios such as delayed allocation in the Lustre file system.

[0050] In the strong synchronization strategy, a layered barrier strategy is adopted to perform the pre-synchronization phase. First, intra-node barriers based on shared memory are executed within the computing nodes, and then network-level barriers are executed between nodes. In addition, non-blocking collective operations are combined to pipeline the synchronization phase and the computing phase, so as to reduce the global synchronization overhead in scenarios with a large number of MPI processes.

[0051] Specifically, the PNetCDF internal file mode flag intercepts PNetCDF key function calls through the LD_PRELOAD mechanism, inserts a state probe without modifying the PNetCDF library source code, records the call sequence and return values ​​of the ncmpi_create and ncmpi_def_var functions, establishes an externally visible state image, and provides the PNetCDF internal file mode flag in the state image to the multi-level cross-validation mechanism.

[0052] The pio_syncfile interface is a file-level synchronization interface provided by the PNetCDF parallel I / O library. After being called, it blocks until all MPI processes have completed the preliminary operations on the target file and the file metadata and data buffer are in a consistent state among all MPI processes.

[0053] The pio_put_var interface is a variable writing interface provided by the PNetCDF parallel I / O library, used to write MPI process local data into a parallel file according to globally consistent variable definitions.

[0054] The state image is a data structure maintained externally through the LD_PRELOAD interception mechanism, reflecting the key initialization state inside the PNetCDF library. It records the call sequence and return values ​​of the ncmpi_create and ncmpi_def_var functions of each MPI process, which can be queried by the multi-level cross-validation mechanism.

[0055] The Jackson queuing network is an open queuing network that satisfies the conditions that the service time of each service node is exponentially distributed and that the service nodes are independent of each other. Its steady-state solution is solved independently by each service node, making it suitable for load modeling and analysis of IO paths with multi-level serial service nodes.

[0056] The preset failure threshold is obtained by running 30 to 50 IO tests containing write failure scenarios on the target platform, recording the number of retries required from the first failure to successful completion of the retry in each test, taking the 75th percentile of the number of retries in all tests as the preset failure threshold, and determining the final preset failure threshold after 3 to 5 rounds of iterative verification.

[0057] In step S02, when calculating the expected completion time variance of the IO path, the expected completion time variance of the IO path is compared with a stability threshold. When the expected completion time variance of the IO path exceeds the stability threshold, warm-up is triggered. At the same time, the expected completion time variance of the IO path is input into the temporal causal convolution and graph attention fusion risk assessment model, and the weights of the reinforcement learning reward signal are adjusted to keep the update direction of the risk score consistent with the real-time queuing state.

[0058] In step S04, when a comparison deviation is detected during the post-synchronization verification stage, the strategy is upgraded to a strong synchronization strategy and a retry is performed. At the same time, the failure event and the corresponding risk score, comprehensive warm-up intensity value, and expected completion time variance of the IO path are recorded in the historical log for use in updating the weights of the strategy decision model in step S05.

[0059] Optionally, the present invention also provides a computer-based approach to form a parallel I / O write consistency guarantee system based on state preheating. The computer is equipped with a readable storage medium that stores program instructions, which execute the above-described method when the computer is run.

[0060] The following is a detailed explanation of all interfaces and functions in the solution:

[0061] The `pio_syncfile` interface is a file-level synchronization interface provided by the PNetCDF parallel I / O library. Its specific process is as follows: First, all MPI processes enter a blocked waiting state when calling the `pio_syncfile` interface. Second, the PNetCDF parallel I / O library initiates a collective communication operation internally to collect the buffer states of all MPI processes, confirming that all MPI processes have completed the pre-write operation on the target file. Third, the PNetCDF parallel I / O library forcibly flushes the data buffer contents held by each MPI process to the underlying parallel file system. Fourth, the PNetCDF parallel I / O library initiates a metadata synchronization request to the underlying parallel file system, waiting for the metadata server to complete the metadata update and return confirmation. Fifth, after the file metadata and data buffers of all MPI processes reach a consistent state, the blocking is lifted, and all MPI processes continue to execute subsequent operations. The calling form of the `pio_syncfile` interface is `pio_syncfile(File)`, where `File` is a file descriptor object maintained internally by the PNetCDF parallel I / O library, pointing to the currently operating parallel file.

[0062] The `pio_put_var` interface is a variable writing interface provided by the PNetCDF parallel I / O library. Its specific process is as follows: First, the calling MPI process submits variable identifiers, data start offsets, data counts, and local data buffer pointers to the PNetCDF parallel I / O library. Second, the PNetCDF parallel I / O library verifies whether the submitted variable identifiers have been registered during the file definition phase based on globally consistent variable definitions; if not registered, it returns an error code. Third, the PNetCDF parallel I / O library aggregates the local data submitted by each MPI process according to the global variable layout, generating a corresponding write request for the underlying parallel file system. Fourth, the PNetCDF parallel I / O library initiates a write operation to the underlying parallel file system in collective I / O mode, with all participating MPI processes collaboratively completing data transmission. Fifth, after the underlying parallel file system completes the data writing, the PNetCDF parallel I / O library returns a write status code to each MPI process, and the caller checks the status code to determine whether the write was successful. The pio_put_var interface is called in the form of pio_put_var(File, VarID, Start, Count, Buf), where File is a file descriptor object, VarID is a variable identifier, Start is an array of data start offsets, Count is an array of data counts for each dimension, and Buf is a pointer to the local data buffer.

[0063] The `ncmpi_create` function is a parallel file creation function provided by the PNetCDF library. Its specific process is as follows: First, the calling MPI process submits the target file path, file creation mode flag, and MPI communication domain to the PNetCDF library. Second, the PNetCDF library, with all MPI processes within the submitted MPI communication domain as participants, initiates a collective operation to collaboratively create the target file on the underlying parallel file system. Third, the PNetCDF library initializes key data structures such as the internal variable ID hash table and dimension metadata buffer, and writes the file creation mode flag into its internal state. Fourth, after the underlying parallel file system returns the file creation result, the PNetCDF library returns a file descriptor object and status code to the calling MPI process, and the file enters definition mode, awaiting subsequent registration of dimensions and variables. The ncmpi_create function is called in the form of ncmpi_create(MPI_Comm, path, cmode, MPI_Info, ncidp), where MPI_Comm is the MPI communication domain, path is the target file path, cmode is the file creation mode flag, MPI_Info is the IO prompt information object, and ncidp is the returned file identifier.

[0064] The `ncmpi_def_var` function is a variable definition function provided by the PNetCDF library. Its specific process is as follows: First, the calling MPI process submits a file identifier, variable name, data type, and dimension identifier list to the PNetCDF library. Second, the PNetCDF library checks if the current file is in definition mode; if not, it returns an error code. Third, the PNetCDF library assigns a unique variable identifier to the submitted variable name in its internal variable ID hash table and writes the data type and dimension identifier list to the dimension metadata buffer. Fourth, the PNetCDF library returns the assigned variable identifier and status code to the calling MPI process, completing the variable definition. The variable identifier is used as a variable index in subsequent `pio_put_var` interface calls. The calling form of the `ncmpi_def_var` function is `ncmpi_def_var(ncid, name, xtype, ndims, dimidsp, varidp)`, where `ncid` is the file identifier, `name` is the variable name, `xtype` is the data type, `ndims` is the number of dimensions, `dimidsp` is a pointer to the dimension identifier list, and `varidp` is the returned variable identifier.

[0065] The LD_PRELOAD mechanism is a runtime library interception mechanism provided by the Linux dynamic linker. Its specific process is as follows: First, before running the parallel I / O program, a custom dynamic link library path is specified through the environment variable LD_PRELOAD. Second, when loading the parallel I / O program, the Linux dynamic linker prioritizes binding function symbols with the same name as the PNetCDF library in the custom dynamic link library to the caller. Third, when the parallel I / O program calls the ncmpi_create or ncmpi_def_var function, it actually executes the wrapper function with the same name in the custom dynamic link library. Fourth, the wrapper function records the call timestamp and return value before and after execution, respectively, and writes them into an externally visible state image data structure. Fifth, the wrapper function calls the actual ncmpi_create or ncmpi_def_var function in the PNetCDF library to complete the original function, ensuring that the normal execution of the parallel I / O program is not affected. The LD_PRELOAD mechanism records the timing of key function calls and return values ​​without modifying the PNetCDF library source code, providing an externally visible source of PNetCDF internal file mode flags for multi-level cross-validation mechanisms.

[0066] The specific implementation of step S01 is as follows: First, the file inode generation time, PNetCDF internal file mode flag, and process local runtime counter of the parallel IO system are collected. A multi-level cross-validation mechanism is used to comprehensively judge these three pieces of information. Only when all three pieces of information indicate the first write operation is the first write result output as true; otherwise, it is false. The PNetCDF internal file mode flag is obtained through the LD_PRELOAD mechanism. This mechanism, without modifying the PNetCDF library source code, specifies a custom dynamic link library through environment variables. This allows the parallel IO program to actually execute the wrapper function when calling the ncmpi_create or ncmpi_def_var function. The wrapper function records the call sequence and return value and writes them to an externally visible state image, thus providing an externally visible source of the PNetCDF internal file mode flag for the multi-level cross-validation mechanism. This effectively avoids misjudgments caused by delayed metadata updates in Lustre file system delayed allocation scenarios. After completing the first write determination, the IO operation timestamp sequence, inter-process communication topology snapshot, and write success rate of each MPI process are collected and input into the temporal causal convolution and graph attention fusion risk assessment model. The single-process temporal feature extraction branch of this model uses stacked causal dilated convolutional layers, with the dilation rate according to... to The model employs a progressively increasing four-layer structure, using causal imputation to ensure predictions are independent of future information. The inter-process graph state aggregation branch models the communication topology as a dynamic graph, with edge weights updated in real-time by the inter-process message passing frequency. A graph attention layer aggregates the hidden states of neighboring processes. The outputs of the two branches are concatenated and then fed into an adaptive jump connection module. When the L2 norm of the single-process temporal feature extraction branch exceeds a dynamic threshold, the model jumps to the abnormal subnet branch for fine-grained analysis. The outputs of the normal path and the abnormal subnet branch are then gated and weighted before being fed into the risk score output layer, where the output risk score ranges from 0 to 1. The model uses a learning rate... The reward discount factor of 0.95 is used to integrate the reinforcement learning reward signal for online parameter updates.

[0067] The specific implementation of step S02 is as follows: Real-time load status of each OST service node, MDS service node, and network link node is collected and input into an adaptive warm-up timing estimation algorithm based on a queuing network. This algorithm models the parallel IO system as an open Jackson queuing network. The service rate of each service node is estimated in real-time using an exponentially weighted moving average based on historical monitoring data. The flow balance equation is solved using the real-time load status to obtain the expected average dwell time and queue length of each service node. The variances of the average dwell time of each service node are summed to obtain the expected completion time variance of the IO path. The stability threshold is obtained by running 100 to 200 IO benchmark tests covering different MPI process sizes and file system types on the target platform. The expected completion time variance of the IO path when the write success rate first falls below 99% is used as a candidate stability threshold. The final stability threshold is determined after 3 to 5 rounds of iterative verification. When the expected completion time variance of the IO path exceeds the stability threshold, the warm-up process is triggered; otherwise, it is skipped. After warm-up, the real-time load status of each node is re-collected and the expected completion time variance of the IO path is recalculated, forming a closed-loop feedback.

[0068] The specific implementation method of step S03 is: using risk scoring Current number of MPI processes and historical write success rate As input, the comprehensive preheating intensity value is calculated using the preheating intensity adjustment function. ,in The maximum number of MPI processes currently supported by the platform, weighted by a certain factor. , , satisfy By running 50 to 100 comparative experiments on the target platform to maximize the write success rate, a grid search method was used to traverse the range of 0.1 to 0.9 with a step size of 0.1. This was determined after 5 rounds of cross-validation. Choose the skip strategy when; When a lightweight verification strategy is selected, the root process performs data verification after the write operation is complete; when When choosing a strong synchronization strategy, the pio_syncfile interface is called to perform full MPI process synchronization before writing.

[0069] The specific implementation of step S04 is as follows: Three stages are executed according to the selected strategy. The pre-synchronization stage calls the `pio_syncfile` interface, which blocks until all MPI processes complete their preliminary operations and forcibly flushes the data buffer content to the underlying parallel file system, completing metadata synchronization. Under the strong synchronization strategy, the pre-synchronization stage adopts a layered barrier strategy, first executing intra-node barriers based on shared memory within the compute nodes, then executing network-level barriers between nodes, and combining this with non-blocking collective operations for pipelined processing, reducing the global synchronization overhead in scenarios with a large number of MPI processes. The atomic write stage calls the `pio_put_var` interface, which aggregates the local data of each MPI process according to globally consistent variable definitions and writes it to the underlying parallel file system in collective IO mode, returning a write status code to each MPI process. The post-synchronization verification stage involves the root process reading the key variable values ​​and comparing them element-by-element with the expected values. If a deviation exists, the strategy is upgraded to a strong synchronization strategy and a retry is performed. Simultaneously, the failure event, corresponding risk score, comprehensive warm-up intensity value, and expected completion time variance of the IO path are recorded in the historical log.

[0070] The specific implementation of step S05 is as follows: Record the policy execution time, write success rate, and expected completion time variance of the IO path for this IO process to the historical log. Use 1 for successful writes and -1 for failed writes as reinforcement learning reward signals, and employ a learning rate... The parameters of the temporal causal convolution and graph attention fusion risk assessment model are updated with a reward discount factor of 0.95, and the weights of the policy decision model are updated simultaneously. The expected completion time variance of the IO path is also input into the risk assessment model to adjust the weights of the reinforcement learning reward signal, so that the update direction of the risk score is consistent with the real-time queuing status.

[0071] The specific implementation of step S06 is as follows: Exceptions captured during the write process are logged. The reference method for obtaining the preset failure threshold is to run 30 to 50 IO tests containing write failure scenarios on the target platform, and take the 75th percentile of the number of retries required from the first failure to successful completion in all tests. This is determined after 3 to 5 rounds of iterative verification. When the cumulative number of failures reaches the preset failure threshold, the system automatically switches to a strong synchronization strategy to re-execute the write operation, and the failure event is recorded in the historical log for use in the next round of strategy decision model weight updates, forming a complete exception self-healing closed loop.

[0072] It should be noted that the key technologies of this invention include: a risk assessment model that combines temporal causal convolution and graph attention. Causal convolution ensures the causality of temporal modeling, while graph attention dynamically captures inter-process state dependencies, making the risk score highly sensitive to abnormal propagation paths. Compared to traditional risk assessment methods based on statistical rules, this model can more accurately identify potential write failure risks in large-scale parallel scenarios. An adaptive warm-up timing estimation algorithm based on queuing networks models the multi-level queuing behavior of the IO system using Jackson queuing networks, transforming warm-up triggering from empirical rules to quantitative inference based on real-time load status. Compared to static threshold mechanisms, this algorithm can trigger warm-up in a timely manner when the load fluctuates significantly and reduces unnecessary synchronization overhead when the system is stable. A warm-up intensity adjustment function, through multi-parameter weighted fusion, maps the risk score, MPI process size, and historical write success rate into a unified comprehensive warm-up intensity value, providing a quantifiable basis for strategy selection. These three key technologies work synergistically to form a mutually constraining adaptive closed loop between risk perception, triggering timing, and strategy intensity, significantly improving the reliability and resource utilization efficiency of parallel IO write consistency assurance.

[0073] It should be noted that this invention also solves the following technical problem: In existing technologies, write failure events in parallel I / O systems typically only trigger simple retries, lacking correlation modeling between failure events and system states, resulting in the policy decision-making model being unable to continuously optimize from historical failures. This invention solves the technical problem in parallel I / O write consistency guarantee methods where the policy decision-making model cannot continuously self-optimize from historical failure events by synchronously recording the risk score, comprehensive warm-up intensity value, and expected completion time variance of each failure event to a historical log, and using reinforcement learning reward signals to drive online updates of model parameters. This allows the policy decision-making model to continuously learn from historical failure events, gradually optimizing the accuracy of risk scores and the rationality of policy selection.

[0074] Specifically, the principle of this invention is as follows: The reason this invention can solve the aforementioned technical problems is that: the risk assessment model ensures the causality of temporal modeling through causal convolution, avoiding information leakage; it dynamically aggregates the hidden states of neighboring processes through a graph attention mechanism, enabling the model to perceive abnormal propagation paths between processes, thereby outputting a risk score with high sensitivity to real abnormal events; the queuing network model transforms the real-time load status of OST, MDS, and network link nodes into the expected completion time variance of IO paths, providing a quantitative basis for triggering preheating; the preheating intensity adjustment function weights and fuses the risk score, MPI process size, and historical write success rate, enabling the comprehensive preheating intensity value to distinguish between three strategy ranges: skip, lightweight verification, and strong synchronization; the reinforcement learning reward signal drives the online update of model parameters, ensuring that the update direction of the risk score is consistent with the real-time queuing status, forming a closed-loop adaptive adjustment mechanism. The logical progression of each of these steps collectively ensures the reliability and adaptive capability of parallel IO write consistency.

[0075] The following provides a specific embodiment 1 of the present invention, and the specific implementation of each step in this embodiment 1 is described in detail below.

[0076] The specific implementation of step S01 involves collecting the IO operation timestamp sequence, inter-process communication topology snapshot, and write success rate of each MPI process, inputting them into a temporal causal convolution and graph attention fusion risk assessment model, and obtaining a risk score for each variable's write operation. This model consists of four parts: a single-process temporal feature extraction branch, an inter-process graph state aggregation branch, an anomaly subnet branch, and a risk score output layer.

[0077] The single-process temporal feature extraction branch is implemented using stacked causal dilated convolutional layers, with a total of 4 layers. The expansion rate of the layer is ( Each convolutional layer has a kernel size of 3, 64 channels, and an output dimension of 64. Causal padding ensures that predictions do not depend on future information. Layer at time Convolution output The calculation formula is expressed as follows:

[0078] ;

[0079] In the formula, For the first Layer at time The output vector, For the first Layer Each convolutional kernel weight, For the first The input vector of the layer at the causal displacement position. The input vector is the original timestamp sequence. It is a linear rectified activation function, i.e. , This is the kernel position index, with values ​​of 0, 1, and 2. For convolutional layer index, For time indexing, all the above quantities are dimensionless.

[0080] Inter-process graph state aggregation branching models the inter-process communication topology as a dynamic graph. ,node For the first Each MPI process, edge weight The graph attention layer is updated in real time based on the inter-process message passing frequency. It has 4 attention heads, each with a dimension of 16, resulting in an output dimension of 64. (Nodes) In the Attention coefficient of each attention head The calculation formula is expressed as follows:

[0081] ;

[0082] In the formula, For the first Nodes in each attention head To the neighbors The attention coefficient is a dimensionless quantity. For the first The linear transformation matrix of each attention head. , , They are nodes , , The hidden state vector. For the first A learnable weight vector for each attention head. The symbol for vector transpose. This represents a vector concatenation operation. For nodes The set of neighboring nodes, , They are nodes With nodes , The edge weights between them are all dimensionless quantities. For a linear rectified function with leakage, For attention head index, , , Indexing nodes. Nodes graph aggregation output The calculation formula is expressed as follows:

[0083] ;

[0084] In the formula, For nodes The output vector after multi-head graph attention aggregation has a dimension of 64. This indicates that the outputs of the four attention heads are concatenated.

[0085] Two-branch output and After concatenation along the feature dimensions, the data is fed into the adaptive skip connection module. When the output of the single-process temporal feature extraction branch... Norm exceeds dynamic threshold When this occurs, the system redirects to the abnormal subnet branch. Dynamic threshold. The calculation formula is expressed as follows:

[0086] ;

[0087] In the formula, The dynamic threshold is a dimensionless quantity. For historical batches of The mean of the norm is a dimensionless quantity. For historical batches of The standard deviation of the norm is a dimensionless quantity. is the sensitivity coefficient, a dimensionless quantity, with an empirical value of 2.0. The anomaly subnet branch consists of two fully connected layers with a hidden layer dimension of 128 and an activation function of . Its output The calculation formula is expressed as follows:

[0088] ;

[0089] In the formula, , The weight matrix for the two fully connected layers of the anomalous subnet branch is shown below. , For the corresponding bias vector The output vectors for the abnormal subnet branches are shown above; all quantities are dimensionless. Normal path output. The calculation formula is expressed as follows:

[0090] ;

[0091] In the formula, The normal path linear transformation matrix, This is the normal path bias vector. The output vector is a linear mapping of the normal path. All of the above quantities are dimensionless. and After gating and weighted fusion, the fused output is... The calculation formula is expressed as follows:

[0092] ;

[0093] In the formula, , For the Sigmoid function, i.e. , For the gated weight matrix, This is the gated bias vector. Here, is the gating coefficient, which is a dimensionless quantity. The output vector after fusion is a dimensionless quantity. The data is fed into the risk scoring output layer to output the risk score. The calculation formula is expressed as follows:

[0094] ;

[0095] In the formula, Risk scoring, with a value range of: , is a dimensionless quantity. This is the output layer weight matrix. The above quantities are dimensionless and represent the output layer bias vector.

[0096] Edge weight matrix of graph state aggregation branch Convolution parameters for the single-process temporal feature extraction branch, allocated to GPU memory, are allocated to CPU memory; the number of CUDA streams... With the number of MPI processes The relationship is:

[0097] ;

[0098] In the formula, For the number of CUDA streams, This represents the current number of MPI processes. It is a rounding function. and All quantities are dimensionless. The model uses mean squared error loss. Gradient loss of reinforcement learning strategy The weighted sum is the total loss. The calculation formula is expressed as follows:

[0099] ;

[0100] In the formula, For the total loss, For mean square error loss, To enhance the learning strategy gradient loss, all three are dimensionless. The calculation formula is expressed as follows:

[0101] ;

[0102] In the formula, This represents the batch sample size; an empirical value is 32. For the first Predicted risk score for each sample For the first Risk score label value for each sample The above quantities are dimensionless and serve as sample indices. The model uses the Adam optimizer with an initial learning rate of... The batch size is 32, the training is conducted for 100 rounds, the reward discount factor is 0.95, and the model parameters are updated online using reinforcement learning reward signals that record successful writes as 1 and failed writes as -1.

[0103] The specific implementation of step S02 is to model the parallel I / O system as an open Jackson queuing network, where each service node corresponds to a queuing node, and the service rate of each service node is estimated in real time using an exponentially weighted moving average. The service node Service rate (estimated) The calculation formula is expressed as follows:

[0104] ;

[0105] In the formula, For the first The service node The estimated service rate is given in times per second. The service rate was estimated previously, in times per second. For the first The measured service rate is expressed as times per second. This is the smoothing coefficient, a dimensionless quantity, with an empirical value of 0.1. For service node indexing, For estimating the frequency index. Solving the flow balance equation based on real-time load status, arrival rate. The calculation formula is expressed as follows:

[0106] ;

[0107] In the formula, For the first Arrival rate of each service node, in times / second. The total external arrival rate of the system, expressed in times per second. For external requests, directly enter the first... The probability of a node is a dimensionless quantity. To request from node Routing to node The transition probability is a dimensionless quantity that satisfies... , The total number of service nodes. Index for service nodes. Average dwell time for each node. The calculation formula is expressed as follows:

[0108] ;

[0109] In the formula, For the first The average dwell time of each node, in seconds. For the first The current estimated service rate of each node, in times / s. For the first The arrival rate of each node, in seconds, is required. Variance of dwell time at each node The calculation formula is expressed as follows:

[0110] ;

[0111] In the formula, For the first The variance of the average dwell time of each node, in units of Expected completion time variance of the IO path The calculation formula is expressed as follows:

[0112] ;

[0113] In the formula, The expected completion time variance of the IO path, in units of Stability threshold The units are the same .when Exceed The preheating process is triggered when the time is right, otherwise it is skipped.

[0114] The specific implementation of step S03 is to comprehensively measure the preheating intensity value. The calculation formula is expressed as follows:

[0115] ;

[0116] In the formula, The comprehensive preheating intensity value is a dimensionless quantity. Risk scoring is a dimensionless quantity. This represents the current number of MPI processes. This represents the maximum number of MPI processes supported by the platform. It is a dimensionless quantity. To write the success rate into history, to be a dimensionless quantity. , , These are weighting coefficients, all of which are dimensionless and satisfy the following conditions: The coefficients were determined by traversing the range of 0.1 to 0.9 using a grid search method with a step size of 0.1, and through five rounds of cross-validation. When choosing a skip strategy, when Choose a lightweight verification strategy when Choose a strong synchronization strategy.

[0117] The specific implementation of steps S04 to S06 involves three stages: pre-synchronization, atomic write, and post-synchronization verification, based on the selected strategy. The strong synchronization strategy employs a layered barrier strategy, first executing shared memory barriers within nodes, then executing network-level barriers between nodes. Non-blocking collective operations are used to pipeline the synchronization and computation stages to reduce overhead. In the post-synchronization verification stage, the root process compares the values ​​of key variables element by element with the expected values. If a discrepancy exists, a strategy upgrade and retry are triggered. Step S05 records the strategy execution time, write success rate, and... Record the data to the historical log and update the risk assessment model parameters and strategy decision model weights with reinforcement learning reward signals. In step S06, the captured anomalies are logged, and retries are automatically triggered based on a preset failure count threshold. The write operation is then re-executed using a strong synchronization strategy. The preset failure count threshold is determined by running 30 to 50 IO tests on the target platform that include write failure scenarios and taking the 75th percentile of the retry count.

[0118] To better understand and implement this invention, the following is a specific application scenario of the invention, Example 2: To verify the effectiveness of the invention, technicians built a test environment and deployed the CESM sea ice coupling mode on the Sunway high-performance computing platform. Using the Lustre file system, the parallel IO write consistency guarantee method based on state preheating of the invention was run with a scale of 4096 MPI processes. The complete process verification of the typical parallel IO write consistency guarantee scenario was carried out.

[0119] The test environment configuration is as follows: Sunway high-performance computing platform, 256 nodes, 16 MPI processes per node, Lustre file system, 32 OST service nodes, 2 MDS service nodes, and network link bandwidth of 100. The IO test task was a checkpoint write in CESM sea ice mode, with 128 variables written in a single operation and a single variable data size of 512. The total amount of data written is approximately 64. .

[0120] First, the technicians ran 150 IO benchmark tests on the target platform to collect the write success rate and the corresponding expected completion time variance of the IO path under different MPI process sizes and file system types. The variance value corresponding to the first time the write success rate is lower than 99% was used as the candidate stability threshold. After four rounds of iterative verification, the stability threshold was determined. The write success rate under different process sizes in the benchmark test is shown in Table 1.

[0121] Table 1. Statistics on IO write success rate under different MPI process sizes

[0122]

[0123] As shown in Table 1, as the size of the MPI process increases, the variance of the expected completion time of the IO path increases significantly, and the write success rate continues to decline. The stability threshold was finally determined to be 0.85.

[0124] Subsequently, technicians conducted 60 comparative experiments, using a grid search method to determine the weighting coefficients of the preheating intensity adjustment function, and finalized the results after 5 rounds of cross-validation. , , This corresponds to the optimal write success rate.

[0125] In the formal test involving 4096 MPI processes, step S01 first intercepts the calls to the `ncmpi_create` and `ncmpi_def_var` functions using the LD_PRELOAD mechanism, recording the call timing and return values ​​to establish a state image. A multi-level cross-validation mechanism, combining file inode generation time, PNetCDF internal file mode flags, and the process's local runtime counter, determines that this is the first write operation, outputting a true first write determination result. A temporal causal convolution and graph attention fusion risk assessment model processes the IO operation timestamp sequence and inter-process communication topology snapshots of the 4096 MPI processes, such as... Figure 2 The figure shows the distribution of risk scores for each variable in the model. It can be seen that the risk scores of some variables are concentrated in the range of 0.6 to 0.8, indicating a moderate to high write risk.

[0126] In step S02, the adaptive preheating timing estimation algorithm based on the queuing network solves the traffic balance equation for the real-time load status of 32 OST service nodes, 2 MDS service nodes, and network link nodes. The calculated expected completion time variance of the IO path is 1.21, which exceeds the stability threshold of 0.85, triggering the preheating process. Figure 3The figure shows the average queue length distribution of each OST service node. There is a significant backlog of queue lengths on some nodes, which is consistent with the result that the variance of the expected completion time of the IO path exceeds the threshold.

[0127] In step S03, the average risk score of 0.67, the current number of MPI processes of 4096, and the historical write success rate of 0.981 are substituted into the preheating intensity adjustment function. Taking 16384, the comprehensive preheating intensity value is calculated. ,fall into For each range, a lightweight verification strategy is selected, and the root process performs data verification after the write operation is completed.

[0128] In step S04, the pre-synchronization phase is executed, calling the pio_syncfile interface to synchronize all 4096 MPI processes before the preceding operations. In the atomic write phase, the pio_put_var interface is called to write the local data of 128 variables to the Lustre file system in collective I / O mode according to globally consistent variable definitions. In the post-synchronization verification phase, the root process reads the values ​​of 8 key variables and compares them element-by-element with the expected values. No discrepancies were found in this comparison, and the write was successful. Figure 4 The figure shows the comparison error distribution of each key variable in the post-synchronization verification stage during this write process. The comparison error of all variables is 0, which verifies the consistency of the write.

[0129] In step S05, the policy execution time, write success rate (1.0), and expected completion time variance (1.21) of this IO process are recorded in the historical log. This is used to update the parameters of the temporal causal convolution and graph attention fusion risk assessment model and the weights of the policy decision model, based on the reinforcement learning reward signal 1. The distribution of the risk score changes after the model parameter update as follows: Figure 5 As shown, the model's sensitivity in identifying high-risk variables has been further improved.

[0130] In 100 consecutive write tests, the technicians recorded the strategy selection and write results for each write, as shown in Table 2, which is a statistical distribution of strategy selection.

[0131] Table 2. Distribution Statistics of Strategy Selection in 100 Consecutive Write Tests

[0132]

[0133] As shown in Table 2, the lightweight verification strategy is frequently selected in medium-risk scenarios, the strong synchronization strategy is accurately triggered in high-risk scenarios, and the skip strategy effectively reduces synchronization overhead in low-risk scenarios. The overall write success rate is 99 times / 100 times, indicating that the adaptive strategy selection mechanism of the present invention can achieve an effective balance between reliability and performance.

[0134] In the test of the write failure scenario, the technicians artificially injected a sudden increase in node load to trigger the write failure. Step S06 automatically switched to the strong synchronization strategy to perform a retry. After the retry was successful, the failure event was recorded to the historical log and the weight of the strategy decision model was updated in the next round.

[0135] Compared to traditional methods, the advancements of this invention are reflected in the following aspects. Traditional fixed global barrier synchronization strategies treat synchronization strength as a constant, failing to perceive changes in real-time system load. This leads to unnecessary synchronization overhead when the system is stable, and underperforms due to insufficient strategy strength during abnormal load conditions, resulting in missed write failure risks. This invention models the IO system as an open Jackson queuing network, providing a quantitative basis for preheating triggering. It can respond promptly during load fluctuations and proactively reduce synchronization overhead when the system is stable, thus overcoming the fundamental deficiency of traditional methods that rely on empirical rules from a mathematical modeling perspective. Furthermore, traditional methods cannot perceive abnormal propagation paths between processes. This invention incorporates inter-process communication topology into risk assessment through a graph attention mechanism, enabling risk scores to reflect abnormal propagation behavior in large-scale parallel scenarios, thereby reducing missed and false positives in preheating triggering decisions. In addition, the strategy decision-making model of traditional methods remains fixed after deployment. This invention uses reinforcement learning reward signals to drive online updates of model parameters, enabling the strategy decision-making model to continuously self-optimize with changes in the operating environment, possessing an adaptive evolution capability not found in traditional methods.

[0136] It should be noted that the variables involved in this invention are explained in detail in Tables 3 and 4.

[0137] Table 3. Variable Explanation Table (Part 1)

[0138]

[0139] Table 4. Variable Explanation Table (Part Two)

[0140]

[0141] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A parallel I / O write consistency guarantee method based on state preheating, characterized in that, Includes the following steps: The system collects the file inode generation time, PNetCDF internal file mode flag, and process local execution counter of the parallel IO system. The first write judgment result is obtained through a multi-level cross-validation mechanism. The system collects the IO operation timestamp sequence, inter-process communication topology snapshot, and write success rate of each MPI process. The data is then input into a risk assessment model that combines temporal causal convolution and graph attention to obtain the risk score of write operations for each variable. Collect the real-time load status of each OST service node, MDS service node, and network link node, input the data into an adaptive preheating timing estimation algorithm based on a queuing network, and calculate the variance of the expected completion time of the IO path. When the variance of the expected completion time of the IO path exceeds the stability threshold, the preheating process is triggered; otherwise, the preheating process is skipped. The comprehensive preheating intensity value is calculated using the risk score, the current number of MPI processes, and the historical write success rate as inputs, through the preheating intensity adjustment function. Based on the range of the comprehensive preheating intensity value, select a strong synchronization strategy, a lightweight verification strategy, or a skip strategy to perform the state preheating operation. Based on the selected strategy, the pre-synchronization phase, atomic write phase, and post-synchronization verification phase are executed. During the pre-synchronization phase, the pio_syncfile interface is called to enable all MPI processes to complete the synchronization of the preceding operations; during the atomic write phase, the pio_put_var interface is called in a globally consistent order to complete the data writing; during the post-synchronization verification phase, the root process reads the values ​​of key variables and compares them element by element with the expected values. If there is a discrepancy in the comparison, the policy upgrade and retry process is triggered. The policy execution time, write success rate, and expected completion time variance of the IO path for this IO process are recorded in the historical log. The parameters of the temporal causal convolution and graph attention fusion risk assessment model are updated with reinforcement learning reward signals that record successful writes as 1 and failed writes as -1, and the weights of the policy decision model are also updated. Exceptions captured during the writing process are logged, and a retry process is automatically triggered based on a preset failure count threshold. The writing process is then switched to a strong synchronization strategy and re-executed. Failure events are recorded in the historical log for use in the next round of strategy decision model weight updates.

2. The parallel I / O write consistency guarantee method based on state preheating according to claim 1, characterized in that, The temporal causal convolution and graph attention fusion risk assessment model consists of four parts: a single-process temporal feature extraction branch, an inter-process graph state aggregation branch, an anomaly subnet branch, and a risk score output layer. The outputs of the two branches are concatenated and then fused with the anomaly subnet branch through an adaptive jump connection module to output the risk score of each variable write operation.

3. The parallel I / O write consistency guarantee method based on state preheating according to claim 2, characterized in that, The single-process temporal feature extraction branch is implemented by stacking causal dilated convolutional layers. Causal padding ensures that the prediction does not depend on future information, and extracts the IO operation temporal pattern within a single process.

4. The parallel I / O write consistency guarantee method based on state preheating according to claim 3, characterized in that, The inter-process graph state aggregation branch models the inter-process communication topology as a dynamic graph, with nodes representing each MPI process. Edge weights are updated in real time by the inter-process message passing frequency, and a graph attention layer is introduced at each time step to aggregate the hidden state information of neighboring processes.

5. The parallel I / O write consistency guarantee method based on state preheating according to claim 4, characterized in that, Specifically, when the L2 norm of the output of the single-process temporal feature extraction branch exceeds the dynamic threshold, the adaptive jump connection module jumps to the abnormal subnet branch for fine-grained analysis. The abnormal subnet branch is composed of fully connected layers with ReLU activation function.

6. The parallel I / O write consistency guarantee method based on state preheating according to claim 5, characterized in that, Regarding the memory allocation of the temporal causal convolution and graph attention fusion risk assessment model, the edge weight matrix of the graph state aggregation branch is allocated in the GPU memory. In terms of memory allocation, the convolution parameters of the single-process temporal feature extraction branch are allocated to CPU memory; the two branches achieve pipelined computation through asynchronous transmission of CUDA streams, and the number of CUDA streams... With the current number of MPI processes The relationship is .

7. The parallel I / O write consistency guarantee method based on state preheating according to claim 6, characterized in that, The adaptive preheating timing estimation algorithm based on queuing network models the parallel IO system as an open Jackson queuing network. It uses the real-time load status of each service node to solve the flow balance equation and obtains the sum of the average dwell time variances of each service node as the expected completion time variance of the IO path. After preheating, the load status is collected again and recalculated to form a closed-loop feedback.

8. The parallel I / O write consistency guarantee method based on state preheating according to claim 7, characterized in that, The stability threshold is obtained by running IO benchmark tests covering different MPI process sizes and file system types on the target platform. The variance of the expected completion time of the IO path when the write success rate first falls below the success rate threshold is used as a candidate stability threshold. The final stability threshold is determined after iterative verification.

9. The parallel I / O write consistency guarantee method based on state preheating according to claim 8, characterized in that, The training dataset for the temporal causal convolution and graph attention fusion risk assessment model is specifically obtained by running the CESM sea ice coupling mode on the Sunway high-performance computing platform, collecting IO operation timestamp sequences, write success rate labels, and inter-process communication topology snapshots under different MPI process sizes and different file system types. The ratio of abnormal samples to normal samples in the first write is 1:

3. The timestamp sequences are normalized based on the maximum timestamp of each process's IO operation.

10. The parallel I / O write consistency guarantee method based on state preheating according to claim 9, characterized in that, In the strong synchronization strategy, a layered barrier strategy is adopted to perform the pre-synchronization phase. First, intra-node barriers based on shared memory are executed within the computing nodes, and then network-level barriers are executed between nodes. The synchronization phase and the computing phase are pipelined by combining non-blocking collective operations.

Images