System fault tolerance method, product, device, equipment and storage medium
By predicting and selecting checkpoint writing methods and dynamically adjusting fault tolerance strategies, checkpoint operations are carried out in advance, and the recovery time extension and overhead problems caused by unfinished writes of checkpoints in the existing technology are solved, and the system fault tolerance and user experience are improved .
Patent Information
- Application Number
- CN202411150051.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-22
AI Technical Summary
When the existing checkpoint technology fails to write, the probability of the latest checkpoint not being written increases, resulting in a longer system recovery time, and the checkpoint strategy cannot effectively reduce overhead and recalculation overhead.
By judging the preset overstep interval, predicting the overall running time of blocking and non-blocking checkpoint writing methods, selecting a smaller value writing method, and dynamically adjusting the fault tolerance strategy based on fault prediction and historical information, and performing checkpoint operations in advance Reduce read and recalculate overhead.
Checkpoint operations in advance before a fault occurs, reducing read response time and recalculation overhead, improving system fault tolerance and user experience, avoiding system pauses, and reducing checkpoint write and recovery time.
Smart Images

Figure CN119003245A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of fault tolerance technology, and in particular to a system fault tolerance method, product, device, equipment and storage medium. Background Art
[0002] In the field of high-performance computing, parallel computing models such as the BSP (Bulk Synchronous Parallel) model are usually used for parallel computing. In the process of parallel computing, in order to ensure the reliability and efficiency of large-scale parallel computing tasks and to deal with failures that occur during the computing process, checkpoint technology is usually used to save the state of the system so that the system can be restored to the previously saved state when a failure occurs, thereby achieving the fault-tolerant purpose of quickly resuming normal operation of the system. In addition, users can specify the superstep interval for checkpoint operations so that the system triggers a write checkpoint operation after the superstep that meets the interval requirements completes the computing task.
[0003] At present, checkpoint technology can be divided into blocking checkpoints and non-blocking checkpoints based on whether it is executed in parallel with the computing task. Among them, blocking checkpoints refer to the system first suspending the current computing task, and then writing the checkpoint to permanent storage before continuing the subsequent computing task. In the absence of faults, blocking checkpoints will cause obvious time overhead in execution time; non-blocking checkpoints allow computing tasks to be performed in parallel with the generation and writing of checkpoints. Although the generation and writing of checkpoints in this way will take longer, parallel operations will ensure that the checkpoint operation does not affect the progress of the computing task, thereby reducing the execution time overhead in the absence of faults. However, due to the longer time for checkpoint generation and writing, the probability that the latest checkpoint has not been completed when the system fails increases, so the system may need to roll back to an earlier checkpoint, which will increase the system recovery time and may also cause additional execution time overhead.
[0004] In summary, a single checkpoint strategy cannot achieve the goal of minimizing the checkpoint overhead. To solve the above problems, a hybrid checkpoint writing method is currently proposed. This method adaptively determines the checkpoint writing strategy by comparing the checkpoint overhead of blocking and non-blocking methods. However, this method does not consider the situation where the failure occurs after the non-blocking checkpoint is completed when calculating the checkpoint cost, which is likely to occur in actual operation. In addition, the checkpoint period setting is also crucial. Too frequent will cause large additional overhead, and too sparse may cause large recalculation overhead when a failure occurs.
[0005] Therefore, how to perform a write checkpoint operation to improve the fault tolerance capability of the system is a technical problem that needs to be further solved in the art. Summary of the invention
[0006] In view of this, the purpose of this application is to provide a system fault tolerance method, product, device, equipment and storage medium, which can perform checkpoint operations before a fault is about to occur, so that when a fault occurs, the checkpoint can be directly obtained from the local to perform fault recovery without transmitting it from the remote end to the local node, thereby reducing the response time of the read operation. At the same time, it can flexibly utilize the system's I / O resources, reduce the time overhead of checkpoint operations and the time overhead of recalculation, and can also avoid system suspension (i.e. interruption) caused by faults, ensure the normal operation of the business, thereby improving the system's fault tolerance capabilities and user experience. The specific scheme is as follows:
[0007] In a first aspect, the present application discloses a system fault tolerance method, comprising:
[0008] Determine whether the preset superstep interval has been reached;
[0009] If the preset superstep interval is reached, the overall running time using the blocking checkpoint writing mode and the non-blocking checkpoint writing mode are predicted respectively to obtain a first overall running time and a second overall running time, and a write checkpoint operation is performed according to the checkpoint writing mode corresponding to the smaller value of the first overall running time and the second overall running time;
[0010] If the preset superstep interval is not reached, whether the system will fail is predicted based on the current log information, as well as the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur, and different fault-tolerant strategies are selected based on the comparison result of the advance fault detection time with a preset duration threshold, so that when a system failure occurs, fault recovery is performed based on the fault-tolerant strategy; the advance fault detection time is the length of time from the predicted fault occurrence time to the current time.
[0011] Optionally, respectively predicting the overall running time using a blocking checkpoint writing mode and a non-blocking checkpoint writing mode to obtain a first overall running time and a second overall running time, and performing a checkpoint writing operation according to a checkpoint writing mode corresponding to a smaller value of the first overall running time and the second overall running time, includes:
[0012] Starting from the starting superstep, the write time overhead of each superstep that has performed a blocking checkpoint operation and a non-blocking checkpoint operation before the current moment is collected to obtain a plurality of blocking time overheads and a plurality of non-blocking time overheads, and the means of the plurality of blocking time overheads and the plurality of non-blocking time overheads are calculated respectively to obtain a blocking time mean and a non-blocking time mean;
[0013] Calculate the sum of local computation time of each superstep between the current superstep and the last superstep in which the checkpoint operation is performed, and obtain a first computation time and value;
[0014] Predicting the sum of local computation times of supersteps between the current superstep and a next superstep requiring a checkpoint operation, to obtain a second computation time and value;
[0015] Inputting the blocking time mean, the first calculation time and value, and the second calculation time and value into a blocking checkpoint operation model created based on a fault distribution function to predict the overall operation time using the blocking checkpoint writing method to obtain a first overall operation time;
[0016] Inputting the non-blocking time mean, the first calculation time and value, and the second calculation time and value into a non-blocking checkpoint operation model created based on the fault distribution function to predict the overall operation time using the non-blocking checkpoint writing method to obtain a second overall operation time;
[0017] The first overall running time and the second overall running time are compared to determine a smaller value of the first overall running time and the second overall running time, and a write checkpoint operation is performed according to a checkpoint writing method corresponding to the smaller value.
[0018] Optionally, the fault distribution function is a uniform distribution function.
[0019] Optionally, the system fault tolerance method further includes:
[0020] Collect historical fault information of the system;
[0021] The failure rate is calculated based on the historical failure information, and the failure rate is input into an exponential distribution function to obtain the failure distribution function.
[0022] Optionally, the write checkpoint mode of the starting superstep is a blocking write mode, and the write checkpoint mode of a subsequent superstep adjacent to the starting superstep that requires a write checkpoint operation is a non-blocking write mode.
[0023] Optionally, the predicting whether a system failure will occur based on the current log information and the corresponding advance fault discovery time when the fault prediction result indicates that a failure will occur include:
[0024] Get the log information of the current system and get the current log information;
[0025] The current log information is input into a trained fault prediction model pre-created based on a neural network model to predict whether a system fault will occur and the corresponding advance fault discovery time when the fault prediction result indicates that a fault will occur.
[0026] Optionally, inputting the current log information into a trained fault prediction model pre-created based on a neural network model includes:
[0027] The current log information is preprocessed to obtain processed log information, and the processed log information is input into a trained fault prediction model pre-created based on a neural network model.
[0028] Optionally, the system fault tolerance method further includes:
[0029] Collect historical log information of the system;
[0030] Tagging the faults in the historical log information to obtain the marked log information;
[0031] The marked log information is input into a time recurrent neural network model for training to obtain the trained fault prediction model.
[0032] Optionally, the training process of the time recurrent neural network model further includes:
[0033] Identify the log chain that causes the fault in the marked log information to obtain the fault chain;
[0034] The time intervals between each event phrase and the last event phrase in the fault chain are calculated respectively, and the order of occurrence of each event phrase in the fault chain is obtained, so as to predict whether a system fault will occur based on the event phrases and the order of occurrence, and predict the corresponding advance fault detection time when a fault occurs based on the time intervals.
[0035] Optionally, preprocessing the current log information to obtain processed log information, and inputting the processed log information into a trained fault prediction model pre-created based on a neural network model, includes:
[0036] Identify each log event in the current log information, and determine whether each log event contains a fault event phrase;
[0037] If the log event does not contain the fault event phrase, the log event that does not contain the fault event phrase is deleted from the log information to obtain processed log information, and the processed log information is input into the trained fault prediction model pre-created based on the time recurrent neural network model.
[0038] Optionally, the system fault tolerance method further includes:
[0039] Collect the system's historical log information to obtain historical log information;
[0040] Fine-tune the decoder model using the historical log information to obtain a fine-tuned model;
[0041] Using the fine-tuned model to perform log prediction, obtain a predicted log, and determine whether the predicted log contains a fault event;
[0042] If the prediction log contains a fault event, identifying a log chain causing the fault in the prediction log containing the fault event to obtain a fault chain;
[0043] A fault sequence is constructed based on the fault chain, and the time difference between each fault event and the last fault event in the fault chain is calculated, so as to predict whether a system will fail based on the fault event vocabulary and the order of occurrence in the fault sequence, and predict the corresponding advance fault detection time when the fault occurs based on the time difference, so as to obtain the trained fault prediction model.
[0044] Optionally, fine-tuning the decoder model using the historical log information to obtain a fine-tuned model includes:
[0045] The decoder model is initialized using the pre-trained language model to obtain an initialized model, and the initialized model is fine-tuned using the historical log information to obtain a fine-tuned model.
[0046] Optionally, the determining whether the prediction log contains a fault event includes:
[0047] Comparing the prediction log with the fault event vocabulary in the fault event vocabulary table to obtain a comparison result;
[0048] If the comparison result indicates that the prediction log is located in the fault event vocabulary, determining that the prediction log contains a fault event;
[0049] If the comparison result indicates that the prediction log is not located in the fault event vocabulary, it is determined that the prediction log does not contain a fault event.
[0050] Optionally, preprocessing the current log information to obtain processed log information, and inputting the processed log information into a trained fault prediction model pre-created based on a neural network model, includes:
[0051] The current log information is converted into a log event phrase sequence based on an ID, and the log event phrase sequence is input into a trained fault prediction model pre-created based on the decoder model.
[0052] Optionally, the selecting of different fault tolerance strategies based on the comparison result between the fault advance discovery time and a preset duration threshold, so as to perform fault recovery based on the fault tolerance strategy when a system fault occurs, includes:
[0053] Determine whether the fault discovery time in advance is greater than a first duration threshold;
[0054] If the fault discovery time in advance is greater than the first duration threshold, each process currently being executed by the computing node is migrated to the standby node;
[0055] If the fault discovery time in advance is not greater than the first duration threshold, determining whether the fault discovery time in advance is greater than a second duration threshold; the second duration threshold is less than the first duration threshold;
[0056] If the fault discovery time in advance is not greater than the first duration threshold and greater than the second duration threshold, suspend the calculation of the current computing node, save the current system status information to a preset checkpoint, and then save the preset checkpoint to the local and parallel file systems in sequence, so that when a system failure occurs, the preset checkpoint can be directly obtained from the local to perform fault recovery;
[0057] If the fault discovery time in advance is not greater than the second duration threshold, the most recently saved checkpoint is obtained from the parallel file system and saved locally, so that when a system failure occurs, the most recently saved checkpoint can be directly obtained locally for fault recovery.
[0058] Optionally, the system fault tolerance method further includes:
[0059] Collect the write time overhead of historical blocking checkpoints to obtain multiple historical time overheads;
[0060] Calculate an average value of a plurality of the historical time expenditures to obtain the second duration threshold;
[0061] The product of the second duration threshold and a preset multiple is calculated to obtain the first duration threshold.
[0062] In a second aspect, the present application discloses a system fault tolerance device, comprising:
[0063] A judgment module, used to judge whether the preset superstep interval is reached;
[0064] A first prediction module is used for respectively predicting the overall running time of using a blocking checkpoint writing mode and a non-blocking checkpoint writing mode if the preset superstep interval is reached, to obtain a first overall running time and a second overall running time;
[0065] A checkpoint writing module, configured to perform a checkpoint writing operation according to a checkpoint writing mode corresponding to a smaller value of the first overall running time and the second overall running time;
[0066] A second prediction module is used to predict whether the system will fail based on the current log information if the preset superstep interval is not reached, and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur; the advance fault detection time is the length of time from the predicted fault occurrence time to the current time;
[0067] The fault-tolerant strategy selection module is used to select different fault-tolerant strategies based on the comparison result between the fault advance discovery time and the preset time threshold, so as to perform fault recovery based on the fault-tolerant strategy when a system fault occurs.
[0068] In a third aspect, the present application discloses a computer program product, including a computer program / instruction, which implements the aforementioned system fault tolerance method when executed by a processor.
[0069] In a fourth aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein the processor implements the aforementioned system fault tolerance method when executing a computer program stored in the memory.
[0070] In a fifth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned system fault tolerance method is implemented.
[0071] It can be seen that the present application first determines whether the preset superstep interval has been reached. If the preset superstep interval has been reached, the overall running time using the blocking checkpoint writing method and the non-blocking checkpoint writing method is predicted respectively to obtain the first overall running time and the second overall running time, and the write checkpoint operation is performed according to the checkpoint writing method corresponding to the smaller value of the first overall running time and the second overall running time; if the preset superstep interval has not been reached, whether the system will fail and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur are predicted based on the current log information, and different fault tolerance strategies are selected based on the comparison result of the fault detection time in advance with the preset duration threshold, so that when a system failure occurs, fault recovery is performed based on the fault tolerance strategy; the fault detection time in advance is the length of time from the predicted fault occurrence time to the current time. The present application first determines whether the preset superstep interval has been reached. If the preset superstep interval has been reached, the overall running time using the blocking checkpoint writing method and the non-blocking checkpoint writing method is predicted respectively, and the write checkpoint operation is performed according to the checkpoint writing method corresponding to the smaller value of the two predicted times. If the preset superstep interval has not been reached, whether the system will fail and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur are predicted based on the current log information, and different fault tolerance strategies are selected based on the comparison result of the advance fault detection time and the preset duration threshold, so that when a system fails, fault recovery is performed based on the fault tolerance strategy. It can be seen that when the present application performs fault tolerance on the system, on the basis of the periodic checkpoint writing method, a dynamic fault tolerance strategy selection mechanism based on fault prediction is combined, so that it can check for deficiencies and make up for them while reducing the frequency of periodic checkpoints, and adopt targeted fault tolerance strategies before the failure occurs according to the advance fault detection time, so that the I / O resources of the system can be flexibly utilized, and the time overhead of checkpoint operations and recalculation can be reduced. In addition, compared with the traditional periodic checkpoint writing method, the present application adopts a blocking and non-blocking mixed checkpoint writing method, which further reduces the time overhead of checkpoint writing and the time overhead of recovery when a failure occurs, thereby improving the system's fault tolerance. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0073] Figure 1 A flow chart of a system fault tolerance method disclosed in this application;
[0074] Figure 2 A specific super-step running time schematic diagram disclosed in this application;
[0075] Figure 3 A specific flow chart of a system fault tolerance method disclosed in this application;
[0076] Figure 4 A schematic diagram of the structure of a system fault-tolerant device disclosed in this application;
[0077] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0078] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0079] The present application embodiment discloses a system fault tolerance method, see Figure 1 As shown, the method includes:
[0080] Step S11: Determine whether the preset superstep interval is reached.
[0081] It should be pointed out that the system fault tolerance solution proposed in this application can be applied to a computer system configured with a parallel computing model, wherein the parallel computing model includes but is not limited to the BSP model, the LogP model (a distributed storage, point-to-point communication multi-processor model), the BDM (distributed storage model, Block Distributed Model) model, the PRAM (Parallel Random Access Machine) model, etc.
[0082] In this embodiment, a superstep interval is preset. When the preset superstep interval is reached, a write checkpoint operation is triggered, that is, the current system status information is obtained and the obtained current status information is saved, such as saving to a local disk. For details, see Figure 2 As shown, is the current superstep, The current superstep The corresponding last superstep that needs to be written to the checkpoint operation, n is the superstep interval for writing the checkpoint operation (ie, the preset superstep interval).
[0083] Step S12: If the preset superstep interval is reached, the overall running time using the blocking checkpoint writing method and the non-blocking checkpoint writing method is predicted respectively to obtain the first overall running time and the second overall running time, and the write checkpoint operation is performed according to the checkpoint writing method corresponding to the smaller value of the first overall running time and the second overall running time.
[0084] In this embodiment, if the current moment reaches the above-mentioned preset superstep interval, it indicates that a write checkpoint operation is required at this time. Specifically, the overall running time using the blocking checkpoint writing mode and the non-blocking checkpoint writing mode is predicted respectively, wherein the overall running time includes not only the write time of the checkpoint, but also the calculation time of the write time and the time of fault recovery, and the corresponding first overall running time and second overall running time are obtained. Then, the smaller value of the first overall running time and the second overall running time is determined to obtain the smaller value of the two times, and the write checkpoint operation is performed according to the checkpoint writing mode corresponding to the smaller value. For example, when the checkpoint writing mode corresponding to the smaller value is the non-blocking checkpoint writing mode, the non-blocking checkpoint writing mode is used to save the current system status information.
[0085] Specifically, the method of respectively predicting the overall running time using a blocking checkpoint writing mode and a non-blocking checkpoint writing mode to obtain a first overall running time and a second overall running time, and performing a write checkpoint operation according to the checkpoint writing mode corresponding to the smaller value of the first overall running time and the second overall running time, may include: starting from a starting superstep, collecting the write time overhead of each superstep that has undergone a blocking checkpoint operation and a non-blocking checkpoint operation before the current moment, obtaining a plurality of blocking time overheads and a plurality of non-blocking time overheads, and respectively calculating the mean of the plurality of blocking time overheads and the plurality of non-blocking time overheads to obtain a blocking time mean and a non-blocking time mean; calculating the sum of the local computing time of each superstep between the current superstep and the previous superstep that has undergone a checkpoint operation to obtain a first computing time and a value; predicting the difference between the current superstep and the next superstep that needs a checkpoint operation The second calculation time and value are obtained by summing up the local calculation time of each superstep between supersteps; the blocking time mean, the first calculation time and value, and the second calculation time and value are input into the blocking checkpoint operation model created based on the fault distribution function to predict the overall operation time using the blocking checkpoint writing method to obtain the first overall operation time; the non-blocking time mean, the first calculation time and value, and the second calculation time and value are input into the non-blocking checkpoint operation model created based on the fault distribution function to predict the overall operation time using the non-blocking checkpoint writing method to obtain the second overall operation time; the first overall operation time and the second overall operation time are compared to determine the smaller value of the first overall operation time and the second overall operation time, and a write checkpoint operation is performed according to the checkpoint writing method corresponding to the smaller value. In this embodiment, if the preset superstep interval is reached, starting from the starting superstep (i.e., superstep 0), the write time overhead of each superstep that has performed blocking checkpoint operations and non-blocking checkpoint operations before the current moment is collected, that is, the time required for superstep writing checkpoints using the blocking checkpoint writing method (i.e., the write time overhead of historical blocking checkpoints) and the time required for superstep writing checkpoints using the non-blocking checkpoint writing method (i.e., the write time overhead of historical non-blocking checkpoints), and multiple blocking time overheads and multiple non-blocking time overheads are obtained, and then the average of the multiple blocking time overheads and the multiple non-blocking time overheads are calculated respectively to obtain the corresponding blocking time average and non-blocking time average. Among them, the calculation formula for the blocking time average is:
[0086] ;
[0087] In the formula, represents the mean blocking time, Indicates the time required for all supersteps that use the blocking checkpoint write mode to write checkpoints. If the superstep does not perform a write checkpoint operation or does not use the blocking checkpoint write mode to perform a write checkpoint operation, is 0, and N represents the number of supersteps using the blocking checkpoint writing method.
[0088] The calculation formula of the non-blocking time mean is:
[0089] ;
[0090] In the formula, represents the mean non-blocking time, Indicates the time required for all supersteps that use non-blocking checkpoint writing. If the superstep does not perform a checkpoint write operation or does not use non-blocking checkpoint writing, then is 0, M represents the number of supersteps using non-blocking checkpoint writing.
[0091] It should be noted that the write checkpoint mode of the starting superstep is a blocking write mode, and the write checkpoint mode of the next superstep adjacent to the starting superstep that needs to perform a write checkpoint operation is a non-blocking write mode. That is, superstep 0 adopts a blocking checkpoint write mode, and superstep 1 adopts a non-blocking checkpoint write mode.
[0092] Next, calculate the sum of the local computation time of each superstep between the current superstep and the previous superstep with the checkpoint operation, and obtain the first computation time and value, see Figure 2 As shown, calculate the current superstep The previous superstep that performed a write checkpoint The sum of the local computation time of each superstep between the two gets the first computation time and value ; Where n is the superstep interval for write checkpoint operation.
[0093] Specifically, calculating the sum of the local calculation time of each superstep between the current superstep and the previous superstep in which the checkpoint operation is performed to obtain the first calculation time and value may include: calculating the sum of the local calculation time of each superstep between the current superstep and the previous superstep in which the checkpoint operation is performed using a preset superstep time statistical formula to obtain the first calculation time and value;
[0094] Wherein, the expression of the superstep time statistical formula is:
[0095] ;
[0096] In the formula, represents the first calculation time and value, i.e., superstep To Superstep The sum of the local computation time of is the previous superstep with a write checkpoint operation, represents the time superstep j is used for local calculation, n is the preset superstep interval, and i is the superstep number corresponding to the current superstep.
[0097] Further, further, predict the above current superstep The next superstep that needs to be checkpointed The sum of the local computation time of each superstep between the two gets the second computation time and value .
[0098] Specifically, predicting the sum of the local computing time of each superstep between the current superstep and the next superstep that needs to be checked, to obtain the second computing time and value may include: predicting the sum of the local computing time of each superstep between the current superstep and the next superstep that needs to be checked, to obtain the second computing time and value, using a preset superstep time prediction formula;
[0099] Wherein, the expression of the superstep time prediction formula is:
[0100] ;
[0101] In the formula, represents the second calculation time and value, i.e., superstep To Superstep The mean of the sum of the local computation times is multiplied by the preset superstep interval n.
[0102] Next, the average of the above blocking time , the first calculation time and value and the second calculation time and value Input into the blocking checkpoint operation model created based on the fault distribution function to predict the overall running time using the blocking checkpoint writing method, and obtain the first overall running time .
[0103] The calculation formula of the blocking checkpoint operation model is:
[0104] ;
[0105] in, ;
[0106] In the formula, represents the first overall running time, represents the first calculation time and value, represents the second calculation time and value, represents the mean value of the blocking time, t and u are both time variables, is the fault distribution function.
[0107] Furthermore, the above non-blocking time average , the first calculation time and value and the second calculation time and value The non-blocking checkpoint operation model created based on the fault distribution function is input to predict the overall operation time of the non-blocking checkpoint writing method, and the second overall operation time is obtained. .
[0108] The calculation formula of the non-blocking checkpoint operation model is:
[0109] ;
[0110] in, ;
[0111] In the formula, represents the second overall running time, represents the first calculation time and value, represents the second calculation time and value, represents the non-blocking time mean, t and u are both time variables, is the fault distribution function.
[0112] Finally, after obtaining the first overall running time and the second overall running time, that is, after obtaining the overall running time using the blocking checkpoint writing method and the overall running time using the non-blocking checkpoint writing method, compare the two to determine the smaller value of the two, that is, the value with the shorter overall running time, and perform a write checkpoint operation according to the checkpoint writing method corresponding to the smaller value, that is, save the status information of the current system.
[0113] For example, when Greater than or equal to When , select non-blocking mode for checkpoint operation; when Less than When the checkpoint operation is performed in blocking mode,
[0114] It should be noted that the fault distribution function refers to the probability that a failure will occur at a certain time u, then its definite integral is represents the probability that a failure will occur between the time periods [a, b]. Failure distribution function You can choose according to the actual situation of the system, such as exponential distribution function, uniform distribution function, etc.
[0115] In a specific implementation, the fault distribution function is a uniform distribution function. In this embodiment, if there is no historical fault information available for reference, a uniform distribution function without additional parameters is selected as the fault distribution function, that is, it is assumed that the probability of a fault occurring at any time during the entire modeling time period is average.
[0116] In another specific implementation, the process of creating the fault distribution function specifically includes: collecting historical fault information of the system; calculating the fault rate based on the historical fault information, and inputting the fault rate into an exponential distribution function to obtain the fault distribution function. In this embodiment, if there is known historical fault information, the fault rate λ can be obtained. , then the exponential distribution function can be used as the fault distribution function. Specifically, the failure rate λ can be calculated based on the collected historical fault information, and then the exponential distribution function can be created based on the calculated failure rate λ. The calculation formula of the failure rate can be specifically: equipment failure rate = number of equipment failures / total equipment operation time.
[0117] Step S13: If the preset superstep interval is not reached, predict whether the system will fail based on the current log information and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur, and select different fault tolerance strategies based on the comparison result of the advance fault detection time and the preset duration threshold, so as to perform fault recovery based on the fault tolerance strategy when the system fails; the advance fault detection time is the length of time from the predicted fault occurrence time to the current time.
[0118] In this embodiment, if the above-mentioned preset superstep interval has not been reached at present, the log information of the current system is collected to obtain the current log information, and then based on the collected current log information, it is predicted whether the system will fail and the length of time from the failure to the current moment when the failure occurs, and a fault prediction result of whether the failure will occur is obtained, and the fault prediction result indicates the corresponding advance fault detection time when the failure occurs, that is, it is predicted how long after the failure will occur, and then the above-mentioned advance fault detection time is compared with the preset time threshold to obtain a corresponding comparison result, and finally a corresponding fault-tolerant strategy is selected based on the comparison result, so that when a system failure occurs, fault recovery is performed based on the fault-tolerant strategy.
[0119] Specifically, the prediction of whether a system will fail based on the current log information and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur may include: obtaining the log information of the current system to obtain the current log information; inputting the current log information into a trained fault prediction model pre-created based on a neural network model to predict whether the system will fail and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur. That is, the prediction of whether the system will fail and the corresponding advance fault detection time when a fault will occur is made through the trained fault prediction model created based on the neural network model and the current log information.
[0120] In a specific embodiment, the inputting of the current log information into a trained fault prediction model pre-created based on a neural network model may specifically include: pre-processing the current log information to obtain processed log information, and inputting the processed log information into a trained fault prediction model pre-created based on a neural network model. It is understandable that log information is usually structured or semi-structured text data. In order to enable the model to receive and process log information, the current log information may be pre-processed first, and then the processed log information may be input into a trained fault prediction model pre-created based on a neural network model. It is understandable that different models have different corresponding pre-processing processes.
[0121] In a specific implementation, the creation process of the trained fault prediction model specifically includes: collecting historical log information of the system; labeling the faults present in the historical log information to obtain labeled log information; inputting the labeled log information into a time recurrent neural network model for training to obtain the trained fault prediction model. In this embodiment, in order to obtain a model that can perform fault prediction and predict the advance time of fault occurrence, the historical log information of the system can be collected first, and then the fault events in the large amount of historical log information collected can be labeled to obtain labeled log information containing fault labels, and then the labeled log information can be input into a time recurrent neural network model, such as an LSTM (Long Short-Term Memory) network model for model training, thereby obtaining a trained fault prediction model.
[0122] It should be pointed out that the training process of the time recurrent neural network model may specifically include: identifying the log chain that causes the fault in the marked log information to obtain the fault chain; respectively calculating the time interval between each event phrase in the fault chain and the last event phrase, and obtaining the order of occurrence of each event phrase in the fault chain, so as to predict whether the system will fail based on the event phrases and the order of occurrence, and predict the corresponding advance fault discovery time when the fault occurs based on the time interval. In this embodiment, in the process of model training, all log chains that can cause faults in the marked log information are first identified to obtain the fault chain (the fault chain contains fault phrases such as ERROR), and then respectively calculating the time interval between each event phrase in each fault chain and the last event phrase, that is, calculating the time interval between each event phrase in the fault chain and the last event phrase (that is, the final event indicating the node fault), and obtaining the order of occurrence of each event phrase in the fault chain, so as to predict whether the system will fail based on the event phrase and the order of occurrence, and predict the advance time relative to the current moment when the fault occurs based on the above time interval, that is, the advance fault discovery time.
[0123] When a time-recurrent neural network model is adopted, the current log information is preprocessed to obtain processed log information, and the processed log information is input into a trained fault prediction model pre-created based on the neural network model. Specifically, the preprocessing may include: identifying each log event in the current log information, and determining whether each log event contains a fault event phrase; if the log event does not contain a fault event phrase, deleting the log event that does not contain the fault event phrase from the log information to obtain processed log information, and inputting the processed log information into a trained fault prediction model pre-created based on the time-recurrent neural network model. In this embodiment, after obtaining the current log information of the current system, each log event in the current log information is identified, and then it is determined whether each log event contains a fault event phrase, for example, it is determined whether it contains a fault event phrase such as ERROR. If the log event does not contain a fault event phrase, it indicates that the log event records a normal log event, and there is no event that can cause a system failure. At this time, the log event that does not contain the fault event phrase can be deleted from the entire log information to obtain the processed log information, that is, the log event that does not contain any fault event phrase in the fault chain is filtered out, and then the filtered log information is input into the trained fault prediction model created based on the time recurrent neural network model for fault prediction. In a specific implementation, the event phrases that are not related to the fault can be first filtered out by regular expressions, and then the filtered log information can be input into the trained fault prediction model to predict the possible fault, and the fault discovery time can be predicted in advance based on the order in which the event phrases appear.
[0124] In another specific implementation, the creation process of the trained fault prediction model specifically includes: collecting historical log information of the system to obtain historical log information; using the historical log information to fine-tune the decoder model to obtain a fine-tuned model; using the fine-tuned model to perform log prediction to obtain a predicted log, and determining whether the predicted log contains a fault event; if the predicted log contains a fault event, identifying the log chain that causes the fault in the predicted log containing the fault event to obtain a fault chain; constructing a fault sequence based on the fault chain, and calculating the time difference between each fault event in the fault chain and the last fault event, so as to predict whether the system will fail based on the fault event vocabulary and the order of occurrence in the fault sequence, and predicting the corresponding advance fault discovery time when the fault occurs based on the time difference, to obtain the trained fault prediction model. In this embodiment, the historical log information of the system is first collected, and then the decoder model is fine-tuned using the above historical log information to obtain a fine-tuned model, and then the fine-tuned model is used to perform log prediction to obtain a predicted log, and it is determined whether the predicted log contains a fault event, such as whether there is an event containing a fault phrase such as ERROR. If the predicted log contains a fault event, the log chain that can cause the fault in the predicted log containing the fault event is further identified to obtain a fault chain, and finally a fault sequence is constructed based on the fault chain, and the time difference between each fault event and the last fault event in the fault chain is calculated, so as to predict whether the system will fail based on the fault event vocabulary contained in the fault sequence and the order of occurrence, and when a fault occurs, the time from the fault to the current moment is predicted based on the above time difference, thereby obtaining a trained fault prediction model.
[0125] In this embodiment, the use of the historical log information to fine-tune the decoder model to obtain a fine-tuned model may specifically include: initializing the decoder model using a pre-trained language model to obtain an initialized model, and fine-tuning the initialized model using the historical log information to obtain a fine-tuned model. For example, a pre-trained language model (PLM) such as BERT, RoBERTa, GPT-3, ALBERT, etc. is used to initialize the Transformer-Decoder model to obtain an initialized model, and then the initialized model is fine-tuned using the historical log information to obtain a fine-tuned model.
[0126] Specifically, the determination of whether the prediction log contains a fault event may include: comparing the prediction log with the fault event vocabulary in the fault event vocabulary to obtain a comparison result; if the comparison result shows that the prediction log is located in the fault event vocabulary, then the prediction log is determined to contain a fault event; if the comparison result shows that the prediction log is not located in the fault event vocabulary, then the prediction log is determined to contain no fault event. In this embodiment, after the prediction log is obtained by performing log prediction through the fine-tuned model, the prediction log is compared with each fault event vocabulary in the fault event vocabulary. If the prediction log is located in the fault event vocabulary, then the prediction log is determined to contain a fault event; if the prediction log is not located in the fault event vocabulary, then the prediction log is determined to contain no fault event. That is, whether the prediction log contains a fault event is determined by determining whether it is located in the fault event vocabulary in the fault event vocabulary.
[0127] In this embodiment, the preprocessing of the current log information to obtain processed log information, and inputting the processed log information into a trained fault prediction model pre-created based on a neural network model, may specifically include: converting the current log information into a log event phrase sequence based on an ID, and inputting the log event phrase sequence into a trained fault prediction model pre-created based on the decoder model. In this embodiment, if a decoder model is used, after obtaining the current log information of the current system, a tokenization technique similar to that in a natural language processing task may be used to convert it into a log event phrase sequence based on an ID (Identity document), thereby facilitating fault prediction processing based on the trained fault prediction model created based on the decoder model.
[0128] It can be seen that the embodiment of the present application first determines whether the preset superstep interval is reached. If the preset superstep interval is reached, the overall running time using the blocking checkpoint writing method and the non-blocking checkpoint writing method is predicted respectively to obtain the first overall running time and the second overall running time, and the write checkpoint operation is performed according to the checkpoint writing method corresponding to the smaller value of the first overall running time and the second overall running time; if the preset superstep interval is not reached, whether the system will fail and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur are predicted based on the current log information, and different fault tolerance strategies are selected based on the comparison result of the fault detection time in advance with the preset duration threshold, so that when the system fails, fault recovery is performed based on the fault tolerance strategy; the fault detection time in advance is the length of time from the predicted fault occurrence time to the current time. The embodiment of the present application first determines whether the preset superstep interval has been reached. If the preset superstep interval has been reached, the overall running time using the blocking checkpoint writing method and the non-blocking checkpoint writing method is predicted respectively, and the write checkpoint operation is performed according to the checkpoint writing method corresponding to the smaller value of the two predicted times. If the preset superstep interval has not been reached, whether the system will fail and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur are predicted based on the current log information, and different fault tolerance strategies are selected based on the comparison result of the advance fault detection time and the preset duration threshold, so that when a system fails, fault recovery is performed based on the fault tolerance strategy. It can be seen that when the system is fault-tolerant, the embodiment of the present application combines a dynamic fault tolerance strategy selection mechanism based on fault prediction on the basis of the periodic checkpoint writing method, so that it can check for deficiencies and make up for them while reducing the frequency of periodic checkpoints, and adopt targeted fault tolerance strategies before the failure occurs according to the advance fault detection time, so that the I / O resources of the system can be flexibly utilized, and the time overhead of checkpoint operations and recalculation can be reduced. In addition, compared with the traditional periodic checkpoint writing method, the embodiment of the present application adopts a blocking and non-blocking mixed checkpoint writing method, which further reduces the time overhead of checkpoint writing and the time overhead of recovery when a failure occurs, thereby improving the fault tolerance of the system.
[0129] The present application embodiment discloses a specific system fault tolerance method, see Figure 3 As shown, the method includes:
[0130] Step S21: Determine whether the preset superstep interval is reached.
[0131] Step S22: If the preset superstep interval is reached, the overall running time using the blocking checkpoint writing method and the non-blocking checkpoint writing method is predicted respectively to obtain the first overall running time and the second overall running time, and the write checkpoint operation is performed according to the checkpoint writing method corresponding to the smaller value of the first overall running time and the second overall running time.
[0132] Step S23: If the preset superstep interval is not reached, predict whether the system will fail based on the current log information and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur, and determine whether the advance fault detection time is greater than the first duration threshold.
[0133] In this embodiment, after obtaining the fault prediction result, if the fault prediction result indicates that a fault will occur, the fault tolerance strategy is further determined based on the fault advance detection time. Specifically, it is first determined whether the fault advance detection time is greater than the preset first duration threshold T1.
[0134] Step S24: If the fault discovery time in advance is greater than the first duration threshold, each process currently being executed by the computing node is migrated to the standby node.
[0135] In this embodiment, if the advance detection time of the above-mentioned fault is greater than the first duration threshold T1, it indicates that there is sufficient time to complete the migration of the entire computing node. At this time, the various processes being executed by the current computing node can be migrated to the backup node to continue data processing, thereby directly avoiding computing interruptions caused by the fault; it should be pointed out that the backup node is a healthy redundant node.
[0136] Step S25: If the fault discovery time in advance is not greater than the first duration threshold, determine whether the fault discovery time in advance is greater than a second duration threshold; the second duration threshold is less than the first duration threshold.
[0137] In this embodiment, if the fault detection time in advance is not greater than the above-mentioned first duration threshold T1, it is further determined whether the fault detection time in advance is greater than the second duration threshold T2; it should be noted that the second duration threshold T2 is less than the first duration threshold T1 and the first duration threshold T1 is greater than the duration for the entire computing node migration, and the second duration threshold T2 can specifically be the average duration for completing the write blocking checkpoint operation.
[0138] Step S26: If the advance fault discovery time is not greater than the first duration threshold and greater than the second duration threshold, the calculation of the current computing node is suspended, and the status information of the current system is saved to a preset checkpoint, and then the preset checkpoint is saved in sequence to the local and parallel file systems, so that when a system failure occurs, the preset checkpoint can be directly obtained from the local for fault recovery.
[0139] In this embodiment, if the fault discovery time in advance is not greater than the first duration threshold T1 and greater than the second duration threshold T2, it indicates that the fault discovery time in advance is greater than the average time to complete the blocking checkpoint but less than the time required for real-time migration. At this time, the blocking checkpoint writing method can be used to write the checkpoint operation, that is, suspend the calculation of the current computing node, and save the current system status information to the preset checkpoint (that is, save the node status). Further, after all the status information of the current system is saved to the preset checkpoint, the preset checkpoint is saved in sequence to the local and parallel file systems (Parallel File System), that is, local and remote storage is performed, so that when a system failure occurs, the preset checkpoint can be directly obtained from the local to perform fault recovery.
[0140] Step S27: If the fault discovery time in advance is not greater than the second time threshold, obtain the most recently saved checkpoint from the parallel file system and save the most recently saved checkpoint locally, so that when a system failure occurs, the most recently saved checkpoint can be directly obtained locally for fault recovery.
[0141] In this embodiment, if the above-mentioned fault advance discovery time is not greater than the second time threshold T2, it indicates that the fault advance discovery time is less than the average time to complete the write blocking checkpoint operation, that is, the fault is about to occur, and the current time is insufficient for the write checkpoint operation. At this time, a checkpoint pre-fetch operation can be performed to directly obtain the checkpoint saved closest to the current moment (that is, the latest checkpoint) from the above-mentioned parallel file system, and then store the most recently saved checkpoint locally. That is, when a fault occurs, a rollback operation is performed by reading the most recently saved complete checkpoint. In this way, once a system fault occurs, the most recently saved checkpoint can be directly obtained locally for corresponding fault recovery, without having to wait for data to be transmitted from the remote end to the local node before performing fault recovery, thereby reducing the response time of the read operation, shortening the time overhead of fault recovery, and improving the efficiency of fault recovery.
[0142] It should be pointed out that the checkpoint obtained from the parallel file system and saved closest to the current moment can be either a checkpoint saved after reaching a preset superstep interval, or a checkpoint saved when the last fault was discovered in advance and the time was between the first duration threshold T1 and the second duration threshold T2.
[0143] In this embodiment, the method may further include: collecting the write time overhead of historical blocking checkpoints to obtain multiple historical time overheads; calculating the average of the multiple historical time overheads to obtain the second duration threshold; calculating the product of the second duration threshold and a preset multiple to obtain the first duration threshold. That is, the average of the write time overhead of historical blocking checkpoints is used as the second duration threshold T2, and the product of the second duration threshold T2 and a preset multiple (such as 3 times) is used as the first duration threshold T1.
[0144] For more specific processing procedures of the above steps S21 and S22, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described in detail here.
[0145] The embodiment of the present application first predicts whether the system will fail and the time from the failure to the current moment (i.e., the advance fault detection time) through a prediction model, and when a failure occurs, determines the relationship between the advance fault detection time and the first duration threshold and the second duration threshold (where the first duration threshold is greater than the second duration threshold). If it is greater than the first duration threshold, it means that there is sufficient time, and the various processes being executed by the current computing node can be migrated to the backup node; if it is between the two thresholds, it means that there is only enough time to complete the checkpoint operation. At this time, a blocking checkpoint operation is performed, and the generated blocking checkpoint is saved to the local and parallel file systems. If it is not greater than the second duration threshold, the most recently saved checkpoint is obtained from the remote end and saved locally. It can be seen that the embodiment of the present application combines a dynamic fault-tolerant strategy selection mechanism based on fault prediction on the basis of periodic write checkpoint operation, and also takes into account the situation that the fault occurs after the non-blocking checkpoint is completed, determines the fault-tolerant strategy by the advance fault discovery time, and performs the checkpoint operation before the fault is about to occur, and saves the checkpoint locally. In this way, when a fault occurs, the checkpoint can be directly obtained from the local to recover the fault without transmitting it from the remote end to the local node, thereby reducing the response time of the read operation. In addition, the dynamic fault-tolerant strategy selection based on fault prediction can flexibly utilize the I / O resources of the system, reduce the time overhead of the checkpoint operation and the time overhead of the recalculation, and better reflect the performance advantages of the non-blocking checkpoint, and can better optimize the overall time overhead of the checkpoint. In addition, the present application can also use the real-time migration technology to migrate the current computing node to the standby node for execution when the advance time is sufficient, thereby avoiding the suspension (i.e., interruption) of the system due to the fault, ensuring the normal operation of the business and the reliable and stable calculation of the system, thereby improving the system fault tolerance capability and user experience.
[0146] In addition, relatively sparse periodic write checkpoint operations can be used to ensure basic availability, and a dynamic fault-tolerant strategy selection mechanism based on fault prediction can be used as a supplement. By monitoring the current system state to predict faults, and selecting different fault-tolerant strategies based on the prediction results, the system recovery and recalculation overhead can be shortened. It is understandable that the use of periodic checkpoint writing requires a balance between the write cycle. Too frequent will cause excessive additional overhead, and too sparse may cause a large recalculation overhead when a fault occurs. In order to effectively reduce the recalculation overhead during fault recovery, this application combines the fault-tolerant strategy selection mechanism based on fault prediction with the periodic hybrid checkpoint writing mechanism, which can not only save the system state as much as possible before the fault occurs, but also occupy as few resources as possible during the normal operation of the system for checkpoint operations, thereby avoiding omissions and false alarms of faults, that is, the combination of the two can also reduce the use of a single periodic checkpoint writing mechanism (that is, when the write frequency is fixed) to check for omissions and fill in the gaps, and effectively reduce the recalculation overhead during fault recovery (that is, reduce the cost of fault recovery), thereby improving the fault tolerance of the system and improving the stability and reliability of the system.
[0147] Correspondingly, the present application embodiment also discloses a system fault tolerance device, see Figure 4 As shown, the device comprises:
[0148] A judging module 11 is used to judge whether a preset superstep interval is currently reached;
[0149] A first prediction module 12 is used to predict the overall running time of using a blocking checkpoint writing mode and a non-blocking checkpoint writing mode respectively if the preset superstep interval is reached, to obtain a first overall running time and a second overall running time;
[0150] A checkpoint writing module 13, configured to perform a checkpoint writing operation according to a checkpoint writing mode corresponding to a smaller value of the first overall running time and the second overall running time;
[0151] The second prediction module 14 is used to predict whether the system will fail based on the current log information if the preset super-step interval is not reached, and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur; the advance fault detection time is the length of time from the predicted fault occurrence time to the current time;
[0152] The fault-tolerant strategy selection module 15 is used to select different fault-tolerant strategies based on the comparison result between the fault advance discovery time and the preset time threshold, so as to perform fault recovery based on the fault-tolerant strategy when a system fault occurs.
[0153] Among them, the specific working processes of the above-mentioned modules can refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.
[0154] It can be seen that in the embodiment of the present application, it is first determined whether the preset superstep interval is reached. If the preset superstep interval is reached, the overall running time using the blocking checkpoint writing method and the non-blocking checkpoint writing method is predicted respectively to obtain the first overall running time and the second overall running time, and the write checkpoint operation is performed according to the checkpoint writing method corresponding to the smaller value of the first overall running time and the second overall running time; if the preset superstep interval is not reached, whether the system will fail and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur are predicted based on the current log information, and different fault tolerance strategies are selected based on the comparison result of the fault detection time in advance with the preset duration threshold, so that when a system failure occurs, fault recovery is performed based on the fault tolerance strategy; the fault detection time in advance is the length of time from the predicted fault occurrence time to the current time. The embodiment of the present application first determines whether the preset superstep interval has been reached. If the preset superstep interval has been reached, the overall running time using the blocking checkpoint writing method and the non-blocking checkpoint writing method is predicted respectively, and the write checkpoint operation is performed according to the checkpoint writing method corresponding to the smaller value of the two predicted times. If the preset superstep interval has not been reached, whether the system will fail and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur are predicted based on the current log information, and different fault tolerance strategies are selected based on the comparison result of the advance fault detection time and the preset duration threshold, so that when a system fails, fault recovery is performed based on the fault tolerance strategy. It can be seen that when the system is fault-tolerant, the embodiment of the present application combines a dynamic fault tolerance strategy selection mechanism based on fault prediction on the basis of the periodic checkpoint writing method, so that it can check for deficiencies and make up for them while reducing the frequency of periodic checkpoints, and adopt targeted fault tolerance strategies before the failure occurs according to the advance fault detection time, so that the I / O resources of the system can be flexibly utilized, and the time overhead of checkpoint operations and recalculation can be reduced. In addition, compared with the traditional periodic checkpoint writing method, the embodiment of the present application adopts a blocking and non-blocking mixed checkpoint writing method, which further reduces the time overhead of checkpoint writing and the time overhead of recovery when a failure occurs, thereby improving the fault tolerance of the system.
[0155] In some specific embodiments, the first prediction module 12 may specifically include:
[0156] The first collection unit is used to collect the write time overhead of each superstep that has performed a blocking checkpoint operation and a non-blocking checkpoint operation before the current moment, starting from the starting superstep, to obtain a plurality of blocking time overheads and a plurality of non-blocking time overheads;
[0157] A first calculation unit, configured to respectively calculate the averages of the plurality of blocking time overheads and the plurality of non-blocking time overheads to obtain a blocking time average and a non-blocking time average;
[0158] A second calculation unit is used to calculate the sum of local calculation times of each superstep between the current superstep and the previous superstep on which the checkpoint operation is performed, to obtain a first calculation time and value;
[0159] A first prediction unit is used to predict the sum of local computing time of each superstep between the current superstep and the next superstep that needs to be checkpointed, to obtain a second computing time and value;
[0160] A second prediction unit is used to input the blocking time mean, the first calculation time and value, and the second calculation time and value into a blocking checkpoint operation model created based on a fault distribution function to predict the overall operation time using the blocking checkpoint writing method to obtain a first overall operation time;
[0161] A third prediction unit is used to input the non-blocking time mean, the first calculation time and value, and the second calculation time and value into a non-blocking checkpoint operation model created based on the fault distribution function to predict the overall operation time using the non-blocking checkpoint writing method to obtain a second overall operation time;
[0162] Accordingly, the checkpoint writing module 13 may specifically include:
[0163] A comparing unit, configured to compare the first overall running time and the second overall running time to determine a smaller value of the first overall running time and the second overall running time;
[0164] The checkpoint writing unit is used to perform a checkpoint writing operation according to the checkpoint writing mode corresponding to the smaller value.
[0165] In some specific embodiments, the fault distribution function is a uniform distribution function.
[0166] In some specific embodiments, the system fault tolerance device may further include:
[0167] The second collection unit is used to collect historical fault information of the system;
[0168] A third calculation unit, configured to calculate a failure rate based on the historical failure information;
[0169] The first input unit is used to input the failure rate into an exponential distribution function to obtain the failure distribution function.
[0170] In some specific embodiments, the write checkpoint mode of the starting superstep is a blocking write mode, and the write checkpoint mode of the subsequent superstep adjacent to the starting superstep that needs to perform a write checkpoint operation is a non-blocking write mode.
[0171] In some specific embodiments, the second prediction module 14 may specifically include:
[0172] An information acquisition unit is used to acquire the log information of the current system and obtain the current log information;
[0173] The second input unit is used to input the current log information into a trained fault prediction model pre-created based on a neural network model to predict whether a system fault will occur and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur.
[0174] In some specific embodiments, the second input unit may specifically include:
[0175] A preprocessing unit, used for preprocessing the current log information to obtain processed log information;
[0176] The third input unit is used to input the processed log information into a trained fault prediction model pre-created based on a neural network model.
[0177] In some specific embodiments, the system fault tolerance device may further include:
[0178] The third collection unit is used to collect historical log information of the system;
[0179] A labeling unit, used for labeling the faults in the historical log information to obtain the labeled log information;
[0180] The fourth input unit is used to input the marked log information into the time recurrent neural network model for training to obtain the trained fault prediction model.
[0181] In some specific embodiments, the system fault tolerance device may further include:
[0182] A first identification unit is used to identify the log chain causing the fault in the marked log information to obtain the fault chain;
[0183] a fourth calculation unit, configured to respectively calculate the time interval between each event phrase and the last event phrase in the fault chain;
[0184] The sequence acquisition unit is used to acquire the order of occurrence of each event phrase in the fault chain, so as to predict whether the system will fail based on the event phrases and the order of occurrence, and to predict the corresponding advance fault discovery time when the fault occurs based on the time interval.
[0185] In some specific embodiments, the pre-processing unit may specifically include:
[0186] A second identification unit, used to identify each log event in the current log information;
[0187] A first judging unit, configured to judge whether each of the log events contains a fault event phrase;
[0188] a deleting unit, configured to delete the log event not containing the fault event phrase from the log information if the log event does not contain the fault event phrase, so as to obtain processed log information;
[0189] Accordingly, the third input unit may specifically include:
[0190] The fifth input unit is used to input the processed log information into a trained fault prediction model pre-created based on the time recurrent neural network model.
[0191] In some specific embodiments, the system fault tolerance device may further include:
[0192] The fourth collection unit is used to collect the historical log information of the system to obtain the historical log information;
[0193] A first model fine-tuning unit, configured to fine-tune the decoder model using the historical log information to obtain a fine-tuned model;
[0194] A log prediction unit, used to perform log prediction using the fine-tuned model to obtain a predicted log, and determine whether the predicted log contains a fault event;
[0195] A third identification unit is configured to identify a log chain causing a fault in the prediction log containing the fault event, if the prediction log contains a fault event, to obtain a fault chain;
[0196] A sequence construction unit, used for constructing a fault sequence based on the fault chain;
[0197] The fifth computing unit is used to calculate the time difference between each fault event and the last fault event in the fault chain, so as to predict whether the system will fail based on the fault event vocabulary and the order of occurrence in the fault sequence, and to predict the corresponding advance fault detection time when the fault occurs based on the time difference, so as to obtain the trained fault prediction model.
[0198] In some specific embodiments, the first model fine-tuning unit may specifically include:
[0199] A model initialization unit, used to initialize the decoder model using a pre-trained language model to obtain an initialized model;
[0200] The second model fine-tuning unit is used to fine-tune the initialized model using the historical log information to obtain a fine-tuned model.
[0201] In some specific embodiments, the first determination unit may specifically include:
[0202] A comparison unit, used for comparing the prediction log with the fault event vocabulary in the fault event vocabulary table to obtain a comparison result;
[0203] a first determination unit, configured to determine that the prediction log contains a fault event if the comparison result indicates that the prediction log is located in the fault event vocabulary;
[0204] The second determination unit is configured to determine that the prediction log does not contain a fault event if the comparison result indicates that the prediction log is not located in the fault event vocabulary.
[0205] In some specific embodiments, the pre-processing unit may specifically include:
[0206] An information conversion unit, used for converting the current log information into a log event phrase sequence based on an ID;
[0207] Accordingly, the third input unit may specifically include:
[0208] The sixth input unit is used to input the log event phrase sequence into a trained fault prediction model pre-created based on the decoder model.
[0209] In some specific embodiments, the fault-tolerant strategy selection module 15 may specifically include:
[0210] A second judgment unit is used to judge whether the fault discovery time in advance is greater than a first time threshold;
[0211] A process migration unit, configured to migrate each process currently being executed by the computing node to a standby node if the fault discovery time in advance is greater than the first duration threshold;
[0212] A third judgment unit is used to judge whether the advance fault detection time is greater than a second time threshold if the advance fault detection time is not greater than the first time threshold; the second time threshold is less than the first time threshold;
[0213] A first saving unit, configured to suspend the calculation of the current computing node and save the state information of the current system to a preset checkpoint if the advance fault discovery time is not greater than the first duration threshold and greater than the second duration threshold;
[0214] A second saving unit is used to save the preset checkpoints to the local file system and the parallel file system in sequence, so that when a system failure occurs, the preset checkpoints can be directly obtained from the local file system for failure recovery;
[0215] A checkpoint acquisition unit, configured to acquire a most recently saved checkpoint from the parallel file system if the fault is discovered in advance at a time not greater than the second time threshold;
[0216] The third saving unit is used to save the most recently saved checkpoint locally, so that when a system failure occurs, the most recently saved checkpoint can be directly obtained locally for failure recovery.
[0217] In some specific embodiments, the system fault tolerance device may further include:
[0218] A time cost collection unit is used to collect the write time cost of historical blocking checkpoints to obtain multiple historical time costs;
[0219] an average value calculation unit, configured to calculate an average value of a plurality of the historical time expenditures to obtain the second duration threshold;
[0220] A product calculation unit is used to calculate the product of the second duration threshold and a preset multiple to obtain the first duration threshold.
[0221] Furthermore, the present application also discloses an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.
[0222] Figure 5 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the system fault tolerance method disclosed in any of the above embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0223] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0224] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0225] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the system fault tolerance method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.
[0226] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed system fault tolerance method. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, and no further description will be given here.
[0227] Furthermore, an embodiment of the present application also discloses a computer program product, including a computer program / instruction, which implements the steps of the system fault tolerance method disclosed above when executed by a processor.
[0228] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0229] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0230] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0231] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0232] The above is a detailed introduction to a system fault-tolerant method, product, device, equipment and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A system fault tolerance method, characterized in that: include: Determine whether the preset superstep interval has been reached; If the preset superstep interval is reached, the overall running time using the blocking checkpoint writing mode and the non-blocking checkpoint writing mode are predicted respectively to obtain a first overall running time and a second overall running time, and a write checkpoint operation is performed according to the checkpoint writing mode corresponding to the smaller value of the first overall running time and the second overall running time; If the preset superstep interval is not reached, predict whether the system will fail based on the current log information and the corresponding advance fault detection time when the fault prediction result indicates that a fault will occur, and select different fault tolerance strategies based on the comparison result of the advance fault detection time and the preset time threshold, so as to perform fault recovery based on the fault tolerance strategy when the system fails; The fault discovery time in advance is the time between the predicted fault occurrence time and the current time.
2. The system fault tolerance method according to claim 1, characterized in that: The predicting the overall running time using the blocking checkpoint writing mode and the non-blocking checkpoint writing mode respectively to obtain a first overall running time and a second overall running time, and performing a checkpoint writing operation according to the checkpoint writing mode corresponding to the smaller value of the first overall running time and the second overall running time, includes: Starting from the starting superstep, the write time overhead of each superstep that has performed a blocking checkpoint operation and a non-blocking checkpoint operation before the current moment is collected to obtain a plurality of blocking time overheads and a plurality of non-blocking time overheads, and the means of the plurality of blocking time overheads and the plurality of non-blocking time overheads are calculated respectively to obtain a blocking time mean and a non-blocking time mean; Calculate the sum of local computation time of each superstep between the current superstep and the last superstep in which the checkpoint operation is performed, and obtain a first computation time and value; Predicting the sum of local computation times of supersteps between the current superstep and a next superstep requiring a checkpoint operation, to obtain a second computation time and value; Inputting the blocking time mean, the first calculation time and value, and the second calculation time and value into a blocking checkpoint operation model created based on a fault distribution function to predict the overall operation time using the blocking checkpoint writing method to obtain a first overall operation time; Inputting the non-blocking time mean, the first calculation time and value, and the second calculation time and value into a non-blocking checkpoint operation model created based on the fault distribution function to predict the overall operation time using the non-blocking checkpoint writing method to obtain a second overall operation time; The first overall running time and the second overall running time are compared to determine a smaller value of the first overall running time and the second overall running time, and a write checkpoint operation is performed according to a checkpoint writing method corresponding to the smaller value.
3. The system fault tolerance method according to claim 2, characterized in that: The fault distribution function is a uniform distribution function.
4. The system fault tolerance method according to claim 2, characterized in that: Also includes: Collect historical fault information of the system; The failure rate is calculated based on the historical failure information, and the failure rate is input into an exponential distribution function to obtain the failure distribution function.
5. The system fault tolerance method according to claim 2, characterized in that: The write checkpoint mode of the initial superstep is a blocking write mode, and the write checkpoint mode of the subsequent superstep adjacent to the initial superstep that needs to perform a write checkpoint operation is a non-blocking write mode.
6. The system fault tolerance method according to claim 1, characterized in that: The prediction of whether a system failure will occur based on the current log information and the corresponding advance fault detection time when the fault prediction result indicates that a failure will occur include: Get the log information of the current system and get the current log information; The current log information is input into a trained fault prediction model pre-created based on a neural network model to predict whether a system fault will occur and the corresponding advance fault discovery time when the fault prediction result indicates that a fault will occur.
7. The system fault tolerance method according to claim 6, characterized in that: The step of inputting the current log information into a trained fault prediction model pre-created based on a neural network model includes: The current log information is preprocessed to obtain processed log information, and the processed log information is input into a trained fault prediction model pre-created based on a neural network model.
8. The system fault tolerance method according to claim 7, characterized in that: Also includes: Collect historical log information of the system; Tagging the faults in the historical log information to obtain the marked log information; The marked log information is input into a time recurrent neural network model for training to obtain the trained fault prediction model.
9. The system fault tolerance method according to claim 8, characterized in that: The training process of the time recurrent neural network model also includes: Identify the log chain that causes the fault in the marked log information to obtain the fault chain; The time intervals between each event phrase and the last event phrase in the fault chain are calculated respectively, and the order of occurrence of each event phrase in the fault chain is obtained, so as to predict whether a system fault will occur based on the event phrases and the order of occurrence, and predict the corresponding advance fault detection time when a fault occurs based on the time intervals.
10. The system fault tolerance method according to claim 9, characterized in that: The preprocessing of the current log information to obtain processed log information, and inputting the processed log information into a trained fault prediction model pre-created based on a neural network model, includes: Identify each log event in the current log information, and determine whether each log event contains a fault event phrase; If the log event does not contain the fault event phrase, the log event that does not contain the fault event phrase is deleted from the log information to obtain processed log information, and the processed log information is input into the trained fault prediction model pre-created based on the time recurrent neural network model.
11. The system fault tolerance method according to claim 7, characterized in that: Also includes: Collect the system's historical log information to obtain historical log information; Fine-tune the decoder model using the historical log information to obtain a fine-tuned model; Using the fine-tuned model to perform log prediction, obtain a predicted log, and determine whether the predicted log contains a fault event; If the prediction log contains a fault event, identifying a log chain causing the fault in the prediction log containing the fault event to obtain a fault chain; A fault sequence is constructed based on the fault chain, and the time difference between each fault event and the last fault event in the fault chain is calculated, so as to predict whether a system will fail based on the fault event vocabulary and the order of occurrence in the fault sequence, and predict the corresponding advance fault detection time when the fault occurs based on the time difference, so as to obtain the trained fault prediction model.
12. The system fault tolerance method according to claim 11, characterized in that: The method of fine-tuning the decoder model by using the historical log information to obtain a fine-tuned model includes: The decoder model is initialized using the pre-trained language model to obtain an initialized model, and the initialized model is fine-tuned using the historical log information to obtain a fine-tuned model.
13. The system fault tolerance method according to claim 11, characterized in that: The determining whether the prediction log contains a fault event includes: Comparing the prediction log with the fault event vocabulary in the fault event vocabulary table to obtain a comparison result; If the comparison result indicates that the prediction log is located in the fault event vocabulary, determining that the prediction log contains a fault event; If the comparison result indicates that the prediction log is not located in the fault event vocabulary, it is determined that the prediction log does not contain a fault event.
14. The system fault tolerance method according to claim 11, characterized in that: The preprocessing of the current log information to obtain processed log information, and inputting the processed log information into a trained fault prediction model pre-created based on a neural network model, includes: The current log information is converted into a log event phrase sequence based on an ID, and the log event phrase sequence is input into a trained fault prediction model pre-created based on the decoder model.
15. The system fault tolerance method according to any one of claims 1 to 14, characterized in that: The selecting of different fault tolerance strategies based on the comparison result between the fault discovery time in advance and the preset duration threshold, so as to perform fault recovery based on the fault tolerance strategy when a system fault occurs, includes: Determine whether the fault discovery time in advance is greater than a first duration threshold; If the fault discovery time in advance is greater than the first duration threshold, each process currently being executed by the computing node is migrated to the standby node; If the fault discovery time in advance is not greater than the first duration threshold, determining whether the fault discovery time in advance is greater than a second duration threshold; the second duration threshold is less than the first duration threshold; If the fault discovery time in advance is not greater than the first duration threshold and greater than the second duration threshold, suspend the calculation of the current computing node, save the current system status information to a preset checkpoint, and then save the preset checkpoint to the local and parallel file systems in sequence, so that when a system failure occurs, the preset checkpoint can be directly obtained from the local to perform fault recovery; If the fault discovery time in advance is not greater than the second duration threshold, the most recently saved checkpoint is obtained from the parallel file system and saved locally, so that when a system failure occurs, the most recently saved checkpoint can be directly obtained locally for fault recovery.
16. The system fault tolerance method according to claim 15, characterized in that: Also includes: Collect the write time overhead of historical blocking checkpoints to obtain multiple historical time overheads; Calculate an average value of a plurality of the historical time expenditures to obtain the second duration threshold; The product of the second duration threshold and a preset multiple is calculated to obtain the first duration threshold.
17. A system fault-tolerant device, characterized in that: include: A judgment module, used to judge whether the preset superstep interval is reached; A first prediction module is used for respectively predicting the overall running time of using a blocking checkpoint writing mode and a non-blocking checkpoint writing mode if the preset superstep interval is reached, to obtain a first overall running time and a second overall running time; A checkpoint writing module, configured to perform a checkpoint writing operation according to a checkpoint writing mode corresponding to a smaller value of the first overall running time and the second overall running time; A second prediction module is used to predict whether a system failure will occur based on current log information if the preset superstep interval is not reached, and a corresponding advance fault discovery time when a fault prediction result indicates that a fault will occur; The fault discovery time in advance is the time between the predicted fault occurrence time and the current time; The fault-tolerant strategy selection module is used to select different fault-tolerant strategies based on the comparison result between the fault advance discovery time and the preset time threshold, so as to perform fault recovery based on the fault-tolerant strategy when a system fault occurs.
18. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the system fault tolerance method according to any one of claims 1 to 16 is implemented.
19. An electronic device, characterized in that: It comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the system fault tolerance method as described in any one of claims 1 to 16 is implemented.
20. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the system fault tolerance method according to any one of claims 1 to 16 is implemented.
Citation Information
Cited By
Fault-tolerant processing method and device, medium and computer program product
CN119376994A
Model training method and device of intelligent calculation center and related equipment
CN120430436A
