Method for predicting task failure rate in cloud computing environment based on Markov chain

Abstract

The present invention discloses a method for predicting task failure rate in a cloud computing environment based on a Markov chain, comprising the following steps: S1, constructing a historical task data set; S2, cleaning and preprocessing the collected historical task data set; S3, constructing an improved Markov chain model using the preprocessed historical task data set, defining different states of task execution and their state transition probabilities; S4, dynamically adjusting the parameters of the improved Markov chain model to adapt to changes in different task types and cloud computing environments; S5, generating task failure rate prediction results based on the constructed improved Markov chain model; S6, providing a real-time prediction function based on the task failure rate prediction results; S7, optimizing the task scheduling strategy based on the task failure rate prediction results. The present invention provides a solution for task scheduling and resource management in a cloud computing environment by using a Markov chain model to predict task failure rates, and combining dynamic adjustment and scheduling optimization mechanisms.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a method for predicting task failure rate in a cloud computing environment based on a Markov chain. Background Art

[0002] As an emerging computing model, cloud computing provides computing resources and services through the Internet and has been widely used in various computing-intensive tasks. Task scheduling and resource management in cloud computing environments are important links to achieve efficient computing, which directly affect the reliability and resource utilization of the system. However, in practical applications, task failure is a common problem, which may be caused by a variety of factors, such as insufficient resources, system failure, or the characteristics of the task itself. Existing task scheduling methods usually find it difficult to accurately and real-time predict the task failure rate, resulting in the inability to adjust the scheduling strategy in a timely manner, thus affecting the overall performance and reliability of the system.

[0003] In the prior art, commonly used task failure rate prediction methods mainly include statistical analysis methods based on historical data and prediction models based on machine learning. The statistical analysis method based on historical data estimates the task failure rate by performing a simple statistical analysis on historical task data. Although this method is intuitive and simple, its prediction accuracy and real-time performance are poor, and it is difficult to cope with the complex and changeable cloud computing environment. Prediction models based on machine learning, such as decision trees and support vector machines, establish prediction models by training historical data. This method improves the accuracy of prediction to a certain extent, but its model complexity is high, it is highly dependent on data, and it has certain limitations when processing tasks with high real-time requirements.

[0004] In cloud computing environments, the optimization of task scheduling strategies is the key to improving system resource utilization and task execution efficiency. However, existing scheduling methods often lack effective prediction of task failure risks, resulting in unreasonable resource allocation, inefficient task execution, and even resource waste. For example, traditional priority-based scheduling methods only consider the priority order of tasks, while ignoring the possibility of task failure, which may cause high-priority tasks to fail frequently, further affecting the overall performance of the system. In addition, existing scheduling methods usually adopt static strategies, which are difficult to dynamically adjust according to the failure situations that occur during task execution, resulting in low system resource utilization.

[0005] In summary, existing technologies have many shortcomings in task failure rate prediction and task scheduling optimization, and it is difficult to meet the needs of efficient and reliable resource management in cloud computing environments. Summary of the invention

[0006] One purpose of the present invention is to propose a method for predicting task failure rate in a cloud computing environment based on Markov chain. The present invention provides an innovative and effective solution for task scheduling and resource management in a cloud computing environment by utilizing a Markov chain model to predict task failure rate and combining it with a dynamic adjustment and scheduling optimization mechanism.

[0007] A method for predicting task failure rate in a cloud computing environment based on a Markov chain according to an embodiment of the present invention includes the following steps:

[0008] S1, automatically collect task execution time, resource usage and task failure times from the cloud computing environment, and build a historical task dataset;

[0009] S2, clean and preprocess the collected historical task data set to remove noise and outliers;

[0010] S3. Using the preprocessed historical task data set to construct an improved Markov chain model, define different states of task execution and their state transition probabilities, where the states include task success, task failure, and task retry;

[0011] S4, dynamically adjust and improve the parameters of the Markov chain model to adapt to changes in different task types and cloud computing environments;

[0012] S5. Based on the constructed improved Markov chain model, the failure probability of future tasks is predicted to generate task failure rate prediction results;

[0013] S6. Provide real-time prediction function based on task failure rate prediction results to help system administrators understand the risk of task failure in a timely manner;

[0014] S7. According to the task failure rate prediction results, optimize the task scheduling strategy, select the best resources and time to execute the task, and reduce the probability of task failure.

[0015] Optionally, the S1 includes the following steps:

[0016] S11, obtaining a task execution log from a task scheduling system in a cloud computing environment, wherein the log records the start time, end time, resource allocation, and task execution result of each task, and extracting the task execution time, resource usage, and number of task failures;

[0017] S12. Define the task execution time T i :

[0018] T i =T end -T start ;

[0019] Among them, T endis the end time of task i, T start is the start time of task i;

[0020] S13. Define resource usage R i , including CPU usage, memory usage, and disk I / O, collects resource monitoring data and calculates the average usage of each resource:

[0021]

[0022] Where n is the number of monitoring data sampling points during the execution of task i, CPU j 、Memory j and I / O j are the CPU usage, memory usage, and disk I / O of the jth sampling point, respectively;

[0023] S14. Define the number of task failures F i , determine whether the task fails based on the task execution result, the failure count is 1, the success count is 0, and the number of failures that occurred during the task execution process is recorded;

[0024] S15. Execution time T of the collected tasks i , Resource usage R i and the number of task failures F i Cleanse to remove incomplete and inaccurate data records;

[0025] S16. Construct a historical task dataset D, which includes the execution time, resource usage, and number of task failures of each task:

[0026] D={(T i ,R i ,F i )|i=1,2,...,n};

[0027] Where n is the total number of tasks.

[0028] Optionally, S2 includes the following steps:

[0029] S21. Perform a preliminary screening on the collected historical task data set D, remove the records with missing key data, and retain the records containing the task execution time T i , Resource usage R i and the number of task failures F i Complete records of;

[0030] S22, task execution time T i Perform statistical analysis to remove extreme values ​​and outliers. Define outliers as values ​​that exceed a set threshold from the median, and determine the threshold by calculating the interquartile range of the data.

[0031] S23. Use z-score standardization method to analyze resource usage R i Normalize the CPU usage, memory usage, and disk I / O in

[0032] S24, the number of task failures F i Perform logic checks to ensure that the recorded number of task failures is consistent with the task execution results, and manually review and correct abnormal values;

[0033] S25. Use the time window technology to segment the historical task data set D to ensure that the data in each time period is consistent and comparable. Define the time window length as W and divide the data set into multiple subsets D according to the time period. k :

[0034] D k ={(T i ,R i ,F i )|T i ∈[t k ,t k+1 )};

[0035] Among them, t k and t k+1 is the start and end time of the kth time window;

[0036] S26. Perform a final check on the preprocessed historical task dataset D to generate a cleaned dataset:

[0037] D′={(T i ,R i ,F i )|i=1,2,...,n′};

[0038] Where n′ is the number of tasks in the cleaned dataset.

[0039] Optionally, S3 includes the following steps:

[0040] S31, using the preprocessed historical task data set D′, construct the task execution status set {S i}, where the status includes the task success status S s , Task failure status S f and task retry status S r ;

[0041] S32, define the state transfer matrix P, matrix element P ij Indicates that from state S i Transfer to state S jThe probability of constructing the initial state transfer matrix P is as follows:

[0042]

[0043] Among them, P ss , P sf , P sr They represent the probability of transferring from the task success state to the task success state, task failure state, and task retry state, respectively. fs , P ff , P fr They represent the probability of transitioning from the task failure state to the task success state, task failure state, and task retry state, respectively. rs , P rf , P rr They represent the probability of transitioning from the task retry state to the task success, task failure, and task retry states respectively;

[0044] S33, according to the task execution records in the historical task data set D', count the transition frequencies between states and calculate the state transition probability P ij :

[0045]

[0046] Among them, N ij Indicates that from state S i Transfer to state S j The number of times, N i Indicates state S i The total number of occurrences of

[0047] S34, combined with the task execution time T i and resource usage R i , introduce time weighting factor and resource weighting factor, make weighted adjustment on the state transfer matrix, and define time weighting factor W t and resource weighting factor W r as follows:

[0048]

[0049] Among them, T max is the maximum value of the task execution time, R max is the maximum value of resource usage;

[0050] S35, construct a weighted state transfer matrix P', matrix element P' ij Represents the state transition probability after weighted adjustment:

[0051] P' ij =P ij ×W t ×Wr ;

[0052] S36. Based on the weighted state transfer matrix P', an adaptive dynamic adjustment mechanism is further introduced to dynamically update the state transfer probability by real-time monitoring the changes in task execution in the cloud computing environment:

[0053]

[0054] Among them, P' ij (t) represents the state transition probability at time t, α is the smoothing coefficient, N ij (t+1) indicates that the state S is changed from state S to state S at time t+1. i Transfer to state S j The number of times, N i (t+1) represents the state S at time t+1 i The total number of occurrences of

[0055] S37, using the Bayesian estimation method to further optimize the dynamically adjusted state transfer matrix P", combining prior knowledge and historical data to obtain the final improved state transfer matrix P";

[0056] S38. Verify the improved state transfer matrix P" to ensure that it can accurately reflect the transfer relationship between states during task execution, and finally construct an improved Markov chain model M, whose state transfer matrix is ​​P".

[0057] Optionally, the S37 includes the following steps:

[0058] S371. Define prior distribution and set state transition probability P ij It obeys Beta distribution, and the prior distribution is:

[0059] P ij ~Beta(α ij ,β ij );

[0060] Among them, α ij and β ij are the shape parameters of the Beta distribution, reflecting the number of successes and failures in prior knowledge;

[0061] S372. Based on the observed data in the historical task data set D', define the likelihood function: Assuming that the number of observed transfers is N ij , then the likelihood function is:

[0062]

[0063] S373. Combining the prior distribution and likelihood function, use the Bayesian formula to update the posterior distribution of the state transition probability:

[0064] P(P ij |N ij )∝L(P ij )·P(P ij );

[0065] Substituting the prior distribution and likelihood function into the Bayesian formula, the posterior distribution is obtained as:

[0066] P(P ij |N ij )=Beta(α ij +N ij ,β ij +N i -N ij );

[0067] S374. Calculate the expected value of the posterior distribution as the estimated value of the final state transition probability:

[0068]

[0069] S375, substituting the calculated estimated value into the state transfer matrix to obtain the optimized state transfer matrix P":

[0070]

[0071] Optionally, S4 includes the following steps:

[0072] S41. Define a task feature set according to different task types and changes in cloud computing environments, where the task features include the task execution time T i , Resource usage R i and the number of task failures F i ;

[0073] S42, construct task type mapping function f(T i ,R i ,F i ), classify tasks into different types according to the task feature set and define a task type set;

[0074] S43, according to different task types C j Historical data, calculate the initial state transition probability matrix P of each type j , the matrix element P j,ij Indicates task type C j From state S i Transfer to state S j probability;

[0075] S44, introduce task type weighting factor The state transition probability is weighted and adjusted according to the importance of the task type, and the weighting factor is defined as follows:

[0076]

[0077] in, For task type C j The number of tasks, N total is the total number of all tasks;

[0078] S45. Weighting factor based on task type Adjust the state transition probability matrix and construct the weighted state transition matrix P' j , the matrix element P' j,ij represents the state transition probability after weighted adjustment, which is defined as follows:

[0079]

[0080] S46. Introduce a dynamic adjustment mechanism to dynamically update the state transition probability matrix by real-time monitoring of task execution changes in the cloud computing environment:

[0081]

[0082] Among them, α is the smoothing coefficient, N j,ij (t+1) indicates the task type C at time t+1 j From state S i Transfer to state S j The number of times, N j,i (t+1) indicates the task type C at time t+1 j Medium state S i The total number of occurrences of

[0083] S47, the dynamically adjusted state transition probability matrix P' j It matches the task execution data in the current cloud computing environment to ensure that the model parameters can adapt to different task types and environmental changes, and finally constructs a dynamically adjusted improved Markov chain model.

[0084] Optionally, S5 includes the following steps:

[0085] S51. Based on the dynamically adjusted improved Markov chain model, define the initial state vector v(t) of the current task, where v(t) represents the state probability distribution of the task execution at time t, and the vector element v i (t) indicates that the task is in state S i probability;

[0086] S52. Use the state transfer matrix P' of the improved Markov chain model to calculate the state probability distribution v(t+1) of the task at the next moment, and define the state transfer formula as follows:

[0087] v(t+1)=v(t)·P ″ ;

[0088] S53, iteratively apply the state transfer formula to predict the state probability distribution of tasks at multiple future moments, and generate the task state probability matrix V. The matrix elements V ij represents the task at time t i Always in state S j Probability:

[0089]

[0090] S54. Extract the task failure probability distribution from the task state probability matrix V, and define the task failure rate F(t+k) as the probability that the task will be in the task failure state S at the future time t+k. f Probability:

[0091] F(t+k)=v f (t+k);

[0092] Among them, F(t+k) means that the task is in the task failure state S at time t+k f probability;

[0093] S55. Generate a task failure rate prediction result according to the task failure rate F(t+k), construct a task failure rate prediction curve, help the system administrator to intuitively understand the task failure risk, and define the task failure rate prediction result set as:

[0094] {F(t+1),F(t+2),...,F(t+n)}.

[0095] Optionally, the S7 includes the following steps:

[0096] S71. According to the task failure rate prediction result, define the task scheduling optimization objective function f(R,T), where R represents the resource allocation, T represents the task execution time, and the objective function is used to minimize the task failure rate;

[0097] S72. Use the task failure rate prediction result set to build a task scheduling optimization model, define resource constraints and time constraints, and ensure that task scheduling optimization is performed under the premise of satisfying resource and time constraints;

[0098] S73. Apply genetic algorithm to optimize task scheduling, initialize the population, define the fitness function as the inverse of the task failure rate, and perform selection, crossover and mutation operations as follows:

[0099]

[0100] Among them, (R i ,T i ) represents the resource allocation and task execution time of the i-th individual, is the sum of the task failure rates of the i-th individual;

[0101] S74, through multiple generations of iterations, select individuals with high fitness, gradually approach the optimal solution, and finally determine the best resource allocation plan and task execution time to generate an optimized task scheduling strategy (R * ,T * ), where R * ,T * ) represents the optimized resource allocation and task execution time;

[0102] S75, optimize the task scheduling strategy (R * ,T * ) is applied in cloud computing environments to dynamically adjust resource allocation and task execution time, reduce the probability of task failure, and improve system resource utilization and task execution efficiency;

[0103] S76. Monitor task execution in real time, and continuously optimize task scheduling strategies based on monitoring data and prediction results to ensure that task scheduling optimization can adapt to dynamic changes in the cloud computing environment.

[0104] The beneficial effects of the present invention are:

[0105] (1) The present invention defines different states of task execution and their state transition probabilities by constructing and improving the Markov chain model, which can accurately describe the transition relationship between the states during task execution. By using the preprocessed historical task data set, the model parameters are dynamically adjusted to adapt to the changes of different task types and cloud computing environments, thereby improving the accuracy of task failure rate prediction. Based on the improved Markov chain model, the failure probability of future tasks can be predicted in real time, and task failure rate prediction results can be generated. System administrators can understand and respond to task failure risks in a timely manner based on real-time prediction results, thereby improving system reliability.

[0106] (2) The present invention introduces an adaptive dynamic adjustment mechanism, which monitors the changes in task execution in the cloud computing environment in real time and dynamically updates the state transition probability, so that the Markov chain model can continuously adapt to different task types and environmental changes. The optimized task scheduling strategy can select the best resources and time to execute tasks based on the latest task failure rate prediction results, reduce the probability of task failure, provide intelligent scheduling suggestions based on the task failure rate prediction results, optimize resource allocation, avoid resource waste caused by task failure, and improve task execution efficiency.

[0107] (3) The method of the present invention can be seamlessly integrated into the existing cloud computing platform and task scheduling system, and is applicable to various cloud computing environments and task types. It supports cross-platform applications, has wide applicability and flexibility, and continuously optimizes model parameters and scheduling strategies by real-time monitoring of task execution, combining prediction results and actual execution results, to ensure that the system can always maintain an efficient and reliable operating state. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0109] Figure 1 This is a flowchart of a method for predicting task failure rate in a cloud computing environment based on Markov chain proposed by the present invention. DETAILED DESCRIPTION

[0110] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0111] refer to Figure 1 , a method for predicting task failure rate in cloud computing environment based on Markov chain, comprising the following steps:

[0112] S1, automatically collect task execution time, resource usage and task failure times from the cloud computing environment, and build a historical task dataset;

[0113] S2, clean and preprocess the collected historical task data set to remove noise and outliers;

[0114] S3. Use the preprocessed historical task data set to build an improved Markov chain model and define different states of task execution and their state transition probabilities. The states include task success, task failure, and task retry.

[0115] S4, dynamically adjust and improve the parameters of the Markov chain model to adapt to changes in different task types and cloud computing environments;

[0116] S5. Based on the constructed improved Markov chain model, the failure probability of future tasks is predicted to generate task failure rate prediction results;

[0117] S6. Provide real-time prediction function based on task failure rate prediction results to help system administrators understand the risk of task failure in a timely manner;

[0118] S7. According to the task failure rate prediction results, optimize the task scheduling strategy, select the best resources and time to execute the task, and reduce the probability of task failure.

[0119] In this implementation, S1 includes the following steps:

[0120] S11, obtaining a task execution log from a task scheduling system in a cloud computing environment, wherein the log records the start time, end time, resource allocation, and task execution result of each task, and extracting the task execution time, resource usage, and number of task failures;

[0121] S12. Define the task execution time T i :

[0122] T i =T end -T start ;

[0123] Among them, T end is the end time of task i, T start is the start time of task i;

[0124] S13. Define resource usage R i , including CPU usage, memory usage, and disk I / O, collects resource monitoring data and calculates the average usage of each resource:

[0125]

[0126] Where n is the number of monitoring data sampling points during the execution of task i, CPU j 、Memory j and I / O j are the CPU usage, memory usage, and disk I / O of the jth sampling point, respectively;

[0127] S14. Define the number of task failures F i , determine whether the task fails based on the task execution result, the failure count is 1, the success count is 0, and the number of failures that occurred during the task execution process is recorded;

[0128] S15. Execution time T of the collected tasks i , Resource usage R i and the number of task failures F i Cleanse to remove incomplete and inaccurate data records;

[0129] S16. Construct a historical task dataset D, which includes the execution time, resource usage, and number of task failures of each task:

[0130] D={(T i , Ri , F i )|=1,2,...,n};

[0131] Where n is the total number of tasks.

[0132] In this implementation, S2 includes the following steps:

[0133] S21. Perform a preliminary screening on the collected historical task data set D, remove the records with missing key data, and retain the records containing the task execution time T i , Resource usage R i and the number of task failures F i Complete records of;

[0134] S22, task execution time T i Perform statistical analysis to remove extreme values ​​and outliers. Define outliers as values ​​that exceed a set threshold from the median, and determine the threshold by calculating the interquartile range of the data.

[0135] S23. Use z-score standardization method to analyze resource usage R i Normalize the CPU usage, memory usage, and disk I / O in

[0136] S24, the number of task failures F i Perform logic checks to ensure that the recorded number of task failures is consistent with the task execution results, and manually review and correct abnormal values;

[0137] S25. Use the time window technology to segment the historical task data set D to ensure that the data in each time period is consistent and comparable. Define the time window length as W and divide the data set into multiple subsets D according to the time period. k :

[0138] D k ={(T i , R i , F i )|T i ∈[t k , t k+1 )};

[0139] Among them, t k and t k+1 is the start and end time of the kth time window;

[0140] S26. Perform a final check on the preprocessed historical task dataset D to generate a cleaned dataset:

[0141] D′={(T i , R i , Fi )|=1, 2, ..., n′};

[0142] Where n′ is the number of tasks in the cleaned dataset.

[0143] In this implementation, S3 includes the following steps:

[0144] S31, using the preprocessed historical task data set D′, construct the task execution status set {S i}, where the status includes the task success status S s , Task failure status S f and task retry status S r ;

[0145] S32, define the state transfer matrix P, matrix element P ij Indicates that from state S i Transfer to state S j The probability of constructing the initial state transfer matrix P is as follows:

[0146]

[0147] Among them, P ss , P sf , P sr They represent the probability of transferring from the task success state to the task success state, task failure state, and task retry state, respectively. fs , P ff , P fr They represent the probability of transitioning from the task failure state to the task success state, task failure state, and task retry state, respectively. rs , P rf , P rr They represent the probability of transitioning from the task retry state to the task success, task failure, and task retry states respectively;

[0148] S33, according to the task execution records in the historical task data set D', count the transition frequencies between states and calculate the state transition probability P ij :

[0149]

[0150] Among them, N ij Indicates that from state S i Transfer to state S j The number of times, N i Indicates state S i The total number of occurrences of

[0151] S34, combined with the task execution time T i and resource usage R i, introduce time weighting factor and resource weighting factor, make weighted adjustment on the state transfer matrix, and define time weighting factor W t and resource weighting factor W r as follows:

[0152]

[0153] Among them, T max is the maximum value of the task execution time, R max is the maximum value of resource usage;

[0154] S35, construct a weighted state transfer matrix P', matrix element P' ij Represents the state transition probability after weighted adjustment:

[0155] P' ij =P ij ×W t ×W r ;

[0156] S36. Based on the weighted state transfer matrix P', an adaptive dynamic adjustment mechanism is further introduced to dynamically update the state transfer probability by real-time monitoring the changes in task execution in the cloud computing environment:

[0157]

[0158] Among them, P' ij (t) represents the state transition probability at time t, α is the smoothing coefficient, N ij (t+1) indicates that the state S is changed from state S to state S at time t+1. i Transfer to state S j The number of times, N i (t+1) represents the state S at time t+1 i The total number of occurrences of

[0159] S37, using the Bayesian estimation method to further optimize the dynamically adjusted state transfer matrix P″, combining prior knowledge and historical data to obtain the final improved state transfer matrix P″;

[0160] S38. Verify the improved state transfer matrix P″ to ensure that it can accurately reflect the transfer relationship between the states during the task execution process, and finally construct an improved Markov chain model M, whose state transfer matrix is ​​P″.

[0161] In this implementation, S37 includes the following steps:

[0162] S371. Define prior distribution and set state transition probability P ij It obeys Beta distribution, and the prior distribution is:

[0163] P ij ~Beta(α ij , β ij );

[0164] Among them, α ij and β ij are the shape parameters of the Beta distribution, reflecting the number of successes and failures in prior knowledge;

[0165] S372. Based on the observed data in the historical task data set D′, define the likelihood function: Assuming that the number of observed transfers is N ij , then the likelihood function is:

[0166]

[0167] S373. Combining the prior distribution and likelihood function, use the Bayesian formula to update the posterior distribution of the state transition probability:

[0168] P(P ij |N ij )∝L(P ij )·P(P ij );

[0169] Substituting the prior distribution and likelihood function into the Bayesian formula, the posterior distribution is obtained as:

[0170] P(P ij |N ij )=Beta(α ij +N ij , β ij +N i -N ij );

[0171] S374. Calculate the expected value of the posterior distribution as the estimated value of the final state transition probability:

[0172]

[0173] S375, substituting the calculated estimated value into the state transfer matrix to obtain the optimized state transfer matrix P":

[0174]

[0175] In this implementation, S4 includes the following steps:

[0176] S41. Define a task feature set according to different task types and changes in cloud computing environments, where the task features include the task execution time T i , Resource usage R i and the number of task failures F i ;

[0177] S42, construct task type mapping function f(T i ,R i ,F i ), classify tasks into different types according to the task feature set and define a task type set;

[0178] S43, according to different task types C j Historical data, calculate the initial state transition probability matrix P of each type j , the matrix element P j , ij Indicates task type C j From state S i Transfer to state S j probability;

[0179] S44, introduce task type weighting factor The state transition probability is weighted and adjusted according to the importance of the task type, and the weighting factor is defined as follows:

[0180]

[0181] in, For task type C j The number of tasks, N total is the total number of all tasks;

[0182] S45. Weighting factor based on task type Adjust the state transition probability matrix and construct the weighted state transition matrix P' j , the matrix element P' j,ij represents the state transition probability after weighted adjustment, which is defined as follows:

[0183]

[0184] S46. Introduce a dynamic adjustment mechanism to dynamically update the state transition probability matrix by real-time monitoring of task execution changes in the cloud computing environment:

[0185]

[0186] Among them, α is the smoothing coefficient, N j,ij (t+1) indicates the task type C at time t+1 j From state S i Transfer to state S j The number of times, N j,i (t+1) indicates the task type C at time t+1 j Medium state S i The total number of occurrences of

[0187] S47, the dynamically adjusted state transition probability matrix P' j It matches the task execution data in the current cloud computing environment to ensure that the model parameters can adapt to different task types and environmental changes, and finally constructs a dynamically adjusted improved Markov chain model.

[0188] In this implementation, S5 includes the following steps:

[0189] S51. Based on the dynamically adjusted improved Markov chain model, define the initial state vector v(t) of the current task, where v(t) represents the state probability distribution of the task execution at time t, and the vector element v i (t) indicates that the task is in state S i probability;

[0190] S52. Use the state transfer matrix P' of the improved Markov chain model to calculate the state probability distribution v(t+1) of the task at the next moment, and define the state transfer formula as follows:

[0191] v(t+1)=v(t)·P";

[0192] S53, iteratively apply the state transfer formula to predict the state probability distribution of tasks at multiple future moments, and generate the task state probability matrix V. The matrix elements V ij represents the task at time t i Always in state S j Probability:

[0193]

[0194] S54. Extract the task failure probability distribution from the task state probability matrix V, and define the task failure rate F(t+k) as the probability that the task will be in the task failure state S at the future time t+k. f Probability:

[0195] F(t+k)=v f (t+k);

[0196] Among them, F(t+k) means that the task is in the task failure state S at time t+k f probability;

[0197] S55. Generate a task failure rate prediction result according to the task failure rate F(t+k), construct a task failure rate prediction curve, help the system administrator to intuitively understand the task failure risk, and define the task failure rate prediction result set as:

[0198] {F(t+1),F(t+2),...,F(t+n)}.

[0199] In this implementation, S7 includes the following steps:

[0200] S71. According to the task failure rate prediction result, define the task scheduling optimization objective function f(R,T), where R represents the resource allocation, T represents the task execution time, and the objective function is used to minimize the task failure rate;

[0201] S72. Use the task failure rate prediction result set to build a task scheduling optimization model, define resource constraints and time constraints, and ensure that task scheduling optimization is performed under the premise of satisfying resource and time constraints;

[0202] S73. Apply genetic algorithm to optimize task scheduling, initialize the population, define the fitness function as the inverse of the task failure rate, and perform selection, crossover and mutation operations as follows:

[0203]

[0204] Among them, (R i ,T i ) represents the resource allocation and task execution time of the i-th individual, is the sum of the task failure rates of the i-th individual;

[0205] S74, through multiple generations of iterations, select individuals with high fitness, gradually approach the optimal solution, and finally determine the best resource allocation plan and task execution time to generate an optimized task scheduling strategy (R * ,T * ), where R * ,T * ) represents the optimized resource allocation and task execution time;

[0206] S75, optimize the task scheduling strategy (R * ,T * ) is applied in cloud computing environments to dynamically adjust resource allocation and task execution time, reduce the probability of task failure, and improve system resource utilization and task execution efficiency;

[0207] S76. Monitor task execution in real time, and continuously optimize task scheduling strategies based on monitoring data and prediction results to ensure that task scheduling optimization can adapt to dynamic changes in the cloud computing environment.

[0208] This embodiment aims at the problems of the prior art. By collecting and preprocessing historical task data, the Markov chain model is used to model and predict the task failure rate, thereby realizing the real-time prediction of the probability of future task failure. The Markov chain model is used to predict the task failure rate, and the complex task scheduling problem is converted into a prediction problem of the probability model, which provides a new solution. The Markov chain model can accurately describe the transition relationship between the states during the task execution process by defining different states of task execution and their state transition probabilities, thereby improving the accuracy of the task failure rate prediction. Secondly, by introducing a dynamic adjustment mechanism, the Markov chain model parameters are dynamically adjusted according to the changes of different task types and cloud computing environments, so that the model can adapt to different task characteristics and environmental changes, further improving the real-time and accuracy of the prediction. Finally, by applying the task failure rate prediction results, the task scheduling strategy is optimized, the best resources and time are selected to execute the task, and the probability of task failure is reduced, which significantly improves the reliability and efficiency of task execution in the cloud computing environment and reduces the waste of resources and time delay caused by task failure.

[0209] Embodiment 1:

[0210] In a large cloud computing data center, the data center processes millions of tasks every day, covering a variety of tasks from data storage, computing to data transmission. As the number of tasks increases, the task failure rate also increases significantly, resulting in a waste of system resources and low efficiency in task execution. In order to improve the reliability and resource utilization of the system, an efficient and accurate task failure rate prediction method is needed. This embodiment introduces a task failure rate prediction method in a cloud computing environment based on a Markov chain, and verifies its superiority through specific data.

[0211] This embodiment applies a task failure rate prediction method based on a Markov chain in the cloud computing data center. The specific steps are as follows:

[0212] First, we obtain the task execution log from the task scheduling system in the data center and extract data such as task execution time, resource usage, and task failure count. Task execution time is defined as the difference between the start time and the end time of the task. Resource usage includes CPU usage, memory usage, and disk I / O. Task failure count records the number of failures that occurred during task execution.

[0213] Secondly, the collected historical task data is cleaned and preprocessed to remove incomplete or abnormal records to ensure the accuracy and reliability of the data. The interquartile range (IQR) of the data is calculated to remove outliers that are beyond the upper and lower quartiles. The resource usage is standardized to ensure that all data are in the same dimension.

[0214] Then, the Markov chain model is constructed using the preprocessed historical task data, and the three states of task success, task failure and task retry and their state transition probabilities are defined. According to the task execution records, the transition frequencies between states are counted, and the state transition probability matrix is ​​calculated. A dynamic adjustment mechanism is introduced to monitor the changes in task execution in the cloud computing environment in real time, and the state transition probability is dynamically updated to ensure that the model can adapt to different task types and environmental changes.

[0215] Next, based on the constructed Markov chain model, the failure probability of future tasks is predicted. By iteratively applying the state transition formula, the state probability distribution of tasks at multiple moments in the future is predicted, and the task failure rate prediction results are generated. System administrators can understand the risk of task failure in a timely manner based on real-time prediction results, and optimize task scheduling strategies based on prediction results, select the best resources and time to execute tasks, and reduce the probability of task failure.

[0216] Concrete data proves our beneficial effects:

[0217] In this embodiment, we selected data from June 2023 to December 2023 as experimental data. The experiment is divided into two parts: one part applies the task failure rate prediction method based on Markov chain of the present invention, and the other part applies the traditional task failure rate prediction method based on statistical analysis. The data comparison results are shown in Table 1 below:

[0218] Table 1 Task failure rate comparison table

[0219]

[0220]

[0221] From the data in Table 1 above, it can be seen that the method of the present invention has significant advantages over the traditional statistical analysis method in predicting task failure rate. Specifically, the task failure rate is reduced by about 2 percentage points, and the task execution efficiency is improved by about 15% to 17%. This improvement is of great significance for large cloud computing data centers, which not only reduces the waste of resources caused by task failures, but also significantly improves the overall performance and reliability of the system.

[0222] Through practical application, the method of the present invention not only improves the accuracy and real-time performance of task failure rate prediction, but also optimizes the task scheduling strategy, improves system resource utilization and task execution efficiency. In summary, the Markov chain-based task failure rate prediction method in a cloud computing environment of the present invention has shown significant beneficial effects in practical applications, and provides an innovative and effective solution for task scheduling and resource management in a cloud computing environment.

[0223] The present invention defines different states of task execution and their state transition probabilities by constructing and improving the Markov chain model, which can accurately describe the transition relationship between the states during task execution. It uses the preprocessed historical task data set to dynamically adjust the model parameters to adapt to the changes of different task types and cloud computing environments, thereby improving the accuracy of task failure rate prediction. Based on the improved Markov chain model, the failure probability of future tasks can be predicted in real time, and task failure rate prediction results can be generated. System administrators can timely understand and respond to task failure risks based on real-time prediction results, thereby improving the reliability of the system.

[0224] The present invention introduces an adaptive dynamic adjustment mechanism, which monitors the changes in task execution in the cloud computing environment in real time and dynamically updates the state transition probability, so that the Markov chain model can continuously adapt to different task types and environmental changes. The optimized task scheduling strategy can select the best resources and time to execute tasks based on the latest task failure rate prediction results, reduce the probability of task failure, provide intelligent scheduling suggestions based on the task failure rate prediction results, optimize resource allocation, avoid resource waste caused by task failure, and improve task execution efficiency.

[0225] The method of the present invention can be seamlessly integrated into the existing cloud computing platform and task scheduling system, and is applicable to various cloud computing environments and task types. It supports cross-platform applications, has wide applicability and flexibility, and continuously optimizes model parameters and scheduling strategies by real-time monitoring of task execution, combining prediction results and actual execution results, to ensure that the system can always maintain an efficient and reliable operating state.

[0226] In summary, the present invention innovatively uses the Markov chain model to predict task failure rates, and combines dynamic adjustment and scheduling optimization mechanisms to achieve efficient and reliable task scheduling and resource management in a cloud computing environment. This method not only improves the accuracy and real-time performance of task failure rate prediction, but also optimizes the task scheduling strategy, significantly improves the overall performance and resource utilization of the system, and provides a solution with important practical value for the field of cloud computing.

[0227] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for predicting task failure rate in cloud computing environment based on Markov chain, characterized in that: The steps include: S1, automatically collect task execution time, resource usage and task failure times from the cloud computing environment, and build a historical task dataset; S2, clean and preprocess the collected historical task data set to remove noise and outliers; S3. Using the preprocessed historical task data set to construct an improved Markov chain model, define different states of task execution and their state transition probabilities, where the states include task success, task failure, and task retry; S4, dynamically adjust and improve the parameters of the Markov chain model to adapt to changes in different task types and cloud computing environments; S5. Based on the constructed improved Markov chain model, the failure probability of future tasks is predicted to generate task failure rate prediction results; S6. Provide real-time prediction function based on task failure rate prediction results to help system administrators understand the risk of task failure in a timely manner; S7. According to the prediction results of task failure rate, optimize the task scheduling strategy, select the best resources and time to execute the task, and reduce the probability of task failure; The S3 comprises the following steps: S31, using the preprocessed historical task data set D′, construct the task execution status set {S i }, where the status includes the task success status S s , Task failure status S f and task retry status S r ; S32, define the state transfer matrix P, matrix element P ij Indicates that from state S i Transfer to state S j The probability of constructing the initial state transfer matrix P is as follows: Among them, P ss , P sf , P sr They represent the probability of transferring from the task success state to the task success state, task failure state, and task retry state, respectively. fs , P ff , P fr They represent the probability of transitioning from the task failure state to the task success state, task failure state, and task retry state, respectively. rs , P rf , P rr They represent the probability of transitioning from the task retry state to the task success, task failure, and task retry states respectively; S33, according to the task execution records in the historical task data set D', count the transition frequencies between states and calculate the state transition probability P ij : Among them, N ij Indicates that from state S i Transfer to state S j The number of times, N i Indicates state S i The total number of occurrences of S34, combined with the task execution time T i and resource usage R i , introduce time weighting factor and resource weighting factor, make weighted adjustment on the state transfer matrix, and define time weighting factor W t and resource weighting factor W r as follows: Among them, T max is the maximum value of the task execution time, R max is the maximum value of resource usage; S35, construct a weighted state transfer matrix P', matrix element P' ij Represents the state transition probability after weighted adjustment: P′ ij =P ij ×W t ×W r ; S36. Based on the weighted state transfer matrix P', an adaptive dynamic adjustment mechanism is further introduced to dynamically update the state transfer probability by real-time monitoring the changes in task execution in the cloud computing environment: Among them, P′ ij (t) represents the state transition probability at time t, α is the smoothing coefficient, N ij (t+1) indicates that the state S is changed from state S to state S at time t+1. i Transfer to state S j The number of times, N i (t+1) represents the state S at time t+1 i The total number of occurrences of S37, using the Bayesian estimation method to further optimize the dynamically adjusted state transfer matrix P", combining prior knowledge and historical data to obtain the final improved state transfer matrix P"; S38. Verify the improved state transfer matrix P" to ensure that it can accurately reflect the transfer relationship between states during task execution, and finally construct an improved Markov chain model M, whose state transfer matrix is ​​P".

2. According to claim 1, a method for predicting task failure rate in a cloud computing environment based on Markov chain is characterized in that: The S1 comprises the following steps: S11, obtaining a task execution log from a task scheduling system in a cloud computing environment, wherein the log records the start time, end time, resource allocation, and task execution result of each task, and extracting the task execution time, resource usage, and number of task failures; S12. Define the task execution time T i : T i =T end -T start ; Among them, T end is the end time of task i, T start is the start time of task i; S13. Define resource usage R i , including CPU usage, memory usage, and disk I / O, collects resource monitoring data and calculates the average usage of each resource: Where n is the number of monitoring data sampling points during the execution of task i, CPU j 、Memory j and I / O j are the CPU usage, memory usage, and disk I / O of the jth sampling point, respectively; S14. Define the number of task failures F i , determine whether the task fails based on the task execution result, the failure count is 1, the success count is 0, and the number of failures that occurred during the task execution process is recorded; S15. Execution time T of the collected tasks i , Resource usage R i and the number of task failures F i Cleanse to remove incomplete and inaccurate data records; S16. Construct a historical task dataset D, which includes the execution time, resource usage, and number of task failures of each task: D={(T i ,R i ,F i )|i=1,2,...,n1}; Where n1 is the total number of tasks.

3. The method for predicting task failure rate in cloud computing environment based on Markov chain according to claim 2 is characterized in that: The S2 comprises the following steps: S21. Perform a preliminary screening on the collected historical task data set D, remove the records with missing key data, and retain the records containing the task execution time T i , Resource usage R i and the number of task failures F i Complete records of S22, task execution time T i Perform statistical analysis to remove extreme values ​​and outliers. Define outliers as values ​​that exceed a set threshold from the median, and determine the threshold by calculating the interquartile range of the data. S23. Use z-score standardization method to analyze resource usage R i Normalize the CPU usage, memory usage, and disk I / O in S24, the number of task failures F i Perform logic checks to ensure that the recorded number of task failures is consistent with the task execution results, and manually review and correct abnormal values; S25. Use the time window technology to segment the historical task data set D to ensure that the data in each time period is consistent and comparable. Define the time window length as W and divide the data set into multiple subsets D according to the time period. k : D k ={(T i ,R i ,F i )|T i ∈[t k ,t k+1 )}; Among them, t k and t k+1 is the start and end time of the kth time window; S26. Perform a final check on the preprocessed historical task dataset D to generate a cleaned dataset: D′={(T i ,R i ,F i )|i=1,2,...,n′}; Where n′ is the number of tasks in the cleaned dataset.

4. According to claim 1, a method for predicting task failure rate in a cloud computing environment based on Markov chain is characterized in that: The S37 comprises the following steps: S371. Define prior distribution and set state transition probability P ij It obeys Beta distribution, and the prior distribution is: P ij ~Beta(α ij ,b ij ); Among them, α ij and β ij are the shape parameters of the Beta distribution, reflecting the number of successes and failures in prior knowledge; S372. Based on the observed data in the historical task data set D', define the likelihood function: Assuming that the number of observed transfers is N ij , then the likelihood function is: S373. Combining the prior distribution and likelihood function, use the Bayesian formula to update the posterior distribution of the state transition probability: P(P ij |N ij )∝L(P ij )·P(P ij ); Substituting the prior distribution and likelihood function into the Bayesian formula, the posterior distribution is obtained as: P(P ij |N ij )=Beta(α ij +N ij ,b ij +N i -N ij ); S374. Calculate the expected value of the posterior distribution as the estimated value of the final state transition probability: S375, substituting the calculated estimated value into the state transfer matrix to obtain the optimized state transfer matrix P":

5. The method for predicting task failure rate in cloud computing environment based on Markov chain according to claim 1, characterized in that: The S4 comprises the following steps: S41. Define a task feature set according to different task types and changes in cloud computing environments, where the task features include the task execution time T i , Resource usage R i and the number of task failures F i ; S42, construct task type mapping function f(T i ,R i ,F i ), classify tasks into different types according to the task feature set and define a task type set; S43, according to different task types C j Historical data, calculate the initial state transition probability matrix P of each type j , the matrix element P j,ij Indicates task type C j From state S i Transfer to state S j The probability of S44, introduce task type weighting factor The state transition probability is weighted and adjusted according to the importance of the task type, and the weighting factor is defined as follows: in, For task type C j The number of tasks, N total is the total number of all tasks; S45. Weighting factor based on task type Adjust the state transition probability matrix and construct the weighted state transition matrix P' j , the matrix element P' j,ij represents the state transition probability after weighted adjustment, which is defined as follows: P' j,ij =P j,ij ×W Cj ; S46. Introduce a dynamic adjustment mechanism to dynamically update the state transition probability matrix by real-time monitoring of task execution changes in the cloud computing environment: Among them, α is the smoothing coefficient, N j,ij (t+1) indicates the task type C at time t+1 j From state S i Transfer to state S j The number of times, N j,i (t+1) indicates the task type C at time t+1 j Medium state S i The total number of occurrences of S47, the dynamically adjusted state transition probability matrix P′ j It matches the task execution data in the current cloud computing environment to ensure that the model parameters can adapt to different task types and environmental changes, and finally constructs a dynamically adjusted improved Markov chain model.

6. The method for predicting task failure rate in cloud computing environment based on Markov chain according to claim 5, characterized in that: The S5 comprises the following steps: S51. Based on the dynamically adjusted improved Markov chain model, define the initial state vector v(t) of the current task, where v(t) represents the state probability distribution of the task execution at time t, and the vector element v i (t) indicates that the task is in state S i probability; S52. Use the state transfer matrix P' of the improved Markov chain model to calculate the state probability distribution v(t+1) of the task at the next moment, and define the state transfer formula as follows: v(t+1)=v(t)·P″; S53, iteratively apply the state transfer formula to predict the state probability distribution of tasks at multiple future moments, and generate the task state probability matrix V. The matrix elements V ij represents the task at time t i Always in state S j Probability: S54. Extract the task failure probability distribution from the task state probability matrix V, and define the task failure rate F(t+k) as the probability that the task will be in the task failure state S at the future time t+k. f Probability: F(t+k)=v f (t+k); Among them, F(t+k) means that the task is in the task failure state S at time t+k f The probability of S55. Generate a task failure rate prediction result according to the task failure rate F(t+k), construct a task failure rate prediction curve, help the system administrator to intuitively understand the task failure risk, and define the task failure rate prediction result set as: {F(t+1),F(t+2),...,F(t+n)}.

7. The method for predicting task failure rate in cloud computing environment based on Markov chain according to claim 6, characterized in that: The S7 comprises the following steps: S71. According to the task failure rate prediction result, define the task scheduling optimization objective function f(R,T), where R represents the resource allocation, T represents the task execution time, and the objective function is used to minimize the task failure rate; S72. Use the task failure rate prediction result set to build a task scheduling optimization model, define resource constraints and time constraints, and ensure that task scheduling optimization is performed under the premise of satisfying resource and time constraints; S73. Apply genetic algorithm to optimize task scheduling, initialize the population, define the fitness function as the inverse of the task failure rate, and perform selection, crossover and mutation operations as follows: Among them, (R i ,T i ) represents the resource allocation and task execution time of the i-th individual, is the sum of the task failure rates of the i-th individual; S74, through multiple generations of iterations, select individuals with high fitness, gradually approach the optimal solution, and finally determine the best resource allocation plan and task execution time to generate an optimized task scheduling strategy (R * ,T * ), where (R * ,T * ) represents the optimized resource allocation and task execution time; S75, optimize the task scheduling strategy (R * ,T * ) is applied in cloud computing environments to dynamically adjust resource allocation and task execution time, reduce the probability of task failure, and improve system resource utilization and task execution efficiency; S76. Monitor task execution in real time, and continuously optimize task scheduling strategies based on monitoring data and prediction results to ensure that task scheduling optimization can adapt to dynamic changes in the cloud computing environment.

Images