Multi-task data analysis method and device and storage medium

Abstract

The invention relates to the technical field of information, in particular to a multi-task data analysis method and device and a storage medium, and the method comprises the steps: constructing a multi-dimensional task state feature vector as the input of a task loss prediction model by collecting a task execution progress, a resource use condition and an abnormal log index in a task operation process; constructing a task dependency graph and a resource conflict graph according to the dependency relationship between the tasks and the resource occupation condition, learning the relevance between the tasks and a resource competition mode through a graph neural network algorithm, and obtaining an influence factor and a propagation path of task loss; the performance of a task loss prediction model is continuously tracked, model parameters are dynamically updated through an incremental learning mechanism, the method adapts to changes of a multi-task environment, prediction accuracy and real-time performance are improved, and a reliable decision basis is provided for task scheduling and resource optimization.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a multi-task data analysis method, device and storage medium. Background Art

[0002] In complex multi-task data analysis business scenarios, the intricate dependencies between tasks lead to low execution efficiency of task processes. Especially on the critical task path, once a task is stuck or lost, it will trigger a chain reaction, and subsequent tasks cannot be completed on time, seriously affecting business continuity. For example, in the order fulfillment process, from order acceptance, stock preparation, packaging to delivery, each link involves multiple subtasks. If the stock preparation task is delayed due to out-of-stock, the packaging and delivery tasks must wait, even if the priority is high. What's more tricky is that the dependencies between tasks may change dynamically. Two originally parallel tasks may generate new dependencies due to resource contention, and two originally serial tasks may remove dependencies due to changes in conditions. How to capture the dynamic changes in the dependencies between tasks in real time and predict the risk of loss of critical tasks based on this has become a difficult point in business continuity risk management. Traditional task monitoring methods can only passively respond to task delays or losses that have already occurred, but lack the ability to predict in advance. There is an urgent need for innovative task operation data collection, calculation, storage and analysis technologies. By comprehensively and real-time collecting the operation status data of each task, we can deeply explore the task dependency patterns hidden behind the data and build a task loss risk prediction model for business continuity. This will enable us to make advance predictions and proactively avoid key tasks that may affect business continuity and their risk levels, eliminate risks in the bud and safeguard business continuity. Summary of the invention

[0003] The technical problem solved by the present invention is to provide a multi-task data analysis method, device and storage medium, which can effectively predict and prevent the risk of task loss, optimize resource allocation and task scheduling, and improve the overall operating efficiency of the multi-task environment.

[0004] The present invention provides a multi-task data analysis method, the method comprising:

[0005] By collecting the task execution progress, resource usage, and abnormal log indicators during the task running process, a multi-dimensional task status feature vector is constructed as the input of the task loss prediction model;

[0006] According to the dependencies between tasks and resource usage, we build a task dependency graph and a resource conflict graph. We use the graph neural network algorithm to learn the correlation between tasks and the resource competition pattern, and obtain the influencing factors and propagation paths of task loss.

[0007] Use long short-term memory network to perform time series modeling on task status feature vectors, capture the dynamic change rules during task operation, and predict the health status and completion possibility of tasks in the future. If the predicted value is lower than the preset threshold, it is judged that the task has a loss risk.

[0008] For tasks with a high risk of loss, the priority and resource allocation strategy are dynamically adjusted through reinforcement learning algorithms to ensure the smooth completion of key tasks while minimizing the impact of task loss on overall performance and obtaining the optimal task scheduling decision.

[0009] Monitor the task running status and resource usage in real time. When abnormal fluctuations occur or exceed the preset threshold, the task loss warning mechanism is triggered and corresponding intervention measures are taken according to the warning level.

[0010] In the task scheduling process, the task loss prediction results, task dependencies, and resource usage factors are comprehensively considered, and the global optimal scheduling strategy is solved through a multi-objective optimization algorithm to balance short-term scheduling benefits and long-term optimization goals;

[0011] Continuously track the performance of the task churn prediction model and dynamically update the model parameters through the incremental learning mechanism.

[0012] The present invention also relates to a multi-task data analysis device, which includes a memory, a processor, and a multi-task data analysis program stored in the memory and executable on the processor, wherein the program implements the steps of the multi-task data analysis method when executed by the processor.

[0013] The present invention also relates to a computer-readable storage medium, on which a multi-task data analysis program is stored, and when the program is executed by a processor, the steps of the multi-task data analysis method are implemented.

[0014] The technical solution of the present invention includes the following beneficial effects:

[0015] The present invention discloses a multi-task data analysis method. The method constructs a feature vector by collecting multi-dimensional indicators during the task operation process, combines the task dependency graph and the resource conflict graph, and uses a graph neural network to learn the correlation between tasks. A long short-term memory network is used to perform time series modeling on the task status to predict the task health status and completion possibility. For tasks with high loss risk, priority and resource allocation are dynamically adjusted through reinforcement learning. The task status is monitored in real time, an early warning mechanism is triggered, and intervention measures are taken. In the scheduling process, the loss prediction results, task dependencies, and resource usage are comprehensively considered, and the global optimal scheduling strategy is solved through a multi-objective optimization algorithm. The present invention also continuously updates the model through incremental learning to improve the prediction accuracy and real-time performance. The method can effectively predict and prevent the risk of task loss, optimize resource allocation and task scheduling, improve the overall operating efficiency of the multi-task environment, and provide a new solution for task management and resource scheduling in a large-scale distributed computing environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the multi-task data analysis method of the present invention;

[0017] Figure 2 It is a structural schematic diagram of the multi-task data analysis system of the present invention. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0019] like Figure 1-2 The multi-task data analysis method involved in this embodiment specifically includes the following steps:

[0020] Step S101, by collecting the task execution progress, resource usage, and abnormal log indicators during the task running process, a multi-dimensional task state feature vector is constructed as the input of the task loss prediction model.

[0021] According to the various indicator data during the task execution process, a multi-dimensional task status feature vector is constructed as the input of the machine learning model. The sliding time window method is used to dynamically obtain the latest task execution progress, resource usage, and abnormal log indicator data. By setting the abnormal threshold, it is determined whether there is an abnormality during the task execution process. If an abnormality occurs, it is marked in the task status feature vector. The collected indicator data of each dimension is preprocessed, including missing value processing, outlier processing, and data normalization. For missing values, deletion or filling is selected according to the characteristics of the data; for outliers, the box plot method is used to identify and process; for numerical indicators, the maximum and minimum value normalization or Z-score normalization method is used for normalization. The filtering method is used to select indicators that have a significant impact on the prediction of task loss as features. The preprocessed multi-dimensional indicator data is combined into a task status feature vector, and each time window corresponds to a feature vector sample. The logistic regression algorithm is used to train a binary classification model with the task status feature vector as input to predict whether the task will be lost. During the model training process, the model is evaluated by cross-validation and hyperparameters are selected. At the same time, for the problem of class imbalance, oversampling, undersampling or adjusting class weights can be used to deal with it. Deploy the trained task loss prediction model to the production environment and monitor the task execution status in real time. When the probability of task loss exceeds the preset threshold, send a task loss warning to relevant personnel through email, SMS or instant messaging tools, prompting them to restore or reschedule the task in time. For abnormal situations during task execution, set anomaly detection rules and thresholds to monitor the execution indicators of tasks in real time. When the indicator data exceeds the normal range, trigger an abnormal alarm, record the abnormal log, and retry, restart or terminate the task according to the type and severity of the abnormality. Continuously collect task execution data, regularly retrain and optimize the task loss prediction model to improve the prediction accuracy and real-time performance. At the same time, dynamically adjust the anomaly detection rules and thresholds to adapt to changes in the task execution environment and improve the accuracy and reliability of anomaly detection.

[0022] Specifically, constructing a multi-dimensional task status feature vector is the key to predicting task churn in machine learning models. Taking e-commerce platform order processing as an example, we can collect indicators such as order processing time, system resource utilization, and number of error logs. A 5-minute sliding window is used to dynamically obtain the latest data. If the order processing time exceeds 30 minutes, the anomaly is marked in the feature vector. Preprocessing the collected indicator data is crucial. For example, when there are missing values ​​in the order processing time, the average value of the period can be used to fill it; the system resource utilization rate exceeding 95% is considered an outlier and replaced by 90%; the number of error logs is normalized to the range of 0-1 using the maximum and minimum values. These preprocessing steps can improve data quality and lay the foundation for subsequent modeling. Feature selection can use filtering methods such as chi-square test. Assuming that the analysis finds that the order processing time and system resource utilization are significantly correlated with task churn, while the number of error logs is not highly correlated, the first two indicators are selected to construct the feature vector. Each 5-minute time window corresponds to a two-dimensional feature vector sample, such as [0.8,0.9], which means that the order processing time is 24 minutes and the system resource utilization rate is 90%. Logistic regression is a commonly used binary classification algorithm. With feature vectors as input, the model is trained to predict whether an order will time out. The optimal regularization parameter can be selected through 5-fold cross validation. Since there are fewer samples of timed orders, the SMOTE oversampling method can be used to balance the ratio of positive and negative samples. These technologies can improve the generalization ability of the model and the prediction accuracy of minority classes. Deploy the trained model to the production environment and monitor the order processing status in real time. When the probability of predicted timeout exceeds 80%, send warning information to relevant personnel through WeChat for Business. At the same time, set rules to detect abnormal situations. For example, if the system resource utilization rate exceeds 95% for three consecutive time windows, an alarm is triggered and server resources are automatically expanded. This real-time monitoring and warning mechanism can detect potential problems in time and improve the reliability of the system. It is also important to continuously optimize models and rules. The model can be retrained every week to adapt to business changes. For example, during the Double Eleven period, the order volume surged, and the timeout threshold and resource utilization threshold needed to be adjusted. Through continuous optimization, the prediction accuracy and anomaly detection reliability are improved, and ultimately the intelligent monitoring and management of the entire process of task execution can be achieved.

[0023] Step S102, construct a task dependency graph and a resource conflict graph based on the dependencies between tasks and resource occupancy, learn the correlation between tasks and resource competition patterns through the graph neural network algorithm, and obtain the influencing factors and propagation paths of task loss.

[0024] After obtaining the task execution records and resource usage logs, the data is first cleaned and preprocessed to remove noise data and outliers and extract key field information. Then feature engineering is performed to construct feature vectors of tasks and resources, including features such as task execution time, resource usage, and task priority. Based on the constructed feature vectors, the Apriori algorithm is used to discover the dependency relationship between tasks and the association pattern of resources, and to construct a task dependency graph and a resource conflict graph. For the constructed task dependency graph and resource conflict graph, a graph convolutional neural network (GCN) is used to represent and learn the graph structure data, and the correlation between tasks and the resource competition pattern are captured through node embedding vectors. In the GCN model training process, the attention mechanism is introduced to dynamically adjust the connection weights between nodes by designing the task importance scoring function and the resource scarcity scoring function, highlighting the importance of key task nodes and key resource nodes. According to the task correlation and resource competition pattern learned by the GCN model, the PageRank algorithm is used to calculate the importance scores of task nodes and resource nodes, identify the key tasks and key resources that have a significant impact on task execution, and set the importance score threshold for screening. Apply the Monte Carlo tree search algorithm on the task dependency graph, simulate the propagation process of task loss multiple times, count the influence of task nodes, and predict the impact range of task loss on downstream tasks and the degree of delay in task completion time. Evaluate the loss risk of identified key tasks and key resources, design task backup strategies based on task importance and resource scarcity, and implement master-slave backup and checkpoint mechanisms for high-risk tasks. At the same time, design a resource scheduling optimization model to minimize task completion time and solve the resource allocation plan. Monitor indicators such as task execution progress and resource utilization rate in real time, and use the isolation forest algorithm to promptly detect task execution anomalies and resource usage anomalies. When task loss occurs, adjust the execution priority and resource allocation of subsequent tasks according to the predicted impact range, start task retry and backup tasks, and ensure the continuous execution of core tasks.

[0025] Specifically, in large-scale distributed systems, task execution and resource management are key challenges. First, the raw data is cleaned and preprocessed, such as detecting outliers on task execution time and removing obviously unreasonable data points. In the feature engineering stage, multi-dimensional feature vectors can be constructed, such as the CPU usage, memory usage, IO read and write speed of the task. The Apriori algorithm is used to discover association rules between tasks. For example, in an e-commerce order processing system, it may be found that the two tasks "payment confirmation" and "inventory deduction" often appear at the same time, and "payment confirmation" is often executed before "inventory deduction". This association information can be used to construct a task dependency graph. Similarly, the resource conflict graph reflects the competition relationship between different tasks for resources, such as multiple computationally intensive tasks may compete for CPU resources. Graph convolutional neural networks (GCNs) can effectively process graph-structured data. In the task scheduling scenario, GCNs can learn the implicit representation of task nodes and capture the complex dependencies between tasks. For example, for an e-commerce system that includes multiple modules such as order processing, inventory management, and logistics distribution, GCN can learn the interaction patterns between different modules, which helps to better understand the overall operation status of the system. The introduction of attention mechanism can highlight important tasks and key resources. For example, during the Double Eleven promotion, the importance of order processing tasks may be given a higher weight. By dynamically adjusting the connection weights, the model can more sensitively capture the state changes of these key nodes. The PageRank algorithm is often used to sort the importance of web pages and can also play a role in task scheduling. By calculating the PageRank score of the task node, the "key tasks" in the system can be identified. For example, in a data analysis pipeline, the data preprocessing task may have a high PageRank score because it affects the execution of all subsequent analysis tasks. Monte Carlo Tree Search (MCTS) can be used to simulate the impact of task loss. Assuming that in a financial trading system, if the order matching task fails, MCTS can predict through multiple simulations that this may cause delays in subsequent clearing and settlement tasks, thereby assessing potential financial risks. For the identified key tasks, a master-slave backup strategy can be adopted. For example, for the transaction confirmation task in the payment system, the same task instance can be run on multiple nodes at the same time to ensure that the continuity of the task can be guaranteed even if a node fails. Resource scheduling optimization needs to consider factors such as task priority and resource availability, such as dynamically increasing the resource quota of order processing tasks during peak hours. In real-time monitoring, the Isolation Forest algorithm can be used for anomaly detection. For example, if the execution time of a data processing task suddenly increases significantly and is far away from the historical average, the Isolation Forest algorithm can quickly detect this anomaly. Once an anomaly is detected, the system can initiate a preset emergency plan, such as switching to a backup task or reallocating resources, to ensure the continued operation of critical business.Through the comprehensive application of this series of technologies, an intelligent, highly reliable task scheduling and resource management system can be built to effectively respond to various challenges in large-scale distributed environments.

[0026] Step S103, using a long short-term memory network to perform time series modeling on the task status feature vector, capture the dynamic change rules during the task operation process, and predict the health status and completion possibility of the task in the future. If the predicted value is lower than the preset threshold, it is judged that the task is at risk of loss.

[0027] Obtain the real-time running parameters, resource occupancy, progress information and other status data of the task, and construct a multi-dimensional task status feature vector. The feature vector can include CPU occupancy, memory occupancy, I / O usage, network bandwidth, and task progress percentage indicators. Preprocess the task status feature vector, including data cleaning, missing value processing, and normalization operations. Organize the preprocessed feature vector according to the time series to form a time series data set. Use the long short-term memory (LSTM) neural network to model the time series data set and learn the dynamic change rules during the task operation process. Introduce the attention mechanism in the hidden layer of the LSTM network, and adaptively pay attention to the task status at different times through weighted summation to highlight the impact of key states. Use the Softmax function to normalize the hidden layer output to obtain the attention weight at each time, and then perform weighted summation with the hidden layer output to obtain the aggregated feature representation. Use the trained LSTM model to predict the new task status feature vector to obtain the health status and completion possibility of the task in the future. The health status is represented by normal / abnormal binary classification, and the completion possibility can be characterized by probability value. According to business needs and historical data, preset thresholds for health status and completion possibility are set. When the probability of the "abnormal" category in the health status classification exceeds a certain threshold, or the completion possibility is lower than a certain threshold, the task is judged to have a loss risk and the early warning mechanism is triggered. For tasks judged to have a high loss risk, statistical analysis and anomaly detection algorithms are used to analyze their state feature vectors and identify the key factors that cause the risk. The combination of abnormal features is found using the isolated forest anomaly detection algorithm. According to the key risk factors, combined with the expert knowledge base and the policy rule base, task retention strategies and optimization suggestions are automatically generated. If it is found that the task CPU usage is too high, it can be recommended to increase the CPU resource quota; if it is found that the task progress is stagnant for a long time, it can be recommended to adjust the task priority or increase the monitoring frequency. Apply the optimization suggestions to the task's operating parameters and resource configuration, dynamically adjust the task's execution environment, improve the task's health and completion possibility, and reduce the risk of loss. At the same time, the task status is continuously monitored, and the retention strategy and parameter configuration are further optimized based on the feedback information to form a closed-loop feedback mechanism.

[0028] Specifically, in large-scale distributed systems, real-time monitoring and prediction of the health status of tasks are crucial. First, it is necessary to construct a multi-dimensional task status feature vector. Taking the order processing system of an e-commerce platform as an example, the feature vector may include indicators such as the CPU occupancy rate, memory usage, database read and write speed, network throughput, and order completion progress of the order processing server. These raw data usually need to be preprocessed, such as correcting or removing outliers, and normalizing indicators of different dimensions to the range of 0-1 for subsequent modeling and analysis. Long short-term memory (LSTM) network is a powerful tool for processing time series data. In the order processing system, LSTM can learn the changing rules of various indicators throughout the life cycle of an order from creation to completion. For example, it may find that under normal circumstances, the CPU occupancy rate of order processing will rise rapidly at the beginning and then gradually decrease; while under abnormal circumstances, it may continue to run at a high level. After introducing the attention mechanism, the model can better capture the state changes at critical moments. For example, in the payment confirmation stage, the system may pay more attention to the two indicators of network bandwidth and database response time. Based on the trained LSTM model, the health status of new order tasks can be predicted. Suppose the model predicts that the probability of an abnormality in a certain order processing task in the next 10 minutes is 0.8, while the probability of normal completion is only 0.5. If the warning thresholds are set to 0.7 and 0.6 respectively, then this task will be judged as high risk and the warning mechanism will be triggered. Next, we need to analyze the reasons for the high risk in depth. Using the Isolation Forest algorithm, it may be found that the database read and write speed of the order task is abnormally slow, far below the normal level. Combined with the expert knowledge base, the system may automatically generate the following optimization suggestions: check the database connection pool configuration and increase the number of connections; or temporarily migrate the order task to a server with a lighter load for processing. After the optimization measures are implemented, the system will continue to monitor the task status. If it is found that the database read and write speed returns to normal and the order processing speed is accelerated, then the optimization measures can be considered effective. This dynamic adjustment and feedback mechanism can continuously improve the system's adaptive ability and reduce the risk of task loss. In this way, not only can potential problems be discovered in a timely manner, but also proactive measures can be taken to optimize them. This is of great significance for maintaining the stability and efficiency of large-scale distributed systems. For example, during a big promotion on an e-commerce platform, it is possible to predict and promptly handle possible order processing bottlenecks, avoiding order backlogs or losses due to system overload, thereby improving user experience and platform reputation.

[0029] Step S104, for tasks with a higher risk of loss, dynamically adjust their priorities and resource allocation strategies through a reinforcement learning algorithm, while ensuring the smooth completion of key tasks, minimizing the impact of task loss on overall performance, and obtaining the optimal task scheduling decision.

[0030] According to the results of the task loss risk assessment, determine whether there is a task loss risk. If there is a loss risk, the priority adjustment and resource reallocation mechanism is triggered. For tasks with higher loss risks, their priorities are increased, and computing resources are allocated preferentially from the resource pool to ensure the smooth completion of the task. In the resource allocation process, the greedy algorithm is used to comprehensively consider the task priority, deadline, and resource demand factors to obtain the optimal resource allocation plan. The greedy algorithm selects the current optimal solution each time and finally obtains the global optimal solution, which can efficiently solve the resource allocation problem. For tasks that have shown signs of loss, that is, the task execution progress is seriously behind expectations, their priority and resource allocation are dynamically adjusted. Increase their priority, give them more computing resources, and reduce the resource occupation of other tasks at the same time, so as to minimize the impact of loss on the overall performance. In the task scheduling process, continuously collect task execution data, including task execution time, resource occupation, and task loss rate. Optimize the scheduling strategy through the Q-learning algorithm to improve the accuracy and efficiency of scheduling decisions. Q-learning learns the optimal scheduling decision by constantly trying and updating strategies. Set the reward function of the Q-learning algorithm according to the task loss situation, average task completion time, and resource utilization. The lower the task loss rate and the better the overall performance index, the higher the reward value will be. The reward function guides the algorithm to optimize the scheduling strategy, minimize the impact of task loss, and achieve the optimal scheduling decision.

[0031] Specifically, in large-scale distributed systems, task churn risk assessment and resource scheduling optimization are crucial. Taking a cloud computing platform as an example, when the system detects that a data analysis task has a high churn risk, it will immediately trigger the priority adjustment and resource reallocation mechanism. Assuming that the task was originally prioritized as 3, the system will upgrade it to priority 1 and allocate additional computing nodes from the resource pool. Greedy algorithms play an important role in resource allocation. Consider that there are 10 tasks to be processed, each with different priorities, deadlines, and resource requirements. The greedy algorithm will first select the task with the highest priority and the most urgent deadline and allocate the required resources to it. Then the remaining tasks will be processed in turn until all resources are allocated or all task requirements are met. Although this method may not be able to obtain the global optimal solution, it can quickly obtain a better solution and is suitable for scenarios with high real-time requirements. For tasks that have shown signs of churn, the system will take more radical measures. For example, a big data processing task was originally planned to be completed within 2 hours, but only 20% was completed after 1 hour. The system will immediately raise its priority to the highest level and reclaim some resources from other low-priority tasks. This dynamic adjustment strategy can effectively reduce task churn and improve overall system performance. The Q-learning algorithm has unique advantages in task scheduling optimization. By continuously collecting task execution data, such as task completion time, resource utilization, and churn rate, the algorithm can continuously learn and improve scheduling strategies. For example, the system may find that increasing resource allocation for specific types of tasks in certain time periods can significantly reduce churn rate. Q-learning will adjust the strategy based on this finding and automatically increase resource allocation in the corresponding time period. The design of the reward function is the core of the Q-learning algorithm. A possible design is: reward = -(task churn rate × 10 + average task completion time / benchmark time + (1-resource utilization rate)). This function comprehensively considers multiple performance indicators and guides the algorithm to optimize task completion time and resource utilization efficiency while reducing churn rate. Through this intelligent resource scheduling and optimization mechanism, the system can effectively deal with the risk of task churn and improve overall operating efficiency. For example, in the order processing system of a large e-commerce platform, this mechanism can automatically adjust resource allocation during peak periods such as Double Eleven, give priority to orders that may be lost, and ensure user experience. At the same time, the system can also predict future resource needs based on historical data, expand resources or schedule tasks in advance, and further improve the stability and reliability of the system. This approach is not only applicable to cloud computing platforms, but can also be extended to other fields. For example, in smart factories, similar mechanisms can be used to optimize production line scheduling to reduce production delays and resource waste. In smart city traffic management systems, signal light timing and lane allocation can be dynamically adjusted to improve traffic efficiency. These applications all reflect the great potential of intelligent scheduling in improving system efficiency and reducing risks.

[0032] Step S105, real-time monitoring of task running status and resource usage. When abnormal fluctuations occur or exceed preset thresholds, the task loss warning mechanism is triggered, and corresponding intervention measures are taken according to the warning level, such as adjusting task priority, dynamically allocating resources, interrupting or restarting tasks, etc.

[0033] Obtain real-time status data and resource consumption data during task execution, including CPU usage, memory usage, and disk I / O indicators. Pass the acquired data to the monitoring and analysis module for real-time monitoring and analysis. The monitoring and analysis module determines whether each indicator exceeds the threshold of the normal range based on the preset abnormal judgment rules. If the monitoring data analysis results show that there is an abnormal situation, such as the CPU usage continues to exceed 80%, the memory usage continues to exceed 90%, etc., the task abnormality warning mechanism is triggered. The warning level is determined according to the preset warning rules. The warning rules are formulated based on historical data and expert experience, and the abnormal situation is divided into three levels: mild, moderate, and severe. According to different warning levels, corresponding task scheduling and resource allocation strategies are adopted. If the warning level is mild, the priority of the abnormal task is lowered by 10% by adjusting the task priority queue to delay its execution; if the warning level is moderate, 20% of CPU and memory resources are dynamically increased on the basis of the original resources to ensure the execution of critical tasks; if the warning level is severe, the execution of the abnormal task is suspended and an additional 50% of computing resources are allocated to other tasks. After adjusting the task priority and dynamically allocating resources, the task execution status and resource consumption are continuously monitored every 5 minutes to determine whether the abnormal indicators have returned to normal. If the abnormal indicators have returned to normal within 3 consecutive monitoring cycles, the warning is determined to be lifted, and the original scheduling and resource allocation strategy of the task is restored; if the warning persists, further measures such as task interruption or restart are taken, and the abnormal task is suspended for 10 minutes before trying to re-execute it. If the abnormal situation still exists after re-execution, a warning is sent to the system administrator via email and SMS, and the administrator remotely logs in to the server for manual processing. During the task execution process, task status data and resource consumption data are continuously collected to construct a training data set. Using the decision tree algorithm, the task execution status and resource consumption indicators are used as features, and whether an abnormality occurs is used as the target variable to train the abnormality prediction model. Using the trained abnormality prediction model, the abnormality judgment threshold is dynamically adjusted to achieve adaptive optimization of the warning threshold. At the same time, the support vector machine algorithm is used to predict the task execution time with task characteristics and execution environment as input, and the task scheduling strategy is dynamically adjusted according to the prediction results to improve the intelligence level of task scheduling.

[0034] Specifically, real-time monitoring of task execution status and resource consumption is the key to ensuring stable operation of the system. Taking a large e-commerce platform as an example, during the Double Eleven Shopping Festival, the order processing system faced tremendous pressure. By collecting indicators such as CPU usage and memory usage in real time, the system can detect potential problems in a timely manner. For example, when the CPU usage of an order processing service continues to exceed 80%, it may indicate that the service is about to face a performance bottleneck. The design of the early warning mechanism is crucial. According to historical data and expert experience, abnormal situations can be divided into three levels: mild, moderate, and severe. In an e-commerce platform, a mild abnormality may be a short-term order peak, a moderate abnormality may be a continuous high load caused by a promotional activity, and a severe abnormality may be a serious failure of the system. For different levels of abnormalities, taking corresponding scheduling strategies can effectively balance the system load. Dynamic resource allocation is an effective means to deal with system load fluctuations. In a cloud computing environment, resource configuration can be flexibly adjusted according to the warning level. For example, for a moderate warning, increasing CPU and memory resources by 20% may be sufficient to cope with a short-term load increase. For a severe warning, some non-critical tasks may need to be suspended to concentrate resources on ensuring the normal operation of the core business. Continuous monitoring and adaptive adjustment are key to maintaining long-term system stability. By monitoring the system status every 5 minutes, it is possible to detect in time whether the anomaly has been alleviated. If the indicator returns to normal within 3 consecutive cycles, it can be considered that the problem has been resolved. This mechanism can avoid over-allocation of resources and improve the overall efficiency of the system. Machine learning algorithms play an important role in optimizing system performance. Decision tree algorithms can help identify key factors that cause system anomalies. For example, by analyzing historical data, it may be found that memory usage exceeding 90% is the main cause of system instability. This insight can guide system administrators to optimize resource allocation strategies. Support vector machine algorithms perform well in predicting task execution time. By analyzing task characteristics and execution environment, the time required for task completion can be estimated more accurately. This is crucial for optimizing task scheduling. For example, in an e-commerce platform, orders that may cause users to wait too long can be prioritized based on the prediction results, thereby improving user experience. Adaptive optimization of the anomaly prediction model can improve the robustness of the system. Over time, the load characteristics of the system may change. By continuously collecting data and updating the model, it can be ensured that the early warning mechanism remains efficient at all times. For example, in an e-commerce platform, it may be found that as user behavior changes, the original CPU usage threshold is no longer applicable and needs to be dynamically adjusted to adapt to the new load pattern. In general, the comprehensive application of real-time monitoring, intelligent early warning, dynamic resource allocation, and machine learning algorithms can significantly improve the stability and efficiency of the system. This is not only applicable to e-commerce platforms, but can also be extended to other high-load scenarios, such as financial trading systems, online gaming platforms, etc. Through continuous optimization and adjustment, the system can better cope with various challenges and provide users with stable and reliable services.

[0035] Step S106, during the task scheduling process, comprehensively consider factors such as task loss prediction results, task dependencies, resource usage, etc., and solve the global optimal scheduling strategy through a multi-objective optimization algorithm to balance short-term scheduling benefits and long-term optimization goals to ensure efficient operation of the multi-task environment.

[0036] Obtain the task list and its related attribute information, including task priority, expected execution time, resource requirements, etc., and build a task dependency graph based on the dependencies between tasks. Obtain the number and performance indicators of currently available resources by monitoring and analyzing resource usage. Combined with the resource requirements of the task, determine whether the resources meet the task scheduling requirements. If the resources do not meet the requirements, the dynamic resource adjustment mechanism is triggered to meet the task scheduling requirements by increasing resources or optimizing resource allocation. Use the logistic regression algorithm to predict task loss. Based on the historical task execution data, extract task attribute features and loss labels, and train the loss prediction model. Extract the attribute features of the current task to be scheduled, input the trained model, and obtain the predicted value of the task loss probability. Use the task loss prediction results, task dependencies, and resource usage as constraints to construct a multi-objective optimization problem. Set the optimization goal to minimize the task loss rate and maximize resource utilization. Use the genetic algorithm to solve the optimization problem and obtain the optimal task scheduling strategy. According to the task scheduling strategy solved by the genetic algorithm, generate a detailed task execution plan, including the execution order of the task, resource allocation plan, and estimated completion time. During the task execution process, the task execution progress and resource usage are monitored in real time. If there is a serious task delay or serious lack of resource utilization, the scheduling strategy is dynamically adjusted. By modifying task priorities, adjusting resource allocation and other measures, the task scheduling strategy is dynamically optimized to improve the quality and efficiency of task completion while meeting task dependency constraints and resource limitations.

[0037] Specifically, the core of the task scheduling system is to efficiently manage resources and optimize task execution. First, the system needs to obtain a list of tasks and their attribute information. Taking a large data processing platform as an example, tasks may include data cleaning, feature extraction, model training, etc. Each task has its priority, expected execution time, and resource requirements. For example, a model training task may require a high-performance GPU, while a data cleaning task mainly relies on CPU and memory. By analyzing the dependencies between tasks, the system can build a task dependency graph to ensure that tasks are executed in the correct order. Resource management is the key to task scheduling. The system needs to monitor the number and performance indicators of available resources in real time. In a cloud computing environment, this may involve monitoring the CPU usage, memory usage, network bandwidth, etc. of virtual machines. When insufficient resources are found, the system triggers a dynamic adjustment mechanism. For example, during peak hours, additional virtual machines may need to be started to handle the increased workload. Task churn prediction is an important means to improve scheduling efficiency. Through the logistic regression algorithm, the system can predict the possibility of task churn based on historical data. For example, analyzing past task execution records, it is found that tasks that take 20% longer than expected have an 80% probability of failure. This insight can help the system take measures in advance, such as allocating more resources to high-risk tasks or adjusting their priorities. Multi-objective optimization is the core challenge of task scheduling. The system needs to find a balance between minimizing task churn and maximizing resource utilization. Genetic algorithms perform well in solving such complex optimization problems. By simulating the natural selection process, the algorithm can find the optimal solution among a large number of possible scheduling schemes. For example, in a scenario with 1,000 tasks, the genetic algorithm may need to evaluate tens of thousands of scheduling schemes to finally find a scheme that can both control the task churn rate below 5% and increase resource utilization to 85%. Generating a detailed task execution plan is a key step in implementing a scheduling strategy. This includes determining the specific execution time, allocated resources, and expected completion time of each task. For example, the system may decide to start a large-scale data processing task at 2 a.m., allocate 50% of the available computing resources, and expect it to be completed within 4 hours. This precise planning helps improve overall system efficiency. Real-time monitoring and dynamic adjustment are key to ensuring the effectiveness of scheduling strategies. During task execution, if it is found that the actual execution time of a task is far longer than expected, the system may need to re-evaluate its priority or allocate more resources. For example, if a data analysis task that is expected to be completed in 2 hours is only 30% complete after 1.5 hours, the system may increase its priority and allocate additional computing resources to ensure that the task can be completed on time. Through this comprehensive task scheduling strategy, the system can effectively manage complex workloads, improve resource utilization, and reduce task loss, thereby significantly improving overall performance and reliability. This is crucial for modern computing environments that need to handle a large number of concurrent tasks, such as cloud computing platforms and big data processing systems.

[0038] Step S107, continuously tracking the performance of the task loss prediction model, dynamically updating the model parameters through the incremental learning mechanism, adapting to the changes in the multi-task environment, improving the prediction accuracy and real-time performance, and providing a reliable decision-making basis for task scheduling and resource optimization.

[0039] Real-time data during task execution, including task attributes and resource usage, is obtained. Data is cleaned, normalized, and other preprocessing is performed, and key features are extracted as model input. The gradient boosting decision tree (GBDT) algorithm is used to incrementally train the prediction model based on new task data and update model parameters in real time. Hyperparameters such as the learning rate and regularization parameter of the model are optimized through grid search and other methods to improve the generalization ability of the model. According to the key attributes of the task, such as task type, priority, resource requirements, etc., the K-means clustering algorithm is used to divide the tasks into multiple categories. A special GBDT sub-model is built for each category to capture the personalized characteristics of the task. When predicting the model, the static attribute characteristics and dynamic execution characteristics of the task, such as the historical execution time of the task, the peak resource usage, etc., are comprehensively considered. These characteristics are input into the corresponding GBDT sub-model to obtain the predicted probability of task loss. According to the predicted probability of task loss, the heuristic algorithm is used to dynamically adjust the priority of the task, and higher execution priority is given to tasks with high loss risks. At the same time, resource scheduling algorithms, such as the minimum heap algorithm, are used to dynamically allocate resources to give priority to meeting the resource requirements of high-priority tasks. The prediction performance of the model is continuously monitored, and the model's precision, recall and other evaluation indicators are calculated using a sliding window method. When the model performance drops to a preset threshold (such as precision below 80%), the model retraining and optimization process is triggered, the GBDT model is retrained using the latest task data, and the model parameters are updated.

[0040] Specifically, in the task scheduling system, the acquisition and processing of real-time data is the key to optimizing decision-making. The system needs to continuously monitor the execution status of the task, including resource indicators such as CPU usage, memory usage, network bandwidth, and attribute information such as task completion progress and execution time. These raw data usually need to be cleaned and normalized to eliminate outliers and unify the data scale. For example, convert the CPU usage from a percentage to a floating point number of 0-1, convert the task execution time to a ratio relative to the expected completion time, etc. The processed data is more suitable as the input of the machine learning model. The gradient boosting decision tree (GBDT) algorithm is a powerful ensemble learning method that is particularly suitable for processing complex nonlinear relationships. In task churn prediction, GBDT can effectively capture the impact of various factors on the success rate of the task. For example, by analyzing historical data, the GBDT model may find that when the CPU usage exceeds 90% and the duration exceeds 30 minutes, the risk of task churn increases significantly. The system can continuously update the model with newly executed task data to improve the prediction accuracy. In order to improve the performance of the model, methods such as grid search can be used to optimize hyperparameters. For example, try different learning rates (such as 0.01, 0.1, 0.5) and tree depths (such as 3, 5, 7) to find the parameter combination that can achieve the best performance on the validation set. This process may find that for the current task churn prediction problem, the model performs best when the learning rate is 0.1 and the tree depth is 5. Considering that different types of tasks may have different characteristics and churn patterns, using the K-means clustering algorithm to classify tasks is an effective strategy. For example, the system may classify tasks into three categories: compute-intensive, memory-intensive, and I / O-intensive. Training a dedicated GBDT model for each type of task can more accurately capture the churn risk of a specific type of task. For example, for compute-intensive tasks, CPU usage may be the most critical indicator; while for I / O-intensive tasks, disk read and write speeds may be more important. When predicting the risk of task churn, the model not only considers static attributes (such as task type, priority), but also dynamic execution characteristics (such as resource usage peak, current progress). For example, if the execution time of a data processing task that was originally of low priority has exceeded 150% of the expected level and the memory usage continues to be high, the system may assess its churn risk as 80%. Based on the prediction results, the system can dynamically adjust task priorities and resource allocation. For tasks predicted to have a high risk of churn, their priority can be increased and more resources can be allocated. For example, a minimum heap algorithm is used to maintain a task queue sorted based on churn risk, giving priority to tasks with the highest risk. This method can effectively reduce the overall task churn rate and improve the reliability of the system. To ensure that the model continues to be effective, the system needs to evaluate model performance regularly. A sliding window method can be used, such as calculating the precision and recall of the model every 1,000 tasks.If the model performance is found to be degraded, for example, the accuracy drops below 75%, the model retraining process is triggered. This mechanism ensures that the prediction model can adapt to the changing task characteristics and system environment and maintain high accuracy. Through this comprehensive task loss prediction and dynamic scheduling strategy, the system can allocate resources more intelligently and improve the task completion rate, thereby significantly improving the overall performance and reliability. This is of great significance for modern computing environments that need to handle a large number of complex tasks, such as cloud computing platforms or large-scale data processing systems.

[0041] The present invention provides a multi-task data analysis system, comprising:

[0042] The task status collection module is used to collect task execution progress, resource usage, and abnormal log indicators during the task running process to build a multi-dimensional task status feature vector as the input of the task loss prediction model;

[0043] The task dependency analysis module is used to build a task dependency graph and a resource conflict graph based on the dependencies between tasks and resource usage. It uses the graph neural network algorithm to learn the correlation between tasks and the resource competition pattern, and obtain the influencing factors and propagation paths of task loss.

[0044] The time series modeling prediction module is used to perform time series modeling on the task status feature vector using a long short-term memory network, capture the dynamic change rules during the task operation process, and predict the health status and completion possibility of the task in the future. If the predicted value is lower than the preset threshold, it is judged that the task has a loss risk;

[0045] The risk task adjustment module is used to dynamically adjust the priority and resource allocation strategy of tasks with high loss risk through reinforcement learning algorithms, while ensuring the smooth completion of key tasks, minimizing the impact of task loss on overall performance and obtaining the optimal task scheduling decision;

[0046] The abnormal monitoring and early warning module is used to monitor the task running status and resource usage in real time. When abnormal fluctuations occur or exceed the preset threshold, the task loss early warning mechanism is triggered, and corresponding intervention measures are taken according to the early warning level, such as adjusting task priority, dynamically allocating resources, interrupting or restarting tasks, etc.;

[0047] The global optimization scheduling module is used to comprehensively consider the task loss prediction results, task dependencies, resource usage and other factors in the task scheduling process, and solve the global optimal scheduling strategy through a multi-objective optimization algorithm to balance short-term scheduling benefits and long-term optimization goals to ensure efficient operation of the multi-task environment;

[0048] The model update and maintenance module is used to continuously track the performance of the task loss prediction model, dynamically update the model parameters through the incremental learning mechanism, adapt to the changes in the multi-task environment, improve the prediction accuracy and real-time performance, and provide a reliable decision-making basis for task scheduling and resource optimization.

[0049] The present invention also relates to a multi-task data analysis device, which includes a memory, a processor, and a multi-task data analysis program stored in the memory and executable on the processor, wherein the program implements the steps of the multi-task data analysis method when executed by the processor.

[0050] The present invention also relates to a computer-readable storage medium, on which a multi-task data analysis program is stored, and when the program is executed by a processor, the steps of the multi-task data analysis method are implemented.

[0051] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-task data analysis method, characterized in that: The method comprises: By collecting the task execution progress, resource usage, and abnormal log indicators during the task running process, a multi-dimensional task status feature vector is constructed as the input of the task loss prediction model; According to the dependencies between tasks and resource usage, we build a task dependency graph and a resource conflict graph. We use the graph neural network algorithm to learn the correlation between tasks and the resource competition pattern, and obtain the influencing factors and propagation paths of task loss. Use long short-term memory network to perform time series modeling on task status feature vectors, capture the dynamic change rules during task operation, and predict the health status and completion possibility of tasks in the future. If the predicted value is lower than the preset threshold, it is judged that the task has the risk of loss. For tasks with a high risk of loss, the priority and resource allocation strategy are dynamically adjusted through reinforcement learning algorithms to ensure the smooth completion of key tasks while minimizing the impact of task loss on overall performance and obtaining the optimal task scheduling decision. Monitor the task running status and resource usage in real time. When abnormal fluctuations occur or exceed the preset threshold, the task loss warning mechanism is triggered and corresponding intervention measures are taken according to the warning level. In the task scheduling process, the task loss prediction results, task dependencies, and resource usage factors are comprehensively considered, and the global optimal scheduling strategy is solved through a multi-objective optimization algorithm to balance short-term scheduling benefits and long-term optimization goals; Continuously track the performance of the task churn prediction model and dynamically update the model parameters through the incremental learning mechanism.

2. The method according to claim 1, characterized in that By collecting the task execution progress, resource usage, and abnormal log indicators during the task running process, a multi-dimensional task status feature vector is constructed as the input of the task loss prediction model, including: According to the various indicator data during the task execution process, a multi-dimensional task state feature vector is constructed as the input of the machine learning model; Use the sliding time window method to dynamically obtain the latest task execution progress, resource usage, and abnormal log indicator data; By setting an abnormality threshold, it is determined whether an abnormality occurs during task execution. If an abnormality occurs, it is marked in the task status feature vector; Preprocess the collected indicator data of each dimension, including missing value processing, outlier processing and data normalization; For missing values, choose to delete or fill them according to the characteristics of the data; Outliers are identified and processed using the box plot method; For numerical indicators, the maximum and minimum value normalization or Z-score normalization method is used for normalization; The indicators that have a significant impact on task loss prediction are selected as features through filtering method; Combine the preprocessed multi-dimensional indicator data into a task status feature vector, with each time window corresponding to a feature vector sample; Using the logistic regression algorithm, we take the task status feature vector as input and train a binary classification model to predict whether a task will be lost. During the model training process, the model is evaluated and hyperparameters are selected through cross-validation method; At the same time, in order to solve the problem of class imbalance, oversampling and undersampling or adjusting class weights are used; Deploy the trained task churn prediction model to the production environment and monitor the task execution status in real time; When the probability of task loss exceeds the preset threshold, a task loss warning is sent to relevant personnel via email, SMS or instant messaging tools, prompting them to restore or reschedule tasks in a timely manner; For abnormal situations during task execution, set anomaly detection rules and thresholds to monitor task execution indicators in real time; When the indicator data exceeds the normal range, an abnormal alarm is triggered, the abnormal log is recorded, and the task is retried, restarted, or terminated according to the abnormal type and severity; Continuously collect task execution data and regularly retrain and optimize the task churn prediction model; At the same time, the anomaly detection rules and thresholds are dynamically adjusted to adapt to changes in the task execution environment.

3. The method according to claim 1, characterized in that According to the dependency relationship and resource occupancy between tasks, a task dependency graph and a resource conflict graph are constructed, and the correlation between tasks and resource competition patterns are learned through a graph neural network algorithm to obtain the influencing factors and propagation paths of task loss, including: After obtaining the task execution records and resource usage logs, first clean and preprocess the data to remove noise data and outliers and extract key field information; Then, feature engineering is performed to construct feature vectors of tasks and resources, including task execution time, resource usage, and task priority features; Based on the constructed feature vector, the Apriori algorithm is used to discover the dependency relationship between tasks and the association pattern of resources, and to construct the task dependency graph and resource conflict graph; For the constructed task dependency graph and resource conflict graph, a graph convolutional neural network (GCN) is used to learn the representation of graph structure data, and the correlation between tasks and resource competition patterns are captured through node embedding vectors; During the GCN model training process, the attention mechanism is introduced to dynamically adjust the connection weights between nodes by designing the task importance scoring function and the resource scarcity scoring function to highlight the importance of key task nodes and key resource nodes. According to the task relevance and resource competition pattern learned by the GCN model, the PageRank algorithm is used to calculate the importance scores of task nodes and resource nodes, identify key tasks and key resources that have a significant impact on task execution, and set importance score thresholds for screening; Apply the Monte Carlo tree search algorithm on the task dependency graph, simulate the propagation process of task loss multiple times, count the influence of task nodes, and predict the impact of task loss on downstream tasks and the degree of delay in task completion time; For the identified key tasks and key resources, assess their loss risk, design task backup strategies based on task importance and resource scarcity, and implement master-slave backup and checkpoint mechanisms for high-risk tasks; At the same time, a resource scheduling optimization model is designed to minimize the task completion time and solve the resource allocation plan; Monitor task execution progress and resource utilization indicators in real time, and use the isolation forest algorithm to promptly detect abnormal task execution and resource utilization; When task loss occurs, the execution priority and resource allocation of subsequent tasks are adjusted according to the predicted impact range, and task retries and backup tasks are initiated to ensure the continuous execution of core tasks.

4. The method according to claim 1, characterized in that: The long short-term memory network is used to perform time series modeling on the task state feature vector, capture the dynamic change rules during the task operation process, and predict the health status and completion possibility of the task in the future. If the predicted value is lower than the preset threshold, it is judged that the task has a loss risk, including: Obtain the real-time running parameters, resource usage and progress information status data of the task, and construct a multi-dimensional task status feature vector; The feature vectors include CPU usage, memory usage, I / O usage, network bandwidth, and task progress percentage indicators; Preprocess the task status feature vector, including data cleaning, missing value processing and normalization operations; Organize the preprocessed feature vectors according to time series to form a time series data set; Use LSTM network to model time series data sets and learn the dynamic changes in the task running process; The attention mechanism is introduced into the hidden layer of the LSTM network. Through weighted summation, it adaptively pays attention to the task status at different times and highlights the impact of key status. Use the Softmax function to normalize the hidden layer output to obtain the attention weight at each moment, and then perform weighted summation with the hidden layer output to obtain the aggregated feature representation; Use the trained LSTM model to predict the new task status feature vector to obtain the health status and completion possibility of the task in the future. The health status is represented by the binary classification of normal / abnormal, and the completion possibility is characterized by the probability value; Set preset thresholds for health status and likelihood of completion based on business needs and historical data; When the probability of the "abnormal" category in the health status classification exceeds a certain threshold, or the completion probability is lower than a certain threshold, the task is judged to be at risk of loss, triggering the early warning mechanism; For tasks that are judged to have a high risk of churn, statistical analysis and anomaly detection algorithms are used to analyze their state feature vectors and identify the key factors that cause the risk; Use the Isolation Forest Anomaly Detection algorithm to find combinations of abnormal features; Automatically generate task retention strategies and optimization suggestions based on key risk factors, combined with expert knowledge base and policy rule base; If the task CPU usage is found to be too high, increase the CPU resource quota; If you find that the task progress has stagnated for a long time, adjust the task priority or increase the monitoring frequency; Apply optimization suggestions to the task's operating parameters and resource configuration, dynamically adjust the task's execution environment, improve the task's health and completion possibility, and reduce the risk of loss; At the same time, the task status is continuously monitored, and the retention strategy and parameter configuration are further optimized based on the feedback information to form a closed-loop feedback mechanism.

5. The method according to claim 1, characterized in that For tasks with a high risk of loss, the priority and resource allocation strategy are dynamically adjusted through a reinforcement learning algorithm to minimize the impact of task loss on overall performance while ensuring the smooth completion of key tasks, and to obtain the optimal task scheduling decision, including: According to the task loss risk assessment results, determine whether there is a task loss risk; If there is a risk of loss, the priority adjustment and resource reallocation mechanism will be triggered; For tasks with a higher risk of loss, their priority is increased and computing resources are allocated preferentially from the resource pool; In the resource allocation process, a greedy algorithm is used to comprehensively consider task priority, deadline and resource demand factors to obtain the optimal resource allocation plan; The greedy algorithm selects the current optimal solution each time and eventually obtains the global optimal solution; Dynamically adjust the priority and resource allocation of tasks that have shown signs of loss, i.e., the progress of task execution is seriously lagging behind expectations; Increase its priority, give it more computing resources, and reduce the resource usage of other tasks; During the task scheduling process, task execution data is continuously collected, including task execution time, resource usage, and task loss rate; Optimize scheduling strategies through Q-learning algorithm; Set the reward function of the Q-learning algorithm based on task loss, average task completion time, and resource utilization; The lower the task loss rate and the better the overall performance index, the higher the reward value will be; The reward function is used to guide the algorithm to optimize the scheduling strategy, minimize the impact of task loss, and achieve the optimal scheduling decision.

6. The method according to claim 1, characterized in that Monitor the task running status and resource usage in real time. When abnormal fluctuations occur or exceed the preset threshold, the task loss warning mechanism is triggered and corresponding intervention measures are taken according to the warning level, including: Obtain real-time status data and resource consumption data during task execution, including CPU usage, memory usage, and disk I / O metrics; The acquired data is passed to the monitoring and analysis module to perform real-time monitoring and analysis on the data; The monitoring and analysis module determines whether each indicator exceeds the threshold of the normal range according to the preset abnormal judgment rules; If the monitoring data analysis results show that there is an abnormal situation, such as the CPU usage rate continues to exceed the preset threshold, the task abnormality warning mechanism will be triggered; Determine the warning level according to the preset warning rules. The warning rules are formulated based on historical data and expert experience, and divide abnormal situations into three levels: mild, moderate and severe; Adopt corresponding task scheduling and resource allocation strategies for different warning levels; If the warning level is mild, the priority of the abnormal task will be lowered by 10% by adjusting the task priority queue to delay its execution; If the warning level is moderate, 20% of CPU and memory resources will be dynamically increased based on the original resources to ensure the execution of key tasks; If the warning level is severe, the execution of the abnormal task is suspended and an additional 50% of computing resources are allocated to other tasks; After adjusting task priorities and dynamically allocating resources, continuously monitor task execution status and resource consumption every 5 minutes to determine whether abnormal indicators have returned to normal; If the abnormal indicators return to normal within three consecutive monitoring cycles, the warning is lifted and the original scheduling and resource allocation strategy of the task is restored; If the warning persists, further measures will be taken to interrupt or restart the task, and the abnormal task will be suspended for 10 minutes before trying to re-execute it; If the abnormal situation still exists after re-execution, a warning will be sent to the system administrator via email and SMS, and the administrator will remotely log in to the server to handle it manually; During the task execution process, task status data and resource consumption data are continuously collected to build a training data set; Using the decision tree algorithm, the anomaly prediction model is trained with task execution status and resource consumption indicators as features and whether an anomaly occurs as the target variable; Using the trained anomaly prediction model, dynamically adjust the anomaly judgment threshold to achieve adaptive optimization of the warning threshold; At the same time, the support vector machine algorithm is used to predict the task execution time with task characteristics and execution environment as input, and the task scheduling strategy is dynamically adjusted according to the prediction results.

7. The method according to claim 1, characterized in that In the task scheduling process, the task loss prediction results, task dependencies, and resource usage factors are comprehensively considered, and the global optimal scheduling strategy is solved through a multi-objective optimization algorithm to balance short-term scheduling benefits and long-term optimization goals, including: Obtain a list of tasks and their related attribute information, including task priority, expected execution time, and resource requirements, and construct a task dependency graph based on the dependencies between tasks; Obtain the quantity and performance indicators of currently available resources by monitoring and analyzing resource usage; Combined with the resource requirements of the task, determine whether the resources meet the task scheduling requirements; If resources do not meet the requirements, the dynamic resource adjustment mechanism is triggered to meet the task scheduling requirements by adding resources or optimizing resource allocation; Use logistic regression algorithm to predict task loss; Based on historical task execution data, extract task attribute features and churn labels, and train a churn prediction model; Extract the attribute features of the current task to be scheduled, input them into the trained model, and obtain the predicted value of the task loss probability; Taking the task loss prediction results, task dependencies, and resource usage as constraints, a multi-objective optimization problem is constructed. The optimization goal is to minimize the task loss rate and maximize the resource utilization rate; Genetic algorithm is used to solve the optimization problem and obtain the optimal task scheduling strategy; Generate a detailed task execution plan based on the task scheduling strategy solved by the genetic algorithm, including the execution order of tasks, resource allocation plan and estimated completion time; During task execution, real-time monitoring of task execution progress and resource usage; If there is a serious task delay or serious underutilization of resources, the scheduling strategy will be dynamically adjusted; By modifying task priorities and adjusting resource allocation measures, the task scheduling strategy is dynamically optimized while satisfying task dependency constraints and resource limitations.

8. The method according to claim 1, characterized in that The performance of the task churn prediction model is continuously tracked, and model parameters are dynamically updated through an incremental learning mechanism, including: Obtain real-time data during task execution, including task attributes and resource usage, clean and normalize the data, and extract key features as model input; Adopting the gradient boosting decision tree algorithm, the prediction model is incrementally trained according to the new task data, and the model parameters are updated in real time; Through the grid search method, the model's learning rate and regularization parameter hyperparameters are optimized to improve the model's generalization ability; Based on the key attributes of the tasks, such as task type, priority, and resource requirements, the K-means clustering algorithm is used to divide the tasks into multiple categories, and a dedicated GBDT sub-model is built for each category to capture the personalized characteristics of the tasks; When predicting the model, the static attribute characteristics and dynamic execution characteristics of the task are comprehensively considered, including the historical execution time and resource usage peak of the task. These characteristics are input into the corresponding GBDT sub-model to obtain the predicted probability of task loss. Based on the predicted probability of task loss, a heuristic algorithm is used to dynamically adjust the priority of tasks, giving higher execution priority to tasks with high risk of loss; At the same time, resource scheduling algorithms, such as the minimum heap algorithm, are used to dynamically allocate resources and give priority to meeting the resource requirements of high-priority tasks; Continuously monitor the prediction performance of the model and use a sliding window approach to calculate the model's precision and recall evaluation indicators; When the model performance drops to a preset threshold (such as an accuracy rate below 80%), the model retraining and optimization process is triggered, the GBDT model is retrained using the latest task data, and the model parameters are updated.

9. A multi-task data analysis device, characterized in that: The device includes a memory, a processor, and a multi-task data analysis program stored in the memory and executable on the processor. When the program is executed by the processor, the steps of the multi-task data analysis method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a multi-task data analysis program, which, when executed by a processor, implements the steps of the multi-task data analysis method according to any one of claims 1 to 8.

Images