Task scheduling and early warning methods and devices

Abstract

This disclosure proposes a task scheduling early warning method and apparatus, relating to the field of computer technology, comprising: using a Monte Carlo simulation method, extracting a portion of samples from the historical execution data of the target task, preprocessing them, and inputting them into a pre-trained LSTM model to obtain the predicted execution time of each sample, and generating an initial confidence interval based on the predicted execution time; collecting real-time execution data generated during the execution of the target task, and dynamically updating the initial confidence interval to obtain a target confidence interval using a Bayesian update mechanism; determining whether to trigger a task early warning signal based on the initial confidence interval and the target confidence interval; using the real-time execution data of the target task as the state space of a reinforcement learning algorithm, and combining it with a preset reward function to output a priority adjustment strategy for the target task; adjusting the priority of the target task based on the priority adjustment strategy, and performing task resource preemption based on the adjusted priority to execute the target task.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a task scheduling early warning method and apparatus. Background Technology

[0002] With the deep integration of 5G communication services and big data technology, network operation and maintenance management systems need to undertake the orderly scheduling of massive data processing tasks. Existing technologies have significant limitations: insufficient real-time performance and early warning accuracy. Current monitoring relies mainly on alarms after task execution failures or periodic inspections, lacking the ability to predict task execution time in advance. Furthermore, early warning thresholds are mostly based on historical averages or static parameters, without considering the dynamic impact of task load fluctuations and resource contention, resulting in a high rate of false alarms and missed alarms due to delay risks. Summary of the Invention

[0003] This application aims to at least partially address one of the technical problems in the related art.

[0004] Therefore, one objective of this application is to propose a task scheduling early warning method, comprising: using Monte Carlo simulation, extracting a portion of samples from the historical execution data of the target task, preprocessing them, and inputting them into a pre-trained Long Short-Term Memory (LSTM) network model to obtain the predicted execution time of each sample, and generating an initial confidence interval based on the predicted execution time, wherein the historical execution data at least covers time features, resource features, and dependency features; collecting real-time execution data generated during the execution of the target task, and dynamically updating the initial confidence interval using a Bayesian update mechanism to obtain the target confidence interval; determining whether to trigger a task early warning signal based on the initial confidence interval and the target confidence interval; during the execution of the target task, using the real-time execution data of the target task as the state space of a reinforcement learning algorithm, and combining it with a preset reward function to output a priority adjustment strategy for the target task; adjusting the priority of the target task based on the priority adjustment strategy, and performing task resource preemption based on the adjusted priority to execute the target task.

[0005] The second objective of this application is to propose a task scheduling early warning device.

[0006] The third objective of this application is to propose an electronic device.

[0007] The fourth objective of this application is to provide a non-transitory computer-readable storage medium.

[0008] The fifth objective of this application is to provide a computer program product.

[0009] To achieve the above objectives, the first aspect of this application proposes a task scheduling early warning method, comprising: using a Monte Carlo simulation method, extracting a portion of samples from the historical execution data of the target task, preprocessing them, and inputting them into a pre-trained Long Short-Term Memory (LSTM) network model to obtain the predicted execution time of each sample, and generating an initial confidence interval based on the predicted execution time, wherein the historical execution data at least covers time features, resource features, and dependency features; collecting real-time execution data generated during the execution of the target task, and dynamically updating the initial confidence interval using a Bayesian update mechanism to obtain a target confidence interval; determining whether a task early warning signal is triggered based on the initial confidence interval and the target confidence interval; during the execution of the target task, using the real-time execution data of the target task as the state space of a reinforcement learning algorithm, and combining it with a preset reward function to output a priority adjustment strategy for the target task; adjusting the priority of the target task based on the priority adjustment strategy, and performing task resource preemption based on the adjusted priority to execute the target task.

[0010] This application's embodiments employ a "Monte Carlo simulation + LSTM model" to generate initial confidence intervals. This utilizes LSTM to capture long-term correlations of time, resource, and dependency features in historical task data, while Monte Carlo simulation covers the uncertainty of data fluctuations, resulting in greater accuracy compared to traditional "historical mean prediction." Combined with a Bayesian update mechanism, the confidence intervals are dynamically calibrated, incorporating real-time data from task execution into the prediction. This allows the confidence intervals to dynamically adjust with the task status, solving the problem of static threshold warnings "failing to adapt to execution fluctuations" and effectively reducing the probability of false alarms and missed warnings. Using real-time task execution data as the state space for reinforcement learning, and combining it with a reward function to output a priority adjustment strategy, this approach, compared to traditional "static priority configuration," can adjust priorities in real-time based on task execution status (such as progress delays or resource shortages). Based on the adjusted priorities, resource preemption is executed, ensuring resource supply for high-risk tasks while avoiding the waste of low-priority tasks occupying resources for extended periods, thus improving the overall utilization efficiency of the resource pool.

[0011] According to one embodiment of this application, the training method of the LSTM model includes: acquiring sample execution data generated during the historical execution of sample tasks, wherein the sample execution data at least covers sample time features, sample resource features, and sample dependency features; preprocessing the sample execution data to obtain standardized data, wherein the preprocessing includes cleaning, denoising, and normalization; inputting the standardized data into an initial LSTM model for training to obtain the predicted execution time of the sample task output by the initial LSTM model, and combining the actual execution time of the sample task to perform iterative training of the model until the loss function converges, thereby obtaining a pre-trained LSTM model.

[0012] According to one embodiment of this application, determining whether to trigger a task warning signal based on an initial confidence interval and a target confidence interval includes: obtaining the task commitment deadline of the target task; calculating the latest start time of the target task based on the task commitment deadline and the upper confidence limit of the initial confidence interval; if it is detected that the current time has reached the latest start time and the target task has not started, then triggering a non-start warning signal for the target task.

[0013] According to one embodiment of this application, determining whether to trigger a task warning signal based on an initial confidence interval and a target confidence interval includes: obtaining the warning buffer time of the target task, and determining the corresponding delayed warning buffer period based on the task commitment deadline and the warning buffer time; within the delayed warning buffer period, collecting the execution time of the target task in real time, and calculating the expected completion time of the target task in combination with the upper confidence limit of the target confidence interval; obtaining the dynamic warning threshold of the target task, and determining the latest completion time of the target task based on the dynamic warning threshold and the task commitment deadline; if the expected completion time is later than the latest completion time, triggering a delayed end warning signal for the target task.

[0014] According to one embodiment of this application, determining whether to trigger a task warning signal based on an initial confidence interval and a target confidence interval includes: if it is detected that the current time exceeds the task's promised deadline and the target task has not been completed, then triggering a target task failure warning signal.

[0015] According to one embodiment of this application, the task scheduling early warning method further includes: obtaining the latest guaranteed completion time of the lower-level task of the target task on the task chain and the predicted execution time of the lower-level task; determining the latest guaranteed completion time of the target task based on the latest guaranteed completion time of the lower-level task and the predicted execution time of the lower-level task, wherein the latest guaranteed completion time refers to the latest completion time point that ensures the upper-level task does not affect the execution of the lower-level task; if it is detected that the current time exceeds the latest guaranteed completion time of the target task and the target task has not been completed, triggering a link-related delay early warning signal, wherein the link-related delay early warning signal contains association information that the delay of the target task will cause the start-up delay of the lower-level task.

[0016] According to one embodiment of this application, task resource preemption based on adjusted priority to execute a target task includes: executing task resource preemption based on adjusted task priority using a modified resource scheduler; the resource scheduler is a modified YARN scheduler, the modification including introducing priority parameters into the YARN task description structure, and realizing priority-based resource preemption adaptation through the pass-through of priority parameters.

[0017] According to one embodiment of this application, the task scheduling early warning method further includes: after the execution time of the target task exceeds the upper confidence limit of the target confidence interval, attempting to reallocate resources to the target task and then restarting the target task; or adjusting the dependencies of the target task and then restarting the target task.

[0018] To achieve the above objectives, a second aspect of this application proposes a task scheduling early warning device, comprising: a generation module, used to extract a portion of samples from the historical execution data of the target task using a Monte Carlo simulation method, preprocess them, and input them into a pre-trained Long Short-Term Memory (LSTM) network model to obtain the predicted execution time of each sample, and generate an initial confidence interval based on the predicted execution time, wherein the historical execution data at least covers time features, resource features, and dependency features; an update module, used to collect real-time execution data generated during the execution of the target task, and dynamically update the initial confidence interval using a Bayesian update mechanism to obtain a target confidence interval; an early warning module, used to determine whether to trigger a task early warning signal based on the initial confidence interval and the target confidence interval; an output module, used to use the real-time execution data of the target task as the state space of a reinforcement learning algorithm during the execution of the target task, and output a priority adjustment strategy for the target task using a preset reward function; and an adjustment module, used to adjust the priority of the target task based on the priority adjustment strategy, and perform task resource preemption based on the adjusted priority to execute the target task.

[0019] According to one embodiment of this application, the task scheduling early warning device further includes a training module, configured to: acquire sample execution data generated during the historical execution of sample tasks, wherein the sample execution data at least covers sample time features, sample resource features, and sample dependency features; preprocess the sample execution data to obtain standardized data, wherein the preprocessing includes cleaning, denoising, and normalization; input the standardized data into an initial LSTM model for training to obtain the predicted execution time of the sample tasks output by the initial LSTM model, and perform iterative training of the model in combination with the actual execution time of the sample tasks until the loss function converges to obtain a pre-trained LSTM model.

[0020] According to one embodiment of this application, the early warning module is further configured to: obtain the task commitment deadline of the target task; calculate the latest start time of the target task based on the task commitment deadline and the upper confidence limit of the initial confidence interval; if it is detected that the current time has reached the latest start time and the target task has not started, then trigger an early warning signal for the target task not started.

[0021] According to one embodiment of this application, the early warning module is further configured to: obtain the early warning buffer time of the target task, and determine the delayed early warning buffer period corresponding to the target task based on the task commitment deadline and the early warning buffer time; within the delayed early warning buffer period, collect the execution time of the target task in real time, and calculate the expected completion time of the target task in combination with the upper confidence limit of the target confidence interval; obtain the dynamic early warning threshold of the target task, and determine the latest completion time of the target task based on the dynamic early warning threshold and the task commitment deadline; if the expected completion time is later than the latest completion time, trigger the delayed end early warning signal of the target task.

[0022] According to one embodiment of this application, the early warning module is further configured to: if it is detected that the current time exceeds the task commitment deadline and the target task has not been completed, trigger a target task failure early warning signal.

[0023] According to one embodiment of this application, the early warning module is further configured to: obtain the latest completion time guaranteed by the link for the lower-level tasks of the target task in the task chain and the predicted execution time of the lower-level tasks; determine the latest completion time guaranteed by the link for the target task based on the latest completion time guaranteed by the link for the lower-level tasks and the predicted execution time of the lower-level tasks, wherein the latest completion time guaranteed by the link refers to the latest completion time point that ensures that the upper-level tasks do not affect the execution of the lower-level tasks; if it is detected that the current time exceeds the latest completion time guaranteed by the link for the target task and the target task has not been completed, trigger a link-related delay early warning signal, wherein the link-related delay early warning signal contains related information that the delay of the target task will cause the start-up delay of the lower-level tasks.

[0024] According to one embodiment of this application, the adjustment module is further configured to: perform task resource preemption to execute the target task based on the adjusted task priority through the modified resource scheduler; the resource scheduler is a modified YARN scheduler, the modification including the introduction of priority parameters into the task description structure of YARN, and the implementation of priority-based resource preemption adaptation through the pass-through of priority parameters.

[0025] According to one embodiment of this application, the adjustment module is further configured to: after the execution time of the target task exceeds the upper confidence limit of the target confidence interval, attempt to reallocate resources to the target task and then restart the target task; or, adjust the dependencies of the target task and then restart the target task.

[0026] To achieve the above objectives, a third aspect of this application provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to implement the task scheduling and early warning method as described in the first aspect of this application.

[0027] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to implement the task scheduling and early warning method as described in the first aspect of this application.

[0028] To achieve the above objectives, a fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the task scheduling and early warning method as described in the first aspect of this application. Attached Figure Description

[0029] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a schematic diagram illustrating an exemplary implementation of a task scheduling early warning method according to one embodiment of this application.

[0030] Figure 2 This is a schematic diagram illustrating an exemplary implementation of a task scheduling early warning method according to one embodiment of this application.

[0031] Figure 3 This is a schematic diagram illustrating an embodiment of the present application for indicating the triggering timing of a warning signal.

[0032] Figure 4 This is a schematic diagram illustrating a link relationship in one embodiment of this application.

[0033] Figure 5 This is a schematic diagram of a task scheduling and early warning device according to one embodiment of this application.

[0034] Figure 6 This is a schematic diagram of an electronic device according to one embodiment of this application. Detailed Implementation

[0035] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0036] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0037] In this application, the core role of the Long Short-Term Memory Network (LSTM) model is to learn the temporal patterns of task execution time and output accurate predictions of task execution duration.

[0038] Figure 1 This is a schematic diagram illustrating an exemplary implementation of a task scheduling early warning method shown in this application, such as... Figure 1 As shown, the task scheduling and early warning method includes the following steps: S101 uses the Monte Carlo simulation method to extract some samples from the historical execution data of the target task, preprocess them, and input them into the pre-trained LSTM model to obtain the task prediction execution time corresponding to each sample, and generate an initial confidence interval based on the task prediction execution time.

[0039] Historical execution data should at least cover time characteristics, resource characteristics, and dependency characteristics.

[0040] The time characteristics include the historical start time and historical end time of the target task.

[0041] Among them, resource characteristics refer to the resource usage of the target task, such as CPU utilization and memory usage.

[0042] Among them, dependency features refer to the relationship between the target task and its upstream and downstream tasks.

[0043] In some feasible ways, historical execution data may also include load characteristics to identify the load characteristics of execution periods such as weekdays, holidays, daytime, and evening.

[0044] In this application, the Monte Carlo simulation method can be used to extract M samples from the historical execution data of the target task, and after preprocessing, input them into the pre-trained LSTM model to obtain the predicted execution time of M tasks. Based on the predicted execution time of M tasks, the initial mean and initial standard deviation are calculated, and the initial confidence interval of the target task's execution time is constructed based on the initial mean and initial standard deviation.

[0045] The formula for calculating the initial mean is as follows:

[0046] The formula for calculating the initial standard deviation is:

[0047] In the above formula, Represents the initial mean. This represents the initial standard deviation, and M represents the number of samples. This represents the actual execution time of the task for the i-th sample out of M samples. This represents the task prediction execution time for the i-th sample out of M samples.

[0048] The initial confidence interval can be represented as:

[0049] S102: Collect real-time execution data generated during the execution of the target task, and dynamically update the initial confidence interval using the Bayesian update mechanism to obtain the target confidence interval.

[0050] In this application, real-time execution data of the target task is continuously collected during the execution of the target task. The real-time execution data includes one or more of the following: the execution duration of the target task, real-time resource usage, real-time progress, dependencies, and the current initial confidence interval (which will be updated to the latest target confidence interval during dynamic updates).

[0051] In this application, based on real-time execution data generated during the execution of the target task, the initial confidence interval is dynamically updated using a Bayesian update mechanism to obtain new target mean and target standard deviation (mean and standard deviation of the posterior probability distribution), and a target confidence interval is constructed based on the target mean and target standard deviation. The formula for the Bayesian update can be expressed as:

[0052] In the above formula, This represents the prior probability distribution corresponding to the initial confidence interval; This represents real-time execution data generated during the execution of the target task; This represents the posterior probability distribution corresponding to the target confidence interval. Represents the likelihood function; This represents the marginal probability of observing data D.

[0053] The new target mean, obtained by dynamically updating the initial confidence interval using the Bayesian update mechanism, can be denoted as: The target standard deviation obtained can be denoted as: The target confidence interval can be represented as:

[0054] S103, determine whether to trigger a task warning signal based on the initial confidence interval and the target confidence interval.

[0055] The initial confidence interval is a priori prediction interval generated based on historical data of the target task, mainly used for risk assessment before task initiation. In this application, the latest start time of the task can be calculated (latest start time = promised deadline of the task - upper confidence limit of the initial confidence interval); if it is detected that the current time has reached the latest start time, but the target task has not yet started, a non-start warning signal is triggered, prompting the user that "if the task is not started immediately, it will not be able to be completed before the promised deadline."

[0056] The target confidence interval is a dynamically predicted interval updated and calibrated using Bayesian methods, primarily used for real-time risk monitoring during task execution. The estimated completion time of the task is calculated as follows: (Estimated completion time = Current time + (Confidence limit of the target confidence interval - Task execution time)). If the estimated completion time is later than the promised deadline, a delayed termination warning signal is triggered, alerting the user that "the current execution progress of the task is lagging behind, and there is a risk of timeout."

[0057] S104: During the execution of the target task, the real-time execution data of the target task is used as the state space of the reinforcement learning algorithm. Combined with the preset reward function, the priority adjustment strategy for the target task is output.

[0058] As described above, real-time execution data includes one or more of the following: the execution duration of the target task, real-time resource usage, real-time progress, dependencies, and the current initial confidence interval (which will be updated to the latest target confidence interval during dynamic updates). In this application, the real-time execution data of the target task is used as the state space S of the reinforcement learning algorithm, and combined with a preset reward function, a priority adjustment strategy for the target task is output.

[0059] That is, the action space A of the reinforcement learning algorithm is the priority adjustment strategy for the target task.

[0060] The preset reward function R is used to evaluate the effect of the adjustment. It can be set to R=1 if the task is completed on time or if the task is not executed due to timeout, otherwise R=0.

[0061] In this application, a Deep Q-Network (DQN) can be used to implement the decision-making process of the reinforcement learning algorithm to dynamically adjust task priorities. DQN is a classic value function-based algorithm in reinforcement learning. Its core is to fit the Q-value (state-action value) through a deep neural network, thereby outputting the optimal action.

[0062] DQN uses the Q-learning algorithm to update the Q-value to optimize the decision-making process. The specific update formula is as follows: ← +

[0063] in, It is the learning rate; It is a discount factor; It represents the current state of the target task; It is the current action of the target task; It is to perform an action The new state following environmental feedback; It's a new state All priority adjustment actions are available below; It is the state-action value; It is a reward function.

[0064] S105, adjust the priority of the target task based on the priority adjustment strategy, and perform task resource preemption to execute the target task based on the adjusted priority.

[0065] In this application, high-priority tasks are allowed to preempt resources from low-priority tasks.

[0066] When resources are scarce, resources are allocated to high-priority tasks first, and resource allocation is dynamically adjusted based on task performance to ensure the smooth execution of critical tasks.

[0067] This application's embodiments employ a "Monte Carlo simulation + LSTM model" to generate initial confidence intervals. This utilizes LSTM to capture long-term correlations of time, resource, and dependency features in historical task data, while Monte Carlo simulation covers the uncertainty of data fluctuations, resulting in greater accuracy compared to traditional "historical mean prediction." Combined with a Bayesian update mechanism, the confidence intervals are dynamically calibrated, incorporating real-time data from task execution into the prediction. This allows the confidence intervals to dynamically adjust with the task status, solving the problem of static threshold warnings "failing to adapt to execution fluctuations" and effectively reducing the probability of false alarms and missed warnings. Using real-time task execution data as the state space for reinforcement learning, and combining it with a reward function to output a priority adjustment strategy, this approach, compared to traditional "static priority configuration," can adjust priorities in real-time based on task execution status (such as progress delays or resource shortages). Based on the adjusted priorities, resource preemption is executed, ensuring resource supply for high-risk tasks while avoiding the waste of low-priority tasks occupying resources for extended periods, thus improving the overall utilization efficiency of the resource pool.

[0068] The training method of the LSTM model used in this application is described below.

[0069] First, obtain the sample execution data generated during the historical execution of the sample task. The sample execution data should at least cover the sample time characteristics, sample resource characteristics, and sample dependency characteristics.

[0070] In some implementations, the sample task can be the target task itself or a variety of similar tasks.

[0071] Among them, the sample time features include the historical start time and historical end time of the sample task.

[0072] Among them, sample resource characteristics refer to the resource usage of sample tasks, such as CPU utilization and memory usage.

[0073] Among them, sample dependency features refer to the relationship between sample tasks and upstream and downstream tasks.

[0074] In some feasible ways, the sample execution data may also include sample load characteristics to identify the load characteristics of execution periods such as weekdays, holidays, daytime, and evening.

[0075] After determining the sample execution data as described above, the sample execution data is preprocessed to obtain standardized data. The preprocessing includes cleaning (removing abnormal failure data and duplicate data), denoising (filtering noise data such as instantaneous fluctuations in resources), and normalization (converting time features into cumulative minutes since the base time and standardizing resource features to the [0,1] interval).

[0076] Finally, standardized data is input into the initial LSTM model for training, obtaining the predicted execution time of the sample task output by the initial LSTM model. The model is then iteratively trained using the actual execution time of the sample task until the loss function converges, resulting in a pre-trained LSTM model. In this application, mean squared error (MSE) is selected as the loss function.

[0077] Figure 2 This is a schematic diagram illustrating an exemplary implementation of a task scheduling early warning method shown in this application, such as... Figure 2 As shown, the task scheduling and early warning method includes the following steps: S201 uses the Monte Carlo simulation method to extract some samples from the historical execution data of the target task, preprocess them, and input them into the pre-trained LSTM model to obtain the task prediction execution time corresponding to each sample, and generate an initial confidence interval based on the task prediction execution time.

[0078] Historical execution data should at least cover time characteristics, resource characteristics, and dependency characteristics.

[0079] For details on the specific implementation of step S201, please refer to the relevant parts of the above embodiments, which will not be repeated here.

[0080] S202, Obtain the task commitment deadline for the target task.

[0081] The Task Commitment Deadline (TCD) refers to the time point that the user promises to complete the target task, which should be completed before this time.

[0082] S203, based on the task commitment deadline and the upper confidence limit of the initial confidence interval, calculate the latest start time of the target task.

[0083] The formula for calculating the latest start time of the target task is as follows:

[0084] In the above formula, This represents the latest possible start time for the target task; that is, the latest possible start time estimated based on the upper confidence limit of the initial confidence interval to ensure the target task is completed on schedule. The deadline for the promised task representing the objective. The upper confidence limit represents the initial confidence interval of the target task.

[0085] S204 If it is detected that the current time has reached the latest start time and the target task has not started, then trigger the target task not started warning signal.

[0086] Figure 3 This application illustrates a schematic diagram for indicating the triggering timing of a warning signal, such as... Figure 3 As shown, assuming the target task is It is 03:30. For 30 minutes, The time limit is 4 minutes, meaning the latest start time for the target task is 02:52. If the target task has not started by 02:52, a warning signal indicating that the target task has not started will be triggered.

[0087] The "Not Started" warning signal for the target task is mainly used to issue a warning to remind the user that the target task is now in a "high risk that it may not be completed by the promised time of 03:30".

[0088] S205: Collect real-time execution data generated during the execution of the target task, and dynamically update the initial confidence interval using a Bayesian update mechanism to obtain the target confidence interval.

[0089] The specific implementation of step S205 can be found in the detailed description of the relevant parts in the above embodiments, and will not be repeated here. The final result is: by dynamically updating the initial confidence interval using the Bayesian update mechanism, the new target mean can be denoted as... The target standard deviation obtained can be denoted as: The target confidence interval can be represented as:

[0090] S206, obtain the warning buffer time of the target task, and determine the corresponding delay warning buffer period based on the task commitment deadline and the warning buffer time.

[0091] Among them, the early warning buffer time for the target task This is set by the user. For example, the warning buffer time can be set to 10 minutes. The delay warning buffer period for the target task is determined to be within the range [TCD-σ, TCD) based on the task commitment deadline and the warning buffer time.

[0092] The delayed warning buffer period can be understood as the period during which the delayed warning signal will be triggered.

[0093] like Figure 3 As shown, continue with the target task. Taking 03:30 as an example, assuming the warning buffer time is set to 10 minutes, the corresponding delay warning buffer time period [03:20, 03:30] can be determined.

[0094] S207: During the delay warning buffer period, the execution time of the target task is collected in real time, and the expected completion time of the target task is calculated by combining the upper confidence limit of the target confidence interval.

[0095] It is not difficult to understand that, as the method for obtaining the target confidence interval has been introduced above, in this application, the execution time of the target task is collected in real time during the delay warning buffer period, and the expected completion time of the target task is dynamically calculated in combination with the upper confidence limit of the target confidence interval.

[0096] like Figure 3 As shown, assuming the delay warning buffer period for the target task is [03:20, 03:30), within the time interval [03:20, 03:30), for example, if it is found at 03:20 that the execution time of the target task is 20 minutes, but the upper limit of the confidence interval of the target task is 36 minutes, that is, the target task still needs to be executed for another 16 minutes, but 16 minutes after 03:20 is 03:36, that is, the expected completion time of the target task is 03:36.

[0097] S208, obtain the dynamic warning threshold of the target task, and determine the latest completion time of the target task based on the dynamic warning threshold and the task commitment deadline.

[0098] Among them, the dynamic early warning threshold of the target task It can be set according to the actual situation.

[0099] In some feasible ways, dynamic early warning thresholds It can be set to 1.

[0100] In some feasible ways, dynamic early warning thresholds It can be set to 1.05.

[0101] In this application, the task commitment deadline can be converted into a duration relative to 00:00 and then compared with the dynamic early warning threshold. Multiply the results and then convert the resulting durations to hours to obtain the latest completion time for the target task.

[0102] S209. If the expected completion time is later than the latest completion time, a delayed completion warning signal for the target task will be triggered.

[0103] Continue with Figure 3 For example, assuming the latest completion time is 03:30 (that is, the dynamic warning threshold is set to 1), if the execution time of the target task is collected in real time and the expected completion time of the target task is dynamically calculated to be 03:36 in combination with the upper confidence limit of the target confidence interval, then the delayed completion warning signal of the target task will be triggered.

[0104] The delayed completion warning signal for the target task is used to inform the user that the target task is highly unlikely to be completed before the latest completion time.

[0105] It is not difficult to understand that during the delay warning buffer period, after determining the latest completion time of the target task based on the dynamic warning threshold and the task commitment deadline, the monitoring process of "real-time collection of the execution time of the target task, and calculation of the expected completion time of the target task in combination with the upper confidence limit of the target confidence interval, and triggering the delay end warning signal of the target task if the expected completion time is later than the latest completion time" will be continuously executed.

[0106] S210 If it is detected that the current time exceeds the task's promised deadline and the target task has not been completed, a warning signal for the target task's failure to meet the deadline will be triggered.

[0107] like Figure 3 As shown, continue with the target task. Taking 03:30 as an example, if the current time is detected to be past 03:30 and the target task has not been completed, a warning signal for the target task being breached will be triggered.

[0108] Among them, the target task failure warning signal is used to inform the user that the target task has not been completed by the promised deadline.

[0109] S211: During the execution of the target task, the real-time execution data of the target task is used as the state space of the reinforcement learning algorithm. Combined with the preset reward function, the priority adjustment strategy for the target task is output.

[0110] For details on the specific implementation of step S211, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0111] S212, adjust the priority of the target task based on the priority adjustment strategy, and perform task resource preemption to execute the target task based on the adjusted priority.

[0112] In this application, a modified resource scheduler is used to perform task resource preemption based on the adjusted task priority in order to execute the target task. The resource scheduler is a modified YARN scheduler. The modification includes introducing priority parameters into the task description structure of YARN. Priority-based resource preemption adaptation is achieved through the pass-through of priority parameters to ensure that resource allocation prioritizes high-priority tasks.

[0113] The resource allocation formula can be defined as:

[0114] In the above formula, This represents the amount of resources allocated to the current task; This represents the total available resources in the resource pool. Represents the priority of the target task; Represents the normalization / resource ratio coefficient; The sum of the priorities of all scheduled tasks.

[0115] This application's embodiments employ a "Monte Carlo simulation + LSTM model" to generate initial confidence intervals. This utilizes LSTM to capture long-term correlations of time, resource, and dependency features in historical task data, while Monte Carlo simulation covers the uncertainty of data fluctuations, resulting in greater accuracy compared to traditional "historical mean prediction." Combined with a Bayesian update mechanism, the confidence intervals are dynamically calibrated, incorporating real-time data from task execution into the prediction. This allows the confidence intervals to dynamically adjust with the task status, solving the problem of static threshold warnings "failing to adapt to execution fluctuations" and effectively reducing the probability of false alarms and missed warnings. Using real-time task execution data as the state space for reinforcement learning, and combining it with a reward function to output a priority adjustment strategy, this approach, compared to traditional "static priority configuration," can adjust priorities in real-time based on task execution status (such as progress delays or resource shortages). Based on the adjusted priorities, resource preemption is executed, ensuring resource supply for high-risk tasks while avoiding the waste of low-priority tasks occupying resources for extended periods, thus improving the overall utilization efficiency of the resource pool.

[0116] Furthermore, in this application, in addition to considering early warning for a single task, it is also necessary to obtain the latest guaranteed completion time of the lower-level tasks and the predicted execution time of the lower-level tasks of the target task in the task chain, considering that if the execution of a certain task is delayed, it may affect the execution of the lower-level tasks. Based on the latest guaranteed completion time of the lower-level tasks and the predicted execution time of the lower-level tasks, the latest guaranteed completion time of the target task is determined. The latest guaranteed completion time refers to the latest completion time that ensures the upper-level tasks will not affect the execution of the lower-level tasks. If it is detected that the current time exceeds the latest guaranteed completion time of the target task and the target task has not been completed, a link-related delay early warning signal is triggered. The link-related delay early warning signal contains the correlation information that the delay of the target task will cause the start delay of the lower-level tasks.

[0117] In this step, the predicted execution time of the lower-level task can also be obtained by combining the Monte Carlo simulation method with the LSTM model. Here, the predicted execution time of the lower-level task can be the average of the predicted execution times of multiple tasks of the lower-level task output by the LSTM model.

[0118] Figure 4 This is a schematic diagram illustrating a link relationship, as shown in this application. Figure 4 As shown, not every task will be monitored and alerted. For example... Figure 4 In this diagram, tasks A, B, C, D, E, and F represent independent jobs with explicit dependencies: for example, tasks D and E depend on task B, and task B in turn depends on task A. This structure is called a "task chain dependency." Figure 4 In this process, monitoring can be performed on tasks as needed. For example, monitoring and early warning can be performed only on tasks A, B, D, and E, while tasks C and F are not subject to this plan.

[0119] In the link relationship, if a parent node corresponds to one child node, then the latest completion time of the link guarantee of the parent node is equal to the latest start time of the link guarantee of its child node. The latest start time of the link guarantee of the child node is equal to the difference between the latest completion time of the link guarantee of the child node and the execution time of the predicted task of the child node.

[0120] In the link relationship, if a parent node corresponds to multiple child nodes, the earliest link guarantee start time among all child nodes is taken as the latest link guarantee completion time of the parent node.

[0121] like Figure 4As shown, if the latest completion time for the link guarantee of lower-level tasks D and E is 5:30 (meaning D and E must complete before 5:30, otherwise it will affect their own lower-level tasks), and if the predicted execution time of lower-level task D is 3 hours, then the latest start time for the link guarantee of task D = 5:30 - 3 hours = 2:30. If the predicted execution time of lower-level task E is 2 hours, then the latest start time for the link guarantee of task E = 5:30 - 2 hours = 3:30. Since task B is the parent node of tasks D and E, the latest completion time for the link guarantee of task B must be the earliest latest start time for the link guarantee of tasks D and E (i.e., 2:30). This means that task B must complete before 2:30 to ensure that task D starts on time at 2:30 and does not affect subsequent execution.

[0122] If the current time is detected to be beyond the latest completion time (2:30) of the link guarantee for task B and task B has not been completed, a link-related delay warning signal will be triggered. This signal contains the related information that "the delay of task B will cause the start-up of the lower-level task D to be delayed".

[0123] For minute-level scheduling tasks (such as high-frequency tasks that are executed every 5 or 10 minutes), the task scheduling early warning method described in this application may not be used. These tasks have short execution cycles, and if an execution failure occurs, the user can directly discover the problem and intervene in a short time (such as within 1-2 scheduling cycles). If the task scheduling early warning method described in this application is used, once the task fails continuously, it will trigger high-frequency alarms. A large number of redundant early warning messages will interfere with the user's monitoring of core tasks and reduce operation and maintenance efficiency.

[0124] Specifically, for hourly-level scheduling tasks (such as tasks executed every 1 hour / 2 hours), the warning time must not be earlier than 0:00 of the current hour. Reason: Hourly-level tasks can only be scheduled at 0:00 at the earliest; issuing an alarm at this time or earlier is meaningless and causes interference. The same applies to daily-level tasks.

[0125] For missions of the highest level, the warning time must not be earlier than 00:00:00 on the same day.

[0126] For monthly tasks, the warning time must not be earlier than 00:00:00 on the 1st of the month.

[0127] For grade-level tasks, the warning time must not be earlier than 00:00:00 on January 1st of that year.

[0128] Reason: These types of tasks are mostly periodic batch processing tasks (such as T+1 data aggregation for daily tasks). The scheduling cycle is based on "natural days / months / years". If the warning time is earlier than the start time of the cycle, it will cause the warning time to be out of sync with the actual execution window of the task (such as setting the warning start time of daily tasks to 23:30 the previous day, at which time the task has not yet entered the scheduling cycle of the day). The warning will lose its practical meaning and is prone to interference.

[0129] When a "dependency offset" scenario is encountered in the task chain (i.e., the actual dependent task of the task is inconsistent with the preset dependent task, or the execution cycle of the dependent task exceeds the scheduling cycle of the current task), the user needs to manually check the dependency configuration problem.

[0130] Furthermore, as mentioned above, during the execution of the target task, resource preemption is considered to ensure its smooth execution. However, if the execution time of the target task exceeds the upper confidence limit of the target confidence interval, the target task can be considered to have failed. At this point, some self-healing strategies can be attempted. For example, try to reallocate resources to the target task and then restart it; or adjust the dependencies of the target task and then restart it (such as temporarily switching to a backup dependency link).

[0131] After the aforementioned target task is completed, or after the self-healing strategy is implemented for the target task, the relevant execution parameters of the target task can be collected to retrain the LSTM model.

[0132] Figure 5 This is a schematic diagram of a task scheduling and early warning device shown in this application, such as... Figure 5 As shown, the task scheduling and early warning device 500 includes a generation module 501, an update module 502, an early warning module 503, an output module 504, and an adjustment module 505, wherein: The generation module 501 is used to extract some samples from the historical execution data of the target task using the Monte Carlo simulation method, input them into the pre-trained Long Short-Term Memory (LSTM) network model after preprocessing, obtain the task prediction execution time corresponding to each sample, and generate an initial confidence interval based on the task prediction execution time. The historical execution data at least covers time features, resource features, and dependency features. The update module 502 is used to collect real-time execution data generated during the execution of the target task, and dynamically update the initial confidence interval in combination with the Bayesian update mechanism to obtain the target confidence interval; The early warning module 503 is used to determine whether to trigger a task early warning signal based on the initial confidence interval and the target confidence interval; The output module 504 is used to take the real-time execution data of the target task as the state space of the reinforcement learning algorithm during the execution of the target task, and output the priority adjustment strategy for the target task in combination with the preset reward function. The adjustment module 505 is used to adjust the priority of the target task based on the priority adjustment strategy, and to perform task resource preemption to execute the target task based on the adjusted priority.

[0133] This device uses a combination of Monte Carlo simulation and an LSTM model to generate initial confidence intervals. It leverages LSTM to capture long-term correlations of time, resource, and dependency features in historical task data, while Monte Carlo simulation covers the uncertainty of data fluctuations, resulting in greater accuracy compared to traditional historical mean prediction. Combined with a Bayesian update mechanism, the confidence interval is dynamically calibrated, incorporating real-time data from task execution into the prediction. This allows the confidence interval to dynamically adjust with the task status, solving the problem of static threshold warnings being unable to adapt to execution fluctuations and effectively reducing the probability of false alarms and missed alarms. Using real-time task execution data as the state space for reinforcement learning, and combining it with a reward function to output a priority adjustment strategy, it can adjust priorities in real-time based on task execution status (such as progress delays or resource shortages), compared to traditional static priority configuration. Based on the adjusted priority, resource preemption is performed, ensuring resource supply for high-risk tasks while avoiding the waste of resources due to low-priority tasks occupying resources for extended periods, thus improving the overall utilization efficiency of the resource pool.

[0134] Furthermore, the task scheduling and early warning device 500 also includes a training module, used to: acquire sample execution data generated during the historical execution of sample tasks, the sample execution data covering at least sample time features, sample resource features, and sample dependency features; preprocess the sample execution data to obtain standardized data, the preprocessing including cleaning, denoising, and normalization; input the standardized data into the initial LSTM model for training, to obtain the predicted execution time of the sample task output by the initial LSTM model, and combine it with the actual execution time of the sample task to perform iterative training of the model until the loss function converges, thus obtaining the pre-trained LSTM model.

[0135] Furthermore, the early warning module 503 is also used to: obtain the task commitment deadline of the target task; calculate the latest start time of the target task based on the task commitment deadline and the upper confidence limit of the initial confidence interval; if it is detected that the current time has reached the latest start time and the target task has not started, then trigger the target task's non-start early warning signal.

[0136] Furthermore, the early warning module 503 is also used to: obtain the early warning buffer time of the target task, and determine the corresponding delayed early warning buffer period based on the task commitment deadline and the early warning buffer time; within the delayed early warning buffer period, collect the execution time of the target task in real time, and calculate the expected completion time of the target task in combination with the upper confidence limit of the target confidence interval; obtain the dynamic early warning threshold of the target task, and determine the latest completion time of the target task based on the dynamic early warning threshold and the task commitment deadline; if the expected completion time is later than the latest completion time, trigger the delayed end early warning signal of the target task.

[0137] Furthermore, the early warning module 503 is also used to: trigger a target task failure early warning signal if it is detected that the current time exceeds the task commitment deadline and the target task has not been completed.

[0138] Furthermore, the early warning module 503 is also used to: obtain the latest completion time of the lower-level tasks of the target task in the task chain and the predicted execution time of the lower-level tasks; determine the latest completion time of the target task based on the latest completion time of the lower-level tasks and the predicted execution time of the lower-level tasks, where the latest completion time of the link guarantee refers to the latest completion time that ensures the upper-level tasks will not affect the execution of the lower-level tasks; if it is detected that the current time exceeds the latest completion time of the target task and the target task has not been completed, trigger a link-related delay early warning signal, which contains the association information that the delay of the target task will cause the start-up delay of the lower-level tasks.

[0139] Furthermore, the adjustment module 505 is also used to: execute the target task by preempting task resources based on the adjusted task priority through the modified resource scheduler; the resource scheduler is the modified YARN scheduler, the modification includes introducing priority parameters into the YARN task description structure, and realizing priority-based resource preemption adaptation through the pass-through of priority parameters.

[0140] Furthermore, the adjustment module 505 is also used to: attempt to reallocate resources to the target task and restart the target task after the execution time of the target task exceeds the upper confidence limit of the target confidence interval; or, adjust the dependencies of the target task and restart the target task.

[0141] To implement the above embodiments, this application also proposes an electronic device 600, such as... Figure 6 As shown, the electronic device 600 includes a processor 601 and a memory 602 communicatively connected to the processor. The memory 602 stores instructions that can be executed by at least one processor. The instructions are executed by at least one processor 601 to implement the task scheduling and early warning method as shown in the above embodiment.

[0142] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to implement the task scheduling and early warning method shown in the above embodiments.

[0143] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the task scheduling and early warning method shown in the above embodiments.

[0144] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0145] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0146] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0147] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A task scheduling early warning method, characterized in that, include: The Monte Carlo simulation method is used to extract some samples from the historical execution data of the target task, preprocess them and input them into the pre-trained Long Short-Term Memory (LSTM) network model to obtain the predicted execution time of the task corresponding to each sample. An initial confidence interval is generated based on the predicted execution time of the task. The historical execution data at least covers time features, resource features and dependency features. Collect real-time execution data generated during the execution of the target task, and dynamically update the initial confidence interval using a Bayesian update mechanism to obtain the target confidence interval; Determine whether to trigger a task warning signal based on the initial confidence interval and the target confidence interval; During the execution of the target task, the real-time execution data of the target task is used as the state space of the reinforcement learning algorithm, and combined with a preset reward function, a priority adjustment strategy for the target task is output. The priority of the target task is adjusted based on the priority adjustment strategy, and the task resource preemption is performed based on the adjusted priority to execute the target task.

2. The method according to claim 1, wherein, The training method for the LSTM model includes: Obtain sample execution data generated during the historical execution of the sample task, wherein the sample execution data at least covers sample time characteristics, sample resource characteristics, and sample dependency characteristics; The sample execution data is preprocessed to obtain standardized data. The preprocessing includes cleaning, denoising, and normalization. The standardized data is input into the initial LSTM model for training, and the predicted execution time of the sample task output by the initial LSTM model is obtained. The model is then iteratively trained in combination with the actual execution time of the sample task until the loss function converges, thus obtaining the pre-trained LSTM model.

3. The method according to claim 2, wherein, The step of determining whether to trigger a task warning signal based on the initial confidence interval and the target confidence interval includes: Obtain the task commitment deadline for the target task; Based on the task commitment deadline and the upper confidence limit of the initial confidence interval, the latest start time of the target task is calculated. If the current time is detected to have reached the latest start time and the target task has not started, then a warning signal for the target task not starting will be triggered.

4. The method according to claim 3, wherein, The step of determining whether to trigger a task warning signal based on the initial confidence interval and the target confidence interval includes: Obtain the warning buffer time of the target task, and determine the delay warning buffer period corresponding to the target task based on the task commitment deadline and the warning buffer time; During the delay warning buffer period, the execution time of the target task is collected in real time, and the expected completion time of the target task is calculated by combining the upper confidence limit of the target confidence interval. Obtain the dynamic warning threshold of the target task, and determine the latest completion time of the target task based on the dynamic warning threshold and the task commitment deadline; If the estimated completion time is later than the latest completion time, a delayed completion warning signal for the target task will be triggered.

5. The method according to claim 4, wherein, The step of determining whether to trigger a task warning signal based on the initial confidence interval and the target confidence interval includes: If the current time exceeds the promised deadline of the task and the target task is not completed, a warning signal for the target task being breached will be triggered.

6. The method according to claim 5, wherein, The method further includes: Obtain the latest guaranteed completion time of the lower-level tasks of the target task in the task chain and the predicted execution time of the lower-level tasks; Based on the latest completion time of the link guarantee for the lower-level task and the predicted execution time of the lower-level task, the latest completion time of the link guarantee for the target task is determined. The latest completion time of the link guarantee refers to the latest completion time point that ensures that the upper-level task will not affect the execution of the lower-level task. If it is detected that the current time exceeds the latest completion time guaranteed by the link for the target task and the target task has not been completed, a link-related delay warning signal is triggered. The link-related delay warning signal contains association information that the delay of the target task will cause the start-up delay of the lower-level task.

7. The method according to claim 1, wherein, The process of preempting task resources based on adjusted priority to execute the target task includes: The modified resource scheduler performs task resource preemption based on the adjusted task priority in order to execute the target task. The resource scheduler is a modified YARN scheduler. The modification includes introducing a priority parameter into the YARN task description structure and realizing priority-based resource preemption adaptation through the pass-through of the priority parameter.

8. The method according to any one of claims 4-7, wherein, The method further includes: After the execution time of the target task exceeds the upper confidence limit of the target confidence interval, attempt to reallocate resources to the target task and restart the target task; or, adjust the dependencies of the target task and restart the target task.

9. A task scheduling and early warning device, characterized in that, include: The generation module is used to extract a portion of samples from the historical execution data of the target task using the Monte Carlo simulation method, preprocess them, and input them into a pre-trained Long Short-Term Memory (LSTM) network model to obtain the predicted execution time of the task corresponding to each sample, and generate an initial confidence interval based on the predicted execution time of the task. The historical execution data at least covers time features, resource features, and dependency features. The update module is used to collect real-time execution data generated during the execution of the target task, and dynamically update the initial confidence interval in combination with the Bayesian update mechanism to obtain the target confidence interval; The early warning module is used to determine whether to trigger a task early warning signal based on the initial confidence interval and the target confidence interval; The output module is used to take the real-time execution data of the target task as the state space of the reinforcement learning algorithm during the execution of the target task, and output the priority adjustment strategy for the target task in combination with the preset reward function. The adjustment module is used to adjust the priority of the target task based on the priority adjustment strategy, and to perform task resource preemption to execute the target task based on the adjusted priority.

10. An electronic device, comprising: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

Images