Task scheduling anomaly monitoring method, device, medium and program product

Abstract

The present application provides a task scheduling anomaly monitoring method, device, medium and program product, which obtains historical task data of one or more historical task cycles according to the work plan of the current task cycle; determines multiple task time-consuming types and scheduling relationship maps based on the historical task data, and the ratio of the amount of data contained in each task time-consuming type to the total amount of time-consuming data meets the preset proportion requirement; determines at least one timed detection task based on multiple task time-consuming types and scheduling relationship maps; and predicts whether the probability of anomalies in the task scheduling of the target system meets the preset warning requirements based on the detection results of each timed detection task at each detection time point; if so, determines and outputs one or more warning information. This solves the technical problems of existing anomaly monitoring, such as poor response time, inflexible configuration, and only logical-level monitoring, which is not highly coupled with actual business.

Description

Technical Field

[0001] The present application relates to the field of financial technology (Fintech), and in particular to a method, device, medium and program product for monitoring task scheduling anomalies. Background Art

[0002] With the development of computer technology, more and more technologies are being applied in the financial sector, and the traditional financial industry is gradually shifting towards Fintech. Currently, the offline data processed daily by the financial and internet industries is characterized by large volumes of data and high timeliness. Financial enterprises, in particular, are involved in processing large amounts of data for regulatory reporting. If data processing tasks are not completed and relevant data is not provided on time, regulatory accountability may be imposed, affecting the company's rating and reputation. Therefore, monitoring and exception response in task scheduling systems are particularly important.

[0003] Currently, open source task scheduling frameworks or tools, such as Azkaban, Airflow, and Oozie, are relatively mature. However, the monitoring functions configured or developed using these frameworks or tools are mostly based on task abnormalities or fixed parameters configured based on experience. When such monitoring alarms are issued, the task is often in an abnormal state or has already caused actual impact.

[0004] That is, the existing exception monitoring has technical problems such as poor response timeliness, inflexible configuration, and only logical-level monitoring, which is not highly coupled with actual business, increasing the difficulty and workload of operation and maintenance work. Summary of the Invention

[0005] The present application provides a task scheduling exception monitoring method, device, medium and program product to solve the technical problems of existing exception monitoring, such as poor response timeliness, inflexible configuration, and only logical level monitoring, which is not highly coupled with actual business.

[0006] In a first aspect, the present application provides a method for monitoring task scheduling anomalies, comprising:

[0007] According to the work plan of the current task cycle, historical task data of one or more historical task cycles are obtained, where the similarity between the historical work plans of the historical task cycles and the work plan of the current task cycle meets the preset requirements, and the historical task data includes: configuration data of each historical task and time consumption data of executing each historical task;

[0008] Using a preset clustering model, cluster each historical task cycle according to the time consumption data until multiple task time consumption types are determined, and the ratio of the amount of data contained in each task time consumption type to the total amount of time consumption data meets the preset ratio requirement;

[0009] Determine a scheduling relationship map based on the configuration data, and determine at least one scheduled detection task based on the time-consuming types of multiple tasks and the scheduling relationship map. The scheduling relationship map is used to represent the dependency relationship between the processing results of each historical task.

[0010] Based on the detection results of each scheduled detection task at each detection time point, it is predicted whether the probability of abnormality in the task scheduling of the target system meets the preset warning requirements; if so, one or more warning information is determined and output.

[0011] In a possible design, the one or more historical task cycles include: the last task cycle closest to the current cycle, or a plurality of consecutive task cycles closest to the current cycle.

[0012] In one possible design, a preset clustering model is used to cluster each historical task cycle based on the time consumption data until multiple task time consumption types are determined, including:

[0013] Randomly extract the time consumption data of multiple historical tasks from all historical tasks as cluster centers;

[0014] Using a preset clustering model, perform a first clustering process on each historical task according to the cluster center to determine one or more first time-consuming types;

[0015] Determine whether the proportion of the data volume in each first time-consuming type meets the preset proportion requirement;

[0016] If so, determining that the first time-consuming type is a task time-consuming type;

[0017] If not, the cluster center is re-determined, and the clustering process is re-performed to re-determine the first time-consuming type, until the data volume ratio corresponding to each first time-consuming type meets the preset ratio requirement;

[0018] The data volume ratio is used to represent the ratio of the data volume of the first time-consuming type to the total data volume of the time-consuming data.

[0019] In one possible design, the preset proportion requirement includes: the data volume proportion is greater than or equal to a first proportion threshold, and less than or equal to a second proportion threshold.

[0020] Optionally, a first value range of the first proportion threshold includes: 1% to 10%, and a second value range of the second proportion threshold includes: 40% to 60%.

[0021] In one possible design, cluster centers are re-determined and clustering is performed again, including:

[0022] Deleting the first time-consuming type whose data volume accounts for less than a first proportion threshold; and / or,

[0023] Randomly select at least two historical tasks from each first time-consuming type whose data volume is greater than a second proportion threshold as new cluster centers;

[0024] For the first time-consuming type that meets the preset proportion requirement, a new cluster center is re-determined according to the preset method;

[0025] Using the preset clustering model, clustering is re-performed according to each new cluster center to determine a new first time-consuming type.

[0026] In one possible design, for the first time-consuming type that meets the preset proportion requirement, a new cluster center is re-determined according to a preset method, including:

[0027] When the first time-consuming type meets the preset proportion requirement, the average time-consuming of the first time-consuming type is used as a new cluster center.

[0028] In one possible design, at least one scheduled detection task is determined based on multiple task time-consuming types and a scheduling relationship graph, including:

[0029] According to the preset screening requirements, determine the first target type and the second target type from the time-consuming types of each task;

[0030] Determine a first fluctuation range and a second fluctuation range based on the respective time-consuming data of the first target type and the second target type using a preset fluctuation algorithm;

[0031] The detection object and detection time of each scheduled detection task are determined according to the scheduling relationship map, the first fluctuation range, the second fluctuation range, and the start time of the historical task execution in the time-consuming data.

[0032] In one possible design, a preset fluctuation algorithm is used to determine the first fluctuation range and the second fluctuation range based on the respective time consumption data of the first target type and the second target type, including:

[0033] Determine a first fluctuation range according to a first average duration and a first standard deviation of all duration data in the first target type;

[0034] A second fluctuation range is determined according to a second average duration and a second standard deviation of all duration data in the two target types.

[0035] In one possible design, determining the first fluctuation range according to a first average duration and a first standard deviation of all duration data in the first target type includes:

[0036] The first fluctuation range is equal to the sum of the first average time and N times the first standard deviation;

[0037] The second fluctuation range is determined according to the second average time consumption and the second standard deviation of all time consumption data in the second target type, including:

[0038] The second fluctuation range is equal to the difference between the second average time consumption and M times the second standard deviation.

[0039] In one possible design, the detection time includes: a first detection time and a second detection time. The first detection time includes: a first fluctuation range superimposed on the starting time, and the second detection time includes: a second fluctuation range superimposed on the starting time.

[0040] In one possible design, based on the test results of each scheduled test task at each test time point, it is predicted whether the probability of abnormal task scheduling of the target system meets the preset warning requirements, including:

[0041] If it is determined according to the detection result that the execution progress of the detection object at the first detection time is incomplete, then a first probability of determining that there is an abnormality in the execution progress of the task meets the early warning requirement;

[0042] If it is determined according to the detection result that the execution progress of the detection object at the second detection time is completed, then it is determined that the second probability that there is an abnormality in the data level of the target system scheduling task meets the early warning requirement.

[0043] In one possible design, determining and outputting one or more warning information includes:

[0044] Calculate the correlation between the previous task and the next task in the scheduling relationship graph according to the preset correlation model;

[0045] If the correlation degree is within the first correlation interval, it is determined that the warning information includes the first warning information and the second warning information, and the warning levels of the first warning information and the second warning information are the same. The first warning information is used to indicate that there is a scheduling anomaly in the previous task and has a correlation impact on the scheduling of the next task. The second warning information is used to indicate that the scheduling anomaly of the next task is caused by the delay of the previous task.

[0046] The first warning information is output to the previous task, and the second warning information is output to the next task.

[0047] In one possible design, determining and outputting one or more warning information includes:

[0048] Calculate the correlation between the previous task and the next task in the scheduling relationship graph according to the preset correlation model;

[0049] If the correlation degree is within the second correlation interval, it is determined that the warning information includes the first warning information and the second warning information, and the first warning level of the first warning information is greater than the second warning level of the second warning information. The first warning information is used to indicate that there is a scheduling anomaly in the previous task and has a correlation impact on the scheduling of the next task. The second warning information is used to indicate that the scheduling anomaly of the next task is caused by the delay of the previous task.

[0050] The first warning information is output to the previous task, and the second warning information is output to the next task.

[0051] In one possible design, determining and outputting one or more warning information includes:

[0052] Calculate the correlation between the previous task and the next task in the scheduling relationship graph according to the preset correlation model;

[0053] If the correlation degree is within the third correlation interval, a warning message is output to the previous task, where the warning message is used to indicate that a scheduling anomaly exists in the previous task.

[0054] In one possible design, the early warning information includes: a weighted feedback link;

[0055] After determining and outputting one or more warning information, the following steps are also included:

[0056] Receive adjustment information input by the user through the weight feedback link;

[0057] Adjust the warning weight of the detection object corresponding to the scheduled detection task according to the adjustment information.

[0058] In one possible design, the method further includes: when a scheduling anomaly is detected in the detection object at the first detection time, determining a third detection time for the detection object according to a preset delay time, the detection object being a currently executing task;

[0059] When it is detected at the third detection time that the currently executed task still has a scheduling anomaly, determining a first warning level of the currently executed task according to the first preset warning weight of the currently executed task and the number of warning triggers;

[0060] Determine whether the first warning level meets the preset warning conditions;

[0061] If so, the warning information is sent again to the currently executing task.

[0062] In a possible design, when it is detected at the third detection time that a scheduling anomaly still exists in the currently executing task, the method further includes:

[0063] Using the preset correlation model and the scheduling relationship map, determine the correlation between the current task and the next task;

[0064] Determine the second warning level of the next task based on the second warning weight, correlation and warning triggering times of the next task;

[0065] Determine whether the second warning level meets the preset warning conditions;

[0066] If so, a warning message is sent to the next task, where the warning message includes a prompt indicating that the scheduling anomaly of the next task is caused by the scheduling delay of the currently executed task.

[0067] In a second aspect, the present application provides a task scheduling anomaly monitoring device, comprising:

[0068] An acquisition module is used to acquire historical task data of one or more historical task cycles according to the work plan of the current task cycle, wherein the similarity between the historical work plan of the historical task cycle and the work plan of the current task cycle meets the preset requirements, and the historical task data includes: configuration data of each historical task and time consumption data of executing each historical task;

[0069] Processing module for:

[0070] Using a preset clustering model, cluster each historical task cycle according to the time consumption data until multiple task time consumption types are determined, and the ratio of the amount of data contained in each task time consumption type to the total amount of time consumption data meets the preset ratio requirement;

[0071] Determine a scheduling relationship map based on the configuration data, and determine at least one scheduled detection task based on the time-consuming types of multiple tasks and the scheduling relationship map. The scheduling relationship map is used to represent the dependency relationship between the processing results of each historical task.

[0072] Based on the test results of each scheduled test task at each test time point, it is predicted whether the probability of abnormal task scheduling of the target system meets the preset warning requirements; if so, one or more warning information is determined;

[0073] The output module is used to output warning information to the detection object of the scheduled detection task.

[0074] In a third aspect, the present application provides an electronic device, comprising:

[0075] a memory for storing program instructions;

[0076] The processor is used to call and execute the program instructions in the memory to perform any possible method provided by the first aspect.

[0077] In a fourth aspect, the present application provides a storage medium, wherein the readable storage medium stores a computer program, and the computer program is used to execute any possible task scheduling exception monitoring method provided in the first aspect.

[0078] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any possible task scheduling exception monitoring method provided in the first aspect.

[0079] The present application provides a task scheduling anomaly monitoring method, device, medium and program product, which obtains historical task data of one or more historical task cycles according to the work plan of the current task cycle, and the similarity between the historical work plan of the historical task cycle and the work plan of the current task cycle meets the preset requirements, and the historical task data includes: configuration data of each historical task and time-consuming data of executing each historical task; using a preset clustering model, clustering processing is performed on each historical task cycle according to the time-consuming data until multiple task time-consuming types are determined, and the ratio of the amount of data contained in each task time-consuming type to the total amount of time-consuming data meets the preset proportion requirement; determining a scheduling relationship map according to the configuration data, and determining at least one timed detection task according to multiple task time-consuming types and the scheduling relationship map, the scheduling relationship map is used to characterize the dependency relationship between the mutual call processing results of each historical task; based on the detection results of each timed detection task at each detection time point, predicting whether the probability of anomaly in the task scheduling of the target system meets the preset warning requirements; if so, determining and outputting one or more warning information. The present application solves the technical problems of poor response timeliness, inflexible configuration, and only logical level monitoring, which is not highly coupled with actual business in existing anomaly monitoring. The early warning can ensure the response time, reserve sufficient time for problem handling, and push warning information based on the dependency relationship between tasks, which is conducive to the rapid location of problems and resource coordination. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0081] Figure 1 A schematic diagram of an application scenario of a task scheduling anomaly monitoring method provided in an embodiment of the present application;

[0082] Figure 2 A flowchart of a task scheduling exception monitoring method provided in this application;

[0083] Figure 3 Provided for the implementation of this application Figure 2A schematic diagram of a process of cyclically determining the time-consuming types of multiple tasks in step S202 of the illustrated embodiment;

[0084] Figure 4 Provided in the embodiments of this application Figure 2 A schematic diagram of a process of determining at least one timed detection task in step S203 of the illustrated embodiment;

[0085] Figure 5 A flowchart of another task scheduling anomaly monitoring method provided for the implementation of this application;

[0086] Figure 6 A schematic diagram of the structure of a task scheduling anomaly monitoring device provided in an embodiment of the present application;

[0087] Figure 7 This is a schematic diagram of the structure of an electronic device provided in this application.

[0088] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0089] To make the purpose, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts, including but not limited to combinations of multiple embodiments, are within the scope of protection of this application.

[0090] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0091] The following is an explanation of the professional terms involved in this application:

[0092] MQ (Message Queue): A first-in, first-out data structure. It places data (also known as messages) in a queue, using the queue mechanism to implement message delivery. Producers create messages and place them in a queue, which is then processed by consumers. Consumers can pull messages from a specific queue or subscribe to a queue to receive push messages from the MQ server.

[0093] Collection cycle: refers to the number of days required for collecting data for analysis and comparison by the early warning model, which can be adjusted according to the data scale.

[0094] Datacheck: Data check refers to the integrity check of dependent data before the scheduling task processes the data.

[0095] Job Server: A job server is a server that receives and executes the specific work content of scheduling tasks.

[0096] The current offline data processing in the financial and internet industries is characterized by large volumes of data and high timeliness. Financial enterprises, in particular, process large amounts of data for regulatory reporting. Failure to complete data processing tasks can lead to delays in timely data delivery and potentially lead to regulatory accountability, impacting corporate ratings and reputation. Therefore, monitoring and exception response within the task scheduling system are crucial.

[0097] Currently, open source task scheduling frameworks or tools, such as Azkaban, Airflow, and Oozie, are relatively mature. However, the monitoring functions configured or developed using these frameworks or tools are mostly based on task abnormality or fixed parameters configured based on experience. When these monitoring alarms are issued, the task is often abnormal or has caused actual impact. The response time is poor and the configuration is inflexible, which increases the difficulty and workload of operation and maintenance. Moreover, the monitoring is at the logical level and is not highly coupled with the actual business.

[0098] To address this situation, existing solutions deploy an engine within a big data cluster to collect cluster health data before and after task scheduling, thereby issuing early warnings. However, these technical solutions often require additional cluster resources. When scheduling large amounts of data, data collection is required virtually throughout the day, increasing the cluster burden and the likelihood of anomalies. Furthermore, the current common early warning implementation algorithm relies on a normal distribution prediction for the entire sample, which lacks discriminatory power and requires a large amount of historical data. This increases the computational time required by the early warning system and results in inaccurate predictions.

[0099] It's important to note that because the normal distribution is established for all historical periods, a large amount of historical data is required. Furthermore, because the tasks performed within each historical period, or within several consecutive historical periods, or over a period of time, have their own phased characteristics, using the entire sample may result in a lack of discrimination.

[0100] In summary, the existing abnormality monitoring has the following technical problems:

[0101] (1) Existing monitoring solutions have poor response times, inflexible configuration, heavy maintenance workload, and low coupling with actual business operations;

[0102] (2) The large amount of data collection required for early warning creates an additional burden on the big data cluster;

[0103] (3) The existing algorithms are highly complex and the data differentiation is not high, the computational cost is high and the prediction results are not accurate enough.

[0104] To improve the existing abnormality monitoring method, the inventors of this application have analyzed and found that there are the following technical obstacles to improving it:

[0105] (1) Most existing monitoring solutions are based on the abnormal resource scheduling of big data clusters and the task status itself. In the case of cross-scheduling of a large amount of business data, the abnormal status of a single task is slowly perceived by downstream tasks, and the impact on downstream tasks cannot be accurately analyzed.

[0106] (2) Real-time data analysis and monitoring requires frequent interaction with the system, which will occupy system resources and increase system computing pressure.

[0107] (3) The application of existing algorithms is not combined with actual scenarios, resulting in additional resource consumption and increased data errors.

[0108] In order to solve the above problems, the invention concept of this application is:

[0109] (1) Without changing the underlying logic of the task scheduling system, the task configuration information, the time taken for task execution, and the MQ sending and receiving time are used to analyze the blood relationship between tasks (i.e., the mutual dependence during scheduling), the expected normal completion time interval of tasks, and the correlation between each blood task (i.e., tasks with a scheduling order relationship between them) (i.e., the correlation degree below). Early warnings are given to upstream and downstream tasks (i.e., two tasks with adjacent execution order) respectively, thereby improving the efficiency of abnormal response and processing time. (2) By analyzing the historical sample data within the cycle, the warning detection time is dynamically adjusted to reduce the manpower of manual configuration of operation and maintenance while increasing the flexibility of warning configuration. (3) An alarm level weight module is added to provide an alarm feedback interface to receive the business attention feedback of operation and maintenance personnel, and generate dynamic alarm push based on the alarm detection results, coupling with actual business needs while avoiding the escalation of the abnormal impact range. (4) By classifying historical data, the complexity of the warning algorithm is reduced, the warning cost is reduced, and the accuracy of the warning is improved. (5) By using existing task configuration information and historical information, the scheduling platform can directly provide it, decoupling it from the big data cluster resources, and avoiding increasing the cluster burden.

[0110] Figure 1 This is a schematic diagram of an application scenario of a task scheduling exception monitoring method provided in this application. Figure 1 As shown, an exception monitoring system 200 is independently set up outside the task scheduling system 100. The exception monitoring system 200 determines multiple timed detection tasks by executing the state task scheduling exception monitoring method provided in this application, and re-determines the detection time of the timed detection task in each task cycle. The exception monitoring system 200 does not issue an alarm only after an exception occurs in the task scheduling. Instead, it monitors the execution progress of each task and sends warning information to both the previous and next tasks with execution order requirements. As far as possible, before the task scheduling exception occurs, it detects in advance through the execution progress of the task that the probability of the exception occurring is greater than the warning requirement, that is, sends the corresponding warning information.

[0111] The following describes in detail the status task scheduling exception monitoring method provided by this application:

[0112] Figure 2 The following is a flow chart of a method for monitoring task scheduling anomalies provided in an embodiment of the present application. Figure 2 As shown, the specific steps of the task scheduling exception monitoring method include:

[0113] S201. Acquire historical task data of one or more historical task cycles according to a work plan of a current task cycle.

[0114] In this step, the similarity between the historical work plan of the historical task cycle and the work plan of the current task cycle meets the preset requirements. The historical task data includes: configuration data of each historical task and time consumption data of executing each historical task.

[0115] It should be noted that the abnormal monitoring method in the embodiment of the present application is different from the prior art in that it requires a normal distribution prediction for the overall sample of all historical task cycles. The present application compares the work plan of each historical task cycle with the work plan of the current task cycle, and when the current task cycle is started, or before it is started, obtains historical task data of one or more historical task cycles whose similarity of work plans meets the preset requirements. In this way, the execution progress of each task can be flexibly monitored in a more targeted manner, that is, the time point of scheduled supervision can be changed, rather than fixed, so that the warning differentiation is higher and more flexible.

[0116] Since the work plans of financial enterprises are phased, that is, they are relatively stable over a period of time, such as multiple task cycles, the historical task cycles include: the historical task cycles of the previous year that are at the same or similar time position as the current task cycle in a year, or the previous task cycle closest to the current cycle, or multiple consecutive task cycles closest to the current cycle.

[0117] S202: Using a preset clustering model, cluster each historical task cycle according to the time consumption data in the historical task data until multiple task time consumption types are determined.

[0118] In this step, the ratio of the amount of data included in each task time-consuming type to the total amount of time-consuming data meets the preset ratio requirement.

[0119] Specifically, according to the requirements of the preset clustering model, a preset number of cluster centers are extracted from the time-consuming data of each historical task. It is worth noting that different preset clustering models may correspond to different numbers of initial cluster centers. Then, the time-consuming data of all historical tasks are clustered using the preset clustering model to obtain the first clustering result, that is, at least one task time-consuming type obtained for the first time. Next, it is necessary to determine whether the ratio of the amount of data contained in each task time-consuming type to the total amount of time-consuming data meets the preset proportion requirement. If so, proceed to the next step S203. Otherwise, it is necessary to reset the cluster centers according to the requirements of the preset clustering model, re-perform clustering, and re-determine whether the ratio of the amount of data contained in the obtained task time-consuming type to the total amount of time-consuming data meets the preset proportion requirement. Repeat multiple cycles until the ratio of the amount of data contained in the task time-consuming type to the total amount of time-consuming data meets the preset proportion requirement.

[0120] It is worth noting that resetting the cluster centers involves two aspects: one is the number of cluster centers, and the other is the replacement of the time-consuming data used as cluster centers. Optionally, the number of cluster centers can be changed (i.e., increased or decreased) or can remain unchanged. Those skilled in the art can set it according to the needs of the actual application scenario.

[0121] It should be noted that, in this embodiment, the preset clustering model for each clustering process may be the same or different, that is, during multiple cycles of clustering process, the same preset clustering model may be used, or a different preset clustering model may be used each time, or one preset clustering model may be used to perform a preset number of clustering processes.

[0122] S203: Determine a scheduling relationship map according to the configuration data, and determine at least one scheduled detection task according to the time-consuming types of the multiple tasks and the scheduling relationship map.

[0123] In this step, the scheduling relationship graph is used to represent the dependency relationship between the processing results of each historical task, or the execution order between each historical task.

[0124] Specifically, according to the task configuration information, the upstream and downstream call relationships of historical tasks are split, including: datacheck, MQ interaction type, and then the blood relationship map of task scheduling, that is, the scheduling relationship map, is generated.

[0125] Determine at least one scheduled detection task based on multiple task time-consuming types and a scheduling relationship graph, including:

[0126] According to the preset screening requirements, a first target type and a second target type are determined from the time-consuming types of each task, for example, the first target type includes types with a longer time consumption, and the second target type includes types with a shorter time consumption;

[0127] Using a preset fluctuation algorithm, a first fluctuation range and a second fluctuation range are determined based on the respective time-consuming data of the first target type and the second target type. It should be noted that the first fluctuation range and the second fluctuation range can be determined based on normal distribution graphs corresponding to the first target type and the second target type.

[0128] The detection object and detection time of each scheduled detection task are determined according to the scheduling relationship map, the first fluctuation range, the second fluctuation range, and the start time of the historical task execution in the time-consuming data.

[0129] It should be noted that there is no strict requirement for the order of "determining the scheduling relationship graph" in step S202 and step S203. These two steps can be executed simultaneously, or any one of them can be executed first.

[0130] S204: Based on the detection results of each scheduled detection task at each detection time point, it is pre-determined whether the probability of abnormality in the task scheduling of the target system meets the preset warning requirements.

[0131] In this step, if yes, then S205 is executed; if no, then it is proved that no abnormality is detected, and the next scheduled detection task is waited for for detection and analysis.

[0132] S205: Determine and output one or more warning information.

[0133] This step includes at least three possible implementation methods.

[0134] 1. The first possible implementation is as follows:

[0135] First, the correlation between the previous task and the next task in the scheduling relationship graph is calculated according to the preset correlation model;

[0136] Then, if the correlation degree is within the first correlation interval, it is determined that the warning information includes the first warning information and the second warning information, and the warning levels of the first warning information and the second warning information are the same. The first warning information is used to indicate that there is a scheduling anomaly in the previous task and has a correlation impact on the scheduling of the next task. The second warning information is used to indicate that the scheduling anomaly of the next task is caused by the delay of the previous task.

[0137] Finally, the first warning information is output to the previous task, and the second warning information is output to the next task.

[0138] 2. The second possible implementation method is as follows:

[0139] First, according to the preset correlation model, the correlation between the previous task and the next task in the scheduling relationship graph is calculated;

[0140] Then, if the correlation degree is within the second correlation interval, it is determined that the warning information includes the first warning information and the second warning information, and the first warning level of the first warning information is greater than the second warning level of the second warning information. The first warning information is used to indicate that there is a scheduling anomaly in the previous task and has a correlation impact on the scheduling of the next task. The second warning information is used to indicate that the scheduling anomaly of the next task is caused by the delay of the previous task.

[0141] Finally, the first warning information is output to the previous task, and the second warning information is output to the next task.

[0142] 3. The third possible implementation method is as follows:

[0143] First, according to the preset correlation model, the correlation between the previous task and the next task in the scheduling relationship graph is calculated;

[0144] Then, if the correlation degree is within the third correlation interval, a warning message is output to the previous task, and the warning message is used to indicate that there is a scheduling anomaly in the previous task.

[0145] In the above three implementation manners, according to a preset correlation model, the correlation degree between the previous task and the next task in the scheduling relationship graph is calculated. In this step, the preset correlation model can be selected according to the actual situation. For example, one implementation manner can be represented by formula (*):

[0146]

[0147] where r represents the correlation degree between the previous task x (also referred to as the upstream task) and the next task y (also referred to as the downstream task), that is, S x [[ID=..]] is the standard deviation of the sample data (i.e., the time-consuming data in each historical data) of task x in the historical task cycle, and S y is the standard deviation of the sample data (i.e., the time-consuming data in each historical data) of task y in the historical task cycle, and cov(X,Y) is the covariance of the sample data of task x and task y in the acquisition cycle.

[0148] It should be noted that in this embodiment, the sample data is time-consuming data, and the historical data obtained in this embodiment is offline data. Because the amount of offline data is very large, most of the processing is based on map / reduce processing. Except for the fluctuation of the data volume, the most intuitive manifestation is the execution time of the task, and this type of data is definitely recorded by the scheduling system and can be directly obtained, avoiding the consumption of cluster resources by additionally deploying a collection module. Here, the time-consuming data includes not only the time for data processing but also the time for waiting for upstream data (i.e., the processing result of the previous task of the current task). In this embodiment, each task (including the tasks in the current task cycle and the historical tasks in the historical task cycle) is a task based on blood relationship (i.e., there is a mutual call of processing results or a dependency relationship in the execution order), and they are definitely related; on this basis, the correlation of the time-consuming fluctuation is more reflected in the layer of dependence of waiting for upstream data between tasks, that is, delay warning.

[0149] Specifically, for the above three implementation manners, when the correlation degree is within the first correlation interval, such as 0.6 < r < 1, it is considered that the two tasks are strongly correlated. Read the task-level weight configuration. Taking the default value 1 of the first warning as an example: Generate the first warning information related to task x, and add the warning content affecting task y to the first warning information. At the same time, generate the second warning information, that is, the delay warning of task y at the same level regarding task x.

[0150] When the correlation degree is within the second correlation interval, such as 0.3 < r ≤ 0.6, it is considered that the two tasks are moderately correlated. The first warning information is generated in the above manner, and at the same time, a secondary warning task, that is, the second warning information, is generated.

[0151] When the correlation degree is within the third correlation interval, such as 0 < r ≤ 0.3, it is considered that the two tasks are weakly correlated. The first warning information is generated in the above manner, and no warning is given to task y. If there is an abnormality when task y is detected, relevant impact content is generated and added to the first warning information.

[0152] It should also be noted that the upstream and downstream relationships are analyzed based on lineage. For the interaction between two tasks or between a task and the upstream and downstream systems, in one possible design, only one layer of interval can be considered because the warning analysis here is for all tasks. For the nodes within a single task, the intervals are multiple layers. However, within the overall task duration, the correlation calculation of the node delays between the upper and lower layers is the same for the whole. Even for the lower-layer check nodes, the duration is calculated starting from the overall task scheduling. Therefore, the longer the waiting time, the closer the fluctuation curve of the node task duration is to the duration of the entire task, that is, the greater the probability that both the upstream and downstream are affected.

[0153] The embodiment of the present application provides a method for monitoring abnormal task scheduling. By obtaining historical task data of one or more historical task cycles according to the work plan of the current task cycle, the similarity between the historical work plan of the historical task cycle and the work plan of the current task cycle meets the preset requirements. The historical task data includes: configuration data of each historical task and the duration data of executing each historical task; using a preset clustering model, clustering each historical task is cycled according to the duration data until multiple task duration types are determined, and the ratio of the amount of data contained in each task duration type to the total amount of duration data meets the preset ratio requirement; determining a scheduling relationship graph according to the configuration data, and determining at least one timed detection task according to the multiple task duration types and the scheduling relationship graph. The scheduling relationship graph is used to represent the dependency relationship of the mutual call processing results between each historical task; according to the detection results of each timed detection task at each detection time point, it is judged whether the probability of abnormal task scheduling in the target system meets the preset warning requirement; if so, one or more warning information are determined and output. This solves the technical problems that the existing abnormal monitoring has poor response timeliness, inflexible configuration, and is only a logical-level monitoring with low coupling with the actual business. It achieves the technical effects that early warning can ensure response timeliness, reserve sufficient time for problem handling, push warning information according to the dependency association between tasks, and is beneficial to quickly locate problems and resource coordination.

[0154] For the convenience of understanding several possible specific implementation manners corresponding to S202, the following will be specifically introduced.

[0155] Figure 3 Provided for the implementation of this application Figure 2 The flowchart of step S202 of the embodiment shown is a flow chart of cyclically determining the time-consuming types of multiple tasks. Figure 3 As shown, the specific steps include:

[0156] S301. Randomly extract the time-consuming data of multiple historical tasks from all historical tasks as cluster centers.

[0157] S302: Using a preset clustering model, perform a first clustering process on each historical task according to the cluster center to determine one or more first time-consuming types.

[0158] In this embodiment, the clustering process of the preset clustering model can be expressed by formula (1):

[0159]

[0160] Among them, C represents the first time-consuming type, k is the number of initial cluster centers, c i is the execution time data of the historical tasks corresponding to each of the k cluster centers, that is, the execution time of the initial cluster center, x j It is the execution time of all samples in one or more historical periods, that is, the execution time of each historical task.

[0161] It is worth noting that formula (1) combines the characteristics of task scheduling and reduces the dimension of conventional kmeans, thereby reducing the complexity of the algorithm while ensuring the accuracy of the data, reducing the burden of early warning deployment and improving computational efficiency.

[0162] S303: Determine whether the proportion of the data volume in each first time-consuming type meets the preset proportion requirement.

[0163] In this step, the data volume ratio is used to represent the ratio of the data volume of the first time-consuming type to the total data volume of the time-consuming data. If yes, step S304 is executed, if not, step S305 is executed.

[0164] In this embodiment, the preset proportion requirement includes: the data volume proportion is greater than or equal to a first proportion threshold and less than or equal to a second proportion threshold. Optionally, the first proportion threshold has a first value range of 1% to 10%, and the second proportion threshold has a second value range of 40% to 60%. Preferably, the first proportion threshold is 5%, and the second proportion threshold is 50%.

[0165] S304: Determine that the first time-consuming type is a task time-consuming type.

[0166] S305: Re-determine the cluster center and re-perform the clustering process to re-determine the first time-consuming type until the data volume ratio corresponding to each first time-consuming type meets the preset ratio requirement.

[0167] In this embodiment, it specifically includes:

[0168] S3051. Delete the first time-consuming type whose data volume accounts for less than the first proportion threshold; and / or randomly select at least two historical tasks as new cluster centers from each first time-consuming type whose data volume accounts for more than the second proportion threshold.

[0169] Specifically, for example, delete the categories with less than 5% of the total number of samples, that is, delete the first time-consuming type with less than 5% of the data volume. Randomly select 2 samples from the categories with more than 50% of the total number of samples as new cluster centers.

[0170] S3052: For the first time-consuming type that meets the preset proportion requirement, a new cluster center is re-determined according to a preset method.

[0171] In a possible implementation, when the first time-consuming type meets a preset proportion requirement, the average time-consuming of the first time-consuming type is used as a new cluster center.

[0172] Specifically, the time consumption data corresponding to the new cluster center can be expressed by formula (2):

[0173]

[0174] Among them, C i is a classification set, that is, the set corresponding to the first time-consuming type, |C i | is the number of samples in the set, x is the time consumption data corresponding to each sample in the set, a i is the time consumption data corresponding to the new cluster center.

[0175] After executing this step, return to step S302, that is, use the preset clustering model to re-perform clustering processing according to each new cluster center to determine a new first time-consuming type until the data volume proportion corresponding to each first time-consuming type meets the preset proportion requirement.

[0176] The method provided in this embodiment, which cyclically determines the time-consuming types of multiple tasks, combines the characteristics of task scheduling and reduces the dimensionality of conventional kmeans, thereby reducing algorithm complexity while ensuring data accuracy, lowering the burden of early warning deployment and improving computational efficiency. By classifying historical data, the complexity of the early warning algorithm is reduced, lowering early warning costs while improving early warning accuracy. Using existing task configuration and historical information, the scheduling platform can directly provide the data needed to construct a scheduling relationship map, as well as classified sample data for task time-consuming types, decoupling it from big data cluster resources and avoiding adding to the cluster's burden.

[0177] In order to facilitate Figure 2 The understanding of possible implementation methods of "determining at least one scheduled detection task according to multiple task time-consuming types and a scheduling relationship map" in step S203 in the illustrated embodiment is introduced below with specific examples.

[0178] Figure 4 Provided in the embodiments of this application Figure 2 The flowchart of determining at least one timing detection task in step S203 of the embodiment shown is as follows. Figure 4 As shown, the specific steps include:

[0179] S401: Determine a first target type and a second target type from various task time-consuming types according to preset screening requirements.

[0180] In this embodiment, the time consumption data corresponding to the cluster center with the largest value is selected from each task time consumption type as the first target type, and the time consumption data with the smallest value is selected as the second target type.

[0181] S402: Determine a first fluctuation range and a second fluctuation range according to each time-consuming data of the first target type and the second target type by using a preset fluctuation algorithm.

[0182] This step specifically includes:

[0183] S4021: Determine a first fluctuation range according to a first average duration and a first standard deviation of all duration data in the first target type.

[0184] In a possible design, the first fluctuation range is equal to the sum of the first average time consumption and N times the first standard deviation. The first fluctuation range B1 is shown in formula (3):

[0185]

[0186] in, represents the first average time consumption, and S1 represents the first standard deviation.

[0187] Preferably, if the sample fluctuation range as a whole obeys a normal distribution, the confidence level within three standard deviations of the mean is 99.6%, and therefore, the value of N can be set to 3. It is understood that those skilled in the art can set the value of N specifically according to the distribution pattern obeyed by the sample fluctuation range, and this is not limited here.

[0188] S4022: Determine a second fluctuation range according to the second average time consumption and the second standard deviation of all time consumption data in the second target type.

[0189] In a possible design, the second fluctuation range is equal to the difference between the second average time and M times the second standard deviation. The second fluctuation range B2 is shown in formula (4):

[0190]

[0191] in, represents the second average time, and S2 represents the second standard deviation.

[0192] Preferably, if the sample fluctuation range as a whole obeys a normal distribution, the confidence level within three standard deviations of the mean is 99.6%, and therefore, the value of M can be set to 3. It is understandable that those skilled in the art can set the value of M specifically according to the distribution pattern obeyed by the sample fluctuation range, and this is not limited here.

[0193] This step selects the clusters with the maximum and minimum time consumption and calculates the task time fluctuation range respectively. This reduces the additional error caused by differences between sample categories. Compared with the normal distribution algorithm of the entire sample, it also reduces the amount of calculation and is more suitable for big data cluster environments with a large number of scheduling tasks.

[0194] S403: Determine the detection object and detection time of each scheduled detection task according to the scheduling relationship map, the first fluctuation range, the second fluctuation range, and the start time of the historical task execution in the time-consuming data.

[0195] In one possible design, the detection time includes: a first detection time and a second detection time. The first detection time includes: a first fluctuation range superimposed on the starting time, and the second detection time includes: a second fluctuation range superimposed on the starting time.

[0196] Specifically, according to the result obtained in S402, the start execution time T of the current scheduled task is read to generate a timed detection job or a timed detection task. The detection times T1 and T2 are respectively as shown in formula (5):

[0197]

[0198] Among them, B1 represents the first fluctuation range, and B2 represents the second fluctuation range.

[0199] It is worth noting that in Figure 4 Based on the embodiment shown, Figure 2 In the illustrated embodiment, S204, based on the detection results of each scheduled detection task at each detection time point, predicting whether the probability of abnormality in the task scheduling of the target system meets the preset warning requirements, includes two aspects:

[0200] If it is determined according to the detection result that the execution progress of the detection object at the first detection time is incomplete, then a first probability of determining that there is an abnormality in the execution progress of the task meets the preset warning requirement;

[0201] If it is determined according to the detection result that the execution progress of the detection object at the second detection time is completed, then it is determined that the second probability that there is an abnormality in the data level of the target system scheduling task meets the preset warning requirement.

[0202] Figure 5 This is a flow chart of another method for monitoring task scheduling anomalies provided by the embodiment of the present application. Figure 5 As shown, the specific steps of the method include:

[0203] S501: Acquire historical task data of one or more historical task cycles according to a work plan of a current task cycle.

[0204] In this step, the similarity between the historical work plan of the historical task cycle and the work plan of the current task cycle meets the preset requirements. The historical task data includes: configuration data of each historical task and time consumption data of executing each historical task.

[0205] S502: Using a preset clustering model, cluster each historical task cycle according to the time consumption data until multiple task time consumption types are determined.

[0206] In this step, the ratio of the amount of data included in each task time-consuming type to the total amount of time-consuming data meets the preset ratio requirement.

[0207] S503: Determine a scheduling relationship map according to the configuration data, and determine at least one scheduled detection task according to the time-consuming types of the multiple tasks and the scheduling relationship map.

[0208] In this step, the scheduling relationship graph is used to represent the dependency relationship between the processing results of each historical task.

[0209] S504: Based on the detection results of each scheduled detection task at each detection time point, it is pre-determined whether the probability of abnormality in the task scheduling of the target system meets the preset warning requirements.

[0210] In this step, if no, then loop this step, that is, continue to perform the next timed detection task. If yes, then specifically include the judgment of the two detection results of the first detection time T1 and the second detection time T2:

[0211] (1) If it is determined according to the detection result that the execution progress of the detection object at the first detection time is incomplete, then the first probability of determining that there is an abnormality in the execution progress of the task meets the early warning requirement.

[0212] (2) If it is determined according to the detection result that the execution progress of the detection object at the second detection time is completed, then it is determined that the second probability that there is an abnormality in the data level of the target system scheduling task meets the early warning requirement.

[0213] It should be noted that, in this embodiment, the first detection time T1 and the second detection time T2 are as shown in formula (5).

[0214] It is worth noting that the embodiment of the present application only illustrates the subsequent processing of the first case, that is, executing steps S505 to S512. For the second case, the processing can refer to the first case, or another warning method can be independently adopted, such as sending a warning message only once.

[0215] S505: When a scheduling anomaly is detected in the detection object at the first detection time, a warning message is sent to the current task.

[0216] In this step, the detection object is the currently executed task. If a scheduling anomaly is detected in the detection object at the first detection time, the type of the warning information is a delayed warning.

[0217] S506: Determine a third detection time for the detection object according to the preset delay time.

[0218] In this step, for delayed warnings, in order to avoid the warning message being ignored, resulting in failure to achieve the expected warning effect, that is, failure to correct the anomaly in time before a large number of task scheduling anomalies occur, it is necessary to add a preset delay time after the delayed warning is issued, that is, after the first detection time T1, to obtain the third detection time, that is, the third detection time T3 = T1 + Td, where Td is the preset delay time. Optionally, the value of Td is equal to S1 in formula (3), that is, T3 = T1 + S1.

[0219] Similarly, after the current task is tested again at the third test time, if a delayed warning is still issued, the process returns to and repeats steps S506 to S512.

[0220] S507: When it is detected at the third detection time that a scheduling anomaly still exists in the currently executed task, a first warning level of the currently executed task is determined according to a first preset warning weight of the currently executed task and the number of warning triggers.

[0221] In this step, assuming that the current task meets the preset warning requirements again at the third detection time, it is necessary to recalculate the warning level of the current task to avoid the situation where warnings with a lower warning level cannot be sent, resulting in small problems becoming big problems and causing serious scheduling accidents.

[0222] In this embodiment, assuming that the current task (also known as the previous task) is task x, the reference value f(x) of the warning level of task x can be calculated by formula (6):

[0223] f(x) = u x t (6)

[0224] where u x is the initial warning weight of task x, defaulting to 1, and t is the number of warnings. The first warning occurs at the first detection time T1, the second warning occurs at the third detection time T3, and so on.

[0225] S508. Determine whether the first warning level meets the preset warning conditions.

[0226] In this step, if so, then execute step S505 again, that is, send a warning message to the current task, and also execute steps S509 - S512. If not, then directly end this process.

[0227] Specifically, if f(x) ≤ 0.3, then determine that the second warning level is a low - level warning and no warning processing is performed; if 0.3 < f(x) ≤ 0.6, then determine that the second warning level is a medium - level warning; if f(x) > 0.6, then determine that the second warning level is a high - level warning.

[0228] After determining the first warning level, the corresponding warning information can be generated by referring to S205, which will not be elaborated here.

[0229] S509. Use a preset association model to determine the association degree between the currently executing task and the next task according to the scheduling relationship graph.

[0230] In this step, the specific calculation method of the association degree r between the currently executing task (i.e., task x) and the next task (i.e., task y) can refer to the formula (*) in S​​​​​In this embodiment, the next task corresponding to the current task, i.e., task x, in the scheduling relationship graph is task y, and the reference value f(y) of the warning level of task y can be calculated by formula (7):

[0233] f(y) = u y tr (7)

[0234] where u y is the initial warning weight of task y, defaulting to 1, r is the correlation degree or correlation coefficient between task x and task y, t is the number of warning times. The first warning occurs at the first detection time T1, the second warning occurs at the third detection time T3, and so on.

[0235] S511. Determine whether the second warning level meets the preset warning conditions.

[0236] In this step, if so, execute step S512.

[0237] Specifically, if f(y) ≤ 0.3, it is determined that the second warning level is a low-level warning and no warning processing is performed; if 0.3 < f(y) ≤  0.6, it is determined that the second warning level is a medium-level warning; if f(y) > 0.6, it is determined that the second warning level is a high-level warning.

[0238] S512. Send a warning message to the next task.

[0239] In this step, the warning message includes a prompt that the scheduling exception of the next task originates from the scheduling delay of the current executing task.

[0240] Specifically, after determining the warning level, the corresponding warning message can be generated by referring to the specific steps of S205.

[0241] It should be noted that S509~S510 can be executed synchronously with S507, and S508 and S511 can also be executed synchronously.

[0242] It should also be noted that in the task scheduling exception monitoring method provided in the above embodiments, the warning message may include: a weight feedback link; after determining and outputting one or more warning messages, it further includes:

[0243] Receiving the adjustment information input by the user through the weight feedback link;

[0244] s Adjusting the warning weight of the detection object corresponding to the timed detection task according to the adjustment information.

[0245] Specifically, the warning information provides a warning level weight feedback link. If the warning level does not meet the business response level, the weight value that needs to be adjusted is fed back through the link. The feedback value is updated and recorded in the warning system database for subsequent warning generation.

[0246] In general, the task scheduling anomaly monitoring method provided by each embodiment of the present application has at least the following beneficial effects:

[0247] (1) Early warning can ensure timely response and reserve sufficient time for problem resolution;

[0248] (2) Pushing dependent tasks together facilitates quick problem location and resource coordination;

[0249] (3) Dynamically adjust the early warning strategy based on the data within the cycle to help detect problems in a timely manner;

[0250] (4) Provide a feedback interface to couple the business’s attention to the scheduled tasks;

[0251] (5) No need to share computing resources with the scheduling server and Job Server, and basically no impact on the system;

[0252] (6) Follow the dependency inversion principle and directly call the system interface implementation.

[0253] Figure 6 This is a schematic diagram of the structure of a task scheduling anomaly monitoring device provided in an embodiment of the present application. The task scheduling anomaly monitoring device 600 can be implemented by software, hardware, or a combination of both.

[0254] like Figure 6 As shown, the task scheduling abnormality monitoring device 600 includes:

[0255] An acquisition module 601 is configured to acquire historical task data of one or more historical task cycles based on a work plan of a current task cycle, wherein the similarity between the historical work plans of the historical task cycles and the work plan of the current task cycle meets a preset requirement, and the historical task data includes configuration data of each historical task and time consumption data of executing each historical task;

[0256] The processing module 602 is configured to:

[0257] Using a preset clustering model, cluster each historical task cycle according to the time consumption data until multiple task time consumption types are determined, and the ratio of the amount of data contained in each task time consumption type to the total amount of time consumption data meets the preset ratio requirement;

[0258] Determine a scheduling relationship map based on the configuration data, and determine at least one scheduled detection task based on the time-consuming types of multiple tasks and the scheduling relationship map. The scheduling relationship map is used to represent the dependency relationship between the processing results of each historical task.

[0259] Based on the test results of each scheduled test task at each test time point, it is predicted whether the probability of abnormal task scheduling of the target system meets the preset warning requirements; if so, one or more warning information is determined;

[0260] The output module 603 is used to output warning information to the detection object of the scheduled detection task.

[0261] In a possible design, the one or more historical task cycles include: the last task cycle closest to the current cycle, or a plurality of consecutive task cycles closest to the current cycle.

[0262] In one possible design, the processing module 602 is configured to:

[0263] Randomly extract the time consumption data of multiple historical tasks from all historical tasks as cluster centers;

[0264] Using a preset clustering model, perform a first clustering process on each historical task according to the cluster center to determine one or more first time-consuming types;

[0265] Determine whether the proportion of the data volume in each first time-consuming type meets the preset proportion requirement;

[0266] If so, determining that the first time-consuming type is a task time-consuming type;

[0267] If not, the cluster center is re-determined, and the clustering process is re-performed to re-determine the first time-consuming type, until the data volume ratio corresponding to each first time-consuming type meets the preset ratio requirement;

[0268] The data volume ratio is used to represent the ratio of the data volume of the first time-consuming type to the total data volume of the time-consuming data.

[0269] In one possible design, the preset proportion requirement includes: the data volume proportion is greater than or equal to a first proportion threshold, and less than or equal to a second proportion threshold.

[0270] Optionally, a first value range of the first proportion threshold includes: 1% to 10%, and a second value range of the second proportion threshold includes: 40% to 60%.

[0271] In one possible design, the processing module 602 is configured to:

[0272] Deleting the first time-consuming type whose data volume accounts for less than a first proportion threshold; and / or,

[0273] Randomly select at least two historical tasks from each first time-consuming type whose data volume is greater than a second proportion threshold as new cluster centers;

[0274] For the first time-consuming type that meets the preset proportion requirement, a new cluster center is re-determined according to the preset method;

[0275] Using the preset clustering model, clustering is re-performed according to each new cluster center to determine a new first time-consuming type.

[0276] In one possible design, the processing module 602 is further configured to:

[0277] When the first time-consuming type meets the preset proportion requirement, the average time-consuming of the first time-consuming type is used as a new cluster center.

[0278] In one possible design, the processing module 602 is further configured to:

[0279] According to the preset screening requirements, determine the first target type and the second target type from the time-consuming types of each task;

[0280] Determine a first fluctuation range and a second fluctuation range based on the respective time-consuming data of the first target type and the second target type using a preset fluctuation algorithm;

[0281] The detection object and detection time of each scheduled detection task are determined according to the scheduling relationship map, the first fluctuation range, the second fluctuation range, and the start time of the historical task execution in the time-consuming data.

[0282] In one possible design, the processing module 602 is further configured to:

[0283] Determine a first fluctuation range according to a first average duration and a first standard deviation of all duration data in the first target type;

[0284] A second fluctuation range is determined according to a second average duration and a second standard deviation of all duration data in the two target types.

[0285] In one possible design, the processing module 602 is used to calculate the first fluctuation range equal to the sum of the first average time and N times the first standard deviation; and calculate the second fluctuation range equal to the difference between the second average time and M times the second standard deviation.

[0286] In one possible design, the detection time includes: a first detection time and a second detection time. The first detection time includes: a first fluctuation range superimposed on the starting time, and the second detection time includes: a second fluctuation range superimposed on the starting time.

[0287] In one possible design, the processing module 602 is configured to:

[0288] If it is determined according to the detection result that the execution progress of the detection object at the first detection time is incomplete, then a first probability of determining that there is an abnormality in the execution progress of the task meets the early warning requirement;

[0289] If it is determined according to the detection result that the execution progress of the detection object at the second detection time is completed, then it is determined that the second probability that there is an abnormality in the data level of the target system scheduling task meets the early warning requirement.

[0290] In one possible design, the output module 603 is configured to:

[0291] Calculate the correlation between the previous task and the next task in the scheduling relationship graph according to the preset correlation model;

[0292] If the correlation degree is within the first correlation interval, it is determined that the warning information includes the first warning information and the second warning information, and the warning levels of the first warning information and the second warning information are the same. The first warning information is used to indicate that there is a scheduling anomaly in the previous task and has a correlation impact on the scheduling of the next task. The second warning information is used to indicate that the scheduling anomaly of the next task is caused by the delay of the previous task.

[0293] The first warning information is output to the previous task, and the second warning information is output to the next task.

[0294] In one possible design, the output module 603 is configured to:

[0295] Calculate the correlation between the previous task and the next task in the scheduling relationship graph according to the preset correlation model;

[0296] If the correlation degree is within the second correlation interval, it is determined that the warning information includes the first warning information and the second warning information, and the first warning level of the first warning information is greater than the second warning level of the second warning information. The first warning information is used to indicate that there is a scheduling anomaly in the previous task and has a correlation impact on the scheduling of the next task. The second warning information is used to indicate that the scheduling anomaly of the next task is caused by the delay of the previous task.

[0297] The first warning information is output to the previous task, and the second warning information is output to the next task.

[0298] In one possible design, the output module 603 is configured to:

[0299] Calculate the correlation between the previous task and the next task in the scheduling relationship graph according to the preset correlation model;

[0300] If the correlation degree is within the third correlation interval, a warning message is output to the previous task, where the warning message is used to indicate that a scheduling anomaly exists in the previous task.

[0301] In one possible design, the early warning information includes: a weighted feedback link;

[0302] The acquisition module 601 is further configured to receive adjustment information input by the user through the weight feedback link;

[0303] The processing module 602 is further configured to adjust the warning weight of the detection object corresponding to the scheduled detection task according to the adjustment information.

[0304] In one possible design, the processing module 602 is further configured to:

[0305] When a scheduling anomaly is detected in the detection object at the first detection time, a third detection time is determined for the detection object according to the preset delay time, where the detection object is the currently executed task;

[0306] When it is detected at the third detection time that the currently executed task still has a scheduling anomaly, determining a first warning level of the currently executed task according to the first preset warning weight of the currently executed task and the number of warning triggers;

[0307] Determine whether the first warning level meets the preset warning conditions;

[0308] If so, the warning information is sent again to the currently executing task.

[0309] In one possible design, the processing module 602 is further configured to:

[0310] Using the preset correlation model and the scheduling relationship map, determine the correlation between the current task and the next task;

[0311] Determine the second warning level of the next task based on the second warning weight, correlation and warning triggering times of the next task;

[0312] Determine whether the second warning level meets the preset warning conditions;

[0313] If so, a warning message is sent to the next task, where the warning message includes a prompt indicating that the scheduling anomaly of the next task is caused by the scheduling delay of the currently executed task.

[0314] It is worth mentioning that Figure 6 The device provided in the illustrated embodiment can execute the method provided in any of the above method embodiments. Its specific implementation principles, technical features, professional terminology explanations and technical effects are similar and will not be repeated here.

[0315] Figure 7This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device 700 may include: at least one processor 701 and a memory 702. Figure 7 An electronic device is shown using a processor as an example.

[0316] The memory 702 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer operation instructions.

[0317] The memory 702 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0318] The processor 701 is configured to execute computer-executable instructions stored in the memory 702 to implement the methods described in the above method embodiments.

[0319] The processor 701 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0320] Optionally, the memory 702 may be independent or integrated with the processor 701. When the memory 702 is a device independent of the processor 701, the electronic device 700 may further include:

[0321] Bus 703 is used to connect the processor 701 and the memory 702. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc., but this does not mean that there is only one bus or only one type of bus.

[0322] Optionally, in a specific implementation, if the memory 702 and the processor 701 are integrated on a chip, the memory 702 and the processor 701 can communicate through an internal interface.

[0323] An embodiment of the present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program code. Specifically, the computer-readable storage medium stores program instructions, and the program instructions are used for the methods in the above-mentioned method embodiments.

[0324] An embodiment of the present application further provides a computer program product, including a computer program, which implements the methods in the above-mentioned method embodiments when executed by a processor.

[0325] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims of the present application.

[0326] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for monitoring task scheduling anomalies, characterized in that: include: According to the work plan of the current task cycle, historical task data of one or more historical task cycles are obtained, where the similarity between the historical work plans of the historical task cycles and the work plan of the current task cycle meets a preset requirement, and the historical task data includes: configuration data of each historical task and time consumption data of executing each historical task; Using a preset clustering model, clustering is performed on each of the historical task cycles according to the time-consuming data until a plurality of task time-consuming types are determined, and the ratio of the amount of data contained in each task time-consuming type to the total amount of the time-consuming data meets a preset ratio requirement; Determining a scheduling relationship map based on the configuration data, and determining at least one timed detection task based on the multiple task time-consuming types and the scheduling relationship map, wherein the scheduling relationship map is used to represent the dependency relationship between the processing results of each of the historical tasks; Based on the detection results of each of the scheduled detection tasks at each detection time point, it is predicted whether the probability of abnormality in the task scheduling of the target system meets the preset warning requirements; if so, one or more warning information is determined and output; The method of using a preset clustering model to cluster each of the historical task cycles according to the time consumption data until multiple task time consumption types are determined includes: Randomly extracting the time-consuming data of a plurality of the historical tasks from all the historical tasks as cluster centers; Using the preset clustering model, and according to the cluster centers, performing the first clustering process on each of the historical tasks to determine one or more first time-consuming types; Determine whether the proportion of the data volume in each of the first time-consuming types meets the preset proportion requirement; If so, determining that the first time-consuming type is a task time-consuming type; If not, re-determine the cluster center and re-perform the clustering process to re-determine the first time-consuming type until the data volume ratio corresponding to each first time-consuming type meets the preset ratio requirement; The data volume ratio is used to represent the ratio of the data volume contained in the first time-consuming type to the total data volume of the time-consuming data.

2. The task scheduling abnormality monitoring method according to claim 1, characterized in that: The one or more historical task cycles include: a task cycle closest to the current task cycle, or a plurality of consecutive task cycles closest to the current task cycle.

3. The task scheduling abnormality monitoring method according to any one of claims 1 to 2, characterized in that: The preset proportion requirement includes: the data volume proportion is greater than or equal to a first proportion threshold and less than or equal to a second proportion threshold.

4. The task scheduling abnormality monitoring method according to claim 3, characterized in that: A first value range of the first proportion threshold includes: 1%~10%, and a second value range of the second proportion threshold includes: 40%~60%.

5. The task scheduling abnormality monitoring method according to claim 3, characterized in that: The re-determining the cluster center and re-performing the clustering process includes: Deleting the first time-consuming type whose data volume accounts for less than the first proportion threshold; and / or, Randomly selecting at least two of the historical tasks from each of the first time-consuming types whose data volume accounts for more than the second proportion threshold as new cluster centers; For the first time-consuming type that meets the preset proportion requirement, re-determine a new cluster center according to a preset method; The clustering process is re-performed according to each new cluster center by using a preset clustering model to determine a new first time-consuming type.

6. The task scheduling abnormality monitoring method according to claim 5, characterized in that: For the first time-consuming type that meets the preset proportion requirement, re-determining a new cluster center according to a preset method includes: When the first time-consuming type meets the preset proportion requirement, the average time-consuming of the first time-consuming type is used as the new cluster center.

7. The task scheduling abnormality monitoring method according to claim 1, characterized in that: The determining of at least one scheduled detection task according to the plurality of task time-consuming types and the scheduling relationship map includes: Determining a first target type and a second target type from each of the task time-consuming types according to preset screening requirements; Determine a first fluctuation range and a second fluctuation range according to each of the time-consuming data in the first target type and the second target type using a preset fluctuation algorithm; The detection object and detection time of each of the scheduled detection tasks are determined according to the scheduling relationship map, the first fluctuation range, the second fluctuation range, and the start time of the execution of the historical tasks in the time-consuming data.

8. The task scheduling abnormality monitoring method according to claim 7, characterized in that: The determining, using a preset fluctuation algorithm and based on the respective time-consuming data of the first target type and the second target type, a first fluctuation range and a second fluctuation range includes: determining the first fluctuation range according to a first average duration and a first standard deviation of all the duration data in the first target type; The second fluctuation range is determined according to a second average duration and a second standard deviation of all the duration data in the two target types.

9. The task scheduling abnormality monitoring method according to claim 8, characterized in that: The determining the first fluctuation range according to the first average time consumption and the first standard deviation of all the time consumption data in the first target type includes: The first fluctuation range is equal to the sum of the first average time consumption and N times the first standard deviation; The determining the second fluctuation range according to the second average time consumption and the second standard deviation of all the time consumption data of the two target types includes: The second fluctuation range is equal to the difference between the second average time consumption and M times the second standard deviation.

10. The method for monitoring task scheduling anomalies according to any one of claims 7 to 9, characterized in that: The detection time includes: a first detection time and a second detection time. The first detection time includes: superimposing the first fluctuation range on the basis of the starting time. The second detection time includes: superimposing the second fluctuation range on the basis of the starting time.

11. The method for monitoring task scheduling anomalies according to claim 10, characterized in that: The step of predicting whether the probability of abnormality in task scheduling of the target system meets a preset warning requirement based on the detection results of each of the scheduled detection tasks at each detection time point includes: If it is determined according to the detection result that the execution progress of the detection object at the first detection time is incomplete, then a first probability of determining that there is an abnormality in the execution progress of the task meets the preset warning requirement; If it is determined according to the detection result that the execution progress of the detection object at the second detection time is completed, then it is determined that the second probability that there is an abnormality in the data level of the task scheduled by the target system meets the preset warning requirement.

12. The method for monitoring task scheduling anomalies according to any one of claims 1-2 and 7-9, characterized in that: The determining and outputting one or more warning information includes: Calculating the correlation between the previous task and the next task in the scheduling relationship graph according to a preset correlation model; If the correlation degree is within a first correlation interval, it is determined that the warning information includes first warning information and second warning information, and the warning levels of the first warning information and the second warning information are the same, the first warning information is used to indicate that a scheduling anomaly exists in the previous task and has an associated impact on the scheduling of the subsequent task, and the second warning information is used to indicate that the scheduling anomaly of the subsequent task is caused by a delay of the previous task; The first warning information is output to the preceding task, and the second warning information is output to the succeeding task.

13. The method for monitoring task scheduling anomalies according to any one of claims 1-2 and 7-9, characterized in that: The determining and outputting one or more warning information includes: Calculating the correlation between the previous task and the next task in the scheduling relationship graph according to a preset correlation model; If the correlation degree is within a second correlation interval, it is determined that the warning information includes first warning information and second warning information, and the first warning level of the first warning information is greater than the second warning level of the second warning information, the first warning information is used to indicate that a scheduling anomaly exists in the previous task and has an associated impact on the scheduling of the subsequent task, and the second warning information is used to indicate that the scheduling anomaly of the subsequent task is caused by a delay of the previous task; The first warning information is output to the preceding task, and the second warning information is output to the succeeding task.

14. The method for monitoring task scheduling anomalies according to any one of claims 1-2 and 7-9, characterized in that: The determining and outputting one or more warning information includes: Calculating the correlation between the previous task and the next task in the scheduling relationship graph according to a preset correlation model; If the correlation degree is within a third correlation interval, the warning information is output to the previous task, where the warning information is used to indicate that a scheduling anomaly exists in the previous task.

15. The method for monitoring task scheduling anomalies according to any one of claims 1-2 and 7-9, characterized in that: The warning information includes: a weight feedback link; After determining and outputting one or more warning information, the method further includes: receiving adjustment information input by the user through the weight feedback link; The warning weight of the detection object corresponding to the scheduled detection task is adjusted according to the adjustment information.

16. The method for monitoring task scheduling anomalies according to claim 10, characterized in that: Also includes: When the scheduling anomaly is detected in the detection object at the first detection time, determining a third detection time for the detection object according to a preset delay time, where the detection object is a currently executing task; When it is detected at the third detection time that the scheduling anomaly still exists in the currently executed task, determining a first warning level of the currently executed task according to the first preset warning weight of the currently executed task and the number of warning triggers; Determining whether the first warning level meets the preset warning conditions; If so, the warning information is sent again to the currently executing task.

17. The method for monitoring task scheduling anomalies according to claim 16, wherein: When it is detected at the third detection time that the scheduling anomaly still exists in the currently executed task, the method further includes: Determine the correlation between the currently executed task and the next task according to the scheduling relationship map using a preset correlation model; determining a second warning level of the next task according to the second warning weight of the next task, the correlation degree, and the number of warning triggering times; Determining whether the second warning level meets the preset warning condition; If so, the warning information is sent to the next task, where the warning information includes a prompt indicating that the scheduling anomaly of the next task is caused by the scheduling delay of the currently executed task.

18. A task scheduling abnormality monitoring device, characterized in that: include: An acquisition module is configured to acquire historical task data of one or more historical task cycles based on a work plan of a current task cycle, wherein the similarity between the historical work plans of the historical task cycles and the work plan of the current task cycle meets a preset requirement, and the historical task data includes configuration data of each historical task and time-consuming data for executing each historical task; Processing module for: Using a preset clustering model, clustering is performed on each of the historical task cycles according to the time-consuming data until a plurality of task time-consuming types are determined, and the ratio of the amount of data contained in each task time-consuming type to the total amount of the time-consuming data meets a preset ratio requirement; Determining a scheduling relationship map based on the configuration data, and determining at least one timed detection task based on the multiple task time-consuming types and the scheduling relationship map, wherein the scheduling relationship map is used to represent the dependency relationship between the processing results of each of the historical tasks; Based on the detection results of each of the scheduled detection tasks at each detection time point, it is predicted whether the probability of abnormality in the task scheduling of the target system meets the preset warning requirements; if so, one or more warning information is determined; An output module, configured to output the warning information to the detection object of the scheduled detection task; The processing module is specifically used to: Randomly extracting the time-consuming data of a plurality of the historical tasks from all the historical tasks as cluster centers; Using the preset clustering model, and according to the cluster centers, performing the first clustering process on each of the historical tasks to determine one or more first time-consuming types; Determine whether the proportion of the data volume in each of the first time-consuming types meets the preset proportion requirement; If so, determining that the first time-consuming type is a task time-consuming type; If not, re-determine the cluster center and re-perform the clustering process to re-determine the first time-consuming type until the data volume ratio corresponding to each first time-consuming type meets the preset ratio requirement; The data volume ratio is used to represent the ratio of the data volume contained in the first time-consuming type to the total data volume of the time-consuming data.

19. An electronic device, characterized in that: include: processor; as well as, a memory for storing a computer program for the processor; The processor is configured to execute the task scheduling exception monitoring method according to any one of claims 1 to 17 by executing the computer program.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the task scheduling exception monitoring method according to any one of claims 1 to 17 is implemented.

21. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the task scheduling exception monitoring method according to any one of claims 1 to 17 is implemented.

Images