A task monitoring method, apparatus, server, and storage medium

Abstract

This invention discloses a task monitoring method, apparatus, server, and storage medium. The method includes: determining the predicted time consumption of a target task based on the data volume growth rate of the target task; determining the configuration information of each data processing task running on the same server as the target task, and determining the predicted start time of the target task based on the configuration information of each data processing task; determining the predicted end time of the target task based on the predicted start time and predicted time consumption; comparing the predicted end time and expected completion time of the target task, and determining the completion status of the target task based on the comparison result. This technical solution determines a more accurate predicted time consumption and predicted start time of the target task to determine a more accurate predicted end time, and determines the completion status of the target task based on the predicted end time and expected end time. By predicting the completion status of the target task before its execution, the method achieves monitoring of the target task.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a task monitoring method, apparatus, server and storage medium. Background Technology

[0002] Data warehousing technology (Extraction-Transformation-Loading, ETL) describes the process of extracting, transforming, and loading data from a source to a destination. As data processing demands in data centers continue to increase, multiple servers within a server cluster are needed to execute a large number of data processing tasks simultaneously, i.e., multiple ETL processes need to be performed concurrently. To ensure the normal operation of these data processing tasks, monitoring each task is extremely important.

[0003] In existing technologies, when server resources are sufficient, the execution status of each data processing task can be monitored in real time, and an alarm can be issued after the task data processing task times out.

[0004] In the process of realizing this invention, the inventors discovered at least the following technical problems in the prior art:

[0005] Alarms can only be issued after a data processing task times out; it cannot predict whether the data processing task will be completed successfully. Summary of the Invention

[0006] This invention provides a task monitoring method, apparatus, server, and storage medium to monitor data processing tasks in real time, so as to determine whether the data processing task can be successfully completed before the data processing task is executed.

[0007] In a first aspect, embodiments of the present invention provide a task monitoring method, including:

[0008] The predicted time consumption of the target task is determined based on the data volume growth rate of the target task;

[0009] Determine the configuration information of each data processing task running on the same server as the target task, and determine the predicted start time of the target task based on the configuration information of each data processing task.

[0010] The predicted end time of the target task is determined based on the predicted start time and predicted duration of the target task.

[0011] The predicted end time and expected completion time of the target task are compared, and the completion status of the target task is determined based on the comparison results.

[0012] Secondly, embodiments of the present invention also provide a task monitoring device, comprising:

[0013] The prediction time determination module is used to determine the predicted time of the target task based on the data volume growth rate of the target task;

[0014] The prediction start time determination module is used to determine the configuration information of each data processing task running on the same server as the target task, and to determine the prediction start time of the target task based on the configuration information of each data processing task.

[0015] The prediction end time determination module is used to determine the prediction end time of the target task based on the prediction start time and prediction duration of the target task.

[0016] The execution module is used to compare the predicted end time and expected completion time of the target task, and determine the completion status of the target task based on the comparison result.

[0017] Thirdly, embodiments of the present invention also provide a server, the server comprising:

[0018] At least one processor; and a memory communicatively connected to said at least one processor;

[0019] The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the task monitoring method described in any one of the first aspects.

[0020] Fourthly, embodiments of the present invention also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform any of the task monitoring methods described in the first aspect.

[0021] The embodiments of the above invention have the following advantages or beneficial effects:

[0022] This invention provides a task monitoring method, comprising: determining the predicted time consumption of a target task based on the data volume growth rate of the target task; determining the configuration information of each data processing task running on the same server as the target task, and determining the predicted start time of the target task based on the configuration information of each data processing task; determining the predicted end time of the target task based on the predicted start time and predicted time consumption of the target task; comparing the predicted end time and expected completion time of the target task, and determining the completion status of the target task based on the comparison result. The above technical solution, on the one hand, can determine a more accurate predicted time consumption of the target task based on the data volume growth rate of the target task; on the other hand, it can determine the execution order of each data processing task based on the configuration information of each data processing task running in the server cluster, and queue each data processing task in each queue of the server cluster according to the execution order, planned start time, node resource usage, and queue resource usage of each data processing task, so as to determine the queuing start time of the target task, and thus determine a more accurate predicted start time of the target task. After determining the predicted time consumption and predicted start time of the target task, the predicted end time of the target task can be determined based on the predicted time consumption and predicted start time of the target task, and the completion status of the target task can be determined based on the predicted end time and expected end time. By predicting the completion status of the target task before the target task is executed, the monitoring of the target task can be realized. Attached Figure Description

[0023] Figure 1 A flowchart of a task monitoring method provided in an embodiment of the present invention;

[0024] Figure 2 A flowchart of another task monitoring method provided in an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of a monitoring system running on a server cluster to perform a task monitoring method, provided in an embodiment of the present invention.

[0026] Figure 4 A flowchart of step 230 in another task monitoring method provided in an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram showing the execution order and resource usage of each data processing task in another task monitoring method provided in an embodiment of the present invention.

[0028] Figure 6 This is a schematic diagram illustrating the queuing of various data processing steps in another task monitoring method provided in an embodiment of the present invention.

[0029] Figure 7This is a schematic diagram illustrating an anomaly in the completion of a data processing task in another task monitoring method provided in an embodiment of the present invention.

[0030] Figure 8 A flowchart illustrating the implementation of another task monitoring method provided in this embodiment of the invention;

[0031] Figure 9 This is a schematic diagram of the structure of a task monitoring device provided in an embodiment of the present invention;

[0032] Figure 10 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. Detailed Implementation

[0033] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0034] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc. Moreover, embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0035] ETL can extract data from distributed, heterogeneous data sources, such as relational data and flat data files, to a temporary intermediate layer for cleaning, transformation, and integration, and finally load it into a data warehouse or data mart. It is the foundation of data processing and data mining. Therefore, monitoring the data processing tasks corresponding to ETL is extremely important.

[0036] As the volume of data and the number of data processing tasks increase, existing technologies can only issue alerts after the current data processing task times out. They cannot predict in advance whether the current data processing task can be completed successfully, nor can they remind staff to handle abnormal data processing tasks in advance. This may cause delays in subsequent data processing tasks, affecting the timeliness of data processing.

[0037] Therefore, this application proposes a task monitoring method to predict the data processing task before its execution in order to determine whether the data processing task can be completed successfully. The task monitoring method proposed in this application will be described in detail below with reference to the figures and embodiments.

[0038] Figure 1 This invention provides a flowchart for task monitoring, applicable to situations requiring accurate prediction of the running status of a data processing task before its execution. The method can be executed by a task monitoring device, which can be implemented in software and / or hardware. Figure 1 The method specifically includes the following steps:

[0039] Step 110: Determine the predicted time consumption of the target task based on the data volume growth rate of the target task.

[0040] Specifically, over time, the amount of data to be processed during data processing tasks may change. This change in data volume affects the execution speed of the data processing tasks, and consequently, the time required to complete them. Therefore, the predicted execution time of the target task can be determined based on the data volume growth rate corresponding to the target task.

[0041] In this embodiment of the invention, when determining the predicted time of the target task, the growth rate of the amount of data that the target task needs to process when performing data processing tasks is fully considered, so that the predicted time of the target task is more accurate. Of course, a more accurate predicted time can determine a more accurate predicted end time, providing a more accurate data reserve for determining the completion status of the target task.

[0042] Step 120: Determine the configuration information of each data processing task running on the same server as the target task, and determine the predicted start time of the target task based on the configuration information of each data processing task.

[0043] Here, "server" can refer to a server cluster consisting of multiple servers. The server cluster contains multiple queues, each capable of executing multiple data processing tasks simultaneously. The memory of the server cluster is referred to as cluster resources, the memory of the queues as queue resources, and the memory of the nodes within the server cluster as node resources. The server cluster may run a scheduling system, a metadata system, and a cluster management system. The scheduling system stores the configuration information of the data processing tasks running in each queue, i.e., the queue and node resources required to execute the data processing tasks. The metadata system stores the scripts or programs for the data processing tasks running in each queue. These scripts or programs record the database table information and partition information read or written during the execution of the data processing task. The cluster management system stores the queue resources of each queue within the server cluster and the node resources within the cluster.

[0044] The configuration information for a data processing task can include the upstream and downstream relationships of the data processing task, the planned start time, the node resource usage, and the queue resource usage.

[0045] Specifically, multiple data processing tasks can be executed in parallel or serially within a server cluster. The scheduling system stores the configuration information of the data processing tasks running in each queue. Therefore, the directed acyclic graph (DAG) corresponding to each data processing task running in the server cluster can be determined based on the upstream and downstream relationships in the configuration information of each data processing task. The DAG can indicate the execution order of each data processing task running in the server cluster. Since the configuration information of each data processing task also includes the planned start time of the data processing task, the node resource usage, and the queue resource usage, the queue start time of each data processing task can also be determined when determining the execution order of each data processing task running in the server cluster.

[0046] In practical applications, the end time of the previous data processing task corresponding to the target task, as well as the planned start time and queue start time of the target task, all affect the start time of the target task. To ensure the accuracy of the predicted start time, the latest of the following can be determined as the predicted start time of the target task: the end time of the previous data processing task corresponding to the target task, the planned start time of the target task, and the queue start time.

[0047] Of course, if the end time of the previous data processing task corresponding to the target task is the same as the planned end time, the remaining queue resources in the queue where the target task is located are sufficient to support the operation of the target task, and the node resources of the nodes in the server cluster are also sufficient to support the operation of the target task, then the predicted start time of the target task is the same as the planned start time.

[0048] In this embodiment of the invention, the execution order of each data processing task can be determined according to the configuration information of each data processing task running in the server cluster. Then, each data processing task can be queued in each queue of the server cluster to determine the queuing start time of the target task. In order to ensure the accuracy of the predicted start time of the target task, the latest time among the end time of the previous data processing task corresponding to the target task, the planned start time of the target task, and the queuing start time can be determined as the predicted start time of the target task.

[0049] Step 130: Determine the predicted end time of the target task based on the predicted start time and predicted duration of the target task.

[0050] Specifically, after determining the prediction start time of the target task, the prediction end time of the target task can be determined by adding the prediction time to the prediction start time.

[0051] In this embodiment of the invention, the predicted end time of the target task can be determined based on the predicted start time and the predicted duration, thereby achieving the prediction of the end time of the target task.

[0052] Step 140: Compare the predicted end time and expected completion time of the target task, and determine the completion status of the target task based on the comparison results.

[0053] The configuration information for data processing tasks may also include the expected completion time of the data processing task.

[0054] Specifically, the predicted end time of the target task is compared with the expected completion time in the target task's configuration information. If the predicted end time is later than the expected completion time, the target task is determined to be abnormal; if the predicted end time of the target task is not later than the expected completion time, the target task is determined to be normal.

[0055] The reasons for the abnormal completion of the target task may include: the data source from which the target task extracts data cannot be connected, the amount of data to be processed for the target task has increased dramatically, or the target task script logic is not rigorous.

[0056] In practical applications, if the target task fails to complete, an alarm message can be sent to the corresponding client in the management center to inform the staff of the target task error, so that the staff can maintain the target task in a timely manner.

[0057] In this embodiment of the invention, after determining the predicted end time of the target task, the completion status of the target task can be predicted based on the comparison result between the predicted end time and the expected end time, thereby enabling the monitoring of the target task and facilitating timely information on the completion status of the target task for staff to take timely action.

[0058] An embodiment of the present invention provides a task monitoring method, comprising: determining the predicted time consumption of the target task based on the data volume growth rate of the target task; determining the configuration information of each data processing task running on the same server as the target task, and determining the predicted start time of the target task based on the configuration information of each data processing task; determining the predicted end time of the target task based on the predicted start time and predicted time consumption of the target task; comparing the predicted end time and expected completion time of the target task, and determining the completion status of the target task based on the comparison result. The above technical solution, on the one hand, can determine a more accurate predicted time consumption of the target task based on the data volume growth rate of the target task; on the other hand, it can determine the execution order of each data processing task based on the configuration information of each data processing task running in the server cluster, and queue each data processing task in each queue of the server cluster according to the execution order, planned start time, node resource usage, and queue resource usage of each data processing task, so as to determine the queuing start time of the target task, and thus determine a more accurate predicted start time of the target task. After determining the predicted time consumption and predicted start time of the target task, the predicted end time of the target task can be determined based on the predicted time consumption and predicted start time of the target task, and the completion status of the target task can be determined based on the predicted end time and expected end time. By predicting the completion status of the target task before the target task is executed, the monitoring of the target task can be realized.

[0059] Figure 2 This is a flowchart of another task monitoring method provided by an embodiment of the present invention. This embodiment is applicable to situations where it is necessary to accurately predict the running status of a data processing task before its execution. Based on the above embodiments, this embodiment adds the following steps before determining the predicted time consumption of the target task according to the data volume growth rate: "Determine the historical time consumption and historical data volume of the target task through its historical execution records; determine the data volume growth rate of the target task based on its current data volume and historical data volume." After determining that the target task has completed abnormally, it adds the following steps: "Determine the idle queue resources based on the queue resource usage of each data processing task; determine the resource skew coefficient based on the idle queue resources and the total queue resources; update the predicted time consumption of the target task based on the resource skew coefficient to obtain the updated predicted time consumption; determine the updated predicted end time of the target task based on the predicted start time and the updated predicted time consumption; compare the updated predicted end time and the expected completion time of the target task; update the completion status of the target task based on the comparison result." Explanations of terms identical or corresponding to those in the above embodiments are not repeated here. See also Figure 2 The task monitoring method provided in this embodiment of the invention includes:

[0060] Step 210: Determine the historical execution time and historical data volume of the target task through the historical execution records of the target task.

[0061] Figure 3 This is a schematic diagram of a monitoring system running on a server cluster to execute a task monitoring method, as provided in an embodiment of the present invention. Figure 3 As shown, the monitoring system communicates with the scheduling system, metadata system, and cluster management system running on the server cluster to obtain the data required for task monitoring from the scheduling system, metadata system, and cluster management system.

[0062] Specifically, such as Figure 3 As shown, the monitoring system can include a first layer and a second layer. The first layer is a summarization module, and the second layer is a prediction module. The monitoring system can obtain historical execution records of the target task from the metadata system based on the summarization module to determine the time spent executing the target task within a historical time period, i.e., historical execution time, and the amount of data processed during the execution of the target task, i.e., historical data volume.

[0063] In practical applications, the historical time period can be 30 days or 7 days prior to the current moment, and the specific historical time period can be determined according to actual needs.

[0064] For example, when the historical time period is 30 days, the average time spent in the 30 days prior to the current moment can be determined. First, the two longest and two shortest times spent in the previous 20 days can be removed. Then, the first average time is calculated based on the time spent in the remaining 16 days. Next, the longest and shortest times spent in the most recent 10 days can be removed. Then, the second average time is calculated based on the time spent in the remaining 8 days. Finally, the first average time can be assigned a weight of 30%, and the second average time can be assigned a weight of 70%, thus determining the historical time spent in the 30 days.

[0065] As time progresses, the functions or scripts for the target task may be optimized, reducing the execution time. Taking this into account, higher weight is given to the execution time of the target task within the time period closest to the current moment, in order to determine a more accurate historical execution time for the target task.

[0066] In this embodiment of the invention, the metadata system running on the server cluster can store the historical execution records of each data processing task running on the server cluster. Then, the historical execution time and historical data volume can be determined based on the historical execution records of the target task, providing data reserves for determining a more accurate predicted execution time of the target task.

[0067] Step 220: Determine the data volume growth rate of the target task based on the current data volume and historical data volume of the target task.

[0068] Specifically, the monitoring system can send the historical time and historical data volume of the target task obtained by the aggregation module to the prediction module. The prediction module can first determine the data volume growth based on the current data volume and historical data volume of the target task, and then determine the data volume growth based on the data volume growth and historical data volume, i.e., the month-on-month growth rate of the data volume.

[0069] In this embodiment of the invention, the increase in the amount of data required to execute the target task in the server cluster will affect the execution speed of the target task, and thus affect the execution time of the target task. Therefore, determining the data volume growth rate of the amount of data required to be processed by the target task will further provide data reserves for determining a more accurate prediction of the execution time of the target task.

[0070] Step 230: Determine the predicted time consumption of the target task based on the data volume growth rate of the target task.

[0071] Specifically, after the prediction module of the monitoring system determines the data volume growth rate of the target task, it can determine the prediction time of the target task based on the data volume growth rate of the target task.

[0072] Figure 4 A flowchart of step 230 in another task monitoring method provided in an embodiment of the present invention is shown below. Figure 4 As shown, in one embodiment, step 230 may specifically include:

[0073] Step 2310: Determine the increase in time consumption of the target task based on the data volume growth rate, historical consumption time, and preset weight of the target task.

[0074] Furthermore, the preset weights include aggregation function weights and cluster resource tilt weights; correspondingly, step 2310 may specifically include:

[0075] The first value is determined by multiplying the weight of the aggregation function with the data volume growth rate of the target task and the historical time consumption; the increase in time consumption of the target task is determined based on the first value and the cluster resource tilt weight.

[0076] The preset weights can include the aggregate function weights corresponding to the aggregate functions executed when the target task is performed, and the cluster resource skew weights corresponding to the cluster resources invoked when the target task is performed. Both the aggregate function weights and the cluster resource skew weights can be preset before the data processing task is executed, and the preset principle is also related to the amount of data to be processed when the data processing task is executed. Then, based on the data volume growth rate and historical latency of the target task, as well as the preset aggregate function weights and cluster resource skew weights, the predicted latency of the target task can be determined. When the aggregate function weight is 1, the predicted latency of the target task is consistent with the historical latency. As the amount of data to be processed by the target task increases, a new aggregate function weight can be determined to improve the execution efficiency of the target task. The new aggregate function weight takes a value between 1 and 2; therefore, the difference between the new aggregate function weight and 1 is between 0 and 1. The cluster resource skew weight can be 1 or the ratio of idle queue resources to total queue resources within the cluster. If the queue containing the target task has not been expanded, the cluster resource skew weight is determined to be 1; otherwise, the cluster resource skew weight can be determined as the ratio of idle queue resources to total queue resources within the cluster.

[0077] Specifically, the first value can be determined by multiplying the aggregation function weight, the data volume growth rate of the target task, and the historical time consumption, and the quotient of the first value and the cluster resource tilt weight can be determined as the increase in the time consumption of the target task.

[0078] Step 2320: Determine the predicted time of the target task based on the historical time consumption and the increase in time consumption of the target task.

[0079] Specifically, after determining the increase in the time consumed by the target task, the sum of the historical time consumed by the target task and the increase in time consumed can be used as the predicted time consumed by the target task.

[0080] In this embodiment of the invention, the increase in execution time of the target task can be determined based on three factors affecting the increase in execution time: cluster resource tilt weight, aggregation function weight, and the data volume growth rate of the target task. The predicted execution time of the target task can be determined based on the historical execution time and the increase in execution time of the target task. Since the increase in execution time of the target task is fully considered, the predicted execution time of the target task is more accurate.

[0081] Step 240: Determine the execution order of each data processing task based on the upstream and downstream relationships in the configuration information of each data processing task.

[0082] The configuration information of a data processing task can include the upstream and downstream relationships of the data processing task. That is, the configuration information of any data processing task can include the previous data processing task and the next data processing task corresponding to that data processing task.

[0083] Specifically, the monitoring system can obtain the upstream and downstream relationships of each data processing task from the configuration information of each data processing task in the scheduling system based on the aggregation module, and determine the DAG corresponding to each data processing task running in the server cluster corresponding to each data task. Furthermore, it can determine the execution order of each data processing task running in the server cluster based on the DAG.

[0084] In this embodiment of the invention, the DAG of the server cluster when executing each data processing task is determined, thereby determining the execution order of each data processing task.

[0085] Step 250: Based on the execution order of each data processing task and the planned start time, node resource usage, and queue resource usage in the configuration information of each data processing task, queue each data processing task is queued within each queue of the cluster resources to determine the queuing start time of the target task.

[0086] Specifically, the monitoring system can determine the execution order of each data processing task based on the aggregation module. It can also obtain the planned start time, queue resource usage, and node resource usage of each data processing task from the scheduling system, and the total queue resource quantity and total node resource quantity within the server cluster from the cluster management system, all based on the aggregation module. The aggregation module can also send the execution order of each data processing task, its planned start time, queue resource usage, and node resource usage, as well as the total queue resource quantity and total node resource quantity within the server cluster, to the prediction module. The prediction module can then queue each data processing task in the various queues of the server cluster based on its planned start time, queue resource usage, node resource usage, and execution order to determine the queuing start time of the target task.

[0087] Figure 5 This is a schematic diagram illustrating the execution order and resource usage of each data processing task in another task monitoring method provided in this embodiment of the invention, as shown below. Figure 5As shown, data processing task A is scheduled to execute between 9:00 and 10:00, consuming 1C (core) node resources and 1G queue resources; data processing task B is scheduled to execute between 10:00 and 11:00, consuming 2C node resources and 2G queue resources; data processing task C is scheduled to execute between 11:00 and 13:00, consuming 1C node resources and 1G queue resources; data processing task D is scheduled to execute between 10:00 and 11:00, consuming 1C node resources and 1G queue resources; data processing task E is scheduled to execute between 12:00 and 13:00, consuming 2C node resources and 2G queue resources; and data processing task F is scheduled to execute between 13:00 and 14:00, consuming 2C node resources and 2G queue resources. The execution order of data processing tasks A, B, C, D, E, and F is "ABC" and "DEF".

[0088] Figure 6 This is a schematic diagram illustrating the queuing of data processing in another task monitoring method provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the total queue resources corresponding to each queue in the server cluster are 3GB. Of course, the total node resources corresponding to each node within the server cluster can be greater than 3 cores to ensure that data processing tasks B and D can be executed simultaneously between 10:00 and 11:00, and data processing tasks C and E can be executed simultaneously between 12:00 and 13:00. Furthermore, the data processing tasks can be queued within the server cluster to obtain... Figure 5 The queuing results shown allow us to determine the start time of the queue for any data processing task within the server cluster. For example, we can determine that the queue start time for data processing task A is 9:00 AM, for data processing task B it is 10:00 AM, for data processing task C it is 11:00 AM, for data processing task D it is 10:00 AM, for data processing task E it is 12:00 PM, and for data processing task F it is 1:00 PM.

[0089] In this embodiment of the invention, by queuing each data processing task in the queue of the server cluster, the queuing start time of each data processing task is determined, that is, the queuing start time of the target task is determined.

[0090] Step 260: Determine the predicted start time of the target task based on the end time of the previous data processing task corresponding to the target task, the planned start time of the target task, and the queuing start time.

[0091] In one implementation, step 260 may specifically include:

[0092] The end time of the previous data processing task corresponding to the target task, the planned start time of the target task, and the queuing start time of the target task are sorted in chronological order; the predicted start time of the target task is determined based on the sorting results.

[0093] Specifically, the end time of the previous data processing task corresponding to the target task will affect the start time of the target task. Data processing tasks are queued in various queues within the server cluster, and the determined queue start time of the target task may differ from its planned start time. To determine a more accurate predicted start time for the target task and ensure its smooth execution, the end time of the previous data processing task, the planned start time of the target task, and the queue start time of the target task can be sorted chronologically. The latest of these three times is then determined as the predicted start time of the target task.

[0094] If the predicted start time is earlier than the end time of the previous data processing task corresponding to the target task, the target task needs to be executed in the queue while the previous data processing task is still occupying queue resources, which may cause the target task to fail.

[0095] In this embodiment of the invention, the dynamic changes of queue resources within the corresponding time period are fully considered, and the latest time among the end time of the previous data processing task corresponding to the target task, the planned start time of the target task, and the queue start time is determined as the predicted start time of the target task, so that the predicted start time of the target task is more accurate.

[0096] Step 270: Determine the predicted end time of the target task based on the predicted start time and predicted duration of the target task.

[0097] Specifically, after determining the prediction start time of the target task, the prediction end time of the target task can be determined by adding the prediction time to the prediction start time.

[0098] In this embodiment of the invention, the end time of the target task is predicted.

[0099] Step 280: Compare the predicted end time and expected completion time of the target task, and determine the completion status of the target task based on the comparison results.

[0100] Specifically, if the predicted end time of the target task is no later than the expected completion time, the target task is determined to be completed normally; if the predicted end time of the target task is later than the expected completion time, the target task is determined to be completed abnormally.

[0101] Of course, after confirming the target task's completion anomaly, the idle queue resources of the server cluster can be determined based on the queue resource usage of each data processing task within the server cluster. This idle queue resource can then be used to expand the memory of the queue containing the target task. Increased memory in the target task's queue will improve its execution efficiency. A resource skew coefficient can be determined based on the idle queue resources and the total queue resources. The predicted execution time of the target task can then be updated based on this coefficient, resulting in an updated predicted execution time. Since the target task's execution efficiency has improved, the updated predicted execution time will be shorter than the original predicted execution time.

[0102] Furthermore, the predicted end time of the target task can be determined based on the predicted start time and the predicted time after the update; the predicted end time and expected completion time of the target task can be compared, and the completion status of the target task can be updated according to the comparison results. After expanding the queue in which the target task is located, the completion status of the target task can be re-determined, thereby realizing continuous monitoring of the target task.

[0103] In practical applications, after determining that the completion status of the target task is normal, the completion status of the target task can be determined again after a five-minute interval; if the completion status of the target task is determined to be abnormal, the queue of the target task can be expanded immediately and the completion status of the target task can be determined again to achieve continuous monitoring of the target task.

[0104] Figure 7 This is a schematic diagram illustrating an anomaly in the completion of a data processing task in another task monitoring method provided by an embodiment of the present invention, corresponding to... Figure 6 The data processing tasks in the middle were completed normally. Figure 7 Data processing task A, originally scheduled to execute between 9:00 and 10:00, is delayed to between 10:00 and 11:00. During this time, data processing tasks D, E, and F complete normally, but the start times of data processing tasks B and C change, causing them to fail. If other queues in the server cluster have idle resources at this time, these queue resources can be expanded to 4GB of memory. Then, the data processing tasks A, B, and D to execute between 9:00 and 10:00 will be A, B, and D. Executing tasks A, B, and D requires 1 core, 2 cores, and 1 core of node resources respectively. Therefore, to satisfy the execution of data processing tasks A, B, and D between 10:00 and 11:00, the server cluster needs more than 4 cores of node resources.

[0105] In this embodiment of the invention, after determining the predicted end time of the target task, the completion status of the target task can be predicted based on the comparison result between the predicted end time and the expected end time, thereby enabling the monitoring of the target task and facilitating timely information on the completion status of the target task for staff to take timely action.

[0106] Of course, when the completion status of the target task is determined to be abnormal, the queue in which the target task is located can be expanded based on the idle queue resources in the server cluster, and the completion status of the target task can be re-determined to further increase the probability that the completion status of the target task is normal.

[0107] The task monitoring method provided in this embodiment of the invention includes: determining the historical execution time and historical data volume of the target task through historical execution records; determining the data volume growth rate of the target task based on the current data volume and historical data volume; determining the predicted execution time of the target task based on the data volume growth rate; determining the execution order of each data processing task based on the upstream and downstream relationships in the configuration information of each data processing task; queuing each data processing task in each queue resource of the cluster resources based on the execution order of each data processing task and the planned start time, node resource usage, and queue resource usage in the configuration information of each data processing task, to determine the queuing start time of the target task; determining the predicted start time of the target task based on the end time of the previous data processing task corresponding to the target task, the planned start time and queuing start time of the target task; determining the predicted end time of the target task based on the predicted start time and predicted execution time; comparing the predicted end time and expected completion time of the target task, and determining the completion status of the target task based on the comparison result. The above technical solution can determine the historical execution time and historical data volume of the target task based on its historical execution records. It can also determine the data volume growth rate based on the current and historical data volume required for the target task, and use this growth rate to determine a more accurate predicted execution time. Furthermore, it can determine the execution order of each data processing task running in the server cluster based on their configuration information. Then, based on the execution order, planned start time, node resource usage, and queue resource usage, it can queue each data processing task in the server cluster to determine the queuing start time of the target task, thereby determining a more accurate predicted start time. After determining the predicted execution time and predicted start time, it can determine the predicted end time of the target task based on these values. Finally, it can determine the completion status of the target task based on the predicted end time and expected end time, thus enabling monitoring of the target task by predicting its completion status before execution.

[0108] In addition, after determining that the target task has failed to complete, and assuming that the node resources are sufficient to support the operation of each data processing task, the queue in which the target task is located is expanded based on the idle queue resources in the server cluster. This is to redetermine the completion status of the target task, further increase the probability of the target task completing normally, and further reduce the occurrence of data processing task delays.

[0109] Figure 8 The flowchart illustrates another task monitoring method provided in this embodiment of the invention, exemplarily showing one implementation method. For example... Figure 8 As shown,

[0110] Step 810: Determine the historical execution time and historical data volume of the target task through the historical execution records of the target task.

[0111] Step 811: Determine the data volume growth rate of the target task based on the current data volume and historical data volume of the target task.

[0112] Step 812: Determine the increase in time consumption of the target task based on the data volume growth rate, historical consumption time, and preset weight of the target task.

[0113] Step 813: Determine the predicted time of the target task based on the historical time consumption and the increase in time consumption of the target task.

[0114] Step 814: Determine the execution order of each data processing task based on the upstream and downstream relationships in the configuration information of each data processing task.

[0115] Step 815: Based on the execution order of each data processing task, as well as the planned start time, node resource usage, and queue resource usage of each data processing task, queue each data processing task is queued within each queue of the cluster resources to determine the queuing start time of the target task.

[0116] Step 816: Sort the end time of the previous data processing task corresponding to the target task, the planned start time of the target task, and the queuing start time of the target task in chronological order, and determine the predicted start time of the target task based on the sorting results.

[0117] Step 817: Determine the predicted end time of the target task based on the predicted start time and predicted duration of the target task.

[0118] Step 818: Compare the predicted end time and expected completion time of the target task.

[0119] If the predicted end time of the target task is no later than the expected completion time, then proceed to step 819; if the predicted end time of the target task is later than the expected completion time, then proceed to step 820.

[0120] Step 819: Confirm that the target task has been completed normally.

[0121] Step 820: Determine that the target task has not been completed.

[0122] Step 821: Determine the available queue resources based on the queue resource usage of each data processing task.

[0123] Step 822: Determine the resource skew coefficient based on the idle queue resources and the total queue resources, and update the predicted time of the target task based on the resource skew coefficient to obtain the updated predicted time.

[0124] After step 822, return to step 817.

[0125] The task monitoring implementation method provided in this embodiment of the invention has the beneficial effects of executing any of the aforementioned task monitoring methods.

[0126] Figure 9 This is a schematic diagram of a task monitoring device provided in an embodiment of the present invention. This device belongs to the same inventive concept as the task monitoring methods in the above embodiments. Details not described in detail in the embodiments of the task monitoring device can be found in the embodiments of the above-described task monitoring methods.

[0127] The specific structure of the task monitoring device is as follows: Figure 9 As shown, it includes:

[0128] The prediction time determination module 910 is used to determine the prediction time of the target task based on the data volume growth rate of the target task;

[0129] The prediction start time determination module 920 is used to determine the configuration information of each data processing task running on the same server as the target task, and to determine the prediction start time of the target task based on the configuration information of each data processing task.

[0130] The prediction end time determination module 930 is used to determine the prediction end time of the target task based on the prediction start time and prediction duration of the target task.

[0131] The execution module 940 is used to compare the predicted end time and expected completion time of the target task, and determine the completion status of the target task based on the comparison result.

[0132] Based on the above embodiments, the device further includes:

[0133] The data volume growth rate determination module is used to determine the historical execution time and historical data volume of the target task through the historical execution records of the target task;

[0134] The data volume growth rate of the target task is determined based on the current data volume and historical data volume of the target task.

[0135] Based on the above embodiments, the predicted time determination module 910 is specifically used for:

[0136] The increase in the time consumption of the target task is determined based on the data volume growth rate of the target task, historical consumption time, and preset weights;

[0137] The predicted time for the target task is determined based on the historical time taken and the increase in time taken.

[0138] In one implementation, the preset weights include aggregation function weights and cluster resource tilt weights. Accordingly, the increase in the time consumption of the target task is determined based on the data volume growth rate of the target task, historical consumption time, and the preset weights, including:

[0139] The first value is determined by multiplying the weight of the aggregation function with the data volume growth rate of the target task and the historical time consumption.

[0140] The increase in time consumption for the target task is determined based on the first value and the cluster resource tilt weight.

[0141] Based on the above embodiments, the prediction start time determination module 920 is specifically used for:

[0142] Determine the configuration information of each data processing task that runs on the same server as the target task;

[0143] The execution order of each data processing task is determined based on the upstream and downstream relationships in the configuration information of each data processing task.

[0144] Based on the execution order of each data processing task and the planned start time, node resource usage, and queue resource usage in the configuration information of each data processing task, each data processing task is queued in each queue of the cluster resources to determine the queuing start time of the target task.

[0145] The predicted start time of the target task is determined based on the end time of the previous data processing task corresponding to the target task, as well as the planned start time and queue start time of the target task.

[0146] In one implementation, determining the predicted start time of the target task based on the end time of the previous data processing task corresponding to the target task, the planned start time of the target task, and the queuing start time of the target task includes:

[0147] Sort the data processing time of the previous data processing task corresponding to the target task, the planned start time of the target task, and the queue start time of the target task in chronological order;

[0148] The predicted start time of the target task is determined based on the sorting results.

[0149] Based on the above embodiments, the execution module 940 is specifically used for:

[0150] If the predicted end time of the target task is not later than the expected completion time, then the target task is determined to have been completed normally.

[0151] If the predicted end time of the target task is later than the expected completion time, then the target task is determined to be abnormal.

[0152] Based on the above embodiments, the device further includes:

[0153] The re-determination module is used to determine the idle queue resources based on the queue resource usage of each data processing task after determining that the target task has completed abnormally; determine the resource skew coefficient based on the idle queue resources and the total queue resources; update the predicted time of the target task based on the resource skew coefficient to obtain the updated predicted time; determine the updated predicted end time of the target task based on the predicted start time and the updated predicted time; compare the updated predicted end time and the expected completion time of the target task, and update the completion status of the target task based on the comparison result.

[0154] The task monitoring device provided in the embodiments of the present invention can execute the task monitoring method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the task monitoring method.

[0155] It is worth noting that in the above-described embodiments of the task monitoring device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0156] Figure 10 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. Figure 10 A block diagram of an exemplary server 10 suitable for implementing embodiments of the present invention is shown. Figure 10The server 10 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0157] like Figure 10 As shown, server 10 is presented in the form of a general-purpose computing server. The components of server 10 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0158] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0159] Server 10 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by server 10, including volatile and non-volatile media, removable and non-removable media.

[0160] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Server 10 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 10 Not shown; usually referred to as a "hard drive"). Although Figure 10 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0161] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0162] Server 10 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable users to interact with server 10, and / or with any device that enables server 10 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, server 10 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. Figure 10 As shown, network adapter 20 communicates with other modules of server 10 via bus 18. It should be understood that, although... Figure 10 As not shown, it can be combined with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0163] Processing unit 16 executes various functional applications and page displays by running programs stored in system memory 28, such as implementing the task monitoring method provided in this embodiment, which includes:

[0164] The predicted time consumption of the target task is determined based on the data volume growth rate of the target task;

[0165] Determine the configuration information of each data processing task running on the same server as the target task, and determine the predicted start time of the target task based on the configuration information of each data processing task.

[0166] The predicted end time of the target task is determined based on the predicted start time and predicted duration of the target task.

[0167] The predicted end time and expected completion time of the target task are compared, and the completion status of the target task is determined based on the comparison results.

[0168] Of course, those skilled in the art will understand that the processor can also implement the technical solutions of the task monitoring method provided in any embodiment of the present invention.

[0169] This invention provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the program implements, for example, the task monitoring method provided in this invention, which includes:

[0170] The predicted time consumption of the target task is determined based on the data volume growth rate of the target task;

[0171] Determine the configuration information of each data processing task running on the same server as the target task, and determine the predicted start time of the target task based on the configuration information of each data processing task.

[0172] The predicted end time of the target task is determined based on the predicted start time and predicted duration of the target task.

[0173] The predicted end time and expected completion time of the target task are compared, and the completion status of the target task is determined based on the comparison results.

[0174] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0175] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0176] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0177] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0178] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0179] Furthermore, the acquisition, storage, use, and processing of data in the technical solution of this invention all comply with the relevant provisions of national laws and regulations.

[0180] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A task monitoring method, characterized in that, include: The increase in the time consumption of the target task is determined based on the data volume growth rate of the target task, historical time consumption, and preset weights. The preset weights include aggregation function weights and cluster resource tilt weights. The aggregation function weights are the weights corresponding to the aggregation functions executed when the target task is executed, and the cluster resource tilt weights are the weights corresponding to the cluster resources invoked when the target task is executed. The predicted time consumption of the target task is determined based on the historical time consumption and the increase in time consumption. Determine the configuration information of each data processing task running on the same server as the target task, and determine the predicted start time of the target task based on the configuration information of each data processing task. The predicted end time of the target task is determined based on the predicted start time and predicted duration of the target task. The predicted end time and expected completion time of the target task are compared, and the completion status of the target task is determined based on the comparison results.

2. The task monitoring method according to claim 1, characterized in that, Before determining the predicted time of the target task based on the data volume growth rate of the target task, the process also includes: The historical execution records of the target task are used to determine the historical time consumption and historical data volume of the target task. The data volume growth rate of the target task is determined based on the current data volume and historical data volume of the target task.

3. The task monitoring method according to claim 2, characterized in that, The increase in the time consumption of the target task is determined based on the data volume growth rate, historical consumption time, and preset weights, including: The first value is determined by multiplying the weight of the aggregation function with the data volume growth rate of the target task and the historical time consumption. The increase in time consumption for the target task is determined based on the first value and the cluster resource tilt weight.

4. The task monitoring method according to claim 1, characterized in that, The prediction start time of the target task is determined based on the configuration information of each of the data processing tasks, including: The execution order of each data processing task is determined based on the upstream and downstream relationships in the configuration information of each data processing task. Based on the execution order of each data processing task and the planned start time, node resource usage, and queue resource usage in the configuration information of each data processing task, each data processing task is queued in each queue of the cluster resources to determine the queuing start time of the target task. The predicted start time of the target task is determined based on the end time of the previous data processing task corresponding to the target task, as well as the planned start time and queue start time of the target task.

5. The task monitoring method according to claim 1, characterized in that, Based on the end time of the previous data processing task corresponding to the target task, and the planned start time and queuing start time of the target task, the predicted start time of the target task is determined, including: Sort the data processing time of the previous data processing task corresponding to the target task, the planned start time of the target task, and the queue start time of the target task in chronological order; The predicted start time of the target task is determined based on the sorting results.

6. The task monitoring method according to claim 5, characterized in that, Comparing the predicted end time and expected completion time of the target task, and determining the completion status of the target task based on the comparison result, includes: If the predicted end time of the target task is not later than the expected completion time, then the target task is determined to have been completed normally. If the predicted end time of the target task is later than the expected completion time, then the target task is determined to be abnormal.

7. The task monitoring method according to claim 6, characterized in that, After determining that the target task has failed to complete, the process also includes: Based on the queue resource usage of each data processing task, determine the available queue resources; The resource skew coefficient is determined based on the idle queue resources and the total queue resources. The predicted time of the target task is updated based on the resource skew coefficient to obtain the updated predicted time. The post-update prediction end time of the target task is determined based on the prediction start time and the post-update prediction time of the target task. The updated predicted end time and expected completion time of the target task are compared, and the completion status of the target task is updated based on the comparison result.

8. A task monitoring device, characterized in that, include: The predicted time consumption determination module is used to determine the increase in time consumption of the target task based on the data volume growth rate of the target task, historical time consumption, and preset weights. The preset weights include aggregation function weights and cluster resource tilt weights. The aggregation function weights are the weights corresponding to the aggregation function executed when the target task is executed, and the cluster resource tilt weights are the weights corresponding to the cluster resources invoked when the target task is executed. The module determines the predicted time consumption of the target task based on its historical time consumption and the increase in time consumption. The prediction start time determination module is used to determine the configuration information of each data processing task running on the same server as the target task, and to determine the prediction start time of the target task based on the configuration information of each data processing task. The prediction end time determination module is used to determine the prediction end time of the target task based on the prediction start time and prediction duration of the target task. The execution module is used to compare the predicted end time and expected completion time of the target task, and determine the completion status of the target task based on the comparison result.

9. A server, characterized in that, The server includes: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the task monitoring method according to any one of claims 1-7.

10. A storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the task monitoring method as described in any one of claims 1-7.

Images