Batch task time monitoring method, device, electronic device and storage medium

Abstract

The present invention provides a batch task time monitoring method, device, electronic device and storage medium, wherein the monitoring method comprises the steps of: obtaining a dependency table of the batch task and starting the batch task; calculating the single task prediction duration of each task in the batch task; calculating multiple process relationships between the tasks according to a historical model of the batch task and the dependency table, and calculating a key path composed of key tasks in each process relationship; obtaining the overall prediction duration of each process relationship according to the single task prediction duration and the key path; comparing whether the overall prediction duration is within the preset duration range of the batch task, and if not, giving an alarm; if yes, sending the start time data of each current task through a data monitoring system, triggering the monitoring of each current task according to the historical model and the start time data, and continuously monitoring.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a batch processing task time monitoring method, device, electronic equipment and computer-readable storage medium. Background Art

[0002] With the development of operation and maintenance services and the business characteristics of the financial industry, more and more systems need to perform batch processing tasks. Batch processing tasks refer to tasks that are executed regularly (for example, every day), such as data backup, business reconciliation, etc., which are generally composed of multiple SQL statements to form a complete process.

[0003] A single task has its own start and completion time. The complete process of a system is regarded as a working group. Most tasks have their own predecessor dependent tasks. Due to the requirements of the process (such as cleaning up expired data and then checking the remaining space) and input (generating a report for the next task), the current task will not start until all the predecessor tasks are completed.

[0004] Batch processing tasks are the most basic and important tasks in the financial industry. If important business cannot be completed on time, it will directly affect the normal business of the next day (such as interest settlement and account reconciliation). Therefore, it is particularly important to monitor the running time of batch processing tasks, which is divided into two parts:

[0005] (1) The running time of a single task cannot be too long, otherwise it will affect the completion of subsequent tasks on schedule; however, if the execution time is too short, it often means that the task reports an error and exits early or does not get the expected input.

[0006] (2) The overall operation task needs to be completed before a certain time node (such as 24:00 or 8:00, etc.).

[0007] There are some research focuses on batch processing tasks in academia, including automatic scaling of batch job runtime prediction in the cloud (AAAI-2020 Cloud Intelligence Workshop), using interactive perception models and simulations to predict the completion time of batch query workloads (EDBT-2011), etc. However, the focus is more on the analysis of resource bottlenecks and reasonable scheduling, or focusing on slow SQL issues, etc. For the direct prediction of batch running time, some relatively basic time series prediction algorithms are used. However, due to the characteristics of batch data, it is difficult for existing algorithms to make better predictions when the time series data has different forms. Due to the importance of batch business, people generally configure machines specifically for batch processing tasks, while allowing a considerable proportion of resource redundancy to prevent task delays or failures due to preemption problems caused by insufficient resources (for example, a state-owned bank has 6 servers dedicated to batch tasks, and the CPU utilization rate generally does not exceed 40%).

[0008] The industry has not yet applied machine learning algorithms on a large scale to solve related problems. The current situation in the financial industry is:

[0009] (1) Set a fixed threshold based on the knowledge of the system operation and maintenance administrator and the average of historical batch runs: such as twice the average, etc.

[0010] (2) Failure to monitor batch runs that are too short;

[0011] (3) Maintain a set of special thresholds for special days;

[0012] (4) For critical systems, thresholds may even be modified daily;

[0013] (5) Lack of historical experience (lack of historical statistics for each business batch);

[0014] In summary, the importance and difficulties of batch time-based prediction and anomaly detection are reflected in the existing academic work and industrial practices, which have some limitations in our actual challenges. Summary of the invention

[0015] The purpose of the present invention is to solve at least one technical problem in the background technology and to provide a batch task time monitoring method, a monitoring device, an electronic device and a computer-readable storage medium.

[0016] To achieve the above object, the present invention provides a batch processing task time monitoring method, comprising the following steps:

[0017] Get the dependency table of the batch processing task and start the batch processing task;

[0018] Calculate the single task prediction duration of each task in the batch task;

[0019] Calculate multiple process relationships between the tasks to be processed based on the historical model of the batch processing tasks and the dependency table, and simultaneously calculate a critical path composed of key tasks in each process relationship;

[0020] Obtaining the overall predicted duration of each process relationship according to the single task predicted duration and the critical path;

[0021] Determine whether the overall predicted duration is within the preset duration range of the batch task, and if not, issue an alarm;

[0022] If yes, the start time data of each current task is sent through the data monitoring system, and the monitoring process of each current task is triggered according to the historical model and the start time data, and the monitoring is continued.

[0023] According to one aspect of the present invention, when a single task is completed, the end time data of the task is sent through the data monitoring system to determine whether the execution time of the task is within a threshold range. If so, training is performed based on the start time data and end time data of the task, the model of the single task is updated, and then the next task is continued. If not, an alarm is generated;

[0024] The single-task predicted duration is a middle value of the threshold range.

[0025] According to one aspect of the present invention, the calculation of the predicted duration of each task in the batch processing task includes: automatically judging the data indicator type of each task in the batch processing task and estimating the batch running time, wherein the data indicator types include: periodic data, trend data, compressed data and other types of data.

[0026] According to one aspect of the present invention, the process of determining the data indicator type is: first determine whether it is periodic data, if not, then continue to determine whether it is trend data or compressed data, if not, it belongs to other types of data.

[0027] According to one aspect of the present invention, the determination of whether the data is periodic data is as follows: using a Fourier transform formula to determine whether the data is periodic data, the Fourier transform formula is:

[0028] ;

[0029] In the formula: ξ represents frequency, x represents time, and -2πixξ is a complex function.

[0030] According to one aspect of the present invention, determining whether the data is trend data is as follows: performing a stationarity test using a unit root test to determine whether the data has a trend, thereby determining whether the data is trend data.

[0031] According to one aspect of the present invention, whether the data is compressed data is determined by bucketing the data and then searching for obvious gaps and performing hypothesis testing to determine whether the data is compressed data.

[0032] According to one aspect of the present invention, a special day in the batch processing task is judged, wherein the special day includes:

[0033] Ordinary holidays except the Spring Festival;

[0034] Spring Festival;

[0035] Special Action Dates;

[0036] One day each month, quarter or year;

[0037] One weekday each month, quarter or year.

[0038] According to one aspect of the present invention, a DTW algorithm is used to determine a special day, wherein the DTW formula is:

[0039] ;

[0040] Where: i and j represent the coordinate points in the time series coordinate x sequence and y sequence respectively, D(i, j) represents the path regularization distance matrix, and Dist(i, j) represents the distance between the i-th point in the x sequence and the j-th point in the y sequence.

[0041] According to one aspect of the present invention, the least squares estimation is used to predict the trend change of the trend data task, and the time threshold is obtained. for:

[0042] ;

[0043] In the formula: X is the predictor variable of trend data, y is the response variable of trend data, X T is the transpose of X.

[0044] According to one aspect of the present invention, for the compression data task, when determining the data indicator type, it is analyzed whether there is a data distribution inflection point. If so, the data size position of the inflection point is the time threshold.

[0045] According to one aspect of the present invention, for the periodic data task, a variational autoencoder is used to learn the fluctuation pattern of historical data, and data that is too far away from the reconstructed data is considered to be abnormal, and the position of the abnormal data is the time threshold.

[0046] According to one aspect of the present invention, for the other types of data tasks, a 3-sigma algorithm is used to obtain the mean mean and standard deviation std of the historical data, and the time threshold is mean±k*std, where k is a multiple of the standard deviation std.

[0047] According to one aspect of the present invention, based on the time threshold and the historical batch running time and the general change amount, the batch running time prediction value of the day can be obtained by using the maximum likelihood estimation of the parameters of the Gaussian distribution. The maximum likelihood estimation formula of the parameters of the Gaussian distribution is:

[0048] ;

[0049] In the formula, is the actual value average of historical batch running time, is the variance of the actual value of the historical batch time, X iis the actual value data set of historical batch running time, and n is the number of actual value data sets of historical batch running time;

[0050] The general variation is the range of the standard deviation.

[0051] According to one aspect of the present invention, during the execution of each task, the execution time of each task is judged, and the judgment results include: task timeout and task early termination.

[0052] According to one aspect of the present invention, when determining that a task has timed out, it is determined whether it is a critical node timeout, and if so, the estimated end time of the monitoring process is adjusted; if not, it is determined whether the critical node has been executed;

[0053] When judging whether the key node has been executed, if so, the estimated end time of the monitoring process is directly adjusted; if not, the key node is continuously observed to be executed, and the estimated end time of the monitoring process is adjusted after the execution is completed.

[0054] According to one aspect of the present invention, when determining whether a task ends ahead of schedule, determine whether other nodes at the same level in the task system have completed execution. If so, reduce the expected end time of the monitoring process. If not, continue to observe the status of other nodes.

[0055] To achieve the above object, the present invention provides a batch processing task time monitoring device, comprising:

[0056] The data reading module is used to obtain the dependency table of the batch processing task and start the batch processing task;

[0057] The single-task calculation prediction module calculates the single-task prediction duration of each task in the batch task;

[0058] A process relationship and critical path calculation module calculates multiple process relationships between the tasks based on the historical model of the batch processing tasks and the dependency table, and simultaneously calculates the critical path composed of the key tasks in each process relationship;

[0059] A process relationship overall calculation prediction module obtains the overall prediction duration of each process relationship according to the single task prediction duration and the critical path;

[0060] A detection and alarm module determines whether the overall prediction duration is within the preset duration range of the batch task, and if not, an alarm is issued;

[0061] If yes, the start time data of each current task is sent through the data monitoring system, and the monitoring of each current task is triggered according to the historical model and the start time data, and the monitoring is continued.

[0062] According to one aspect of the present invention, it further includes a detection control module. When a single task is executed, the detection control module sends the end time data of the task through the data monitoring system to determine whether the execution time of the task is within a threshold range. If so, training is performed according to the start time data and the end time data of the task, the model of the single task is updated, and then the next task is continued. If not, an alarm is generated;

[0063] The single-task predicted duration is a middle value of the threshold range.

[0064] To achieve the above objectives, the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the above batch task time monitoring method when executed by the processor.

[0065] To achieve the above object, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above batch task time monitoring method is implemented.

[0066] The beneficial effects of the present invention are:

[0067] 1. The present invention adopts a combination of several algorithms to solve the technical problem that the existing technology does not judge different types of data, and cannot predict the batch duration of the task according to the above different types of data, resulting in inaccurate prediction results. It is no longer necessary to combine with the expert knowledge of the system, which significantly reduces the factors to be considered, shortens the monitoring time, and can recognize special dates, making the monitoring effect more accurate and reliable.

[0068] 2. After accurately predicting the batch time of a single task, it can be reflected in the batch time prediction of the entire system task, so not only the single task prediction time is fast and accurate, but also the batch time prediction of each process formed between tasks and the overall batch task system of the system is fast and accurate, and there is no missed report and it can effectively reduce the false alarm rate and occupy very few resources. At the same time, it can accurately detect various problems in the system, solve the problems in a timely and rapid manner, and update the correct data, so that subsequent applications are accurate and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flowchart schematically showing a batch processing task time monitoring method according to the present invention;

[0070] Figures 2 to 5 Line graphs schematically showing periodic data, cyclical data, compressed data, and trend data;

[0071] Figure 6 and Figure 7 Schematic representation of two special daily bar charts;

[0072] Figure 8 The functional structure diagram of a batch task time monitoring device of the present invention is schematically shown. DETAILED DESCRIPTION

[0073] The content of the present invention will now be discussed with reference to exemplary embodiments. It should be understood that the embodiments discussed are only to enable those skilled in the art to better understand and thus implement the content of the present invention, rather than implying any limitation on the scope of the present invention.

[0074] As used herein, the term “including” and variations thereof are to be interpreted as open-ended terms meaning “including, but not limited to.” The term “based on” is to be interpreted as “based, at least in part, on.” The terms “one embodiment” and “an embodiment” are to be interpreted as “at least one embodiment.”

[0075] In view of the defects existing in the prior art described in the above background technology, the present invention provides a batch task time monitoring method, which can realize the time prediction of batch tasks and the anomaly detection of batch tasks, and update the task model or generate an alarm according to the prediction and detection results.

[0076] Figure 1 The flowchart of the batch processing task time monitoring method according to the present invention is schematically shown. Figure 1 As shown, the batch processing task time monitoring method according to the present invention comprises the following steps:

[0077] 101. Get the dependency table of the batch processing task and start the batch processing task (for example, the batch processing task for the current day);

[0078] 102. Calculate the predicted duration of each task in the batch task;

[0079] 103. Calculate multiple process relationships between the tasks according to the historical model of the batch processing tasks and the dependency table, and simultaneously calculate a critical path composed of key tasks in each process relationship;

[0080] 104. Obtaining the overall predicted duration of each process relationship according to the single task predicted duration and the critical path;

[0081] 105. Determine whether the overall predicted duration is within the preset duration range of the batch task, and if not, issue an alarm;

[0082] If yes, the start time data of each current task is sent through the data monitoring system, and the monitoring process of each current task is triggered according to the historical model and the start time data, and the monitoring is continued;

[0083] The batch processing task time monitoring method provided by the present invention can accurately predict the batch time of a single task and then reflect it to the batch time prediction of the entire system task. Therefore, it not only makes the single task prediction time fast and accurate, but also makes the batch time prediction of each process formed between each task and the overall batch task system of the system fast and accurate, and does not miss any report and can effectively reduce the false alarm rate, occupying very few resources. At the same time, it can accurately detect various problems existing in the system, solve the problems in a timely and rapid manner, and update the correct data, so that subsequent applications are accurate and reliable.

[0084] According to an embodiment of the present invention, 106. when a single task is executed, the end time data of the task is sent through the data monitoring system to determine whether the execution time of the task is within a threshold range. If so, training is performed based on the start time data and the end time data of the task, the model of the single task is updated, and then the next task is continued. If not, an alarm is generated;

[0085] The single-task predicted duration is a middle value of the threshold range.

[0086] According to one embodiment of the present invention, the data monitoring system adopts the Kafka system.

[0087] Figures 2 to 5 Line graphs schematically represent periodic data, cyclical data, compressed data, and trend data. As shown in the figures, when predicting the batch running time of a single task, the data has different forms. Some data has obvious periodicity, such as historical statistical analysis, and weekly analysis is performed every weekend. Some data has holiday characteristics, which may be tasks related to online business, such as report data generation. Some data has certain trends, such as data backup tasks. Some data is very stable, and occasionally there are some very long and time-consuming tasks, which may be abnormal situations. In the figure, the horizontal axis represents time, the vertical axis represents the batch running time, the gray area represents the reasonable execution time range considered by the algorithm, and the dots represent the abnormal execution time considered by the algorithm.

[0088] In fact, the batch processing task time monitoring in the prior art does not make judgments on different types of data, and therefore cannot predict the batch running time of the task based on the above different types of data, which makes the prediction result inaccurate.

[0089] Furthermore, traditional monitoring methods need to be combined with expert knowledge of the system, taking into account:

[0090] (1) For core systems, system administrators can even set thresholds on a daily basis. There is no need to use algorithms to increase the risk of errors, or at least they can refer to manually set thresholds.

[0091] (2) Some systems have special requirements, such as "must not be completed later than 12 o'clock";

[0092] (3) Some batch runs will affect the opening of the door, regulatory reporting, etc., and special consideration needs to be given to the abnormal classification and abnormal sensitivity.

[0093] In addition, since there are many "special days" in batch running, the algorithm needs to support complex multi-cycle recognition. Special days may include but are not limited to:

[0094] (1) The first day and the second to last day of each month, etc.;

[0095] (2) The 20th day of the last month of each quarter;

[0096] (3) The last day of each year;

[0097] (4) The first working day of each month.

[0098] Figure 6 This is a typical batch running task chart with special days. In addition to the fixed batch running every month, there are also fixed batch running every quarter.

[0099] (5) Affected by other factors, such as:

[0100] The amount of data is a very important factor affecting the batch running time;

[0101] There are various reasons for the anomaly, such as performance problems caused by hardware damage and resource preemption by other businesses.

[0102] As can be seen from the above, there are many factors in the prior art that affect batch prediction and prediction results, which directly lead to the inability of the existing monitoring method to accurately monitor batch processing tasks. In order to solve such technical problems, the present invention proposes the following specific embodiments:

[0103] According to one embodiment of the present invention, the data indicator type of each task is automatically judged and the batch running time is estimated, wherein the data indicator type includes four types, namely periodic data, trend data, compressed data and other types of data, and the judgment process is: first judge whether it is periodic data, if not, then continue to judge whether it is trend data or compressed data, if not, it belongs to other types of data.

[0104] In this embodiment, for periodic data, because the time consumption of weekly data is not much different from the corresponding time of last week, such as the time consumption every weekend is less, this indicator often appears in batch tasks related to business volume. The method adopted by the present invention is Fourier transform, which detects whether the time series has a significant height in certain key positions (such as the corresponding frequency domain of weekly and monthly cycles). The Fourier transform formula is:

[0105] ;

[0106] In the formula: ξ represents frequency, x represents time, and -2πixξ is a complex function.

[0107] In this embodiment, a unit root test is used to perform a stationarity test to determine whether the data has a trend, thereby determining whether the data is trend data.

[0108] After the data is bucketed, look for obvious gaps and / or hypothesis testing to determine whether the data is compressed data.

[0109] Figure 6 and Figure 7 Schematic representation of two special daily line charts.

[0110] According to one embodiment of the present invention, an algorithm is used to judge special days in a task, wherein special days include: ordinary holidays other than the Spring Festival, the Spring Festival, special behavior dates, one day of each month, each quarter, and each year, and one working day of each month, each quarter, and each year.

[0111] In this embodiment, the DTW algorithm is used to determine the special day, wherein the DTW formula is:

[0112] ;

[0113] Where: i and j represent the coordinate points in the time series coordinate x sequence and y sequence respectively, D(i, j) represents the path regularization distance matrix, and Dist(i, j) represents the distance between the i-th point in the x sequence and the j-th point in the y sequence.

[0114] As can be seen from the above, the present invention adopts the combination of the above algorithms to solve the technical problem that the existing technology does not judge different types of data, and the batch duration of the task cannot be predicted based on the above different types of data, resulting in inaccurate prediction results. It is no longer necessary to combine with the expert knowledge of the system, which significantly reduces the factors to be considered, shortens the monitoring time, and can recognize special dates, making the monitoring effect more accurate and reliable.

[0115] Furthermore, according to an embodiment of the present invention, because trend data refers to the batch running time being in a unilateral rising / falling state most of the time, it often appears in tasks whose task volume increases and changes over time (such as data backup tasks). In this embodiment, regression analysis is used to obtain the trend change of indicator data. Specifically, the least squares estimation method is used to estimate the trend change of trend data tasks and obtain the time threshold. The least squares estimation method is used to estimate the trend change of trend data tasks and obtain the time threshold. for:

[0116] ;

[0117] In the formula: X is the predictor variable of trend data, y is the response variable of trend data, X T is the transpose of X.

[0118] Furthermore, for compression data tasks, when determining the data indicator type, it is analyzed whether there is a data distribution inflection point. If so, the data size position of the inflection point is the time threshold.

[0119] For the periodic data task, a variational autoencoder is used to learn the fluctuation pattern of historical data, and data that is too far away from the reconstructed data is considered to be abnormal. The position of the abnormal data is the time threshold.

[0120] For the other types of data tasks, the 3-sigma algorithm is used to obtain the mean and standard deviation std of the historical data, and the time threshold is mean±k*std, where k is a multiple of the standard deviation std.

[0121] In this embodiment, based on the above time thresholds, the batch running time of the previous day, and the general change (the general change is the range of the standard deviation), the maximum likelihood estimation of the parameters of the Gaussian distribution can be used to obtain the batch running time prediction value of each task on the day. The maximum likelihood estimation formula of the parameters of the Gaussian distribution is:

[0122] ;

[0123] In the formula, is the actual value average of historical batch running time, is the variance of the actual value of the historical batch time, X i is the actual value data set of historical batch running time, and n is the number of actual value data sets of historical batch running time;

[0124] The general variation is the range of the standard deviation std. In addition, during the running of each task, the execution time of each task is judged, and the judgment results include: task timeout and task early termination.

[0125] In this embodiment, when determining that a task has timed out, it is determined whether it is a critical node timeout. If so, the estimated end time of the monitoring process is increased; if not, it is determined whether the critical node has been executed;

[0126] When judging whether the key node has been executed, if it has, directly adjust the expected end time of the monitoring process (for example, if the key node is executed ahead of schedule, then reduce the expected end time); if not, continue to observe whether the key node has been executed, and adjust the expected end time of the monitoring process after completion.

[0127] In this embodiment, when determining whether a task ends ahead of schedule, it is determined whether other nodes at the same level in the task system have completed execution. If so, the estimated end time of the monitoring process is reduced. If not, the status of other nodes is continuously observed.

[0128] It can be seen from the above embodiments of the present invention that the overall implementation of the batch task time monitoring method of the present invention is a spark-streaming streaming task. In the above step 103, the process relationship refers to multiple connected graphs (processes) that may be composed of all batch tasks in a system, and each connected graph independently triggers the start of the process.

[0129] In addition, in the above steps 105-106, Spark-streaming obtains the start and end time of each task, and then retrieves the corresponding task history execution status from the database, and then writes the training model into the MongoDB database for storage and backup.

[0130] According to the above-mentioned embodiment of the present invention, after the single task batch time is accurately predicted, it can be reflected in the batch time prediction of the entire system task, so not only the single task prediction time is fast and accurate, but also the batch time prediction of each process formed between the tasks and the overall batch task system of the system is fast and accurate, without missing any reports and can effectively reduce the false alarm rate, occupying very few resources.

[0131] Furthermore, in order to achieve the above-mentioned object of the invention, the present invention also provides a batch processing task time monitoring device, which comprises:

[0132] The data reading module is used to obtain the dependency table of the tasks to be batch processed and start the batch processing tasks;

[0133] The single-task calculation prediction module calculates the single-task prediction duration of each task in the batch task;

[0134] The process relationship and critical path calculation module calculates multiple process relationships between tasks based on the historical model and dependency table of batch tasks, and also calculates the critical path composed of key tasks in each process relationship;

[0135] The overall calculation and prediction module of process relationships obtains the overall prediction duration of each process relationship based on the single task prediction duration and critical path;

[0136] The detection and alarm module determines whether the overall prediction duration is within the preset duration of the batch task. If not, an alarm is issued.

[0137] If so, the start time data of each current task is sent through the data monitoring system, and the monitoring of each current task is triggered according to the historical model and the start time data, and the monitoring is continued.

[0138] Furthermore, according to an embodiment of the batch processing task time monitoring device of the present invention, it also includes:

[0139] Detection control module, when a single task is completed, the detection control module sends the end time data of the task through the data monitoring system to determine whether the execution time of the task is within the threshold range. If so, it is trained according to the start time data and end time data of the task, updates the model of the single task, and then continues to the next task. If not, an alarm is generated;

[0140] Among them, the single-task prediction duration is the middle value of the threshold range.

[0141] According to one embodiment of the present invention, the calculation prediction module automatically determines the data indicator type of each task in the batch processing task and estimates the batch running time, wherein the data indicator types include: periodic data, trend data and compressed data, and the judgment process is: first determine whether it is periodic data, if not, then continue to determine whether it is trend data or compressed data, if not, then it belongs to other types of data.

[0142] In this embodiment, the Fourier transform formula is used to determine whether the data is periodic data. The Fourier transform formula is:

[0143] ;

[0144] In the formula: ξ represents frequency, x represents time, and -2πixξ is a complex function.

[0145] The unit root test is used to conduct a stationarity test to determine whether the data has a trend, and thus to determine whether the data is trend data.

[0146] After the data is bucketed, we use the method of looking for obvious gaps and hypothesis testing to determine whether the data is compressed data.

[0147] In addition, in this embodiment, the calculation prediction module judges the special days in the batch processing tasks, where the special days include: ordinary holidays, Spring Festival, special behavior dates, one day of each month, each quarter, and each year, and one working day of each month, each quarter, and each year.

[0148] Use the DTW algorithm to determine special days, where the DTW formula is:

[0149] ;

[0150] Where: i and j represent the coordinate points in the time series coordinate x sequence and y sequence respectively, D(i, j) represents the path regularization distance matrix, and Dist(i, j) represents the distance between the i-th point in the x sequence and the j-th point in the y sequence.

[0151] Use least squares estimation to predict the trend changes of trend data tasks, obtain time thresholds, and use least squares estimation for:

[0152] ;

[0153] In the formula: X is the predictor variable of trend data, y is the response variable of trend data, X T is the transpose of X.

[0154] For the compression data task, when determining the data indicator type, it is analyzed whether there is a data distribution inflection point. If so, the data size position of the inflection point is the time threshold.

[0155] For the periodic data task, a variational autoencoder is used to learn the fluctuation pattern of historical data, and data that is too far away from the reconstructed data is considered to be abnormal. The position of the abnormal data is the time threshold.

[0156] For other types of data tasks, the 3-sigma algorithm is used to obtain the mean and standard deviation std of historical data. The time threshold is mean±k*std, where k is a multiple of the standard deviation std.

[0157] Furthermore, the detection control module uses the maximum likelihood estimation of the parameters of the Gaussian distribution to obtain the predicted value of the batch running time of the day based on the threshold, the batch running time of the previous day, and the general change (the general change is the range of the standard deviation). The maximum likelihood estimation formula of the parameters of the Gaussian distribution is:

[0158] ;

[0159] In the formula, is the actual value average of historical batch running time, is the variance of the actual value of the historical batch time, Xi is the actual value data set of historical batch running time, and n is the number of actual value data sets of historical batch running time;

[0160] The general variation is the range of the standard deviation std.

[0161] During the running of each task, the detection control module determines the execution time of each task, and the determination results include: task timeout and task premature termination.

[0162] When determining that the task has timed out, determine whether it is a key node timeout. If so, increase the estimated end time of the monitoring process; if not, determine whether the key node has been executed;

[0163] When judging whether the key node has been executed, if it has been executed, the estimated end time of the monitoring process is directly increased; if not, the key node is continuously observed to be executed, and the estimated end time of the monitoring process is increased after the execution is completed.

[0164] When judging that the task ends ahead of schedule, judge whether other nodes at the same level in the task system have completed execution. If so, reduce the expected end time of the monitoring process. If not, continue to observe the status of other nodes.

[0165] The beneficial effects of this embodiment are:

[0166] After accurately predicting the batch time of a single task, it can be reflected in the batch time prediction of the entire system task, so not only the single task prediction time is fast and accurate, but also the batch time prediction of each process formed between tasks and the overall batch task system of the system is fast and accurate, without missing any reports and can effectively reduce the false alarm rate, occupying very few resources. At the same time, it can accurately detect various problems in the system, solve the problems in a timely and rapid manner, and update the correct data, making subsequent applications accurate and reliable.

[0167] In addition, the combination of several algorithms solves the technical problem that the existing technology does not judge different types of data and cannot predict the batch duration of tasks based on the above different types of data, resulting in inaccurate prediction results. It is no longer necessary to combine with the expert knowledge of the system, which significantly reduces the factors to be considered, shortens the monitoring time, and can recognize special dates, making the monitoring effect more accurate and reliable.

[0168] To achieve the above-mentioned purpose of the invention, the present invention also provides an electronic device, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, and the computer program implements the above-mentioned batch task time monitoring method when executed by the processor.

[0169] To achieve the above-mentioned purpose of the invention, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned batch task time monitoring method is implemented.

[0170] According to the above scheme of the present invention, when it is actually applied, it is deployed in the actual production environment of a national joint-stock commercial bank, and the threshold setting standards between different systems are standardized. During the 6-month operation statistical period, no underreporting occurs, the false alarm rate is less than 33%, and the resources are extremely small. The batch task monitoring of the whole bank only requires a 6-core 12G configuration. In the offline data tests of several other commercial banks, significant results have been achieved, and known production events can be accurately reported, and the alarm is on average 23 minutes in advance (due to the more accurate threshold), and the false alarm is compressed by more than 50%. It can be seen that the batch task time monitoring method, device, electronic device and storage medium provided by the present invention effectively solve multiple technical problems in the above-mentioned prior art, and realize the accurate prediction of the execution time of single tasks and batch tasks. At the same time, various problems in the system can be accurately detected, and the problems can be solved promptly and quickly, and the correct data can be updated, so that subsequent applications are accurate and reliable.

[0171] Those of ordinary skill in the art will appreciate that the modules and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0173] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0174] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0175] In addition, each functional module in the embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0176] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the energy-saving signal sending / receiving method of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0177] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the present application.

[0178] It should be understood that the size of the serial numbers of each step in the content of the invention and the embodiments of the present invention does not absolutely mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

Claims

1. A batch processing task time monitoring method, characterized in that: The following steps are involved: Get the dependency table of the batch processing task and start the batch processing task; Calculate the single task prediction duration of each task in the batch task; Use least squares estimation to predict the trend changes of trend data tasks, obtain time thresholds, and use least squares estimation for, In the formula: X is the predictor variable of trend data, y is the response variable of trend data, X T is the transpose of X; For compressed data tasks, when determining the data indicator type, analyze whether there is a data distribution inflection point. If so, the data size at the inflection point is the time threshold. For periodic data tasks, a variational autoencoder is used to learn the fluctuation pattern of historical data, and data that is too far away from the reconstructed data is considered abnormal. The position of the abnormal data is the time threshold. For other types of data tasks, the 3-sigma algorithm is used to obtain the mean and standard deviation std of historical data. The time threshold is mean±k*std, where k is a multiple of the standard deviation std. According to the time threshold, historical batch running time and general changes, the maximum likelihood estimation of the parameters of the Gaussian distribution can be used to obtain the predicted value of the batch running time for the day. The maximum likelihood estimation formula of the parameters of the Gaussian distribution is: In the formula, is the actual average value of the historical batch running time, is the variance of the actual value of the historical batch running time, X i is the actual value data set of historical batch running time, and n is the number of actual value data sets of historical batch running time; The general variation is the range of the standard deviation std; Calculate multiple process relationships between the tasks based on the historical model of the batch processing tasks and the dependency table, and simultaneously calculate a critical path composed of key tasks in each process relationship; Obtaining the overall predicted duration of each process relationship according to the single task predicted duration and the critical path; Determine whether the overall predicted duration is within the preset duration range of the batch task, and if not, issue an alarm; If yes, the start time data of each current task is sent through the data monitoring system, and the monitoring process of each current task is triggered according to the historical model and the start time data, and the monitoring is continued; When a single task is completed, the end time data of the task is sent through the data monitoring system to determine whether the execution time of the task is within the threshold range. If so, training is performed based on the start time data and end time data of the task, the model of the single task is updated, and then the next task is continued. If not, an alarm is generated; Wherein, the single task prediction duration is the middle value of the threshold range; The calculation to obtain the predicted duration of each task in the batch task includes: The data indicator type of each task in the batch processing task is automatically determined and the batch running time is estimated, wherein the data indicator types include: periodic data, trend data, compressed data and other types of data.

2. The batch processing task time monitoring method according to claim 1 is characterized in that: The process of determining the data indicator type is as follows: first determine whether it is periodic data. If not, then continue to determine whether it is trend data or compression data. If neither is the case, then it belongs to other types of data.

3. The batch processing task time monitoring method according to claim 2 is characterized in that: The method of judging whether the data is periodic data is as follows: judging whether the data is periodic data by using the Fourier transform formula, the Fourier transform formula is: In the formula: ξ represents frequency, x represents time, and -2πixξ is a complex function.

4. The batch processing task time monitoring method according to claim 2 is characterized in that: To determine whether it is trend data: use the unit root test to perform a stationarity test to determine whether the data has a trend, and thus determine whether the data is trend data.

5. The batch processing task time monitoring method according to claim 2 is characterized in that: To determine whether the data is compressed data, we divide the data into buckets and then look for obvious gaps and conduct hypothesis testing to determine whether the data is compressed data.

6. The batch processing task time monitoring method according to claim 1 is characterized in that: Determine the special days in the batch processing task, where the special days include: Holidays other than the Spring Festival; Spring Festival; Special Action Dates; One day each month, quarter or year; One weekday each month, quarter or year.

7. The batch processing task time monitoring method according to claim 6 is characterized in that: Use the DTW algorithm to determine special days, where the DTW formula is: D(i,j)=Dist(i,j)+min[D(i-1,j),D(i,j-1),D(i-1,j-1)]; Where: i and j represent the coordinate points in the time series coordinates x and y, respectively, D(i, j) represents the path regularization distance matrix, and Dist(i, j) represents the distance between the i-th point in the x-sequence and the j-th point in the y-sequence.

8. The batch processing task time monitoring method according to claim 7, characterized in that: During the running of each task, the execution time of each task is judged, and the judgment results include: task timeout and task early termination.

9. The batch processing task time monitoring method according to claim 8, characterized in that: When determining that the task has timed out, determine whether it is a key node timeout. If so, adjust the estimated end time of the monitoring process; if not, determine whether the key node has been executed; When judging whether the key node has been executed, if so, the estimated end time of the monitoring process is directly adjusted; if not, the key node is continuously observed to be executed, and the estimated end time of the monitoring process is adjusted after the execution is completed.

10. The batch processing task time monitoring method according to claim 9, characterized in that: When judging that the task ends ahead of schedule, judge whether other nodes at the same level in the task system have completed execution. If so, reduce the expected end time of the monitoring process. If not, continue to observe the status of other nodes.

11. A batch processing task time monitoring device, characterized in that: include: The data reading module is used to obtain the dependency table of the batch processing task and start the batch processing task; The single-task calculation prediction module calculates the single-task prediction duration of each task in the batch task, automatically determines the data indicator type of each task in the batch task and estimates the batch running duration, where the data indicator types include: periodic data, trend data, compressed data and other types of data; Use least squares estimation to predict the trend changes of trend data tasks, obtain time thresholds, and use least squares estimation for, In the formula: X is the predictor variable of trend data, y is the response variable of trend data, X T is the transpose of X; For compressed data tasks, when determining the data indicator type, analyze whether there is a data distribution inflection point. If so, the data size at the inflection point is the time threshold. For periodic data tasks, a variational autoencoder is used to learn the fluctuation pattern of historical data, and data that is too far away from the reconstructed data is considered abnormal. The position of the abnormal data is the time threshold. For other types of data tasks, the 3-sigma algorithm is used to obtain the mean and standard deviation std of historical data. The time threshold is mean±k*std, where k is a multiple of the standard deviation std. According to the time threshold, historical batch running time and general changes, the maximum likelihood estimation of the parameters of the Gaussian distribution can be used to obtain the predicted value of the batch running time for the day. The maximum likelihood estimation formula of the parameters of the Gaussian distribution is: In the formula, is the actual average value of the historical batch running time, is the variance of the actual value of the historical batch running time, X i is the actual value data set of historical batch running time, and n is the number of actual value data sets of historical batch running time; The general variation is the range of the standard deviation std; A process relationship and critical path calculation module calculates multiple process relationships between the tasks based on the historical model of the batch processing tasks and the dependency table, and simultaneously calculates the critical path composed of the key tasks in each process relationship; A process relationship overall calculation prediction module obtains the overall prediction duration of each process relationship according to the single task prediction duration and the critical path; A detection and alarm module determines whether the overall prediction duration is within the preset duration range of the batch task, and if not, an alarm is issued; If yes, the start time data of each current task is sent through the data monitoring system, and the monitoring of each current task is triggered according to the historical model and the start time data, and the monitoring is continued; A detection control module, when a single task is completed, the detection control module sends the end time data of the task through the data monitoring system to determine whether the execution time of the task is within the threshold range. If so, training is performed according to the start time data and end time data of the task, the model of the single task is updated, and then the next task is continued. If not, an alarm is generated; The single-task predicted duration is a middle value of the threshold range.

12. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method for monitoring the time of batch processing tasks as claimed in any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the batch processing task time monitoring method according to any one of claims 1 to 10 is implemented.

Images