Batch task time consumption early warning method and device, medium and program product

Abstract

The invention discloses a batch task time consumption early warning method and device, a medium and a program product, and the method comprises the steps: firstly obtaining the historical time sequence data of batch tasks, taking the execution starting time and task time consumption of the batch tasks as a timestamp and a time sequence value, and generating a time sequence corresponding to the batch tasks; multi-dimensional analysis processing is carried out on the time sequence corresponding to the batch tasks to obtain a multi-dimensional analysis result, and the multi-dimensional analysis comprises concept drift analysis, holiday influence analysis and regular high point analysis; according to a multi-dimensional analysis result, predicting a task time consumption early warning threshold value of each task node of the batch tasks; and finally, performing task time consumption early warning in the subsequent execution process of the batch tasks by utilizing the task time consumption early warning threshold. According to the invention, through multi-dimensional analysis, the time-consuming early-warning threshold values of the batch tasks are automatically and efficiently provided, the accuracy of the threshold values is effectively improved, and the labor cost of manually setting a large number of threshold values is reduced.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to a batch task time-consuming early warning method, equipment, medium and program product. Background Art

[0002] In banking systems, batch execution of tasks is a common way of large-scale data processing, which often occurs in the following scenarios: batch transfer, batch payroll, batch account opening, batch query, and batch report generation. In these scenarios, banks will batch process a large number of tasks to improve processing efficiency, reduce system resource usage, and reduce labor costs. Through batch processing, banks can process a large number of requests at the same time, improve operational efficiency and accuracy, and thus meet customer needs.

[0003] For a good batch task execution system, in addition to excellent batch task design and task scheduling design in the early stage, a task time-consuming warning system is essential. In banking business, batch tasks often involve a large amount of data processing and complex business logic, so the execution time of individual tasks may be long. In order to promptly detect abnormal task execution time or timeout, banks need a task time-consuming warning system that can monitor the execution time of tasks and send an alarm to notify relevant personnel when the task execution time exceeds the preset threshold. In this way, banks can take timely measures, such as optimizing task design, increasing resource allocation, etc., to ensure that tasks can be completed on time.

[0004] As the number of tasks increases, manually setting time thresholds for all tasks will consume a lot of manpower costs.

[0005] At present, the existing technical solutions for providing time-consuming thresholds mostly perform anomaly detection on time series from a single dimension, and use the results of anomaly detection to automatically set the threshold. The defects of this method are: 1. The rationality of the threshold is directly affected by the anomaly detection results; 2. The threshold often does not change periodically over time, and has poor flexibility; 3. It is difficult to interact with domain experts in design, and the display results are single; 4. The task time consumption may occur according to actual business needs, such as concept drift, data missing and other abnormal situations. For the anomaly detection method using a neural network structure, the interpretability is poor, the training occupies a lot of resources, and the data quality requirements are high. It is difficult to control the effect of the model on the validation set, which may lead to the problem of over-adjustment. If the system adjusts the changes in task execution time too sensitively, the time-consuming threshold may be changed frequently, causing unnecessary interference and warnings to business personnel.

[0006] Therefore, it is urgent to invent a new batch task time-consuming warning method to solve the problem that the existing batch task time-consuming warning technology requires manual setting of a large number of warning thresholds and the poor accuracy of the warning thresholds. Summary of the invention

[0007] In view of this, embodiments of the present invention provide a batch task time-consuming warning method, device, medium and program product, which at least partially solve the problems existing in the prior art.

[0008] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0009] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0010] According to a first aspect of an embodiment of the present invention, a batch task time-consuming warning method is provided, the method comprising:

[0011] Obtain historical time series data of batch tasks, where the historical time series data includes the start execution time and the task duration;

[0012] The start execution time of the batch task and the task duration are used as the timestamp and the timing value respectively to generate a time series corresponding to the batch task;

[0013] Performing multi-dimensional analysis on the time series corresponding to the batch tasks to obtain multi-dimensional analysis results, wherein the multi-dimensional analysis includes concept drift analysis, holiday impact analysis, and regularity high point analysis;

[0014] According to the multi-dimensional analysis results, the task time warning threshold of each task node of the batch task is predicted;

[0015] The task time consumption warning threshold is used to issue a task time consumption warning during the subsequent execution of batch tasks.

[0016] Furthermore, the concept drift analysis includes:

[0017] Generate an ordered array according to the time series corresponding to the batch task, including:

[0018] Extracting the timestamps in the time series and performing difference calculation to obtain a first ordered array representing the task triggering interval;

[0019] Extracting the time series value in the time series to generate a second ordered array representing the time consumption of the task;

[0020] For the first ordered array and the second ordered array, determining whether the array length is greater than a first preset length;

[0021] If the length of the array is greater than the first preset length, the array is segmented using the sliding time window according to the preset sliding step length to obtain segmented sub-arrays;

[0022] Based on the ruptures algorithm, a concept drift recognition process is performed on each sub-array to obtain a first concept drift recognition result corresponding to each sub-array;

[0023] The first concept drift recognition results are spliced ​​to obtain a segmented time series based on concept drift;

[0024] If the length of the array is less than or equal to the first preset length, a concept drift recognition process is performed on the array based on the ruptures algorithm to obtain a second concept drift recognition result;

[0025] According to the second concept drift recognition result, a segmented time series based on concept drift is obtained.

[0026] Furthermore, the multi-dimensional analysis also includes task triggering interval analysis;

[0027] The task trigger interval analysis includes:

[0028] Taking the last drift data of the segmented time series as the first time series to be analyzed;

[0029] Extracting the timestamps in the first time series to be analyzed and performing difference calculation to obtain trigger interval data;

[0030] For each preset interval type, calculating the interval difference between the trigger interval data after unit conversion and the discrete points set by the preset interval type;

[0031] For each preset tolerance error, determining whether the interval difference is less than or equal to the preset tolerance error;

[0032] If the interval difference is less than or equal to the preset tolerance error, the trigger interval data corresponding to the interval difference is used as a nearby point of the discrete point;

[0033] If the interval difference is greater than the preset tolerance error, the trigger interval data corresponding to the interval difference is not used as a nearby point of the discrete point;

[0034] According to all the nearby points, a nearby rate of the preset interval type under the preset tolerance error is calculated;

[0035] Determining whether the proximity rate is greater than a preset proximity rate threshold;

[0036] If the proximity rate is greater than a preset proximity rate threshold, the preset interval type under the preset tolerance error is used as a passing interval type;

[0037] If the proximity rate is less than or equal to a preset proximity rate threshold, the preset interval type under the preset tolerance error is not used as a passing interval type;

[0038] Determine whether the number of types of passed intervals is greater than or equal to 1;

[0039] If the number of the passing interval types is greater than or equal to 1, then selecting the passing interval type with the smallest preset tolerance error from all the passing interval types as the optimal interval type corresponding to the time series, and obtaining the task triggering interval analysis result;

[0040] If the number of the passing interval types is less than 1, the corresponding interval type cannot be matched for the time series.

[0041] Furthermore, the multi-dimensional analysis also includes task time-consuming periodicity analysis;

[0042] The task time-consuming periodic analysis includes:

[0043] Using the last drift data of the segmented time series as the second time series to be analyzed;

[0044] Performing a seasonal difference test on the second time series to be analyzed using preset periods of different lengths to obtain a seasonal offset that can best predict out-of-sample changes;

[0045] Determining whether the second time series to be analyzed has periodicity according to the seasonal offset;

[0046] If the second time series to be analyzed is periodic, obtaining the main period corresponding to the second time series to be analyzed according to the seasonal offset;

[0047] Determine whether the length of the main cycle is greater than the first preset cycle length and the length of the main cycle is not a multiple of the second preset cycle length;

[0048] If the length of the main cycle is greater than the length of the first preset cycle and the length of the main cycle is not a multiple of the length of the second preset cycle, then the second time series to be analyzed has no periodicity;

[0049] If the length of the main period is less than or equal to the first preset period length and / or the length of the main period is a multiple of the second preset period length, the main period is used as the periodicity analysis result of the time series;

[0050] If the second time series to be analyzed is not periodic, the result of the task time consumption periodicity analysis is that it is not periodic.

[0051] Furthermore, the holiday impact analysis includes:

[0052] The last drift data of the segmented time series is used as the third time series to be analyzed;

[0053] Performing grouping processing on the third time series to be analyzed, and obtaining holiday data and other data by grouping, wherein the holiday data includes Saturday data, Sunday data, and national statutory holiday data;

[0054] Based on a preset hypothesis testing method, the holiday data and the other data are used to perform hypothesis testing to obtain a hypothesis testing result, wherein the preset hypothesis testing method includes a KS test, an ES test, and a T test;

[0055] According to the hypothesis test result, it is determined whether holidays have an impact on the task duration of batch tasks, and the time sequence points of task duration affected by holidays in the segmented time series are taken as holiday time sequence points to obtain holiday impact analysis results.

[0056] Furthermore, the regularity high point analysis includes:

[0057] For the time series, using a sliding window, calculate the percentile value of each time point in the time series in the sliding window;

[0058] According to the box plot method, the time series points whose percentile values ​​are higher than the preset percentile threshold are marked as abnormal points, and the abnormal point set is obtained;

[0059] Create a set of rules;

[0060] For each preset rule in the rule set, a periodic high point set is obtained from the abnormal point set according to the preset rule;

[0061] Obtaining a score corresponding to the preset rule according to the quantile average of each point in the periodic high point set in the segmented time series;

[0062] According to the scores corresponding to the preset rules, a preset number of preset rules with the highest scores are taken as the first top rules;

[0063] Perform fuzzy matching on the first top rule to obtain a second top rule;

[0064] Based on a preset post-processing rule, post-process the second top rule, discard the second top rule that does not meet the requirements, and obtain a third top rule;

[0065] The points corresponding to the periodic high point set corresponding to the third top regularity in the time series are taken as regular high points.

[0066] Furthermore, based on the multi-dimensional analysis results, the task time warning thresholds of each task node of the batch task are predicted, including:

[0067] According to the elimination points provided by the user, the corresponding points in the segmented time series are deleted to obtain the eliminated segmented time series;

[0068] According to the multi-dimensional analysis result, the time points in the eliminated segmented time series are divided into ordinary time points, holiday time points and regularity high points;

[0069] For all time series points, the last drift data of the removed segmented time series is used as the predicted time series;

[0070] The forecast time series is processed for trend forecasting using the RANSA robust linear regression algorithm, and an inner point set and an outer point set are recorded;

[0071] According to the upper and lower extreme values ​​of the interior point set and the standard deviation of the interior point set data, the first upper and lower thresholds are obtained;

[0072] Determine whether the external point set data is empty;

[0073] If the external point set data is empty, the first upper and lower thresholds are used as the second upper and lower thresholds;

[0074] If the external point set data is not empty, then the external point threshold is obtained according to the external point set extreme value and the external point set data standard deviation;

[0075] In a preset external point influence time period, the first upper and lower thresholds and the external point thresholds are used to perform a union process to obtain second upper and lower thresholds of the timing point;

[0076] For each common timing point, the second upper and lower thresholds corresponding to the common timing point are used as the first warning threshold of the common timing point;

[0077] For each regular high point, a corresponding high point prediction threshold is set for the regular high point;

[0078] According to the preset threshold merging rule, the second upper and lower thresholds corresponding to the regular high point and the high point prediction threshold are used for merging processing to obtain the first warning threshold of the regular high point;

[0079] For each holiday timing point, a corresponding holiday prediction threshold is set for the holiday timing point;

[0080] According to the preset threshold merging rule, the second upper and lower thresholds corresponding to the holiday timing point and the holiday prediction threshold are used to perform merging processing to obtain the first warning threshold of the holiday timing point;

[0081] Performing a first adjustment process on the first warning threshold according to the false alarm point fed back by the user to obtain a second warning threshold;

[0082] Based on the upper threshold and / or lower threshold additionally set by the user according to the Blocky rule, performing a second adjustment process on the second warning threshold to obtain a third warning threshold;

[0083] Based on the upper threshold and / or lower threshold additionally set by the user according to the manual rule, a third adjustment process is performed on the third warning threshold to obtain a fourth warning threshold.

[0084] According to a second aspect of an embodiment of the present invention, a batch task time-consuming warning device is provided, the device comprising: a processor and a memory;

[0085] The memory is used to store one or more program instructions;

[0086] The processor is used to run one or more program instructions to execute the steps of a batch task time-consuming warning method as described in any one of the above items.

[0087] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a batch task time-consuming warning method as described in any one of the above items are implemented.

[0088] According to a fourth aspect of an embodiment of the present invention, a computer program product is provided, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer implements the steps of a batch task time-consuming warning method as described in any one of the above items.

[0089] The present invention discloses a batch task time-consuming warning method, equipment, medium and program product, the method comprising: firstly obtaining the historical time series data of the batch task, taking the start execution time and task time of the batch task as the timestamp and time series value respectively, generating the time series corresponding to the batch task; then performing multi-dimensional analysis and processing on the time series corresponding to the batch task to obtain the multi-dimensional analysis result, the multi-dimensional analysis includes concept drift analysis, holiday impact analysis and regularity high point analysis; according to the multi-dimensional analysis result, predicting the task time-consuming warning threshold of each task node of the batch task; finally using the task time-consuming warning threshold, performing the task time-consuming warning in the subsequent execution process of the batch task. The present invention realizes the automatic and efficient provision of the batch task time-consuming warning threshold through multi-dimensional analysis, effectively improves the accuracy of the threshold, and reduces the manpower cost of manually setting a large number of thresholds. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0091] Figure 1 A flow chart of a batch task time-consuming warning method provided by an embodiment of the present invention;

[0092] Figure 2 A schematic block diagram of the process of concept drift analysis provided by an embodiment of the present invention;

[0093] Figure 3 A schematic block diagram of the process of task trigger interval analysis provided by an embodiment of the present invention;

[0094] Figure 4 A schematic block diagram of the process of periodic analysis of task time consumption provided by an embodiment of the present invention;

[0095] Figure 5 A schematic flowchart of the holiday impact analysis process provided by an embodiment of the present invention;

[0096] Figure 6 A schematic block diagram of the process of regular high point analysis provided by an embodiment of the present invention;

[0097] Figure 7 A schematic block diagram of the principles of a batch task time-consuming early warning method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0098] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0099] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0100] Figure 1 The figure shows a flow chart of a batch task time-consuming warning method according to an embodiment of the present invention.

[0101] like Figure 1 As shown, the batch task time-consuming warning method according to an embodiment of the present invention may include step S100, step S200, step S300, step S400 and step S500.

[0102] In step S100, historical time series data of batch tasks are obtained, where the historical time series data includes the start execution time and the task duration.

[0103] Specifically, the above steps include:

[0104] In the batch task execution scenario, tasks executed at different time points are classified according to the system name and task name to which each batch task belongs. All comprehensive multi-dimensional analysis and automatic threshold algorithms are based on the time series data composed of such batch tasks as the smallest unit for analysis.

[0105] The historical time series data of batch tasks with the same system name and the same task name are extracted, wherein the historical time series data includes the character start execution time and the task duration.

[0106] Next, in step S200 , the start execution time of the batch task and the task duration are used as a timestamp and a time series value respectively to generate a time series corresponding to the batch task.

[0107] Specifically, the above steps include:

[0108] The historical time series data of batch tasks are sorted in ascending order according to the start execution time of the task, and the start execution time and task duration are used as timestamps and time series values ​​respectively to generate a one-dimensional time series.

[0109] Then, in step S300, a multi-dimensional analysis process is performed on the time series corresponding to the batch tasks to obtain a multi-dimensional analysis result. The multi-dimensional analysis includes concept drift analysis, holiday impact analysis, and regularity high point analysis.

[0110] Specifically, refer to Figure 2 , the above concept drift analysis includes:

[0111] First, extract the timestamps in the time series to perform difference calculation to obtain a first ordered array representing the task triggering interval; extract the timing values ​​in the time series to generate a second ordered array representing the task time consumption.

[0112] In order to reduce the concept drift time of a single time series, for the first ordered array and the second ordered array, it is determined whether the array length is greater than a first preset length (1500 points).

[0113] If the array length is greater than the first preset length, the original array is segmented using the sliding time window according to the preset sliding step (500) to obtain sub-arrays with a sliding step of 500 and an overlapping length of 1000. Based on the ruptures change point penalty detection algorithm, concept drift recognition is performed on each segmented sub-array to obtain the first concept drift recognition result corresponding to each sub-array.

[0114] The first concept drift recognition results are spliced. In order to prevent the algorithm from making inconsistent judgments on the same original point under different input data, the new recognition points with a time stamp interval of less than 2 days from the previous recognition point in the time series are deleted in ascending order of timestamps during the splicing process, and finally a segmented time series based on concept drift is obtained.

[0115] If the array length is less than or equal to the first preset length, the array is subjected to concept drift recognition processing based on the ruptures change point penalty detection algorithm to obtain a second concept drift recognition result, and then a segmented time series based on concept drift is obtained according to the second concept drift recognition result.

[0116] Among them, change point penalty detection is an algorithm for detecting concept drift and realizing signal segmentation. Based on the signal mutation point detection algorithm, the signal can be segmented into different paragraphs or mutation points. The principle of this algorithm is based on the dynamic programming method. For a given model and penalty level, the segmentation that minimizes the constrained approximation error and is calculated.

[0117] Since concept drift often occurs in the trigger interval and task time during the execution of batch tasks, common causes of concept drift include changes in business processes, technology upgrades or replacements, and increased data volume. The embodiment of the present invention uses the last segment of the segmented time series based on concept drift for subsequent time series analysis, which can significantly enhance the effect of the threshold prediction algorithm.

[0118] Specifically, refer to Figure 5 The above holiday impact analysis includes:

[0119] The last drift data of the above segmented time series is taken as the third time series to be analyzed.

[0120] The third time series to be analyzed is grouped to obtain holiday data and other data, wherein the holiday data includes Saturday data, Sunday data and national statutory holiday data.

[0121] Based on the preset hypothesis testing method, holiday data and other data are used to perform hypothesis testing to determine whether there are differences between the two groups of data. The maximum p-value of the hypothesis test is taken as the hypothesis test result. The above-mentioned preset hypothesis testing methods include KS test (two-sample Kolmogorov-Smirnov test), ES test (Epps-Singleton) and T test (T-test).

[0122] If the p-value of the hypothesis test result is less than 0.01, it means that holidays have an impact on the task duration of batch tasks. In the segmented time series, the time points where the task duration is affected by holidays are taken as holiday time points to obtain the holiday impact analysis results.

[0123] In order to solve the problem that the time consumption of some batch tasks on holidays may be different from that on weekdays, the embodiment of the present invention analyzes the impact of holidays to find out the timing points of tasks that will be affected by holidays, thereby helping to enhance business personnel's summary of the time consumption patterns of a certain type of batch tasks, and will also help subsequent warning threshold predictions to provide adaptive and more accurate upper and lower thresholds based on date characteristics.

[0124] Specifically, refer to Figure 6 , the above regular high point analysis includes:

[0125] From a business perspective, the time consumption of some batch tasks will appear high points on regular dates, and the time consumption performance will be different from the usual time consumption, but logically it is not an abnormal point. Therefore, it is necessary to automatically identify multiple groups of possible regular high points.

[0126] First, for the time series of batch tasks, the sliding window is used to calculate the percentile value of each time point in the time series in the sliding window. Then, according to the box plot method, the time points with percentile values ​​higher than the preset percentile threshold are marked as outliers, and the outlier point set A is obtained.

[0127] Based on all possible rules, create a rule set, the rule types include: a. Week x every n weeks, , b. The xth day of every m months, , c. The xth week y of every m months, .

[0128] For each preset rule in the rule set, the points corresponding to the abnormal point set in the preset rule are screened from back to front according to the time sequence. If more than 1 / 2 of the first k points in the preset rule are not in the abnormal point set A, and k>=3, the rule is abandoned in advance. Otherwise, until there are 4 consecutive points in the preset rule that are not in the abnormal point set, the sequence before these 4 points is taken to determine whether the sequence is less than 3. If so, the sequence is abandoned. If greater than or equal to 3, a periodic high point set is obtained based on the sequence, and the average value of the quantile of each point in the point set in its concept drift is taken as the score corresponding to the periodic high point set corresponding to the preset rule.

[0129] The preset rules are sorted from high to low according to the scores, and the rules with the rule type of "the xth week y of every m months" and the point set length less than or equal to 4 are discarded.

[0130] The periodic high point sets with subset relationship are fused. For example, the point set every 14 days is a subset of the point set every 7 days, and the fusion retains the subset of the point set every 7 days.

[0131] After filtering and fusion, the top ten preset rules with the highest scores are taken as the first top rules.

[0132] Perform fuzzy matching on the first top rule mentioned above. For regular cycles of more than one month, determine whether there is a higher point within the error range (two days before and after) and connect it in. For regular cycles of less than one month, determine whether there is a higher point within the error range (one day before and after) and connect it in to obtain the second top rule.

[0133] Based on the preset post-processing rules, the second top rule is post-processed, the rules that do not meet the requirements are discarded, and the third top rule is obtained.

[0134] The above-mentioned preset post-processing rules include discarding rules and merging rules.

[0135] The above discarding rules are to discard the point sets that meet the following rules: i. The number of point sets is 3 but the interval is weekly; ii. The number of point sets is 3 but the interval is less than 3 unit regular periods (such as 2 months, 2 weeks); iii. After excluding missing points, the number is less than 3; iv. There are only 4 points but there is more than one non-abnormal point; v. The number of time series points above the linear piecewise function composed of adjacent points exceeds 10 and the number of time series points above in the data of the past six months exceeds 5; vi. More than 30% of the linear segments rise / fall more than 75% of the overall amplitude and the number of points with the percentage of peripheral points below 80% exceeds 30% of the total number of point sets.

[0136] The above fusion rule is to fuse the point sets that meet the following rules: i. In different periodic properties, point set Q is completely covered by point set M (one point error is allowed); ii. In different periodic properties, the latest three or more consecutive points of point set Q are in point set M.

[0137] Through the above steps, at most 10 groups and at least 0 groups of different third top rules are obtained, and the points corresponding to the periodic high point sets corresponding to the third top rules in the time series are taken as regular high points.

[0138] The embodiment of the present invention effectively enhances the business personnel's summary of the time-consuming regularity of a certain type of batch tasks by analyzing the regular high points of the batch task time series, and also helps the subsequent warning threshold prediction to provide adaptive and more accurate upper and lower thresholds based on the characteristics of the regular high points.

[0139] Preferably, the above multi-dimensional analysis also includes task triggering interval analysis.

[0140] refer to Figure 3 , the above task trigger interval analysis includes:

[0141] For batch tasks, the last drift data of the above segmented time series is used as the first time series to be analyzed.

[0142] The timestamps in the first time series to be analyzed are extracted to perform difference calculation to obtain trigger interval data.

[0143] According to the business logic, 10 interval types that meet the main business scenarios and the corresponding data ratios near discrete points are preset. For each preset interval type, the interval difference between the trigger interval data after unit conversion and the discrete points set by the preset interval type is calculated. For example, if the preset interval type is executed once a day, the interval data should be concentrated near discrete point 1 after the unit is converted into unit days. If the preset interval type is executed six times a week, the interval data should be concentrated near the two set discrete points 1 and 2 after the unit is converted into unit days, and the ratio of the number of data distributed near the two discrete points should be close to 5:1.

[0144] The actual triggering events of batch tasks on the same time series are affected by the environment, which results in that after the time series time interval units are converted into unit days, most of them are close to but not equal to the set discrete points. Therefore, preset tolerances of 0.1%, 0.5%, 1%, and 2% before and after the discrete points are provided in segments. For each preset tolerance, it is determined whether the interval difference is less than or equal to the preset tolerance. If the interval difference is less than or equal to the preset tolerance, the trigger interval data corresponding to the interval difference is used as the nearby point of the discrete point. If the interval difference is greater than the preset tolerance, the trigger interval data corresponding to the interval difference is not used as the nearby point of the discrete point.

[0145] Then, based on all nearby points, the proximity rate of the preset interval type under the preset tolerance is calculated, and the calculation formula of the proximity rate is: proximity rate = (proportion of discrete point 1 * number of nearby points of discrete point 1 + proportion of discrete point 2 * number of nearby points of discrete point 2 + ...) / total number of trigger interval data.

[0146] Determine whether the proximity rate is greater than a preset proximity rate threshold (0.9); if the proximity rate is greater than the preset proximity rate threshold, the preset interval type under the preset tolerance is used as a passing interval type; if the proximity rate is less than or equal to the preset proximity rate threshold, the preset interval type under the preset tolerance is not used as a passing interval type.

[0147] Finally, determine whether the number of passing interval types is greater than or equal to 1; if the number of passing interval types is greater than or equal to 1, select the passing interval type with the smallest preset tolerance error from all passing interval types as the optimal interval type corresponding to the time series to obtain the task triggering interval analysis result; if the number of passing interval types is less than 1, the interval type of the time series cannot be determined.

[0148] For batch task time series, the embodiment of the present invention automatically detects whether the intervals between time series timestamps are uniform. If they are uniform, the time interval followed by the interval is returned, such as 'one execution per day' (indicating that such batch tasks are normally triggered once a day), 'two executions per day', 'one execution per week', 'six executions per week', 'four executions per day', etc. If they are not uniform, it is returned that the departure interval cannot be determined.

[0149] Preferably, the above multi-dimensional analysis also includes task time-consuming periodicity analysis.

[0150] refer to Figure 4 ,The above task time-consuming periodic analysis includes:

[0151] The last drift data of the above segmented time series is taken as the second time series to be analyzed.

[0152] Based on the robustness method, the seasonal difference test is performed on the second time series to be analyzed using preset periods of different lengths to obtain the seasonal offset that can best predict out-of-sample changes.

[0153] According to the seasonal offset, determine whether the second time series to be analyzed is periodic; if the second time series to be analyzed is not periodic, the result of the task time-consuming periodic analysis is non-periodic; if the second time series to be analyzed is periodic, estimate the main cycle corresponding to the second time series to be analyzed according to the seasonal offset, and determine whether the length of the main cycle is greater than the first preset cycle length (120 days) and the length of the main cycle is not a multiple of the second preset cycle length (30 days); if so, the main cycle is invalid and the second time series to be analyzed is non-periodic; if not, based on the main cycle, according to the corresponding timestamp in the time series, the periodic analysis result of the time series is obtained, and the above periodic analysis result is in days / months.

[0154] The embodiment of the present invention identifies whether the task time consumption has periodicity by periodicity analysis of the time consumption of batch tasks.

[0155] Next, in step S400, based on the multi-dimensional analysis results, the task time-consuming warning threshold of each task node of the batch task is predicted.

[0156] Specifically, refer to Figure 7 , the above steps include:

[0157] First, according to the elimination points provided by the user, the points corresponding to the elimination points in the segmented time series are deleted to obtain the eliminated segmented time series.

[0158] According to the above multi-dimensional analysis results, the time points in the segmented time series after elimination are divided into ordinary time points, holiday time points and regular high points.

[0159] For any type of time series point, the last drift data of the removed segmented time series is used as the predicted time series.

[0160] Use the RANSA robust linear regressor to perform trend forecasting on the forecast time series, and record the inliers and outliers generated by the RANSA algorithm process.

[0161] Taking the trend prediction point as the center, the basic first upper and lower thresholds are obtained according to the upper and lower extreme values ​​of the interior point set and n times the standard deviation of the interior point set data, and the trend prediction value is provided using the interior point median.

[0162] Then determine whether the outlier set data is empty; if the outlier set data is empty, directly use the first upper and lower thresholds as the second upper and lower thresholds; if the outlier set data is not empty, the outlier threshold is obtained based on the outlier set extreme value and n times the standard deviation of the outlier set data. Within the preset outlier impact time period (the default is 60 days), the first upper and lower thresholds and the outlier threshold are used to perform a union process to obtain the second upper and lower thresholds of the time series point. It should be noted that the default range of the outlier point set's impact on the future threshold is 60 days, that is, the outlier data at point 1 will not affect the data on the 62nd day in the future.

[0163] For each common timing point, the second upper and lower thresholds corresponding to the common timing point are used as the first warning thresholds of the common timing point.

[0164] For each regular high point, if the number of regular high points is greater than 5, the high point prediction threshold corresponding to the regular high point is calculated using the above-mentioned RANSA robust linear regressor. If the number of regular high points is less than or equal to 5, the corresponding high point prediction threshold is set for the regular high point based on common statistical methods. According to the preset threshold merging rule, the second upper and lower thresholds corresponding to the regular high point and the high point prediction threshold are merged to obtain the first warning threshold of the regular high point.

[0165] For each holiday sequence point, if the number of holiday sequence points is greater than 5, the above-mentioned RANSA robust linear regressor is used to calculate the holiday prediction threshold corresponding to the holiday sequence point. If the number of holiday sequence points is less than or equal to 5, the corresponding holiday prediction threshold is set for the holiday sequence point based on common statistical methods. According to the preset threshold merging rule, the second upper and lower thresholds corresponding to the holiday sequence point and the holiday prediction threshold are merged to obtain the first warning threshold of the holiday sequence point.

[0166] The first warning threshold is adaptively adjusted according to the false alarm points reported by users, and the thresholds for all future dates are adjusted, and the upper threshold is increased or the lower threshold is lowered so that no points with the same time consumption level will be alarmed again, thereby obtaining the second warning threshold.

[0167] Based on the upper threshold and / or lower threshold set by the user according to the Blocky rule, the second warning threshold is adjusted to adjust the upper or lower threshold or both of the upper and lower thresholds of all future date thresholds to cover the second warning threshold, thereby obtaining a third warning threshold.

[0168] Finally, based on the upper threshold and / or lower threshold additionally set by the user according to the manual rules, the third warning threshold is subjected to a third adjustment process to adjust the upper or lower or upper and lower thresholds of the corresponding date threshold to obtain a fourth warning threshold.

[0169] The embodiment of the present invention automatically predicts the reasonable thresholds of each task node based on the results of multi-dimensional analysis of batch tasks, manually marked elimination points, manually set upper and lower thresholds, and user feedback on algorithm results. For example, for time series that are affected by holidays, different levels of thresholds are provided for holidays and non-holidays when predicting upper and lower thresholds in the future. For time series with regular high points, the corresponding regular high point dates in future predictions will provide a threshold range with a larger span than the ordinary time-consuming duration threshold. For time series points where users raise false alarms, feedback should be generated to self-adjust the prediction accuracy of subsequent thresholds.

[0170] The embodiment of the present invention adopts a multi-dimensional analysis method to automatically provide a task time warning threshold. Compared with a single-dimensional anomaly detection method, the present invention can more comprehensively consider multiple factors such as task trigger interval, task time periodicity, task time concept drift, task time holiday impact, and task time regularity peaks, thereby setting the warning threshold more accurately.

[0171] Finally, in step S500, the task time warning threshold is used to issue a task time warning during the subsequent execution of the batch task.

[0172] The embodiment of the present invention provides upper and lower thresholds for the next 60 days at most for each time series consisting of the same system name and task name through an automatic threshold algorithm. Exceeding the upper limit of the threshold or falling below the lower limit of the threshold will trigger an alarm prompt.

[0173] In addition, an embodiment of the present invention also provides a batch task time-consuming warning device, which includes: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of a batch task time-consuming warning method as described above.

[0174] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the batch task time-consuming warning method as described above are implemented.

[0175] In addition, an embodiment of the present invention further provides a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the steps of the batch task time-consuming warning method as described above are implemented.

[0176] The batch task time-consuming warning method, device, medium and program product provided by the embodiment of the present invention have the following advantages:

[0177] The rationality of the warning threshold provided by the present invention is no longer directly affected by the results of an anomaly detection method. The present invention can more accurately determine the threshold and reduce the impact of anomaly detection results on the threshold by comprehensively analyzing data from multiple dimensions such as task trigger intervals, task duration periodicity, task duration concept drift, task duration holiday impact, and task duration regularity peaks.

[0178] The warning threshold provided by the present invention can change periodically over time, can detect periodic changes in task duration, and automatically adjust the warning threshold according to the periodic changes, so that the threshold can adapt to changes in task duration in different periods.

[0179] By displaying the multi-dimensional analysis results, the present invention enables experts in the field to more intuitively understand the time consumption of tasks and manually set thresholds as needed, thereby improving interactivity with users.

[0180] Compared with the abnormality detection method using a neural network structure, the present invention can provide better interpretability.

[0181] The present invention reduces the sensitivity of the warning threshold to changes in task execution time, avoids the algorithm from significantly changing the time-consuming warning threshold, and reduces interference and warnings to business personnel.

[0182] The present invention can timely discover abnormal situations of task time consumption through concept drift analysis of task time consumption, identify drift data as normal data processing, reduce the influence of other historical data, thereby avoiding unnecessary warnings and improving the accuracy of warnings.

[0183] In the embodiment of the present invention, the processor may be an integrated circuit chip having the ability to process signals. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware. The storage medium may be a memory, for example, a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM).The storage medium described in the embodiment of the present invention is intended to include but is not limited to these and any other suitable types of memory. Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by a combination of hardware and software. When the software is applied, the corresponding function can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication medium includes any medium that is convenient for transmitting a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access. Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to it on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed in the present invention.

[0184] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Those skilled in the art may make some simple modifications, equivalent changes or modifications using the technical contents disclosed above, which all fall within the protection scope of the present invention.

Claims

1. A batch task time-consuming warning method, characterized in that: The method comprises: Obtain historical time series data of batch tasks, where the historical time series data includes the start execution time and the task duration; The start execution time of the batch task and the task duration are used as the timestamp and the timing value respectively to generate a time series corresponding to the batch task; Performing multi-dimensional analysis on the time series corresponding to the batch tasks to obtain multi-dimensional analysis results, wherein the multi-dimensional analysis includes concept drift analysis, holiday impact analysis, and regularity high point analysis; According to the multi-dimensional analysis results, the task time warning threshold of each task node of the batch task is predicted; The task time consumption warning threshold is used to issue a task time consumption warning during the subsequent execution of batch tasks.

2. A batch task time-consuming warning method according to claim 1, characterized in that: The concept drift analysis includes: Generate an ordered array according to the time series corresponding to the batch task, including: Extracting the timestamps in the time series and performing difference calculation to obtain a first ordered array representing the task triggering interval; Extracting the time series value in the time series to generate a second ordered array representing the time consumption of the task; For the first ordered array and the second ordered array, determining whether the array length is greater than a first preset length; If the length of the array is greater than the first preset length, the array is segmented using the sliding time window according to the preset sliding step length to obtain segmented sub-arrays; Based on the ruptures algorithm, a concept drift recognition process is performed on each sub-array to obtain a first concept drift recognition result corresponding to each sub-array; The first concept drift recognition results are spliced ​​to obtain a segmented time series based on concept drift; If the length of the array is less than or equal to the first preset length, a concept drift recognition process is performed on the array based on the ruptures algorithm to obtain a second concept drift recognition result; According to the second concept drift recognition result, a segmented time series based on concept drift is obtained.

3. A batch task time-consuming warning method according to claim 2, characterized in that: The multi-dimensional analysis also includes task triggering interval analysis; The task trigger interval analysis includes: Taking the last drift data of the segmented time series as the first time series to be analyzed; Extracting the timestamps in the first time series to be analyzed and performing difference calculation to obtain trigger interval data; For each preset interval type, calculating the interval difference between the trigger interval data after unit conversion and the discrete points set by the preset interval type; For each preset tolerance error, determining whether the interval difference is less than or equal to the preset tolerance error; If the interval difference is less than or equal to the preset tolerance error, the trigger interval data corresponding to the interval difference is used as a nearby point of the discrete point; If the interval difference is greater than the preset tolerance error, the trigger interval data corresponding to the interval difference is not used as a nearby point of the discrete point; According to all the nearby points, a nearby rate of the preset interval type under the preset tolerance error is calculated; Determining whether the proximity rate is greater than a preset proximity rate threshold; If the proximity rate is greater than a preset proximity rate threshold, the preset interval type under the preset tolerance error is used as a passing interval type; If the proximity rate is less than or equal to a preset proximity rate threshold, the preset interval type under the preset tolerance error is not used as a passing interval type; Determine whether the number of types of passed intervals is greater than or equal to 1; If the number of the passing interval types is greater than or equal to 1, then selecting the passing interval type with the smallest preset tolerance error from all the passing interval types as the optimal interval type corresponding to the time series, and obtaining the task triggering interval analysis result; If the number of the passing interval types is less than 1, the corresponding interval type cannot be matched for the time series.

4. A batch task time-consuming warning method according to claim 2, characterized in that: The multi-dimensional analysis also includes task time-consuming periodicity analysis; The task time-consuming periodic analysis includes: Using the last drift data of the segmented time series as the second time series to be analyzed; Performing a seasonal difference test on the second time series to be analyzed using preset periods of different lengths to obtain a seasonal offset that can best predict out-of-sample changes; Determining whether the second time series to be analyzed has periodicity according to the seasonal offset; If the second time series to be analyzed is periodic, obtaining the main period corresponding to the second time series to be analyzed according to the seasonal offset; Determine whether the length of the main cycle is greater than the first preset cycle length and the length of the main cycle is not a multiple of the second preset cycle length; If the length of the main cycle is greater than the length of the first preset cycle and the length of the main cycle is not a multiple of the length of the second preset cycle, then the second time series to be analyzed has no periodicity; If the length of the main period is less than or equal to the first preset period length and / or the length of the main period is a multiple of the second preset period length, the main period is used as the periodicity analysis result of the time series; If the second time series to be analyzed is not periodic, the result of the task time consumption periodicity analysis is that it is not periodic.

5. A batch task time-consuming warning method according to claim 2, characterized in that: The holiday impact analysis includes: The last drift data of the segmented time series is used as the third time series to be analyzed; Performing grouping processing on the third time series to be analyzed, and obtaining holiday data and other data by grouping, wherein the holiday data includes Saturday data, Sunday data, and national statutory holiday data; Based on a preset hypothesis testing method, the holiday data and the other data are used to perform hypothesis testing to obtain a hypothesis testing result, wherein the preset hypothesis testing method includes a KS test, an ES test, and a T test; According to the hypothesis test result, it is determined whether holidays have an impact on the task duration of batch tasks, and the time sequence points of task duration affected by holidays in the segmented time series are taken as holiday time sequence points to obtain holiday impact analysis results.

6. A batch task time-consuming warning method according to claim 5, characterized in that: The regular high point analysis includes: For the time series, using a sliding window, calculate the percentile value of each time point in the time series in the sliding window; According to the box plot method, the time series points whose percentile values ​​are higher than the preset percentile threshold are marked as abnormal points, and the abnormal point set is obtained; Create a set of rules; For each preset rule in the rule set, a periodic high point set is obtained from the abnormal point set according to the preset rule; Obtaining a score corresponding to the preset rule according to the quantile average of each point in the periodic high point set in the segmented time series; According to the scores corresponding to the preset rules, a preset number of preset rules with the highest scores are taken as the first top rules; Perform fuzzy matching on the first top rule to obtain a second top rule; Based on a preset post-processing rule, post-process the second top rule, discard the second top rule that does not meet the requirements, and obtain a third top rule; The points corresponding to the periodic high point set corresponding to the third top regularity in the time series are taken as regular high points.

7. A batch task time-consuming warning method according to claim 6, characterized in that: According to the multi-dimensional analysis results, the task time warning thresholds of each task node of the batch task are predicted, including: According to the elimination points provided by the user, the corresponding points in the segmented time series are deleted to obtain the eliminated segmented time series; According to the multi-dimensional analysis result, the time points in the eliminated segmented time series are divided into ordinary time points, holiday time points and regularity high points; For all time series points, the last drift data of the removed segmented time series is used as the predicted time series; The forecast time series is processed for trend forecasting using the RANSA robust linear regression algorithm, and an inner point set and an outer point set are recorded; According to the upper and lower extreme values ​​of the interior point set and the standard deviation of the interior point set data, the first upper and lower thresholds are obtained; Determine whether the external point set data is empty; If the external point set data is empty, the first upper and lower thresholds are used as the second upper and lower thresholds; If the external point set data is not empty, then the external point threshold is obtained according to the external point set extreme value and the external point set data standard deviation; In a preset external point influence time period, the first upper and lower thresholds and the external point thresholds are used to perform a union process to obtain second upper and lower thresholds of the timing point; For each common timing point, the second upper and lower thresholds corresponding to the common timing point are used as the first warning threshold of the common timing point; For each regular high point, a corresponding high point prediction threshold is set for the regular high point; According to the preset threshold merging rule, the second upper and lower thresholds corresponding to the regular high point and the high point prediction threshold are used for merging processing to obtain the first warning threshold of the regular high point; For each holiday timing point, a corresponding holiday prediction threshold is set for the holiday timing point; According to the preset threshold merging rule, the second upper and lower thresholds corresponding to the holiday timing point and the holiday prediction threshold are used to perform merging processing to obtain the first warning threshold of the holiday timing point; Performing a first adjustment process on the first warning threshold according to the false alarm point fed back by the user to obtain a second warning threshold; Based on the upper threshold and / or lower threshold additionally set by the user according to the Blocky rule, performing a second adjustment process on the second warning threshold to obtain a third warning threshold; Based on the upper threshold and / or lower threshold additionally set by the user according to the manual rule, a third adjustment process is performed on the third warning threshold to obtain a fourth warning threshold.

8. A batch task time-consuming warning device, characterized in that: The device comprises: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of a batch task time-consuming warning method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a batch task time-consuming warning method as described in any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product includes computer program instructions, which, when executed by a processor, implement the steps of a batch task time-consuming warning method as described in any one of claims 1 to 7.

Images