Batch job completion time early warning method, device, equipment and medium

Abstract

The invention provides a batch job completion time early warning method which can be applied to the technical field of cloud computing and artificial intelligence. The method comprises the following steps: acquiring execution state information of batch jobs; determining a first predicted completion time range of the to-be-executed job set based on the historical execution data of the to-be-executed jobs, and generating first early warning information according to a comparison result of the first predicted completion time range and the service remaining execution time; acquiring system operation state data, and judging whether the performance of the batch job operation environment is abnormal or not in combination with the actual execution duration of completed jobs; and if the performance is abnormal, correcting the first prediction completion time range based on the deviation between the actual execution duration and the historical data to obtain a second prediction completion time range, and generating second early warning information according to a comparison result between the second prediction completion time range and the service remaining execution time. The invention further provides a batch job completion time early warning device, equipment, a medium and a program product.

Description

Technical Field

[0001] This application relates to the fields of cloud computing and artificial intelligence technology, specifically to a method, apparatus, device, medium, and program product for early warning of batch job completion time. Background Technology

[0002] In critical business systems such as finance and telecommunications, batch jobs must be completed before strict business deadlines to ensure accounting accuracy and data consistency. However, their actual execution is susceptible to dynamic factors such as upstream dependency delays and system resource performance fluctuations. Without accurate and forward-looking dynamic prediction and early warning of the overall completion time of batch jobs, it will be difficult to identify potential timeout risks in a timely manner, leading to delayed operational intervention. Therefore, improving the accuracy of batch job completion time warnings has become a pressing technical problem to be solved in this field. Summary of the Invention

[0003] In view of the above problems, this application provides a method, apparatus, equipment, medium and program product for early warning of batch operation completion time, which aims to improve the accuracy of early warning.

[0004] According to a first aspect of this application, a batch job completion time early warning method is provided, comprising: acquiring execution status information of batch jobs, the execution status information including the current time, the business deadline, a set of completed jobs, and a set of jobs to be executed, wherein the set of completed jobs includes the actual execution time of each completed job, and the set of jobs to be executed includes the historical execution data of each job to be executed; determining a first predicted completion time range for the set of jobs to be executed based on the historical execution data of each job in the set of jobs to be executed, and generating a first early warning message based on the comparison result between the first predicted completion time range and the remaining execution time of the business, wherein the remaining execution time of the business is determined based on the comparison between the current time and the business deadline. The time point is determined, and the first early warning information is used to indicate whether the batch job can be completed before the business deadline. The system operation status data of the batch job running environment is obtained, and combined with the actual execution time in the set of completed jobs, it is determined whether there is a performance anomaly in the batch job running environment that leads to a decrease in overall execution efficiency. If there is a performance anomaly, the first predicted completion time range is corrected based on the deviation between the actual execution time and the corresponding historical execution data to obtain the second predicted completion time range. The second early warning information is generated based on the comparison result between the second predicted completion time range and the remaining execution time of the business. The second early warning information is used to indicate whether the batch job can be completed before the business deadline.

[0005] According to an embodiment of this application, determining the first predicted completion time range of the set of jobs to be executed based on the historical execution data of each job in the set of jobs to be executed includes: constructing a job dependency topology in the form of a directed acyclic graph based on the pre-stored dependency relationships between each batch of jobs; identifying the critical path from the start job to the end job based on the job dependency topology, wherein the critical path is the job path with the longest total execution time; determining the corresponding predicted execution time based on the historical execution data of each job to be executed on the critical path, and accumulating the predicted execution times to obtain the first predicted completion time range.

[0006] According to the embodiments of this application, determining whether there is a performance anomaly in the batch job running environment that leads to a decrease in overall execution efficiency includes: obtaining the historical average execution time corresponding to each job in the completed job set, and determining the average deviation rate of the actual execution time relative to the historical average execution time; if the average deviation rate is greater than a first preset threshold, and at least one resource indicator in the system running status data exceeds the corresponding second preset threshold, then it is determined that there is a performance anomaly.

[0007] According to an embodiment of this application, the resource indicators include at least one of processor utilization, memory utilization, disk read / write latency, and network bandwidth utilization; the second preset threshold is set based on the statistical distribution of the corresponding resource indicators during historical normal operation.

[0008] According to an embodiment of this application, correcting the first predicted completion time range includes: using a static average model or a dynamic time series model, and correcting the first predicted completion time range based on the deviation between the actual execution time and the corresponding historical execution data.

[0009] According to an embodiment of this application, correcting the first predicted completion time range using a static average model includes: calculating the average deviation between the actual execution time of completed tasks and their historical average execution time, and determining the expected increase in total execution time based on the average deviation and the number of tasks to be executed; adding the product of the expected increase in total execution time and a first weighting coefficient to the lower limit of the first predicted completion time range, and adding the product of the expected increase in total execution time and a second weighting coefficient to the upper limit of the first predicted completion time range to obtain a second predicted completion time range; wherein the first weighting coefficient and the second weighting coefficient are obtained by fitting based on historical running data, and the first weighting coefficient is greater than 0 and less than or equal to the second weighting coefficient, and the second weighting coefficient is less than or equal to 1.

[0010] According to an embodiment of this application, the correction of the first predicted completion time range using a dynamic time series model includes: performing exponential smoothing filtering on the execution time deviation sequence of completed jobs to obtain a smoothed deviation trend value; determining the degree of execution efficiency decay of the job to be executed based on the deviation trend value; and adjusting the first predicted completion time range according to the degree of execution efficiency decay to obtain a second predicted completion time range.

[0011] A second aspect of this application provides a batch job completion time early warning device, comprising: an acquisition module, configured to acquire execution status information of batch jobs, the execution status information including the current time, the business deadline, a set of completed jobs, and a set of jobs to be executed, wherein the set of completed jobs includes the actual execution time of each completed job, and the set of jobs to be executed includes the historical execution data of each job to be executed; and a first generation module, configured to determine a first predicted completion time range for the set of jobs to be executed based on the historical execution data of each job in the set of jobs to be executed, and generate first early warning information based on a comparison between the first predicted completion time range and the remaining execution time of the business, wherein the remaining execution time of the business is determined by the comparison between the current time and the business deadline. The system is configured as follows: First, a warning message is used to indicate whether the batch job can be completed before the business deadline; a judgment module is used to obtain system operation status data of the batch job's operating environment and, in conjunction with the actual execution time in the set of completed jobs, determine whether there is a performance anomaly in the batch job's operating environment that would cause a decrease in overall execution efficiency; second, a generation module is used to, in the event of a performance anomaly, correct the first predicted completion time range based on the deviation between the actual execution time and the corresponding historical execution data to obtain a second predicted completion time range, and generate a second warning message based on the comparison result between the second predicted completion time range and the remaining execution time of the business. The second warning message is used to indicate whether the batch job can be completed before the business deadline.

[0012] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0013] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0014] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0015] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0016] Figure 1 The illustration schematically depicts an application scenario of a batch job completion time early warning method, apparatus, device, medium, and program product according to embodiments of this application;

[0017] Figure 2 A flowchart illustrating a batch job completion time warning method according to an embodiment of this application is shown schematically.

[0018] Figure 3 A flowchart illustrating a first prediction completion time range determination method according to an embodiment of this application is shown schematically.

[0019] Figure 4 A flowchart illustrating a method for determining performance anomalies in a batch job execution environment according to an embodiment of this application is shown schematically.

[0020] Figure 5 A flowchart illustrating a first prediction completion time range correction method according to an embodiment of this application is shown schematically.

[0021] Figure 6 This schematic diagram illustrates the structural block diagram of a batch job completion time early warning device according to an embodiment of this application;

[0022] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a batch job completion time warning method according to an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0028] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0029] Figure 1 The illustration schematically depicts an application scenario of a batch job completion time warning method, apparatus, device, medium, and program product according to embodiments of this application.

[0030] like Figure 1As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0031] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various operation and maintenance monitoring client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as alarm push applications, visual dashboards, instant messaging tools, etc. (for example only).

[0032] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and supporting data interaction, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0033] Server 105 can be a server that provides batch job scheduling and monitoring services, such as a batch job management platform in a bank data center or an intelligent operation and maintenance server in a cloud computing environment. This server can collect the execution status of batch jobs and system resource indicators in real time, execute the early warning method of this application, and push the early warning results to the operation and maintenance personnel's terminals.

[0034] It should be noted that the batch job completion time warning method provided in this application embodiment can generally be executed by server 105. Correspondingly, the batch job completion time warning device provided in this application embodiment can generally be set in server 105. The batch job completion time warning method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the batch job completion time warning device provided in this application embodiment can also be set in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0035] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0036] It should be noted that the batch job completion time warning method and apparatus of this application are deeply adaptable to the fields of cloud computing and artificial intelligence technology, and are particularly suitable for critical business systems in banking, securities, telecommunications, and other industries with stringent requirements for batch processing timeliness. In daily transaction scenarios in the financial industry, online transaction data needs to undergo hundreds of batch jobs to generate reconciliation files, complete accounting settlements, and update general ledger account balances. If the job chain times out due to upstream file delays or resource bottlenecks such as server central processing unit (CPU) or disk input / output (I / O), it will directly lead to accounting discrepancies or a decline in customer experience.

[0037] For example, during the end-of-day batch processing of a bank's core system, this method can obtain the actual execution time of currently completed jobs and historical data of pending jobs in real time, preliminarily predict the completion time range, and generate a first warning. At the same time, it monitors the server resource status. When it finds that disk I / O is continuously under high load and jobs are generally slowing down, it determines that there is a performance anomaly. Then, based on the execution deviation of the day, it dynamically corrects the prediction range, generates a more accurate second warning, and pushes an alarm "may not be completed before the end of the day" to the operation and maintenance personnel through the terminal device, so that they can start emergency measures in advance (such as resource expansion and job priority adjustment) to effectively avoid production accidents.

[0038] It should be noted that the batch operation completion time warning method and device of this application embodiment are not only applicable to the financial technology field, but can also be widely applied to any scenario that relies on batch operations and is sensitive to completion time limits, such as e-commerce promotion order settlement, logistics nighttime sorting and scheduling, etc., and its application field is not limited.

[0039] The following will be based on Figure 1 The described scene, through Figures 2-5 The batch job completion time warning method according to the embodiments of this application will be described in detail.

[0040] Figure 2 A flowchart illustrating a batch job completion time warning method according to an embodiment of this application is shown schematically.

[0041] like Figure 2 As shown, the batch job completion time warning method of this embodiment includes operations S210 to S240, and the batch job completion time warning method can be executed by the server 105.

[0042] Operation S210 retrieves the execution status information of batch jobs.

[0043] Execution status information includes the current time, the business deadline, the set of completed jobs, and the set of jobs to be executed. The business deadline can be the daily cutoff time or other critical business time nodes, requiring all jobs to be completed before this time. The set of completed jobs contains the actual execution time of each completed job; this data can be used for subsequent performance anomaly identification and prediction correction. The set of jobs to be executed contains the historical execution data of each job to be executed; this historical data can serve as the basis for initial predictions. As an example, data collection can be implemented through timed polling or event-driven methods to ensure real-time performance and accuracy. For large-scale batch job chains, distributed storage and parallel processing technologies can be used to efficiently manage massive amounts of job data.

[0044] In operation S220, based on the historical execution data of each job in the set of jobs to be executed, a first predicted completion time range for the set of jobs to be executed is determined, and a first warning message is generated based on the comparison result between the first predicted completion time range and the remaining execution time of the business.

[0045] For each pending job, server 105 can extract its historical execution time data from historical records. This data may include, for example, the shortest historical execution time (representing optimal efficiency), the longest historical execution time (representing worst efficiency), or statistical measures such as multi-day averages and quantiles. This data can be filtered by dimensions such as date, business type, and data volume to improve the relevance of the prediction. The first predicted completion time range includes the earliest and latest possible completion times of the batch job chain under current conditions. The shortest historical execution times of all pending jobs are summed to obtain the theoretical earliest possible completion time offset; the longest historical execution times of all pending jobs are summed to obtain the theoretical latest possible completion time offset. Starting from the current time, adding these two offsets respectively yields a first predicted completion time range in interval form.

[0046] The remaining execution time of a task is determined by the current time and the task's deadline. Specifically, the remaining execution time is obtained by subtracting the current time from the task's deadline, and this time can serve as a baseline window for judging whether the task can be completed on time. By comparing the first predicted completion time range with the remaining execution time, a first warning message can be obtained to indicate whether the batch job can be completed before the task's deadline. For example, if even at the fastest execution speed, the task cannot be completed before the deadline (T_min > remaining time), a "cannot complete" warning can be issued; if the task can be completed at the fastest speed but may time out at the slowest speed (T_min ≤ remaining time < T_max), a "may not complete" warning can be issued; if the task can be completed even at the slowest speed (T_max ≤ remaining time), a "expected to be completed on time" message can be displayed. Here, T_min and T_max are the minimum and maximum time values ​​of the first predicted completion time range, respectively. It is important to emphasize that the prediction at this stage assumes that the system operating environment is normal and the execution efficiency is consistent with history, i.e., it does not consider whether there are performance anomalies such as CPU overload or I / O bottlenecks. Therefore, this warning (i.e., the first warning information) is mainly used to deal with the progress delay caused by external dependency delays (such as upstream files not arriving).

[0047] In operation S230, the system running status data of the batch job running environment is obtained, and combined with the actual execution time of the completed job set, it is determined whether there is any performance abnormality in the batch job running environment that leads to a decrease in overall execution efficiency.

[0048] On one hand, server 105 can collect basic resource metrics at the server or container level in real time, reflecting the health status of the underlying infrastructure. On the other hand, server 105 can obtain the actual execution time of completed jobs for the day from the batch scheduling platform and compare it with its historical benchmark to calculate the degree of deviation. Then, by fusing these two types of heterogeneous data—system operation status data and the actual execution performance of completed jobs—server 105 collaboratively determines whether there are performance anomalies affecting overall efficiency in the current batch job execution environment. This determination is a key prerequisite for triggering subsequent dynamic corrections. It is important to emphasize that the object of the determination is not a single job or a single resource, but the overall execution efficiency of the batch job chain.

[0049] In operation S240, in the event of a performance anomaly, the first predicted completion time range is corrected based on the deviation between the actual execution time and the corresponding historical execution data to obtain a second predicted completion time range, and a second warning message is generated based on the comparison result between the second predicted completion time range and the remaining execution time of the business.

[0050] Once operation S230 confirms a performance anomaly causing a decrease in overall execution efficiency, the system no longer relies on the original historical execution data for prediction. Instead, it dynamically adjusts the first predicted completion time range based on the efficiency deviation reflected in the actual performance of completed tasks that day. This generates a second predicted completion time range that more closely reflects the current operating state, and outputs a second warning message accordingly. For example, this deviation can be an absolute deviation, a relative deviation, or a standardized comprehensive deviation index. This deviation reflects the overall degree of degradation in job execution efficiency under the current abnormal environment. Based on this deviation, server 105 can make an overall adjustment to the first predicted completion time range. For example, it can calculate the total additional time that the remaining tasks may consume based on the total deviation, and add this additional time to the upper and lower limits of the original prediction range. The corrected result is the second predicted completion time range, which truly reflects the expected completion window under the current performance anomaly conditions. Then, the second predicted completion time range is compared again with the remaining execution time of the business to generate an updated warning message, i.e., the second warning message. The second warning may be more severe than the first warning, for example, it may be upgraded from "may not be able to complete" to "cannot be completed", or it may maintain the original judgment due to the small deviation. The key is that it has integrated real-time operation performance, so it is more credible.

[0051] It's important to emphasize that in cases of performance anomalies, the second warning can be pushed to relevant personnel as the final warning; conversely, in cases where no performance anomalies exist, the first warning can be pushed to relevant personnel as the final warning. For example, the final warning can be pushed to relevant personnel via SMS, email, monitoring dashboards, or office software robots. Understandably, the first warning is a baseline prediction assuming normal system performance. Its value lies in the fact that if the first warning indicates a timeout, but system performance is normal, the problem may lie in non-resource aspects such as job dependencies or a surge in data volume; it can provide early warnings based on progress delays before performance anomalies occur.

[0052] The batch job completion time early warning method in this embodiment first generates a first warning based on historical data, which can effectively identify risks caused by upstream dependency delays. Then, it combines the system operating status with the actual execution deviation of completed jobs to determine whether there are performance anomalies, and dynamically corrects the prediction range accordingly to generate a second warning that is closer to reality. Because this solution not only focuses on the progress of the job itself, but also incorporates real-time performance changes in the operating environment, it can distinguish the root causes of delays and maintain high prediction accuracy even in complex scenarios such as system resource degradation. This provides maintenance personnel with more decision-making-valuable forward-looking warnings, effectively avoiding the risk of accounting discrepancies or business interruptions caused by batch timeouts.

[0053] Figure 3A flowchart illustrating a first prediction completion time range determination method according to an embodiment of this application is shown.

[0054] like Figure 3 As shown, the first prediction completion time range determination method of this embodiment, namely the above-mentioned operation S220, may include operations S310 to S330.

[0055] In operation S310, a job dependency topology in the form of a directed acyclic graph is constructed based on the pre-stored dependencies between batch jobs.

[0056] For example, all batch jobs and their dependencies can be retrieved from a database. These dependencies can be direct or indirect preconditions, data flow requirements, etc. Then, this information can be used to construct a Directed Acyclic Graph (DAG), where each node represents a single job, and edges represent dependencies between jobs. This graph structure not only clearly shows the logical sequence of the entire batch job chain but also provides a foundation for subsequent critical path identification. By constructing a DAG, complex batch job processes can be effectively managed and visualized, helping to quickly locate and understand the relationships between individual jobs. Furthermore, DAGs can provide an intuitive data structure for automation tools, facilitating further analysis and optimization.

[0057] When operating S320, based on job dependency topology, the critical path from the start job to the end job is identified. The critical path is the job path with the longest total execution time.

[0058] Based on the constructed DAG, algorithms can be used to find all possible job execution paths and calculate the total execution time for each path. Algorithms can include, for example, Kahn's algorithm (for topology sorting) or depth-first search, or they can be combined with dynamic programming to calculate path length. The critical path is the path consisting of a series of consecutive jobs that take the longest time from start to finish in the entire job flow. This path determines the shortest possible completion time for the entire batch job.

[0059] In operation S330, based on the historical execution data of each job to be executed on the critical path, the corresponding predicted execution time is determined, and the predicted execution times are accumulated to obtain the first predicted completion time range.

[0060] For each job to be executed on the critical path, its historical execution records can be referenced, including but not limited to statistics such as average execution time, shortest execution time, and longest execution time. Based on this data, the expected execution time of each job can be estimated. Then, these predicted values ​​are accumulated along the critical path to form a preliminary time estimation interval, i.e., the first predicted completion time range.

[0061] This embodiment identifies critical paths using directed acyclic graphs, which helps to focus on the tasks that have the greatest impact on the overall schedule. This allows operations personnel to monitor and optimize these critical points in a targeted manner, thereby improving resource utilization and task processing efficiency.

[0062] Figure 4 The flowchart illustrates a method for judging performance anomalies in a batch job execution environment according to an embodiment of this application.

[0063] like Figure 4 As shown, the performance anomaly judgment method in the batch job running environment of this embodiment, namely, the above-mentioned operation S230 includes operations S410 to S420.

[0064] In operation S410, the historical average execution time of each job in the completed job set is obtained, and the average deviation rate of the actual execution time relative to the historical average execution time is determined.

[0065] In this embodiment, the historical average execution time of each job in the completed job set under similar business scenarios during the same period can first be extracted from the historical execution database as a benchmark for its normal execution efficiency. Then, the actual execution time of each completed job on the current day is compared with the corresponding historical average to calculate the execution deviation rate of a single job, for example: Execution deviation rate = (Actual time - Historical average time) / Historical average time. Finally, the arithmetic mean of the deviation rates of all completed jobs is taken to obtain the average deviation rate reflecting the overall change in execution efficiency.

[0066] If, during operation S420, the average deviation rate is greater than the first preset threshold, and at least one resource indicator in the system operation status data exceeds the corresponding second preset threshold, then a performance anomaly is determined to exist.

[0067] In this embodiment, the first condition is that the average deviation rate exceeds a preset first threshold, indicating that the operation has generally slowed down significantly; the second condition is that at least one item in the system operation status data continuously exceeds its corresponding second preset threshold. If both conditions are met simultaneously, i.e., the logical relationship is "AND", then a performance anomaly is determined to exist.

[0068] This embodiment significantly improves the accuracy of performance anomaly detection by introducing a dual-condition joint criterion of job execution deviation rate and system resource indicators. Specifically, a performance anomaly causing a decrease in overall efficiency is determined only when the overall execution time of completed jobs significantly deviates from the historical average level (i.e., the average deviation rate exceeds a first preset threshold) and at least one abnormal system resource indicator is detected simultaneously (exceeding a second preset threshold). Because this mechanism requires that job behavior anomalies and system state anomalies occur simultaneously, it effectively eliminates misjudgments caused by isolated factors such as sudden increases in individual job data volume or instantaneous resource fluctuations, resulting in anomaly detection results with high confidence.

[0069] In some embodiments, resource metrics include at least one of processor utilization, memory utilization, disk read / write latency, and network bandwidth utilization. These metrics comprehensively reflect the load status of the batch job execution environment from four core dimensions: computing, storage, I / O, and network. Processor utilization characterizes the busyness of CPU resources; excessively high utilization may lead to job scheduling delays. Excessively high memory utilization may cause frequent paging, affecting job stability. Disk read / write latency directly reflects storage I / O performance bottlenecks; in this paper, disk read / write latency and disk input / output time are interchangeable. Network bandwidth utilization relates to cross-node data transmission efficiency, which is crucial, especially in distributed job environments. In this embodiment, the second preset threshold is not a static constant but is dynamically generated based on the statistical distribution of resource metrics during the system's historical normal operation, such as using the 95th percentile, the mean plus twice the standard deviation, etc., thereby adapting to the normal fluctuation range under different business cycles, hardware configurations, and load modes.

[0070] This embodiment combines multi-dimensional key resource indicators with a dynamic threshold mechanism based on historical statistics, which not only ensures high sensitivity to real performance anomalies but also effectively suppresses false alarms caused by business peaks or environmental differences. Compared with the traditional method that relies on manually configuring fixed thresholds, it significantly improves the adaptability, robustness, and intelligent operation and maintenance level of anomaly detection.

[0071] Figure 5 A flowchart illustrating a first prediction completion time range correction method according to an embodiment of this application is shown.

[0072] like Figure 5 As shown, in the first prediction completion time range correction method of this embodiment, a static average model 501 or a dynamic time series model 502 can be used to correct the first prediction completion time range based on the deviation between the actual execution time and the corresponding historical execution data, so as to obtain the second prediction completion time range.

[0073] In this embodiment, the choice between using a static average model 501 or a dynamic time series model 502 to correct the first prediction completion time range can be made based on the actual operating scenario and data characteristics. Both models use the actual execution deviation as the core input, but they are designed for steady-state and changing situational environments, respectively, providing differentiated correction strategies to balance accuracy and computational efficiency.

[0074] The static average model 501 is suitable for scenarios where job execution deviations are relatively stable and show no obvious trend changes. This model can estimate the overall additional time by calculating the average deviation between the actual execution time of completed jobs and their historical average execution time, and then extending this deviation proportionally to the remaining jobs to be executed. This is then added to the upper and lower limits of the original prediction interval, achieving a simple and efficient correction.

[0075] The Dynamic Time Series Model 502 is suitable for complex scenarios where execution efficiency exhibits a continuous trend of deterioration or improvement. This model treats the execution time deviation of completed tasks as a time series, employs lightweight time series analysis methods such as exponential smoothing and moving average to extract the smoothing trend value of the deviation, and then predicts the degree of execution efficiency decay of subsequent tasks, dynamically adjusting the predicted completion time range accordingly.

[0076] This embodiment introduces two types of correction models, static and dynamic, and supports selection on demand. This enables the early warning system to not only cope with the rapid response requirements under sudden resource fluctuations, but also to capture complex trends such as continuous performance degradation. It significantly improves the predictive robustness and early warning foreseeability of batch job completion time in real operation and maintenance environments, and provides flexible and reliable technical support for intelligent scheduling and risk intervention in different business scenarios.

[0077] In some embodiments, revising the first predicted completion time range using a static average model 501 includes: First, calculating the average deviation between the actual execution time of completed jobs and their historical average execution time. This deviation reflects the degree of difference between the current execution environment and the historical average level. Then, determining the expected increase in total execution time based on this average deviation and the number of jobs to be executed. Next, obtaining a more accurate second predicted completion time range by adding the product of the expected increase in total execution time and a first weighting coefficient to the lower limit of the first predicted completion time range, and adding the same product of the increase in total execution time and a second weighting coefficient to the upper limit. The first and second weighting coefficients are obtained by fitting historical operating data, such as linear regression or historical deviation regression, to ensure the rationality and accuracy of the adjustment. The weighting coefficients can also be obtained by fitting historical deviations and actual timeouts using the least squares method to ensure that the correction magnitude does not excessively amplify noise. The first weighting coefficient is greater than 0 and less than or equal to the second weighting coefficient, while the second weighting coefficient does not exceed 1. For example, by collecting the relationship between the average deviation of completed jobs and the total actual deviation of subsequent jobs to be executed in historical data, a linear regression model is used to determine the first weighting coefficient as 0.7 and the second weighting coefficient as 0.9.

[0078] This embodiment quantifies the execution deviation of completed tasks and estimates the additional time required by combining the number of tasks to be executed. At the same time, it introduces weighting coefficients to adapt to different error tolerances, which effectively improves the accuracy and reliability of the predicted completion time range, and is especially suitable for scenarios where the execution deviation is relatively stable.

[0079] In some embodiments, revising the first predicted completion time range using the dynamic time series model 502 includes: First, applying exponential smoothing filtering to the execution time deviation sequence of completed tasks to eliminate random noise and extract trend components, obtaining smoothed deviation trend values. Based on these trend values, the potential performance degradation of the tasks to be executed can be assessed. For example, the degree of degradation can be a coefficient between 0.8 and 1.2, where 1.0 represents normal performance and a value greater than 1.0 represents decreased efficiency (increased time required). Next, the initial predicted completion time range is adjusted according to the identified efficiency change trend, resulting in a second predicted completion time range that more closely reflects the actual progress.

[0080] This embodiment captures potential trends in execution time rather than relying solely on average deviation, enabling it to more accurately reflect changes and trends during execution. It is particularly suitable for complex scenarios where execution efficiency is unstable or exhibits significant trend changes.

[0081] Based on the above-mentioned batch job completion time early warning method, this application also provides a batch job completion time early warning device. The following will combine... Figure 6The device is described in detail.

[0082] Figure 6 A schematic block diagram of a batch job completion time warning device according to an embodiment of this application is shown.

[0083] like Figure 6 As shown, the batch job completion time early warning device 600 of this embodiment includes an acquisition module 610, a first generation module 620, a judgment module 630, and a second generation module 640.

[0084] The acquisition module 610 can be used to acquire the execution status information of batch jobs. The execution status information includes the current time, the business deadline, the set of completed jobs, and the set of jobs to be executed. The set of completed jobs includes the actual execution time of each completed job, and the set of jobs to be executed includes the historical execution data of each job to be executed. In one embodiment, the acquisition module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0085] The first generation module 620 can be used to determine a first predicted completion time range for the set of jobs to be executed based on the historical execution data of each job in the set of jobs to be executed, and generate a first warning message based on the comparison result between the first predicted completion time range and the remaining execution time of the business. The remaining execution time of the business is determined based on the current time and the business deadline. The first warning message is used to indicate whether the batch jobs can be completed before the business deadline. In one embodiment, the first generation module 620 can be used to execute the operation S220 described above, which will not be repeated here.

[0086] The judgment module 630 can be used to obtain system operation status data of the batch job execution environment and, in conjunction with the actual execution time of the completed job set, determine whether there are performance anomalies in the batch job execution environment that cause a decrease in overall execution efficiency. In one embodiment, the judgment module 630 can be used to execute the operation S230 described above, which will not be repeated here.

[0087] The second generation module 640 can be used to correct the first predicted completion time range based on the deviation between the actual execution time and the corresponding historical execution data in the event of performance anomalies, thereby obtaining a second predicted completion time range. It then generates a second warning message based on a comparison between the second predicted completion time range and the remaining execution time of the business process. The second warning message indicates whether the batch job can be completed before the business deadline. In one embodiment, the second generation module 640 can be used to execute the operation S240 described above, which will not be repeated here.

[0088] According to an embodiment of this application, the first generation module 620 can also be used to construct a job dependency topology in the form of a directed acyclic graph based on the pre-stored dependency relationships between batch jobs; based on the job dependency topology, identify the critical path from the start job to the end job, the critical path being the job path with the longest total execution time; based on the historical execution data of each job to be executed on the critical path, determine the corresponding predicted execution time, and accumulate the predicted execution times to obtain a first predicted completion time range.

[0089] According to an embodiment of this application, the judgment module 630 can also be used to obtain the historical average execution time of each job in the completed job set, and determine the average deviation rate of the actual execution time relative to the historical average execution time; if the average deviation rate is greater than the first preset threshold, and at least one resource indicator in the system operation status data exceeds the corresponding second preset threshold, then it is determined that there is a performance abnormality.

[0090] According to an embodiment of this application, the resource indicators include at least one of processor utilization, memory utilization, disk read / write latency, and network bandwidth utilization; the second preset threshold is set based on the statistical distribution of the corresponding resource indicators during historical normal operation.

[0091] According to an embodiment of this application, the second generation module 640 can also be used to correct the first predicted completion time range based on the deviation between the actual execution time and the corresponding historical execution data by employing a static average model or a dynamic time series model.

[0092] According to an embodiment of this application, the second generation module 640 can also be used to calculate the average deviation between the actual execution time of the completed job and its historical average execution time, and determine the expected increase in total execution time based on the average deviation and the number of jobs to be executed; add the product of the expected increase in total execution time and the first weight coefficient to the lower limit of the first predicted completion time range, and add the product of the expected increase in total execution time and the second weight coefficient to the upper limit of the first predicted completion time range to obtain the second predicted completion time range; wherein the first weight coefficient and the second weight coefficient are obtained by fitting based on historical running data, and the first weight coefficient is greater than 0 and less than or equal to the second weight coefficient, and the second weight coefficient is less than or equal to 1.

[0093] According to an embodiment of this application, the second generation module 640 can also be used to perform exponential smoothing filtering on the execution time deviation sequence of completed jobs to obtain a smoothed deviation trend value; determine the degree of execution efficiency decay of the job to be executed based on the deviation trend value, and adjust the first predicted completion time range according to the degree of execution efficiency decay to obtain a second predicted completion time range.

[0094] According to embodiments of this application, any plurality of the above modules can be combined into one module, or any one of the modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the above modules can be at least partially implemented as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the above modules can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0095] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a batch job completion time warning method according to an embodiment of this application.

[0096] like Figure 7 As shown, an electronic device 700 according to an embodiment of this application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0097] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0098] According to embodiments of this application, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0099] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0100] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0101] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the batch job completion time warning method provided in the embodiments of this application.

[0102] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0103] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0104] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0105] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0107] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A method for early warning of batch job completion time, characterized in that, The method includes: Obtain the execution status information of batch jobs. The execution status information includes the current time, the business deadline, the set of completed jobs, and the set of jobs to be executed. The set of completed jobs includes the actual execution time of each completed job, and the set of jobs to be executed includes the historical execution data of each job to be executed. Based on the historical execution data of each job in the set of jobs to be executed, a first predicted completion time range for the set of jobs to be executed is determined, and a first warning message is generated based on the comparison result between the first predicted completion time range and the remaining execution time of the business. The remaining execution time of the business is determined based on the current time and the business deadline. The first warning message is used to indicate whether the batch job can be completed before the business deadline. Obtain the system running status data of the batch job running environment, and combine it with the actual execution time in the set of completed jobs to determine whether there is any performance abnormality in the batch job running environment that leads to a decrease in overall execution efficiency; In the event of the performance anomaly, the first predicted completion time range is corrected based on the deviation between the actual execution time and the corresponding historical execution data to obtain a second predicted completion time range. A second warning message is generated based on the comparison between the second predicted completion time range and the remaining execution time of the business. The second warning message is used to indicate whether the batch job can be completed before the business deadline.

2. The batch operation completion time early warning method according to claim 1, characterized in that, Based on the historical execution data of each job in the set of jobs to be executed, the first predicted completion time range of the set of jobs to be executed is determined as follows: Based on the pre-stored dependencies between batch jobs, a job dependency topology in the form of a directed acyclic graph is constructed. Based on the job dependency topology, the critical path from the start job to the end job is identified, and the critical path is the job path with the longest total execution time. Based on the historical execution data of each job to be executed on the critical path, the corresponding predicted execution time is determined, and the predicted execution times are accumulated to obtain the first predicted completion time range.

3. The batch operation completion time early warning method according to claim 1, characterized in that, The determination of whether the batch job execution environment has performance anomalies that cause a decrease in overall execution efficiency includes: Obtain the historical average execution time for each job in the completed job set, and determine the average deviation rate of the actual execution time relative to the historical average execution time; If the average deviation rate is greater than the first preset threshold, and at least one resource indicator in the system operation status data exceeds the corresponding second preset threshold, then the performance abnormality is determined to exist.

4. The batch operation completion time early warning method according to claim 3, characterized in that, The resource indicators include at least one of processor utilization, memory utilization, disk read / write latency, and network bandwidth utilization; the second preset threshold is set based on the statistical distribution of the corresponding resource indicators during historical normal operation.

5. The batch operation completion time early warning method according to claim 1, characterized in that, The correction of the first prediction completion time range includes: The first predicted completion time range is corrected by using a static average model or a dynamic time series model, based on the deviation between the actual execution time and the corresponding historical execution data.

6. The batch operation completion time early warning method according to claim 5, characterized in that, The correction of the first prediction completion time range using the static average model includes: Calculate the average deviation between the actual execution time of completed tasks and their historical average execution time, and determine the expected increase in total execution time based on the average deviation and the number of tasks to be executed; The second predicted completion time range is obtained by adding the product of the expected increase in total execution time and the first weight coefficient to the lower limit of the first predicted completion time range, and by adding the product of the expected increase in total execution time and the second weight coefficient to the upper limit of the first predicted completion time range. The first weighting coefficient and the second weighting coefficient are obtained by fitting historical operating data, and the first weighting coefficient is greater than 0 and less than or equal to the second weighting coefficient, while the second weighting coefficient is less than or equal to 1.

7. The batch operation completion time early warning method according to claim 5, characterized in that, The correction of the first prediction completion time range using the dynamic time series model includes: The execution time deviation sequence of completed tasks is subjected to exponential smoothing filtering to obtain the smoothed deviation trend value; The degree of efficiency decay of the task to be executed is determined based on the deviation trend value, and the first predicted completion time range is adjusted according to the degree of efficiency decay to obtain the second predicted completion time range.

8. A batch operation completion time early warning device, characterized in that, The device includes: The acquisition module is used to acquire the execution status information of batch jobs. The execution status information includes the current time, the business deadline, the set of completed jobs, and the set of jobs to be executed. The set of completed jobs includes the actual execution time of each completed job, and the set of jobs to be executed includes the historical execution data of each job to be executed. The first generation module is used to determine a first predicted completion time range for the set of jobs to be executed based on the historical execution data of each job in the set of jobs to be executed, and to generate a first warning message based on the comparison result between the first predicted completion time range and the remaining execution time of the business. The remaining execution time of the business is determined based on the current time and the business deadline. The first warning message is used to indicate whether the batch job can be completed before the business deadline. The judgment module is used to obtain the system operation status data of the batch job running environment, and in combination with the actual execution time in the set of completed jobs, to determine whether there is a performance abnormality in the batch job running environment that causes a decrease in overall execution efficiency. The second generation module is used to, in the event of the performance anomaly, correct the first predicted completion time range based on the deviation between the actual execution time and the corresponding historical execution data to obtain a second predicted completion time range, and generate a second warning message based on the comparison result between the second predicted completion time range and the remaining execution time of the business. The second warning message is used to indicate whether the batch job can be completed before the business deadline.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images