Job scheduling and monitoring system and methods, storage media, computer equipment

By building a job scheduling and monitoring system, generating an overview of job operations, and performing anomaly identification and compliance alarm detection, the stability and reliability issues of the banking industry's job scheduling system were resolved, early warning capabilities and monitoring observability were improved, and business risks were reduced.

CN123114884APending Publication Date: 2026-11-17LONGYING ZHIDA (BEIJING) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611244714.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-17
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Under the traditional manual inspection and passive alarm model, the stability, reliability and monitoring observability of the banking industry's operation scheduling system are insufficient, making it difficult to detect abnormal operation problems in a timely manner, which may lead to business risks and systemic financial risks.

Method used

A job scheduling and monitoring system is provided, including a data processing module, a UI module, an intelligent algorithm anomaly detection module, a rule engine module, and an alarm gateway module. By constructing a fusion job scheduling adjacency matrix, it generates a job operation overview map, performs anomaly identification and compliance alarm detection, and visualizes and aggregates the monitoring results.

Benefits of technology

It enhanced the early warning capability of the job scheduling system, enabled real-time monitoring of job nodes and flows, reduced alarm noise, improved the sensitivity and accuracy of monitoring, and ensured the stability and reliability of job scheduling in the banking industry.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114884A_ABST
    Figure CN123114884A_ABST
Patent Text Reader

Abstract

This application discloses a job scheduling and monitoring system and method, storage medium, and computer equipment. The method includes: a data processing module that traverses the job flow generated by each job in the job scheduling system, collects runtime metadata generated by the job nodes under each job flow, and constructs a fused job scheduling adjacency matrix; a UI module that automatically generates a job operation overview map based on the fused job scheduling adjacency matrix for visualization; an intelligent algorithm anomaly detection module that identifies job operation anomalies based on runtime metadata; a rule engine module that detects job operation compliance alarms based on runtime metadata; and an alarm gateway module that aggregates and suppresses deduplication of anomaly identification results and alarm detection results to obtain effective monitoring data for display. This separate monitoring system is specifically designed for monitoring the job scheduling system of the banking industry, and it not only uses a single alarm indicator but also visualizes the monitoring results, thus improving early warning capabilities.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a job scheduling and monitoring system and method, storage medium, and computer equipment. Background Technology

[0002] With the rapid development of distributed systems and big data businesses, the scale of batch operations and scheduled tasks in the banking industry continues to grow, placing higher demands on the stability, reliability, execution efficiency, and monitoring observability of scheduling. These scheduling systems support dozens of core business scenarios, including core accounting, customer group operations, regulatory data reporting, and end-of-day batch clearing. The daily scheduling volume often reaches tens or even hundreds of thousands, covering complex dependency chains across clusters, levels, and systems. Any scheduling anomaly could trigger a chain of business risks.

[0003] In the traditional manual inspection and passive alarm model, problems such as operation timeouts, operation failures, and scheduling chain disorder are often only discovered after the impact has expanded. This may directly lead to accounting errors, service interruptions for users, and delays in regulatory reporting. At best, it may cause customer complaints and financial losses; at worst, it may trigger regulatory compliance penalties and even trigger systemic financial risks. Summary of the Invention

[0004] In view of this, this application provides a job scheduling monitoring system and method, storage medium, and computer equipment. Through a separate monitoring system, it is specifically used to monitor the job scheduling system of the banking industry. In addition to using a single alarm indicator, it can also visualize the monitoring results, thereby improving the early warning capability.

[0005] According to one aspect of this application, a job scheduling monitoring system is provided for monitoring job scheduling systems in the banking industry, the system comprising: Data processing module, UI module, intelligent algorithm anomaly detection module, rule engine module, and alarm gateway module; The data processing module is used to traverse the job flow generated by each job when scheduling each job in the job scheduling system, collect the runtime metadata generated by the job nodes under each job flow when they run, and construct a fusion job scheduling adjacency matrix that simultaneously covers the job node dependency relationship and the global scheduling dependency relationship of the job flow based on the runtime metadata. The jobs include clearing jobs and daily settlement jobs. The UI module is used to automatically generate a job operation overview map based on the integrated job scheduling adjacency matrix. The job operation overview map is used to visually display the real-time running status of job nodes and the operation link of job flow on the front-end page. The UI module corresponds to the front-end page. The intelligent algorithm anomaly detection module is used to identify anomalies in the job operation of the job scheduling system based on the runtime metadata, and output the anomaly identification results. The rules engine module is used to perform compliance alarm detection on the job operation of the job scheduling system based on the runtime metadata, and output the alarm detection results. The alarm gateway module is used to aggregate and suppress deduplication of anomaly identification results and alarm detection results to obtain effective monitoring data, and to push the effective monitoring data to the UI module so that the UI module can display the monitoring results on the front-end page based on the effective monitoring data.

[0006] According to another aspect of this application, a job scheduling monitoring method is provided for monitoring a banking industry job scheduling system, the method comprising: The job scheduling system traverses the job flows generated by each job when scheduling each job, collects the runtime metadata generated by the job nodes under each job flow, and constructs a fusion job scheduling adjacency matrix that simultaneously covers job node dependencies and global job flow scheduling dependencies. The jobs include clearing jobs and daily settlement jobs. Based on the fusion job scheduling adjacency matrix, a job operation overview map is automatically generated, which visually displays the real-time operation status of job nodes and the operation link of job flow. Based on the aforementioned runtime metadata, anomalies are identified in the job execution of the job scheduling system, and the anomaly identification results are output. Based on the aforementioned runtime metadata, compliance alarm detection is performed on the job operation of the job scheduling system, and the alarm detection results are output. Aggregation and deduplication are performed on the anomaly identification results and alarm detection results to obtain effective monitoring data, and the monitoring results are displayed based on the effective monitoring data.

[0007] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described job scheduling and monitoring method.

[0008] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described job scheduling and monitoring method.

[0009] By means of the above technical solutions, the job scheduling monitoring system and method, storage medium and computer equipment provided in this application, through a separate monitoring system, are specifically used to monitor the job scheduling system of the banking industry. At the same time, it not only uses a single alarm indicator, but also visualizes the monitoring results, thereby improving the early warning capability.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This illustration shows a schematic diagram of the overall implementation framework of a job scheduling and monitoring system provided in an embodiment of this application; Figure 2 This paper shows a schematic diagram of the structure of a job scheduling and monitoring system provided in an embodiment of this application; Figure 3 A flowchart illustrating a job scheduling and monitoring method provided in an embodiment of this application is shown. Detailed Implementation

[0012] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0013] This embodiment provides a job scheduling and monitoring system, the overall implementation framework of which is as follows: Figure 1 As shown, Figure 1 In this context, Prometheus is a widely used open-source cloud-native monitoring system responsible for collecting system resource data such as CPU utilization, memory utilization, and I / O throughput of job nodes. Kafka, a high-throughput distributed message queue, serves as a data buffer and handles asynchronous data transfer. The massive runtime metadata from the job scheduling API and Prometheus can be first written to Kafka for peak smoothing before being synchronously distributed to downstream data processing modules, preventing system overload caused by high-concurrency data surges. Together with the job scheduling API, these two components form the runtime data acquisition and access layer of the entire system, serving as the data source entry point for all subsequent adjacency matrix construction and anomaly detection processes.

[0014] Specifically, such as Figure 2 As shown, the job scheduling and monitoring system includes: Data processing module 101, UI module 102, intelligent algorithm anomaly detection module 103, rule engine module 104, and alarm gateway module 105; Data processing module 101 is used to traverse the job flow generated by each job when scheduling each job in the job scheduling system, collect the running metadata generated by the job nodes under each job flow when they run, and construct a fusion job scheduling adjacency matrix that simultaneously covers job node dependencies and job flow global scheduling dependencies based on the running metadata. The jobs include clearing jobs and daily settlement jobs. UI module 102 is used to automatically generate a job operation overview map based on the fusion job scheduling adjacency matrix. The job operation overview map is used to visually display the real-time running status of job nodes and the operation link of job flow on the front-end page. The UI module corresponds to the front-end page. The intelligent algorithm anomaly detection module 103 is used to identify anomalies in the operation of the job scheduling system based on the running metadata, and output the anomaly identification results. The rule engine module 104 is used to perform compliance alarm detection on the job operation of the job scheduling system based on the running metadata, and output the alarm detection results. The alarm gateway module 105 is used to aggregate and suppress deduplication of anomaly identification results and alarm detection results to obtain effective monitoring data, and synchronously push the effective monitoring data to the UI module so that the UI module can display the monitoring results on the front-end page based on the effective monitoring data; The data storage module 106 is used to store runtime metadata, the fusion job scheduling adjacency matrix, anomaly identification results, alarm detection results, and effective monitoring data.

[0015] In the above embodiments of this application, the UI module 102 is used to manage system configuration data such as workflows, jobs, and dependencies, alarm rules, and maintenance periods. It supports viewing job running status data, alarm data, and historical execution data of the workflow. Based on the fused job scheduling adjacency matrix, it automatically generates a job running overview map and visualizes the real-time running status of job nodes and the running link of the workflow on the corresponding front-end page. At the same time, it presents the running dynamics and key performance indicators, which is completely consistent with the logic of adjacency matrix-driven visualization in the core architecture.

[0016] The data processing module 101 is used to traverse the job flow generated by each job when scheduling each job in the job scheduling system, collect the runtime metadata generated by the job nodes under each job flow, and simultaneously access the system resource data (CPU, memory, disk I / O, disk remaining space) of the job nodes. It extracts the vector features of jobs and job flows, and constructs a fusion job scheduling adjacency matrix that covers both job node dependencies and global job flow scheduling dependencies, which is then used by the UI module to generate a visual overview. Based on historical runtime metadata, it predicts the completion time of the running system, job flow, and job, and provides job and job flow runtime data to the rule engine module 104 for compliance alarm detection. It also provides job and job flow runtime data and historical data to the intelligent algorithm anomaly detection module 103 for anomaly identification. At the same time, it uniformly stores alarm data, alarm notification records, job and job flow runtime data, upper and lower bound data, and other process and result data into the database.

[0017] The intelligent algorithm anomaly detection module 103 is used to identify anomalies in running and recently completed jobs based on the runtime metadata output by the data processing module 101. It employs a dual mechanism of timers and triggers to identify anomalies and output anomaly detection results. For example, at 22:00 every day, the feature descriptor analyzes the historical execution data of jobs and job flows to extract key operational characteristics. The upper and lower bound statistical algorithms are used to calculate the execution time confidence intervals of all job flows and jobs on day T+1 and store them in the database for anomaly detection on day T+1. The timed detector scans the execution time deviation of currently executing jobs every minute to generate detection results. It triggers anomaly alarms for jobs that have timed out. Whenever a job is completed, the instant detector is triggered to generate detection results for the execution time of that job. It triggers anomaly alarms for jobs that are too short to execute.

[0018] The rules engine module 104 is used to perform compliance alarm detection on the job operation of the job scheduling system based on the running metadata output by the data processing module 101 and combined with the alarm rules pre-configured by the user, and output the alarm detection results and push them to the alarm gateway module 105.

[0019] The alarm gateway module 105 is used to aggregate and suppress duplicates of the anomaly identification results output by the intelligent algorithm anomaly detection module 103 and the alarm detection results output by the rule engine module 104, and filter redundant alarms to obtain effective monitoring data. On the one hand, the effective monitoring data is synchronously pushed to the UI module 102 so that the monitoring results are displayed on the front-end page. On the other hand, it can also be synchronously pushed to a designated third-party interface to complete cross-system alarm distribution.

[0020] Data storage module 106, as the underlying support module, uniformly stores real-time running metadata of jobs and job flows, full historical execution data, upper and lower bounds of job and job flow execution time confidence intervals, scheduling dependency data, full alarm data, system resource data and other full-link data, providing persistent data support for the operation, visualization and detection logic of the other five modules.

[0021] To address this, an automatic job overview diagram clarifies the complex dependencies between jobs and workflows, making execution progress, status, and remaining time readily apparent. A more sensitive, accurate, and convenient alarm strategy is employed; a single, universal strategy monitors all job timeouts and excessively short execution times, triggering an alarm when execution time deviates significantly from the confidence interval, thus enhancing early warning capabilities. Intelligent algorithms, combined with the topology of jobs and workflows, can also predict system and workflow completion times. Alarm aggregation merges duplicate or related alarms into a single alarm, improving alarm quality, reducing noise, and eliminating alarm fatigue.

[0022] In particular, the job scheduling system is the underlying platform that carries all scheduling capabilities. It is the main control body of the entire system, responsible for the unified management of resource allocation, task triggering and process orchestration of all job nodes, providing the operating environment for job nodes and providing global scheduling rules support for job workflows.

[0023] A job node is the smallest execution unit in a job scheduling system. It is deployed on the system's hardware nodes and carries the single-step clearing business logic. The system resource data and running status data of each node are uniformly collected and managed by the job scheduling system, and it is a basic component of the job flow.

[0024] A workflow is a complete business link formed by multiple job nodes connected through directed dependencies in a job scheduling system. It is defined by a DAG (Directed Acyclic Graph) to indicate the order of execution between nodes and is the core scheduling unit of the job scheduling system. A complete clearing process in the banking industry usually corresponds to an independent workflow.

[0025] To this end, the job scheduling system serves as a carrier, arranging discrete job nodes into job flows according to the dependency rules of clearing business. Through the data processing module, it collects the resource and operation data of all job nodes, and finally constructs a complete job flow matrix to achieve full-link visual monitoring and scheduling control.

[0026] Optionally, the runtime metadata includes system resource data, job runtime data, and job flow runtime data. The system resource data includes CPU utilization, memory utilization, and I / O throughput. The system also includes: The data processing module 101 is also used to extract global scheduling constraints between independent job flows based on the job flow running status data, and to establish the global scheduling dependency relationship between independent job flows through the global scheduling constraints. The data processing module 101 is also used to extract the sequential execution constraints between each work node within the same work flow based on the work running state data, and to establish the work node dependency relationship between each work node within the same work flow through the sequential execution constraints. The data processing module 101 is further configured to construct a node-level adjacency matrix of the job flow, with all job nodes within the same job flow as rows and columns, and the job dependency relationship between job nodes as matrix elements, wherein the node-level adjacency matrix is ​​used to carry the job node dependency relationship within the same job flow. The data processing module 101 is further configured to construct a job flow-level adjacency matrix with all independent job flows as rows and columns, and the job flow global scheduling dependency relationship between each independent job flow as matrix elements, wherein the job flow-level adjacency matrix is ​​used to carry the job flow global scheduling dependency relationship across job flows. The data processing module 101 is further configured to uniformly map the node-level adjacency matrix and the job flow-level adjacency matrix to the topological vertex index space of the global topology graph. By assigning globally unique topological vertex indices to all job nodes and independent job flows, the matrix elements of the node-level adjacency matrix and the job flow-level adjacency matrix are synchronously mapped to the same topological vertex index space, resulting in a fused job scheduling adjacency matrix that simultaneously covers job node dependencies and job flow global scheduling dependencies. Here, the topological vertex is the job node and independent job flow that have been assigned a globally unique topological vertex index, the global topology graph is an entity graph structure that carries all job nodes, job flows, job node dependencies and job flow global scheduling dependencies, and the fused job scheduling adjacency matrix is ​​a mathematical matrix representation of the global topology graph in a unified topological vertex index space.

[0027] In the above embodiments of this application, the processing procedure of the data processing module 101 in the scenario of banking clearing operation scheduling is as follows: A bank's end-of-day clearing system (corresponding to a job scheduling system) has two independent workflows: Workflow A: Account reconciliation workflow, which includes 3 work nodes: A1 (data loading), A2 (detail verification), and A3 (balance update).

[0028] Workflow B: Funds settlement workflow, which includes two work nodes: B1 (position collection) and B2 (batch payment).

[0029] The collected runtime metadata includes, for example: System resource data, including CPU / memory / I / O metrics; Job runtime data, including the start / end time and execution status of each node; Job workflow runtime data, the overall runtime sequence of two job workflows, and cross-flow triggering rules.

[0030] Based on the job flow runtime data, a global scheduling constraint is obtained: job flow A must be fully executed before job flow B can start, thus establishing the global scheduling dependency relationship between A and B.

[0031] Based on the job runtime data, the sequential execution constraints within the flow are obtained: A2 can only start after A1 is completed, A3 can only start after A2 is completed, and B2 can only start after B1 is completed, thus establishing the job node dependencies of A1→A2, A2→A3, and B1→B2.

[0032] Using the three nodes of job flow A as rows and columns, generate a 3×3 matrix, marking only the matrix elements corresponding to A1→A2 and A2→A3 as 1, and the rest as 0, to carry the node dependencies within the flow; similarly, generate a 2×2 node-level adjacency matrix for job flow B.

[0033] Construct a job flow-level adjacency matrix with two independent job flows as rows and columns, generating a 2×2 matrix. Only the matrix elements corresponding to A→B are marked as 1, and the rest are 0, which carries the global scheduling dependency across job flows.

[0034] Assign globally unique topology indices to all entities: A1=0, A2=1, A3=2, B1=3, B2=4, job flow A=5, job flow B=6. Synchronously map the elements of the two adjacency matrices to a 7×7 unified index space, and finally obtain a fusion matrix that covers both node dependencies and cross-flow dependencies, thus completing the mathematical representation of the global topology graph.

[0035] Optionally, the system also includes: The UI module 102 is also used to read all globally unique topological vertex indices in the fused job scheduling adjacency matrix, and based on the topological vertex indices, automatically distinguish the job flow layer aggregate vertices that represent globally unique job flows, and the job node layer detailed vertices that represent globally unique job nodes. The UI module 102 is also used to traverse all non-zero matrix elements in the fusion job scheduling adjacency matrix, extract two types of associated edges based on the non-zero matrix elements, and bind different edge style identifiers to the two types of associated edges respectively. The associated edges include the execution order edges between job nodes within the same job flow and the global scheduling dependency edges across job flows. The UI module 102 is also used to automatically gather the detailed vertices of the job node layer belonging to the same job flow into the corresponding outer container, using the job flow layer aggregate vertex as the outer container, automatically arrange the job nodes in the outer container based on the force-oriented layout algorithm, and simultaneously arrange all job flow layer aggregate vertices in a horizontal time sequence based on the layered arrangement algorithm to generate an initial double-layer nested topology layout framework. The UI module 102 is also used to extract the running status of the job node based on the job running status data, extract the overall running progress of the job flow based on the job flow running status data, and in the initial double-layer nested topology layout framework, map the running status of the job node to the fill color of the detailed vertices of the job node layer, and map the overall running progress of the job flow to the border progress bar of the aggregate vertices of the job flow layer, so as to obtain the double-layer nested topology layout framework. The running status of the job node includes running, running successfully, running failed and waiting for scheduling, and the running link of the job flow is represented by the overall running progress of the job flow. The UI module 102 is also used to overlay the edge style identifier of the associated edge on the basis of the double-layer nested topology layout framework, as well as the state visualization encoding for the aggregated vertices of the job flow layer and the detailed vertices of the job node layer, to automatically generate a complete job operation overview map. The job operation overview map supports zooming, dragging and clicking drill-down interaction on the front-end page. Clicking the aggregated vertex of the job flow layer can expand to view the operation link of all job nodes inside the corresponding job flow. Clicking the detailed vertex of the job node layer can pop up a window to display the system resource data corresponding to the job node.

[0036] In the above embodiments of this application, the execution process of the UI module 102 is as follows: Assuming the fused job scheduling adjacency matrix contains 32 globally unique topological vertex indices, indices 01, 05, and 12 correspond to three independent clearing job flows: "Batch Reconciliation," "Funds Netting," and "Report Generation." The remaining 29 indices correspond to clearing nodes within each job flow, such as data loading and verification nodes. The UI module 102 automatically identifies the three job flow layer aggregate vertices, and the remaining 29 are automatically categorized as detailed vertices in the job node layer.

[0037] After traversing all non-zero elements of the adjacency matrix of the integrated job scheduling, two types of edges are extracted: within the same "batch reconciliation" job flow, the edge from "pre-data loading" to "account validity verification" is bound with a solid blue line; the global dependency edge from "batch reconciliation" across job flows to "fund netting" is bound with a dashed red line. The two types of edges are visually distinguishable.

[0038] Using three job flow aggregation vertices as outer containers, the detailed vertices of the job node layer in the corresponding job flow are gathered into the containers. The nodes in the containers are arranged in a constrained force-directed algorithm to avoid overlap. Then, the three job flow containers are arranged horizontally from left to right according to the order of the clearing business to obtain a nested initial topology framework.

[0039] Next, the status of each node is extracted from the running data. For example, nodes in operation are filled with yellow, successful nodes are filled with green, failed nodes are filled with red, and waiting nodes are filled with gray. At the same time, the completion progress of the job flow is mapped to the border of the aggregation vertex. For example, if "batch reconciliation" is 60% complete, the border of its aggregation vertex will show a green progress bar of 60% length.

[0040] By overlaying pre-bound edge style identifiers, a complete overview diagram of the liquidation job operation is generated. The front end supports zooming and dragging; clicking on aggregated vertices expands to view the full node chain of the job flow; clicking on a single liquidation node displays real-time system resource data such as CPU and memory in a pop-up window.

[0041] In particular, the force-directed layout algorithm is a graph layout algorithm based on physical mechanics simulation. It treats nodes as particles with repulsive forces and the connections between nodes as spring attraction. Through iterative calculation, the total energy of the system is gradually reduced until a stable equilibrium state is reached, achieving uniform arrangement of nodes and avoiding overlap. It can be used for the arrangement of job nodes inside a workflow aggregation container, enabling related nodes within the same workflow to automatically cluster and unrelated nodes to disperse, ensuring a clear and uncrowded distribution of nodes within the container.

[0042] The hierarchical layout algorithm, also known as the layered arrangement algorithm, belongs to the category of directed graph layout. Its core is to complete three steps: hierarchical division, sorting of nodes within a layer, and coordinate allocation based on the dependencies of nodes. This ensures that nodes are arranged hierarchically according to the specified logic, reducing edge intersections. It can be used for the horizontal temporal arrangement of aggregated vertices of all workflow layers. It can arrange workflows such as "batch reconciliation", "fund netting", and "report generation" from left to right according to the order of clearing business, clearly presenting the temporal logic of global scheduling.

[0043] Optionally, the system further includes: The intelligent algorithm anomaly detection module 103 is also used to obtain the real-time value of any type of system resource data of the current work node, calculate the deviation between the real-time value and the preset baseline value corresponding to the system resource data, and determine that the work node has a system resource anomaly of the type of system resource data when the deviation exceeds the preset threshold. The intelligent algorithm anomaly detection module 103 is also used to extract single job node execution features based on the job running status data of the current job node, and to identify whether there is a job node running anomaly in the current job node through the single job node execution features. The job node running anomaly includes abnormal execution time, retry number exceeding a reasonable range, and output data volume missing anomaly. The single job node execution features include job execution time, retry number, output data volume, and return status code. The intelligent algorithm anomaly detection module 103 is also used to extract global scheduling features based on the job flow running status data of the current job node, and to construct a job execution link probability graph model by combining the job node dependency relationship and the job flow global scheduling dependency relationship recorded in the fused job scheduling adjacency matrix. Based on the job execution link probability graph model and global scheduling features, it identifies whether there is a job flow scheduling anomaly in the current job node. The job flow scheduling anomaly includes job nodes not executing in a preset order, job nodes not triggering downstream blocking rules, and the overall completion progress of the job flow lagging behind the overall completion progress of the same period in history.

[0044] The intelligent algorithm anomaly detection module 103 is also used to generate anomaly identification results based on the anomaly type, anomaly content, the job node that caused the anomaly, and the job flow to which the job node that caused the anomaly belongs. The anomaly types include system resource anomalies, job node operation anomalies, and job flow scheduling anomalies. When a job node operation anomaly occurs, the anomaly content includes execution time anomalies, retries exceeding a reasonable range, and output data missing anomalies. When a job flow scheduling anomaly occurs, the anomaly content includes job nodes not executing in a preset order, job nodes not triggering downstream blocking rules, and the overall completion progress of the job flow lagging behind the overall completion progress of the same period in history.

[0045] In the above embodiments of this application, the overall completion progress in the same historical period is, for example, last year.

[0046] Furthermore, for example, targeting the core server cluster of city commercial banks in the early morning of the settlement day, the intelligent algorithm anomaly detection module 103 accesses four real-time indicator streams: CPU utilization, memory usage, disk I / O, and network bandwidth, and uses a "three-stage joint detection" mechanism to complete the identification: The first stage uses the 3-sigma statistical method to quickly screen out preliminary anomalies where CPU utilization exceeds 95% for 3 consecutive minutes. The second stage uses the isolated forest algorithm to filter out false alarms of normal high load during the clearing peak period. The third stage uses a large model combined with the business context of 80 million batch reconciliations on the same day for secondary verification to confirm that the node's continuous high CPU load is a real resource anomaly that exceeds the historical baseline. At the same time, it is marked that the node's remaining available computing power can no longer support subsequent batch payment tasks.

[0047] Using the "TA Share Verification" node in the clearing process as the detection target, the intelligent algorithm anomaly detection module 103 collects the node's operational data in real time, including task start time, number of processed data entries, current execution progress, and key fields in the returned logs. When the node processes the 3.2 millionth share data entry, and there is no progress update for 15 consecutive seconds, with the logs displaying the keyword "database connection timeout," the algorithm, combining the node's historical average processing rate and the preset timeout threshold, determines that the node has experienced an operational deadlock anomaly. Simultaneously, it eliminates interference from normal operations such as "manual task pause," accurately pinpointing the node's execution failure.

[0048] Using the standard clearing workflow of "batch reconciliation → fund transfer → clearing result archiving" as the detection target, the intelligent algorithm anomaly detection module 103 performs verification based on the preset dependencies of the workflow. After the "batch reconciliation" node has completed execution and returned a success flag, the scheduling system should have triggered the downstream "fund transfer" node within 10 seconds. However, a situation arose where the scheduling queue was backed up and the downstream node was never started. The algorithm, combined with the topology rules of the workflow and the historical scheduling latency baseline, identified a deadlock-like anomaly in the scheduling link. At the same time, it correlated with the upstream marked system resource anomalies, confirming that the scheduling link blockage was caused by insufficient cluster computing power, rather than a configuration error of the job node itself.

[0049] The three types of abnormal information mentioned above are aggregated in a structured manner to generate a complete and traceable abnormal event report. The report includes the timestamp of the abnormality, the associated server node number, the batch number of the corresponding clearing operation, the abnormality level label, the root cause determination conclusion, and the associated historical handling plan. At the same time, it can also be automatically synchronized to the clearing monitoring dashboard and the mobile alarm channel of the operation and maintenance personnel, providing accurate basis for subsequent fault isolation, computing power expansion, and task rescheduling. All abnormal records are automatically stored in the ledger to meet the compliance audit requirements of the banking clearing industry.

[0050] Optionally, the runtime metadata includes system resource data, job runtime data, and job flow runtime data. The system resource data includes CPU utilization, memory utilization, and I / O throughput. The system also includes: The rule engine module 104 is also used to match the real-time collected system resource data, job running status data and job flow running status data one by one with their respective preset alarm rule bases, identify explicit compliance anomalies, and output structured alarm detection results based on explicit compliance anomalies. Explicit compliance anomalies include job timeout failure to start, job node execution failure and system resource data exceeding limits. The structured alarm detection results include alarm level, violation alarm rules and related job information.

[0051] In the above embodiments of this application, taking a bank's daily batch clearing and reconciliation scenario as an example, the rule engine module 104 first collects system resource data such as the CPU and memory usage of the clearing server in real time, job operation status data such as the start time and execution status of each clearing task, and job flow operation status data such as the node transfer progress of the entire clearing job flow.

[0052] The rule engine module 104 matches the three types of data one by one with their respective preset alarm rule libraries. For example, when matching the rule "clearing nodes must start before 23:00 every day", it finds that a batch of reconciliation operations has not started for 15 minutes, thus identifying the corresponding explicit compliance anomaly and outputting a structured alarm detection result that includes a high alarm level, the corresponding violation alarm rule, and the associated reconciliation operation's full-link information. This result is then pushed to the operation and maintenance platform to trigger the handling process.

[0053] The above process can achieve hot deployment of rules based on a mature rule engine, and can quickly iterate alarm rules without restarting the system, adapting to the dynamic adjustment of compliance requirements of clearing business, and ensuring the stable and compliant operation of batch clearing operations.

[0054] Optionally, the system further includes: The alarm gateway module 105 is also used to aggregate the anomaly identification results and alarm detection results into the same event cluster, filter out duplicate results in the event cluster, and generate effective monitoring data with root cause type, impact range and fault level.

[0055] In the above embodiments of this application, taking a payment link failure scenario as an example, the specific execution process is as follows: When a payment service node experiences an OutOfMemory (OOM) exception, the exception identification results will capture characteristics such as continuously exceeding the instance's memory usage limit and the process being about to be evicted. Simultaneously, the alarm detection results will generate CPU utilization alarms, database connection pool exhaustion alarms, and downstream payment interface timeout alarms. The alarm gateway module 105, using the DBSCAN clustering algorithm, aggregates all these results from different dimensions, within a 30-second time window, and related to the same payment link into a single event cluster.

[0056] This event cluster may contain a large number of duplicate alarms of the same type, such as 100 duplicate records of "payment interface timeout" from different probe nodes. The alarm gateway module 105 can deduplicate the alarms according to the dimensions of device ID, alarm type, and precise timestamp, retaining only one core original record and removing the rest of the redundant entries.

[0057] The alarm gateway module 105 automatically labels the root cause type as "application instance memory overflow" based on the deduplicated event cluster information. The impact range covers the three core microservice nodes of the entire payment transaction chain, and the fault level is judged as a P1 level core fault. Finally, it generates structured and effective monitoring data containing these three types of key information.

[0058] Currently, most existing job scheduling and monitoring solutions rely on built-in monitoring functions, such as the domestic taskctl job scheduling management platform. These solutions suffer from issues like rigid policies, limited monitoring metrics, insufficient monitoring and early warning capabilities, weak visualization, and a lack of global topology, dependency link, and trend prediction capabilities. The monitoring pages of existing solutions are unfriendly, failing to clearly and intuitively display the topology, progress, and status of all jobs on a single page. Custom development is required for new monitoring pages, and adding or deleting jobs necessitates code modifications. Alarm policy settings are also inflexible. For example, job timeout alarm policies require fixed time thresholds, preventing the setting of different thresholds for different jobs. Applying these policies to a single job can lead to an excessive number of monitoring policies, resulting in significant maintenance costs, and fixed-threshold policies have a high false alarm rate. Furthermore, there is no alarm aggregation function; existing monitoring solutions cannot merge a large number of identical, repetitive, or related alarms into a single alarm.

[0059] By applying the technical solution of this embodiment, a global job topology map is automatically generated based on vector data such as jobs, job flows, and dependencies. This map presents the dependencies between and within systems, including job flows and jobs. The colors of jobs and job flows change according to their status, thus displaying the global job operation dynamics. Dynamic threshold alarm strategies are supported; a general global strategy can be created to monitor all job execution timeouts / insufficient execution times. An alarm is triggered when the job execution time deviates from the specified confidence interval. Key characteristics of jobs and job flows are obtained by analyzing historical data, and algorithms such as upper and lower bounds are used to calculate the execution time confidence intervals for all job flows and jobs on day T+1. This information is applied to intelligent alarm strategies and system / job flow completion time prediction. Alarm aggregation solves the problem of excessive alarms by merging a large number of duplicate or related alarms into a single alarm data point.

[0060] Furthermore, embodiments of this application provide a job scheduling and monitoring method, such as... Figure 3 As shown, the method includes: Step 201: Traverse the job flow generated by each job when scheduling each job in the job scheduling system, collect the runtime metadata generated by the job nodes under each job flow when they run, and construct a fusion job scheduling adjacency matrix that simultaneously covers job node dependencies and job flow global scheduling dependencies based on the runtime metadata. The jobs include clearing jobs and daily settlement jobs. Step 202: Based on the fusion job scheduling adjacency matrix, automatically generate a job operation overview map, and visualize the real-time operation status of job nodes and the operation link of job flow through the job operation overview map; Step 203: Based on the runtime metadata, perform anomaly identification on the job execution of the job scheduling system and output the anomaly identification results; Step 204: Based on the runtime metadata, perform compliance alarm detection on the job operation of the job scheduling system and output the alarm detection results; Step 205: Aggregate and suppress duplicates of the anomaly identification results and alarm detection results to obtain effective monitoring data, and display the monitoring results based on the effective monitoring data.

[0061] It should be noted that the corresponding descriptions of the job scheduling and monitoring method provided in this application embodiment can be found by referring to... Figure 2 The corresponding descriptions in the system will not be repeated here.

[0062] Based on the above, Figure 3 Accordingly, this application also provides a storage medium storing a computer program, which, when executed by a processor, implements the above-described method. Figure 3 The job scheduling and monitoring method shown.

[0063] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), and includes several instructions to cause a computer device (such as a personal computer, server, or network device) to execute the methods described in the various implementation scenarios of this application.

[0064] Based on the above, Figure 3 The method shown, and Figure 2 To achieve the above objectives, the virtual system embodiment shown in this application also provides a computer device, specifically a personal computer, server, network device, etc. This computer device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to achieve the above-described objectives. Figure 3 The job scheduling and monitoring method shown.

[0065] Optionally, the computer device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB ports, card reader ports, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Bluetooth interfaces, Wi-Fi interfaces), etc.

[0066] Those skilled in the art will understand that the computer device structure provided in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0067] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and stores the hardware and software resources of a computer device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software within the physical device.

[0068] Through the above description of the implementation methods, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms. Alternatively, it can be implemented using hardware. The data processing module traverses the job scheduling system to schedule each job, collecting the runtime metadata generated by the job nodes under each job flow. This data is then used to construct a fused job scheduling adjacency matrix. The UI module automatically generates a job operation overview diagram based on the fused job scheduling adjacency matrix for visualization. The intelligent algorithm anomaly detection module identifies job operation anomalies based on the runtime metadata. The rule engine module detects job operation compliance alarms based on the runtime metadata. The alarm gateway module aggregates and suppresses deduplication of the anomaly identification results and alarm detection results to obtain effective monitoring data for display. A separate monitoring system is specifically designed to monitor the banking industry's job scheduling system. Furthermore, it not only uses a single alarm indicator but also visualizes the monitoring results, thus improving early warning capabilities.

[0069] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0070] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any modifications that can be made by those skilled in the art should fall within the protection scope of this application.

Claims

1. A job scheduling monitoring system, characterized by, A work scheduling system for monitoring the banking industry, the system comprising: Data processing module, UI module, intelligent algorithm anomaly detection module, rule engine module, and alarm gateway module; The data processing module is used to traverse the job flow generated by each job when scheduling each job in the job scheduling system, collect the runtime metadata generated by the job nodes under each job flow when they run, and construct a fusion job scheduling adjacency matrix that simultaneously covers the job node dependency relationship and the global scheduling dependency relationship of the job flow based on the runtime metadata. The jobs include clearing jobs and daily settlement jobs. The UI module is used to automatically generate a job operation overview map based on the integrated job scheduling adjacency matrix. The job operation overview map is used to visually display the real-time running status of job nodes and the operation link of job flow on the front-end page. The UI module corresponds to the front-end page. The intelligent algorithm anomaly detection module is used to identify anomalies in the job operation of the job scheduling system based on the runtime metadata, and output the anomaly identification results. The rules engine module is used to perform compliance alarm detection on the job operation of the job scheduling system based on the runtime metadata, and output the alarm detection results. The alarm gateway module is used to aggregate and suppress deduplication of anomaly identification results and alarm detection results to obtain effective monitoring data, and to push the effective monitoring data to the UI module so that the UI module can display the monitoring results on the front-end page based on the effective monitoring data.

2. The system of claim 1, wherein, The runtime metadata includes system resource data, job runtime data, and job flow runtime data. System resource data includes CPU utilization, memory utilization, and I / O throughput. The system also includes: The data processing module is also used to extract global scheduling constraints between independent job flows based on the job flow running status data, and to establish the global scheduling dependency relationship between independent job flows through the global scheduling constraints. The data processing module is also used to extract the sequential execution constraints between each job node within the same job flow based on the job running state data, and to establish the job node dependency relationship between each job node within the same job flow through the sequential execution constraints. The data processing module is further configured to construct a node-level adjacency matrix of the workflow, with all the work nodes within the same workflow as rows and columns, and the work dependencies between work nodes as matrix elements, wherein the node-level adjacency matrix is ​​used to carry the work node dependencies within the same workflow. The data processing module is further configured to construct a job flow-level adjacency matrix with all independent job flows as rows and columns, and the job flow global scheduling dependency relationship between each independent job flow as matrix elements. The job flow-level adjacency matrix is ​​used to carry the job flow global scheduling dependency relationship across job flows. The data processing module is further configured to uniformly map the node-level adjacency matrix and the job flow-level adjacency matrix to the topological vertex index space of the global topology graph. By assigning globally unique topological vertex indices to all job nodes and independent job flows, the matrix elements of the node-level adjacency matrix and the job flow-level adjacency matrix are synchronously mapped to the same topological vertex index space, resulting in a fused job scheduling adjacency matrix that simultaneously covers job node dependencies and job flow global scheduling dependencies. Here, the topological vertex is the job node and independent job flow that have been assigned a globally unique topological vertex index, the global topology graph is an entity graph structure that carries all job nodes, job flows, job node dependencies, and job flow global scheduling dependencies, and the fused job scheduling adjacency matrix is ​​a mathematical matrix representation of the global topology graph in a unified topological vertex index space.

3. The system according to claim 2, characterized in that, The system also includes: The UI module is also used to read all globally unique topological vertex indices in the fused job scheduling adjacency matrix, and based on the topological vertex indices, automatically distinguish the job flow layer aggregate vertices that represent globally unique job flows, and the job node layer detailed vertices that represent globally unique job nodes. The UI module is also used to traverse all non-zero matrix elements in the fusion job scheduling adjacency matrix, extract two types of associated edges based on the non-zero matrix elements, and bind different edge style identifiers to the two types of associated edges respectively. The associated edges include the execution order edges between job nodes within the same job flow and the global scheduling dependency edges across job flows. The UI module is also used to automatically gather the detailed vertices of the job node layer belonging to the same job flow into the corresponding outer container, using the aggregated vertices of the job flow layer as the outer container, automatically arrange the job nodes in the outer container based on the force-oriented layout algorithm, and simultaneously arrange all the aggregated vertices of the job flow layer in a horizontal time sequence based on the layered arrangement algorithm to generate an initial double-layer nested topology layout framework. The UI module is also used to extract the running status of job nodes based on job running status data, and to extract the overall running progress of the job flow based on job flow running status data. In the initial double-layer nested topology layout framework, the running status of job nodes is mapped to the fill color of the detailed vertices of the job node layer, and the overall running progress of the job flow is mapped to the border progress bar of the aggregate vertices of the job flow layer, thus obtaining the double-layer nested topology layout framework. The running status of job nodes includes running, running successfully, running failed, and waiting for scheduling. The running link of the job flow is represented by the overall running progress of the job flow. The UI module is also used to overlay edge style identifiers of associated edges on the basis of a double-nested topology layout framework, as well as state visualization encoding for aggregated vertices of the job flow layer and detailed vertices of the job node layer, to automatically generate a complete job operation overview map. The job operation overview map supports zooming, dragging, and click drill-down interaction on the front-end page. Clicking on an aggregated vertex of the job flow layer can expand to view the operation link of all job nodes inside the corresponding job flow. Clicking on a detailed vertex of the job node layer can pop up a window to display the system resource data corresponding to the job node.

4. The system according to claim 1, characterized in that, The runtime metadata includes system resource data, job runtime data, and job flow runtime data. System resource data includes CPU utilization, memory utilization, and I / O throughput. The system also includes: The intelligent algorithm anomaly detection module is also used to obtain the real-time value of any type of system resource data of the current work node, calculate the deviation between the real-time value and the preset baseline value corresponding to the system resource data, and determine that the work node has a system resource anomaly of the type of system resource data when the deviation exceeds the preset threshold. The intelligent algorithm anomaly detection module is also used to extract single job node execution features based on the job running status data of the current job node, and to identify whether there are job node operation anomalies in the current job node through the single job node execution features. The job node operation anomalies include abnormal execution time, retry number exceeding a reasonable range, and output data volume missing anomalies. The single job node execution features include job execution time, retry number, output data volume, and return status code. The intelligent algorithm anomaly detection module is also used to extract global scheduling features based on the job flow running status data of the current job node, and to construct a job execution link probabilistic graph model by combining the job node dependency relationship recorded in the fused job scheduling adjacency matrix and the job flow global scheduling dependency relationship. Based on the job execution link probabilistic graph model and global scheduling features, it identifies whether there is a job flow scheduling anomaly in the current job node. The job flow scheduling anomaly includes job nodes not executing in a preset order, job nodes not triggering downstream blocking rules, and the overall completion progress of the job flow lagging behind the overall completion progress of the same period in history. The intelligent algorithm anomaly detection module is also used to generate anomaly identification results based on the anomaly type, anomaly content, the job node that caused the anomaly, and the job flow to which the job node that caused the anomaly belongs. The anomaly types include system resource anomalies, job node operation anomalies, and job flow scheduling anomalies. When a job node operation anomaly occurs, the anomaly content includes execution time anomalies, retries exceeding a reasonable range, and output data missing anomalies. When a job flow scheduling anomaly occurs, the anomaly content includes job nodes not executing in a preset order, job nodes not triggering downstream blocking rules, and the overall completion progress of the job flow lagging behind the overall completion progress of the same period in history.

5. The system according to claim 1, characterized in that, The runtime metadata includes system resource data, job runtime data, and job flow runtime data. System resource data includes CPU utilization, memory utilization, and I / O throughput. The system also includes: The rule engine module is also used to match the real-time collected system resource data, job running status data and job flow running status data one by one with their respective preset alarm rule bases, identify explicit compliance anomalies, and output structured alarm detection results based on explicit compliance anomalies. Explicit compliance anomalies include job timeout failure to start, job node execution failure and system resource data exceeding limits. Structured alarm detection results include alarm level, violation alarm rule and related job information.

6. The system according to claim 1, characterized in that, The system also includes: The alarm gateway module is also used to aggregate the anomaly identification results and alarm detection results into the same event cluster, filter out duplicate results in the event cluster, and generate effective monitoring data with root cause type, scope of impact and fault level.

7. The system according to claim 1, characterized in that, The system also includes: The data storage module is used to store runtime metadata, the fusion job scheduling adjacency matrix, anomaly identification results, alarm detection results, and effective monitoring data.

8. A job scheduling and monitoring method, characterized in that, The method for monitoring a banking industry job scheduling system includes: The job scheduling system traverses the job flows generated by each job when scheduling each job, collects the runtime metadata generated by the job nodes under each job flow, and constructs a fusion job scheduling adjacency matrix that simultaneously covers job node dependencies and global job flow scheduling dependencies. The jobs include clearing jobs and daily settlement jobs. Based on the fusion job scheduling adjacency matrix, a job operation overview map is automatically generated, which visually displays the real-time operation status of job nodes and the operation link of job flow. Based on the aforementioned runtime metadata, anomalies are identified in the job execution of the job scheduling system, and the anomaly identification results are output. Based on the aforementioned runtime metadata, compliance alarm detection is performed on the job operation of the job scheduling system, and the alarm detection results are output. Aggregation and deduplication are performed on the anomaly identification results and alarm detection results to obtain effective monitoring data, and the monitoring results are displayed based on the effective monitoring data.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the job scheduling and monitoring method of claim 8.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the job scheduling and monitoring method of claim 8.