Batch job scheduling method and system based on shell script and database

Abstract

The application discloses a batch job scheduling method and system based on a Shell script and a database. The method realizes multiple technical breakthroughs such as decoupling of scheduling logic and business logic, process self-recovery daemon, fine concurrent control, multi-type job compatibility and automatic monitoring and alarming by adopting a Shell script lightweight program architecture combined with core design of database metadata management and control. The application provides a lightweight, highly adaptable, highly reliable, self-recoverable, easy-to-maintain and multi-channel alarming financial-level batch scheduling method and system, which is suitable for heterogeneous databases and multi-business scenarios and meets the high-standard scheduling requirements of financial IT systems.

Description

Technical Field

[0001] This application relates to the field of computer information technology, and in particular to a batch job scheduling method and system based on Shell scripts and databases, which is suitable for lightweight, reliable and heterogeneous adaptive automated scheduling of large-scale, highly dependent and time-sensitive batch jobs in financial enterprises such as banks, securities, and telecommunications. Background Technology

[0002] In core IT systems of financial institutions such as banks, securities firms, and telecommunications companies, batch jobs are a fundamental support for the normal operation of daytime business, widely covering key business scenarios such as end-of-day accounting batch processing, ETL data extraction, interest rate calculation, cross-institutional reconciliation, and business report generation. Unlike ordinary internet task scheduling, financial-grade batch jobs are characterized by massive task volume, complex upstream and downstream dependencies, stringent execution time windows, and extremely low business fault tolerance. These batch jobs are often executed automatically during unattended periods such as late night and early morning. If an error, interruption, or failure occurs during job execution, and if it cannot be detected, self-healed, and handled in a timely manner, it will directly lead to missing basic data, data corruption, or reconciliation anomalies required for the next day's business, causing major problems such as business interruption, incorrect customer fund display, and transaction failures, severely impacting the continuity of financial services and user experience. Therefore, building a highly reliable, controllable, observable, and easily maintainable large-scale automated batch job scheduling system is a core technical requirement for the construction of financial-grade IT systems.

[0003] Currently, the mainstream technical solutions for batch job scheduling in the financial industry are mainly divided into two categories: traditional script-based scheduling solutions and professional scheduling platform solutions. The specific technical architecture and existing technical defects are as follows:

[0004] The first category is the traditional script-based scheduling scheme, which mainly uses the Crontab timer mechanism combined with shell scripts to achieve job scheduling. This scheme relies on the native timer capabilities of operating systems such as Linux and Unix, eliminating the need for dedicated scheduling software. Its overall architecture is simple and easy to deploy; it only requires configuring timed trigger rules via Crontab to call the main control script, which then serially calls the sub-job scripts at each level in a hard-coded fixed order to complete the batch task loop. This scheme is suitable for lightweight scheduling scenarios with fewer than 50 tasks, simple dependencies, and a small business volume, and was a common solution for batch scheduling in early small financial systems.

[0005] However, this solution has many inherent technical defects in large-scale, high-reliability scheduling scenarios of medium and large-sized financial core systems, and cannot meet the requirements of financial-grade business: First, the system robustness is extremely poor. The overall scheduling process relies on the serial execution of a single script process. Once the process exits due to factors such as memory overflow, human error termination, host restart, or system abnormality, the entire batch job will be directly interrupted. Moreover, faults during unattended periods cannot be automatically detected and are usually only discovered after the business starts the next day, resulting in large-scale business data loss. Second, the operation and maintenance iteration costs are high. The execution order and dependency constraints of all jobs are hard-coded in the script source code. When adding, deleting, adjusting jobs, or optimizing scheduling logic, the script code must be manually modified, making the change process cumbersome and iteration-intensive. The script suffers from several drawbacks: 1) Long execution cycles and the ease with which manual modifications can introduce logical vulnerabilities, making it unsuitable for the rapid iteration needs of financial businesses; 2) Lack of resource management capabilities, with no concurrency restrictions or resource threshold constraints on child processes during script execution, easily exhausting host CPU and memory resources in high-concurrency scenarios, causing system lag and crashes, leading to widespread failures of batch tasks; 3) Lack of logging and maintenance capabilities, with job execution logs scattered across various server directories without a unified log collection and status recording mechanism, resulting in extremely low efficiency in fault diagnosis and problem localization; and 4) Lack of proactive alarm mechanisms, with no automated notification channels for job anomalies, interruptions, or failures, relying entirely on manual log inspections by maintenance personnel, leading to severe delays in fault detection and an inability to promptly handle faults at night.

[0006] The second category comprises commercial scheduling software and open-source scheduling platform solutions, with mainstream products including Taskctl, MOIA, and XXL-JOB. Compared to traditional script-based scheduling solutions, this type of solution features an upgraded architecture, a graphical web configuration interface, and supports visual task orchestration, job dependency configuration, and runtime status monitoring. It allows for fine-grained configuration such as job mutual exclusion, forced dependencies, and custom execution plans, while also possessing basic runtime monitoring capabilities. These solutions can meet the fundamental scheduling needs of core batch jobs in medium to large enterprises and are currently widely used in the core business scheduling scenarios of medium to large financial institutions.

[0007] However, despite meeting the core requirements of lightweight, highly adaptable, efficient, and easy-to-maintain financial-grade systems, these professional scheduling platforms still suffer from significant technical shortcomings. Specifically, these shortcomings include: First, cumbersome deployment architecture and high implementation costs. Commercial scheduling software requires independent deployment of servers, web containers, and dedicated metadata databases, resulting in complex installation and configuration processes, strict compatibility requirements with server hardware environments and system versions, long deployment cycles, and high maintenance thresholds. Second, strong technology stack binding exists. Most commercial scheduling platforms are deeply coupled with specific ETL tools and dedicated databases, leading to poor compatibility and significant challenges in adapting to the heterogeneous environments of multiple databases and business systems within the financial industry, resulting in insufficient versatility. Third, low scheduling execution efficiency. The platform itself incurs heavy scheduling overhead, significantly increasing efficiency compared to manual execution of scripts and database stored procedures. The platform schedules job execution times, which increase significantly, with performance losses reaching tens or even hundreds of times in some scenarios, making it unsuitable for the batch processing needs of financial institutions with narrow time windows. Fourth, the configuration synchronization timeliness is poor. After job scheduling processes and parameter configurations are modified, manual restarting of the service and loading of the configuration are required for them to take effect, making real-time configuration synchronization impossible and resulting in low iteration efficiency. Fifth, the alarm capabilities are weak, with a single alarm channel that only supports email and platform in-site message notifications, and does not support native alarms from mainstream instant messaging tools such as WeChat and DingTalk. In nighttime unattended scenarios, fault notifications are not delivered in a timely manner, resulting in low response efficiency. Sixth, the operation and maintenance are complex. The platform has many functions and professional operation logic, and non-dedicated operation and maintenance personnel cannot independently complete configuration, troubleshooting, and operation and maintenance work. Specialized skills training is required, resulting in high human operation and maintenance costs.

[0008] In summary, the two existing mainstream batch scheduling technologies share common technical bottlenecks, making it difficult to simultaneously address the reliability, flexibility, versatility, and ease of operation and maintenance required for financial-grade scheduling scenarios. Existing technologies generally suffer from a series of technical problems, including bloated architecture, complex deployment, poor adaptability to heterogeneous environments, lack of process protection and self-healing capabilities, inability to automatically recover from abnormally interrupted tasks, lack of granular single-machine concurrency control, high risk of resource contention and system downtime, difficulty in troubleshooting, and delayed alarm response. These issues prevent them from meeting the current financial industry's demands for large-scale, highly dependent, time-sensitive, and unattended automated batch job scheduling. Summary of the Invention

[0009] This application provides a batch job scheduling method and system based on Shell scripts and databases to address many shortcomings of existing technologies. It offers a lightweight, highly adaptable, highly reliable, self-healing, easy-to-maintain, and multi-channel alarm financial-grade batch scheduling solution that is compatible with heterogeneous databases and multiple business scenarios, meeting the high-standard scheduling requirements of financial IT systems.

[0010] The embodiments of this application adopt the following technical solutions:

[0011] In a first aspect, embodiments of this application provide a batch job scheduling method based on Shell scripts and a database, including:

[0012] The control program is triggered by the timed triggering module according to a preset cycle. The control program, execution program and monitoring and alarm program are all implemented by Shell scripts. The database is used to store job metadata and job dependencies to decouple the scheduling engine from the business logic and eliminate the need to install a scheduling client on the target server.

[0013] The control program calls a common module to perform a process protection check: it detects the process status of the control program based on the process identifier (PID) file, and restarts the control program when the process of the control program is detected to have died, so that the control program can be automatically restarted by the timed trigger module in the next trigger cycle after the control program is unexpectedly killed.

[0014] The control program reads job definition information, job dependencies, and parameter configuration information from the database, filters executable jobs that meet the execution conditions from the jobs to be executed according to the job dependencies, and controls the number of jobs to be executed concurrently in a single batch according to the parameter configuration information.

[0015] The control program asynchronously calls the execution program and passes job parameters to the execution program. The execution program updates the status of the corresponding job to "in execution" and calls the corresponding execution unit to execute the job according to the job type. The job types include Shell script type, database stored procedure type and file check type.

[0016] After the job is completed, the execution program writes the execution result of the job back to the database and records the execution log.

[0017] The monitoring and alarm program determines alarms based on the execution status of jobs in the database and sends alarm messages when alarm conditions are met.

[0018] In some embodiments, detecting the process state of the control program based on the process identifier (PID) file includes:

[0019] Read the process identifier recorded in the PID file, check whether the process corresponding to the process identifier exists, and whether the process name corresponding to the process matches the program name of the control program;

[0020] When the process does not exist or the process name does not match, it is determined that the control program has been terminated, the invalid PID file is cleaned up, the control program is restarted and a new PID file is written.

[0021] When the process exists and the process name matches, the control program is determined to be running normally, and the program exits immediately upon triggering to avoid repeated execution.

[0022] In some embodiments, controlling the number of jobs executed concurrently in a single batch according to the parameter configuration information includes:

[0023] The configuration value of the maximum number of tasks to be processed in a single batch is read from the parameter configuration table of the database. The configuration value supports online dynamic adjustment and takes effect without restarting the scheduling process after adjustment.

[0024] When the number of executable jobs that meet the execution conditions exceeds the configured value, jobs are dispatched according to the configured value. Undispatched jobs are left to participate in the screening and dispatch in the next trigger cycle, so as to achieve macro-serial and micro-concurrency controlled resource management.

[0025] In some embodiments, the execution conditions include dependency conditions and time window conditions:

[0026] The dependency condition includes: all the preceding dependent jobs of the job have been successfully executed within the current batch cycle;

[0027] The time window condition includes: the current time is within the daily execution time window configured for the job;

[0028] For jobs configured with prerequisite file checks, the execution conditions also include: the prerequisite files required for the job are detected by the job with the file check type.

[0029] In some embodiments, the public module provides a log writing interface, a unified database call interface, and a process verification interface in the form of public functions;

[0030] The unified database call interface encapsulates connection and execution logic for various databases, selects the corresponding connection and execution methods according to the type of the target database, supports the mixed scheduling of Shell scripts and database stored procedures, and adapts to various heterogeneous databases.

[0031] In some embodiments, the executor is further configured to: retry the job according to the retry policy configured for the job when the job fails to execute, the retry policy including the number of retries and the retry interval; and update the status of the job to failure when the number of retries has been exhausted.

[0032] The job status includes pending execution, executing, successful, and failed, and the job status transitions are persistently recorded in the database.

[0033] In some embodiments, the method further includes:

[0034] According to the job dependencies, the start program is triggered first before the batch jobs are executed as a whole, and the start program performs the initialization process of the whole batch.

[0035] After the batch job is completed, a termination procedure is triggered, which performs final processing such as overall batch result verification, temporary file deletion, log cleanup, and accounting date update.

[0036] In some embodiments, the monitoring and alarm program determines alarms based on the execution status of jobs in the database and sends an alarm message when alarm conditions are met, including:

[0037] The monitoring and alarm program periodically scans the execution status of jobs in the database and determines whether a job needs an alarm based on the alarm policy configuration. The alarm policy configuration includes alarm recipients, alarm levels, and alarm convergence policies. The alarm convergence policy is used to merge or suppress duplicate alarms for the same job.

[0038] When the alarm conditions are met, the interface is invoked to send an alarm message to the alarm recipient.

[0039] In some embodiments, the method further includes:

[0040] The visualization module enables configuration and management of job definition information, job dependencies, and parameter configuration information in the database, and displays the execution status of the jobs.

[0041] In response to a manual rerun command or a specified date rollback command initiated through the visualization module, the status of the target job in the database is adjusted, and the execution program is invoked to re-execute the target job; wherein, the specified date rollback command carries a specified business date, which is used to re-execute the target job with the specified business date as the job parameter.

[0042] Secondly, embodiments of this application also provide a batch job scheduling system based on Shell scripts and databases, including a timed triggering module, a control module, a common module, a first execution module, a second execution module, and a monitoring and alarm module;

[0043] The timed triggering module is used to trigger the control program according to a preset period. The control program, execution program and monitoring alarm program are all implemented by Shell scripts. The database is used to store job metadata and job dependencies to decouple the scheduling engine from the business logic and eliminate the need to install a scheduling client on the target server.

[0044] The control module is used by the control program to call the common module to perform process protection checks on the control program: detect the process status of the control program based on the process identifier (PID) file, and restart the control program when the process of the control program is detected to have died, so that the control program can be automatically restarted by the timed trigger module in the next trigger cycle after the control program is unexpectedly killed;

[0045] The common module is used by the control program to read job definition information, job dependencies and parameter configuration information from the database, filter executable jobs that meet the execution conditions from the jobs to be executed according to the job dependencies, and control the number of jobs to be executed concurrently in a single batch according to the parameter configuration information.

[0046] The first execution module is used by the control program to asynchronously call the execution program and pass job parameters to the execution program. The execution program updates the status of the corresponding job to be executing and calls the corresponding execution unit to execute the job according to the job type. The job types include Shell script type, database stored procedure type and file check type.

[0047] The second execution module is used by the execution program to write back the execution result of the job to the database and record the execution log after the job is completed;

[0048] The monitoring and alarm module is used to monitor and alarm programs to make alarm judgments based on the execution status of jobs in the database, and send alarm messages when alarm conditions are met.

[0049] The above-mentioned technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: the batch job scheduling method based on Shell script and database, by adopting a lightweight Shell script program architecture combined with the core design of database metadata management, achieves multiple technical breakthroughs such as decoupling of scheduling logic and business logic, process self-healing protection, fine-grained concurrency management, compatibility with multiple types of jobs, and automated monitoring and alarms, effectively solving the industry shortcomings such as bloated technical architecture, poor adaptability, low reliability, high operation and maintenance costs, and delayed fault response.

[0050] The control program is triggered by a timed trigger module according to a preset cycle. The control program, execution program, and monitoring and alarm program are all implemented using Shell scripts. The database stores job metadata and job dependencies, decoupling the scheduling engine from the business logic. This eliminates the need to install a scheduling client on the target server, achieving a lightweight architecture, non-intrusive deployment, significantly reducing deployment and maintenance costs, and adapting to various heterogeneous server environments. The core control program, execution program, and monitoring and alarm program of this invention are all implemented based on Shell scripts, eliminating the need to install a dedicated scheduling client, web container, or middleware on the target server. This overcomes the drawbacks of traditional commercial scheduling software, such as cumbersome deployment, high environment dependencies, and bloated systems. Furthermore, by centrally storing job metadata, dependencies, and parameter configurations in the database, the scheduling engine and business logic are completely decoupled. Job configuration iterations can be completed without modifying local script code, adapting to the heterogeneous operating environment of multiple databases and business systems in the financial industry. It boasts strong compatibility and versatility, simple deployment, and extremely low resource consumption, perfectly meeting the lightweight and efficient scheduling requirements of the financial sector.

[0051] The control program invokes a common module to perform process protection checks: it detects the process status of the control program based on the process identifier (PID) file. When the process of the control program is detected as terminated, it is restarted. This allows the control program to be automatically restarted by the timed trigger module in the next trigger cycle after an unexpected termination, introducing a process protection self-checking mechanism to achieve self-healing of the scheduler and significantly improve the reliability of batch job scheduling. This invention performs real-time process protection checks on the control program through an independent common module, accurately detecting the running status of the control program based on the PID file. When the program is unexpectedly terminated due to system abnormalities, memory overflow, or human error, it can be automatically restarted and resumed in the next trigger cycle by the timed trigger module. This completely solves the problem in traditional Crontab+Shell script scheduling schemes where a single process abnormality leads to the interruption of all batch jobs and the inability to automatically recover from failures during unattended periods. It effectively avoids business failures caused by batch job stagnation and data loss at night, ensuring stable 24 / 7 unattended operation of financial batch jobs.

[0052] The control program reads job definition information, job dependencies, and parameter configuration information from the database. Based on the job dependencies, it filters executable jobs that meet the execution conditions from the pending jobs. It then controls the number of jobs executed concurrently in a single batch based on the parameter configuration information, achieving visualized and configurable job scheduling and management, supporting fine-grained concurrency control and flexible task orchestration. This invention stores job definitions, upstream and downstream dependencies, and execution parameters uniformly in the database. The control program dynamically reads configuration information to filter executable jobs, eliminating the need for hard-coded scheduling logic. Adding, modifying, or deleting jobs, and adjusting dependencies and concurrency parameters, only requires modifying the database configuration. Changes take effect immediately without requiring a service restart, avoiding the cumbersome, error-prone, and inefficient iteration of traditional script scheduling. Simultaneously, by precisely controlling the number of concurrent jobs executed in a single batch through parameter configuration, it solves problems such as host CPU and memory resource exhaustion, system crashes, and task contention caused by the lack of concurrency control in traditional scripts, achieving reasonable allocation and efficient utilization of server resources.

[0053] The control program asynchronously calls the execution program and passes job parameters to it. The execution program updates the status of the corresponding job to "in execution" and calls the corresponding execution unit to execute the job according to its job type. The job types include Shell scripts, database stored procedures, and file checks, supporting unified scheduling of multiple job types and greatly expanding the adaptability of batch scheduling scenarios. This application embodiment is compatible with three mainstream financial batch job types: Shell scripts, database stored procedures, and file checks. It completes differentiated job scheduling and execution by matching the corresponding execution unit with the execution program, breaking the limitations of traditional scheduling tools that have only a single job type and limited adaptability. It can comprehensively cover various core batch scenarios in the financial industry, such as end-of-day batch processing, data ETL, interest rate calculation, file verification, reconciliation, and report generation. A single architecture can meet the scheduling needs of all categories of batch jobs, significantly improving system reusability.

[0054] After the job is completed, the execution program writes the execution result back to the database and records the execution log, achieving full-chain traceability of job data and significantly improving troubleshooting efficiency. In this embodiment, after job execution, the execution program automatically writes the job execution status, execution result, and running log back to the database, achieving centralized collection and unified storage of job lifecycle data. Compared to the shortcomings of traditional script scheduling logs being scattered and lacking unified status recording, this solution allows for quick querying of execution details and anomaly information for any job, enabling precise fault location and rapid source tracing, significantly reducing the difficulty of operation and maintenance troubleshooting, and improving the efficiency of handling batch problems.

[0055] The monitoring and alarm program determines alarms based on the execution status of jobs in the database and sends alarm messages when alarm conditions are met, thus constructing an automated proactive alarm mechanism to solve the industry problem of delayed fault response at night. This embodiment uses an independent monitoring and alarm program to read the job execution status in the database in real time, automatically completing anomaly judgment and alarm triggering. It can promptly push alarm messages in abnormal scenarios such as job execution failure, timeout, and interruption, changing the passive operation and maintenance mode of traditional script scheduling that lacks alarm capabilities and relies entirely on manual inspection. It overcomes the shortcomings of traditional commercial scheduling platforms, such as single alarm channels and untimely nighttime reach, enabling proactive detection and rapid response to batch faults. This effectively avoids daytime business anomalies caused by faults remaining during unattended nighttime periods, comprehensively ensuring the continuity of financial services and customer service experience.

[0056] In summary, the embodiments of this application take into account multiple advantages such as lightweight deployment, high heterogeneous adaptability, high scheduling reliability, refined resource management, full-process traceability and intelligent alarm. It retains the characteristics of traditional script scheduling, such as lightweight, non-redundant and low cost, while solving its defects such as poor reliability, difficult maintenance and lack of control. At the same time, it avoids the problems of commercial scheduling platforms, such as bloated complexity, high coupling and high operation and maintenance threshold. It is extremely suitable for financial-grade high-precision, high-reliability, unattended large-scale batch operation scheduling scenarios such as banking and securities. Attached Figure Description

[0057] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0058] Figure 1 This is a schematic diagram of the system architecture of the batch job scheduling method based on Shell scripts and database in the embodiments of this application;

[0059] Figure 2 This is a flowchart illustrating the batch job scheduling method based on Shell scripts and database in the embodiments of this application;

[0060] Figure 3 This is a schematic diagram of the structure of the batch job scheduling device based on Shell scripts and database in the embodiments of this application;

[0061] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0064] like Figure 1 The diagram shown illustrates the system architecture of the batch job scheduling method based on Shell scripts and a database in this embodiment of the application. It details the data relationships between the modules of the batch task scheduling system, which consists of eight parts: a timed triggering module, a control module, a common module, a database, an execution module, execution units, a monitoring and alarm module, and a visualization module. Each module is organized in a loosely coupled manner with the database tables. All scheduling logic is carried out by the Shell scripts, while the database only stores job metadata and dependencies, thus decoupling the scheduling engine from the business logic. The following sections describe each module and its connections.

[0065] (1) Timed Trigger Module (Crontab): This is the starting point for the system's automated scheduling. It uses the Crontab timed task built into the operating system to generate a "timed trigger" signal according to a preset period to start the control module. The target server does not need to install any scheduling client.

[0066] (2) Control module (batch_start.sh): This is the core scheduling engine of the system, which contains six sub-functions in sequence: ① process protection check, ② obtaining jobs to be executed, ③ dependency judgment, ④ concurrency control, ⑤ calling the execution program (nohup), and ⑥ loop sleep. Among them, the process protection check is based on the PID file to realize process status detection and self-healing, ensuring the continuous operation of batch jobs 24 / 7.

[0067] (3) Public module (batch_pub.sh): Provides public capabilities for the system. It contains global variables and public functions. The public functions cover log writing, SQL execution (unified database call interface) and process verification, which can be reused by the control module, execution module and execution unit.

[0068] (4) Database: Used to persistently store job metadata and running status. It contains a batch date table, job definition table, file job relationship table, dependency relationship table, parameter configuration table and log table, which respectively implement the functions of job definition, dependency maintenance, parameter configuration and execution log recording.

[0069] (5) Execution module (batch_deal.sh): This is the executor of the specific task. It contains five sub-functions in sequence: ① parameter parsing, ② status update, ③ execution by type (supports three types: Shell script, stored procedure and file check), ④ retry processing, and ⑤ status write-back.

[0070] (6) Execution Unit: This is the set of underlying jobs to be scheduled for execution, including the start program (deal_start.sh), job 1, job 2, ..., job n, and the end program (deal_end.sh). The start program is triggered first before the batch jobs are executed as a whole to perform the initialization of the batch; the end program is triggered last after the batch jobs are executed as a whole to perform the final processing such as batch result verification, file deletion, log cleanup, and accounting date update.

[0071] (7) Monitoring and alarm module (batch_alert.sh): It is used for monitoring the task execution status and proactive alarms. It includes functions such as task execution status monitoring, alarm policy judgment, alarm recipient setting, alarm message sending and enterprise WeChat interface calling.

[0072] (8) Visualization module: Provides human-computer interaction entry for operation and maintenance personnel, with four types of operation capabilities: query, modification, trigger, and configuration. It can realize functions such as task configuration, status monitoring, status adjustment, and manual rerun on a specified date.

[0073] like Figure 1 As shown, the connections and data flow between the various modules of the system are as follows:

[0074] (1) The timed trigger module connects to the control module through the "timed trigger" signal, periodically starts the control module, and forms the starting point of automated scheduling; (2) The control module connects to the public module through "call public function", and after starting, it first calls the process verification function of the public module to perform process protection checks; (3) The control module connects to the database through "read configuration / dependency", obtains the jobs to be executed from the job definition table, dependency table and parameter configuration table of the database, filters the executable jobs according to the dependency, and controls the maximum number of processing jobs in a single batch through the parameter configuration table; (4) The control module connects to the execution module through "asynchronous call nohup", asynchronously calls the execution module in the nohup mode for jobs that meet the conditions and passes the job parameters; (5) The execution module connects to the database through "get information". The database reads job-related information and writes the execution process and results back to the database log table through "update log"; (6) The execution module connects to the execution unit through "call execution unit" and calls the underlying execution unit according to the job type (Shell / stored procedure / file check). The execution unit also reuses the capabilities of the public module through "call public function" during the execution process; (7) The monitoring and alarm module connects to the database through "get task status", reads the execution status of the job in the database and makes alarm judgment accordingly, and then calls the enterprise WeChat interface to send alarm messages; (8) The visualization module connects to the database through "query update", queries and updates the configuration and status in the database, and connects to the execution module through "trigger execution program" to realize manual rerun or rerun on a specified date.

[0075] exist Figure 1 A complete scheduling process in the system is as follows: The timed trigger module periodically triggers the control module → The control module first completes process protection checks through the common module, then reads the configuration and dependencies from the database, filters executable jobs, and performs concurrency control → The executable job is asynchronously called to the execution module → The execution module updates the job status and calls the execution unit to execute according to the type, and the execution result is written back to the database → The monitoring and alarm module judges the alarm based on the database status and sends alarms → The visualization module performs configuration management and status query of the database, and can trigger the execution module to perform manual rerun or rerun on a specified date.

[0076] This application provides a batch job scheduling method based on Shell scripts and a database, such as... Figure 2 The diagram illustrates a batch job scheduling method based on Shell scripts and a database, as shown in this application embodiment. The method includes at least the following steps S210 to S260:

[0077] This application provides a batch job scheduling system based on Shell scripts and a database. The system implements the scheduling logic using pure Shell scripts, and the database only stores task metadata and dependencies, thus completely decoupling the two. The system consists of a common module batch_pub.sh, a control program batch_start.sh, an execution program batch_deal.sh, a monitoring and alarm program batch_alert.sh, a start program deal_start.sh, and an end program deal_end.sh. The database side includes six tables: a batch date table BATCH_CTRL, a job definition table JOB_DEF, a file job relationship table JOB_FILE, a dependency relationship table JOB_DEP, a parameter configuration table BATCH_PARAM, and a log table JOB_LOG. These tables are used for batch accounting date control, job definition, maintenance of the correspondence between prerequisite files and jobs, dependency maintenance, parameter configuration, and execution log recording, respectively. The batch date table BATCH_CTRL, job definition table JOB_DEF, file job relationship table JOB_FILE, dependency relationship table JOB_DEP, parameter configuration table BATCH_PARAM, and log table JOB_LOG mentioned above are merely examples used to illustrate the following specific implementation methods and are not intended to limit the scope of protection in the embodiments of this application.

[0078] Step S210: The timed triggering module triggers the control program according to a preset cycle. The control program, execution program and monitoring alarm program are all implemented by Shell scripts. The database is used to store job metadata and job dependencies to decouple the scheduling engine from the business logic and eliminate the need to install a scheduling client on the target server.

[0079] The control program is triggered by a timed triggering module according to a preset cycle. This timed triggering module is the operating system's built-in Crontab (e.g., configured to trigger every 30 minutes as "* / 30**** / home / batch / bin / batch_start.sh"). The control program batch_start.sh, the execution program batch_deal.sh, and the monitoring and alarm program batch_alert.sh are all implemented using shell scripts. The database is only used to store job metadata and job dependencies. The scheduling engine is completely decoupled from the business logic, and the target server does not need to install any scheduling client, thus possessing the characteristics of being lightweight, deployment-free, and easily portable.

[0080] In step S220, the control program calls the common module to perform a process protection check on the control program: based on the process identifier (PID) file, the process status of the control program is detected, and when the process of the control program is detected to have died, the control program is restarted so that the control program can be automatically restarted by the timed trigger module in the next trigger cycle after the control program is unexpectedly killed.

[0081] The control program calls a common module to perform process protection checks: it detects the process status of the control program based on the process identifier PID file (such as / var / run / batch_start.pid). When the process of the control program is detected to have died, the control program is restarted so that the timed trigger module can automatically start the control program in the next trigger cycle after the control program dies unexpectedly, ensuring the 24 / 7 continuity of batch operations.

[0082] In step S230, the control program reads job definition information, job dependencies and parameter configuration information from the database, selects executable jobs that meet the execution conditions from the jobs to be executed according to the job dependencies, and controls the number of jobs to be executed concurrently in a single batch according to the parameter configuration information.

[0083] The control program reads job definition information (JOB_DEF), job dependencies (JOB_DEP), and parameter configuration information (BATCH_PARAM) from the database. Based on the job dependencies, it selects executable jobs that meet the execution conditions from the jobs to be executed, and controls the number of jobs to be executed concurrently in a single batch based on the parameter configuration information.

[0084] In step S240, the control program asynchronously calls the execution program and passes job parameters to the execution program. The execution program updates the status of the corresponding job to "in execution" and calls the corresponding execution unit to execute the job according to the job type. The job types include Shell script type, database stored procedure type, and file check type.

[0085] The control program operates asynchronously.

[0086] (e.g., "nohupbatch_deal.sh")<JOB_ID><ACCT_DATE> &” calls the executor and passes job parameters to it; the executor updates the status of the corresponding job to “running” and calls the corresponding execution unit to execute the job according to the job type, which includes Shell script type, database stored procedure type and file check type.

[0087] In step S250, after the job is completed, the execution program writes the execution result of the job back to the database and records the execution log.

[0088] After the job is completed, the execution program writes the execution result back to the database (JOB_LOG) and records the execution log.

[0089] Step S260: The monitoring and alarm program makes an alarm judgment based on the execution status of the jobs in the database, and sends an alarm message when the alarm conditions are met.

[0090] The monitoring and alarm program determines alarms based on the execution status of jobs in the database and sends alarm messages when alarm conditions are met.

[0091] Compared with the prior art, the batch job scheduling method and system based on Shell scripts and databases provided in this application have the following beneficial effects:

[0092] 1) Lightweight architecture, low deployment and maintenance costs

[0093] In this embodiment, a lightweight, decoupled architecture is adopted, which implements scheduling logic using pure Shell scripts and stores only job metadata and dependencies in the database. Compared with heavy Java open-source schedulers and commercial scheduling software, it does not require the installation of a dedicated client or specific operating environment on the target server, making deployment simple. At the same time, the scheduling code logic is clear and highly versatile, and internal IT personnel can easily maintain it without special training, significantly reducing the system's deployment cost and long-term operation and maintenance cost.

[0094] (2) Process self-healing ensures business continuity.

[0095] In this embodiment, the process protection check based on PID file and the timed triggering mechanism are combined to overcome the defect of traditional script-based scheduling that "if a process dies, the whole batch stops". When the scheduling process exits abnormally, the process can be automatically detected and rebuilt in the next triggering cycle, realizing the continuous operation of batch jobs 24 / 7 in unattended scenarios and ensuring business continuity.

[0096] (3) Concurrency is controllable, protecting the stability of the production system.

[0097] In this embodiment, the number of concurrent jobs executed in a single batch is dynamically controlled through a parameter configuration table, realizing a resource scheduling mode of "macro-serialization and micro-concurrency control". The configuration values ​​can be adjusted online and take effect immediately, effectively avoiding the risk of host load spikes and resource exhaustion caused by excessive job concurrency, and protecting the stability of the production system.

[0098] (4) Database-independent, supports heterogeneous hybrid scheduling

[0099] This application embodiment uses a unified database call interface encapsulated by public functions to completely decouple the scheduling engine from the underlying database type, which can be adapted to various mainstream relational databases such as Gaussian, Oracle, and MySQL. In heterogeneous environments, only configuration needs to be modified to switch. At the same time, it supports the mixed orchestration of shell scripts and database stored procedures in the same workflow, overcoming the limitation of most schedulers that only support a single execution type (such as pure Java or pure SQL).

[0100] (5) Flexible rerunning and return runs improve operation and maintenance efficiency.

[0101] This application embodiment supports multiple compensation mechanisms, such as specified business date rollback, pre-file readiness check, automatic retry upon failure, and manual retry via a visual page. These mechanisms can cover various abnormal scenarios, including data correction, file delay arrival, and instantaneous failures, significantly improving the efficiency of batch operation and maintenance.

[0102] (6) Real-time proactive alarms to form a closed loop of operation and maintenance.

[0103] Compared to traditional scheduling software that relies on email alerts (which are delayed and easily overwhelmed), this embodiment pushes alerts through an instant messaging interface (such as WeChat for Business), enabling maintenance personnel to perceive operational anomalies in real time on their mobile terminals and intervene immediately. This forms a closed-loop maintenance system of "detecting failures and responding to them," significantly shortening the time for fault discovery and recovery, and especially ensuring the reliability of batch operations during unattended nighttime hours.

[0104] In one embodiment of this application, detecting the process status of the control program based on the process identifier (PID) file includes: reading the process identifier recorded in the PID file; detecting whether the process corresponding to the process identifier exists and whether the process name corresponding to the process matches the program name of the control program; when the process does not exist or the process name does not match, determining that the control program has been terminated, cleaning up the invalid PID file, restarting the control program and writing a new PID file; when the process exists and the process name matches, determining that the control program is running normally, and exiting directly in this trigger to avoid the control program running repeatedly.

[0105] In its implementation, the public module provides the process verification function `check_pid`: if the PID file does not exist, it is considered the first run, and the current process ID is directly written and the process continues; if the PID file exists, the integer process ID is read from it and processed via " / proc / <pid>"Directory existence (Linux) or "ps -p" <pid>"It determines whether the process is alive and further verifies whether the process's command line contains "batch_start.sh" to rule out misjudgments caused by the operating system's PID reuse. Since Crontab triggers the control program every cycle, even if it is not started immediately this time, it will automatically restart in the next cycle, realizing the self-healing of the scheduling process."

[0106] In one embodiment of this application, controlling the number of concurrently executed jobs in a single batch according to the parameter configuration information includes: reading the configuration value of the maximum processing task volume in a single batch from the parameter configuration table of the database, wherein the configuration value supports online dynamic adjustment and takes effect without restarting the scheduling process after adjustment; when the number of executable jobs that meet the execution conditions exceeds the configuration value, jobs are dispatched according to the configuration value, and undispatched jobs are reserved for screening and dispatch in the next trigger cycle, so as to achieve macro-serial and micro-concurrency controlled resource management.

[0107] The configuration value of the maximum number of tasks to be processed in a single batch is read from the parameter configuration table BATCH_PARAM in the database (e.g., PARAM_NAME='MAX_BATCH', PARAM_VALUE=10). This configuration value can be dynamically adjusted online, and the adjustment takes effect without restarting the scheduling process. When the number of executable jobs that meet the execution conditions exceeds the configuration value, jobs are dispatched according to the configuration value. Undispatched jobs are left to participate in the screening and dispatch in the next trigger cycle, so as to achieve macro-serial and micro-concurrency controlled resource management.

[0108] For example, if there are currently 25 jobs that meet the execution conditions and MAX_BATCH=10, then 10 jobs will be dispatched in the first cycle, 10 jobs in the second cycle, and 5 jobs in the third cycle. As each job completes, its downstream dependent jobs will be released and executed in subsequent cycles. This mechanism avoids both the resource exhaustion caused by the lack of concurrency control in traditional scripts and the high scheduling overhead of commercial platforms.

[0109] In one embodiment of this application, the execution conditions include dependency conditions and time window conditions: the dependency conditions include: all the preceding dependent jobs of the job are executed successfully in the current batch cycle; the time window conditions include: the current time is within the daily execution time window configured for the job; wherein, for jobs configured with preceding file checks, the execution conditions also include: the preceding files required by the job are detected by a file check type job.

[0110] The dependency condition is that all preceding dependent jobs of the job have been successfully executed within the current batch period (identified by the accounting date ACCT_DATE); the time window condition is that the current system time is within the daily execution time window configured for the job (JOB_DEF.TIME_WINDOW, such as 02:00–04:00); for jobs configured with preceding file checks, the execution condition also includes detecting that the preceding files required by the job are ready through a file check type job (e.g., checking if / data / inbound / ${ACCT_DATE} / *.dat exists and its size is non-zero), used to verify the arrival of monthly / daily batch data files. The control program sequentially verifies that all the above conditions are met before including each job in the executable set.

[0111] In one embodiment of this application, the public module provides a log writing interface, a unified database call interface, and a process verification interface in the form of public functions; wherein, the unified database call interface encapsulates connection and execution logic for multiple databases, selects the corresponding connection method and execution method according to the type of the target database, so as to support the mixed scheduling of Shell scripts and database stored procedures and adapt to multiple heterogeneous databases.

[0112] The common module batch_pub.sh provides a log writing interface write_log, a unified database call interface exec_sql, and a process verification interface check_pid in the form of common functions. Among them, the unified database call interface encapsulates the connection and execution logic of various databases. It selects the corresponding connection method and execution method according to the type of the target database (global variable DB_TYPE) to support the mixed scheduling of shell scripts and database stored procedures, and adapts to various heterogeneous databases such as Gaussian, Oracle, and MySQL.

[0113] In one embodiment of this application, the execution program is further configured to: retry the job according to the retry policy configured for the job when the job fails to execute, the retry policy including the number of retries and the retry interval; when the number of retries is exhausted and the job still fails to execute, update the status of the job to failure; wherein the status of the job includes pending execution, executing, success and failure, and the status transition of the job is persistently recorded in the database.

[0114] When a job fails to execute, it is retried according to the retry policy configured in the job configuration. The retry policy includes the number of retries (JOB_DEF.RETRY_TIMES) and the retry interval (JOB_DEF.RETRY_INTERVAL). If the job still fails after the number of retries has been exhausted, the job status is updated to "failed". The job status includes WAIT, RUNNING, SUCCESS, and FAILED. The job status transitions are persistently recorded in the database (JOB_LOG.STATUS) for downstream dependent jobs and monitoring and alarm modules to read.

[0115] In one embodiment of this application, the method further includes: according to the job dependency relationship, first triggering a start program before the batch job is executed as a whole, and having the start program perform the initialization process of the whole batch; after the batch job is executed as a whole, triggering a stop program, and having the stop program perform the final processing of the whole batch result verification, temporary file deletion, log cleanup and accounting date update.

[0116] Based on job dependencies, the start program deal_start.sh is triggered before the batch job is executed as a whole. It performs the initialization of the entire batch (creating and verifying the run lock directory, preparing the temporary working directory, backing up the snapshot of the previous cycle, setting the accounting date ACCT_DATE to the current business date and marking the batch as "in progress"). After the batch job is executed as a whole, the end program deal_end.sh is triggered. It performs the final processing of the entire batch result verification, temporary file deletion, log cleanup and accounting date update (verifying the number of successful critical jobs and the data row threshold, cleaning up intermediate artifacts in / tmp / batch / ${ACCT_DATE}, deleting historical logs from N days ago, and advancing BATCH_CTRL.ACCT_DATE to the next business date).

[0117] In one embodiment of this application, the monitoring and alarm program makes an alarm judgment based on the execution status of jobs in the database and sends an alarm message when the alarm conditions are met. This includes: the monitoring and alarm program periodically scanning the execution status of jobs in the database, determining whether a job needs an alarm based on an alarm policy configuration, wherein the alarm policy configuration includes an alarm recipient, an alarm level, and an alarm convergence policy, and the alarm convergence policy is used to merge or suppress duplicate alarms for the same job; and when the alarm conditions are met, calling an interface to send an alarm message to the alarm recipient.

[0118] The monitoring and alarm program periodically (e.g., every 5 minutes) scans the execution status of jobs in the database and determines whether a job needs to be alarmed based on the alarm policy configuration (ALERT_POLICY: alarm recipient, alarm level, alarm convergence strategy). The alarm convergence strategy is used to merge or suppress duplicate alarms for the same job (e.g., the same job alarms only once within 30 minutes). When the alarm conditions are met, an interface (e.g., WeChat webhook) is called to send an alarm message to the alarm recipient, realizing immediate and proactive notification of faults at night.

[0119] In one embodiment of this application, the method further includes: configuring and managing job definition information, job dependencies, and parameter configuration information in the database through a visualization module, and displaying the execution status of the job; responding to a manual rerun instruction or a specified date rerun instruction initiated through the visualization module, adjusting the status of the target job in the database, and calling the execution program to re-execute the target job; wherein the specified date rerun instruction carries a specified business date, used to re-execute the target job with the specified business date as a job parameter.

[0120] The visualization module manages and configures job definitions, dependencies, and parameter settings in the database, and displays the job execution status. Responding to manual rerun or specified date rollback commands initiated through the visualization module, it adjusts the status of the target job in the database (e.g., setting JOB_LOG.STATUS back to pending execution) and calls the execution program to re-execute the target job. The specified date rollback command carries a specified business date (BIZ_DATE) and is used to re-execute the target job with this specified business date as the job parameter (ACCT_DATE), commonly used for data correction. All configuration changes are written back to the database via a unified interface, and the scheduling engine automatically detects and executes them in the next trigger cycle.

[0121] It should be noted that the core concept of the embodiments of this application lies in the decoupling of the scheduling engine from business logic and the lightweight batch scheduling architecture based on the database. The specific implementation methods in the aforementioned embodiments (such as Crontab, PID file, nohup, Shell public functions, Web front-end, single-machine deployment, etc.) are all preferred implementations. Without departing from the core concept of the embodiments of this application, those skilled in the art can adopt the following alternative implementation methods, which also fall within the protection scope of this application.

[0122] (a) Alternative to process protection: In the aforementioned embodiments, the control program is protected and self-healed through Crontab timed triggering combined with a PID file. In other embodiments of this application, the Linux system-level systemd service or supervisor process management tool can also be used to monitor the scheduling master process: the control program is registered as a system service, and the system service manager automatically restarts it when the process exits abnormally. In this case, the timed triggering module and the process protection check function are handled by the system service manager, which also achieves automatic restart after the scheduling process unexpectedly dies.

[0123] (II) Alternative Concurrency Control Methods: In the aforementioned embodiments, the maximum number of concurrent processes per batch on a single host is controlled through a parameter configuration table. In other embodiments of this application, a message queue (such as Redis or RabbitMQ) can also be introduced: the control program is responsible for enqueuing jobs that meet the execution conditions, multiple execution program instances consume concurrently as consumers, and the maximum concurrency is controlled through a semaphore mechanism; the scheduling logic can also be implemented in other programming languages ​​as a multi-threaded version, and the concurrency limit is controlled through the thread pool size. The above alternative methods also achieve the resource management effect of "macro-serialization and micro-concurrency control".

[0124] (III) Alternative Implementation Language for Public Functions: In the foregoing embodiments, the public modules are implemented using Shell scripts. In other embodiments of this application, core public logic such as log writing and SQL execution can also be encapsulated as Python scripts or compiled into binary executable files, with the control program and execution program called via command line. This alternative approach also decouples the scheduling engine from the database access logic and can run across platforms (such as Windows environments).

[0125] (iv) Alternatives to Front-End Interaction: In the foregoing embodiments, the visualization module provides task configuration and status monitoring via a web front-end. In other embodiments of this application, Excel import / export functionality can also be provided in scenarios without a front-end interface: operations personnel can edit Excel files to batch modify job dependencies and then import the changes into the database for effect; or a standard REST API interface can be provided for external systems to call to complete configuration management and status queries. These alternative methods also achieve the capability to manage job definition information, job dependencies, and parameter configuration information in the database.

[0126] (V) Distributed Deployment and Alternatives to Anti-Duplicate Methods: In the aforementioned embodiments, the system is deployed on a single machine. In other embodiments of this application, the system can also be distributed across multiple scheduling nodes: Before executing a task, each scheduling node implements anti-duplicate measures by updating the task status in the database—updating the task status from "pending execution" to "in execution," with the update condition being "current status is pending execution." The atomicity of a single SQL update operation in the database ensures that the same task is successfully updated by only one node. This method does not rely on additional distributed lock components and only utilizes the atomicity of a single SQL statement to achieve anti-duplicate measures, making it the most lightweight approach. Alternatively, it can be implemented based on a Redis distributed lock: Before executing a task, the scheduling node writes key-value pairs with expiration times to Redis, and the node that successfully writes the pair gains execution rights. This method can reduce database connection pressure in high-concurrency scenarios, preventing database connections from becoming a bottleneck.

[0127] This application also provides a batch job scheduling system 300 based on Shell scripts and a database, such as... Figure 3 The diagram shows a schematic of a batch job scheduling system based on Shell scripts and a database, as illustrated in this application embodiment. The system 300 includes at least: a timed triggering module 310, a control module 320, a common module 330, a first execution module 340, a second execution module 350, and a monitoring and alarm module 360, wherein:

[0128] In one embodiment of this application, the timed triggering module 310 is specifically used to: trigger a control program according to a preset period, wherein the control program, execution program and monitoring alarm program are all implemented by Shell scripts, and the database is used to store job metadata and job dependencies to achieve decoupling between the scheduling engine and business logic, and without the need to install a scheduling client on the target server.

[0129] The control program is triggered by a timed triggering module according to a preset cycle. The timed triggering module is the operating system's built-in Crontab (e.g., configured to trigger every 30 minutes with "* / 30**** / home / batch / bin / batch_start.sh"). The control program batch_start.sh, the execution program batch_deal.sh, and the monitoring and alarm program batch_alert.sh are all implemented using shell scripts. The database is only used to store job metadata and job dependencies. The scheduling engine is completely decoupled from the business logic, and the target server does not need to install any scheduling client, thus possessing the characteristics of being lightweight, deployment-free, and easily portable.

[0130] In one embodiment of this application, the control module 320 is specifically used for: the control program reading job definition information, job dependency relationship and parameter configuration information from the database, filtering executable jobs that meet the execution conditions from the jobs to be executed according to the job dependency relationship, and controlling the number of jobs to be executed concurrently in a single batch according to the parameter configuration information.

[0131] The control program reads job definition information (JOB_DEF), job dependencies (JOB_DEP), and parameter configuration information (BATCH_PARAM) from the database. Based on the job dependencies, it selects executable jobs that meet the execution conditions from the jobs to be executed, and controls the number of jobs to be executed concurrently in a single batch based on the parameter configuration information.

[0132] In one embodiment of this application, the common module 330 is specifically used for: the control program reading job definition information, job dependency relationship and parameter configuration information from the database, filtering executable jobs that meet the execution conditions from the jobs to be executed according to the job dependency relationship, and controlling the number of jobs to be executed concurrently in a single batch according to the parameter configuration information.

[0133] The control program reads job definition information (JOB_DEF), job dependencies (JOB_DEP), and parameter configuration information (BATCH_PARAM) from the database. Based on the job dependencies, it selects executable jobs that meet the execution conditions from the jobs to be executed, and controls the number of jobs to be executed concurrently in a single batch based on the parameter configuration information.

[0134] In one embodiment of this application, the first execution module 340 is specifically used for: the control program asynchronously calling the execution program and passing job parameters to the execution program; the execution program updating the status of the corresponding job to be executing; and calling the corresponding execution unit to execute the job according to the job type of the job; the job type includes Shell script type, database stored procedure type and file check type.

[0135] The control program operates asynchronously.

[0136] (e.g., "nohupbatch_deal.sh")<JOB_ID><ACCT_DATE> &” calls the executor and passes job parameters to it; the executor updates the status of the corresponding job to “running” and calls the corresponding execution unit to execute the job according to the job type, which includes Shell script type, database stored procedure type and file check type.

[0137] In one embodiment of this application, the second execution module 350 is specifically used to: after the job is executed, write the execution result of the job back to the database and record the execution log.

[0138] After the job is completed, the execution program writes the execution result back to the database (JOB_LOG) and records the execution log.

[0139] In one embodiment of this application, the monitoring and alarm module 360 ​​is specifically used for: the monitoring and alarm program to make alarm judgments based on the execution status of the jobs in the database, and to send an alarm message when the alarm conditions are met.

[0140] The monitoring and alarm program determines alarms based on the execution status of jobs in the database and sends alarm messages when alarm conditions are met.

[0141] In one embodiment of this application, the control module 320 is further configured to:

[0142] Read the process identifier recorded in the PID file, check whether the process corresponding to the process identifier exists, and whether the process name corresponding to the process matches the program name of the control program;

[0143] When the process does not exist or the process name does not match, it is determined that the control program has been terminated, the invalid PID file is cleaned up, the control program is restarted and a new PID file is written.

[0144] When the process exists and the process name matches, the control program is determined to be running normally, and the program exits immediately upon triggering to avoid repeated execution.

[0145] In one embodiment of this application, the common module 330 is further configured to:

[0146] The configuration value of the maximum number of tasks to be processed in a single batch is read from the parameter configuration table of the database. The configuration value supports online dynamic adjustment and takes effect without restarting the scheduling process after adjustment.

[0147] When the number of executable jobs that meet the execution conditions exceeds the configured value, jobs are dispatched according to the configured value. Undispatched jobs are left to participate in the screening and dispatch in the next trigger cycle, so as to achieve macro-serial and micro-concurrency controlled resource management.

[0148] In one embodiment of this application, the execution conditions include dependency conditions and time window conditions:

[0149] The dependency condition includes: all the preceding dependent jobs of the job have been successfully executed within the current batch cycle;

[0150] The time window condition includes: the current time is within the daily execution time window configured for the job;

[0151] For jobs configured with prerequisite file checks, the execution conditions also include: the prerequisite files required for the job are detected by the job with the file check type.

[0152] In one embodiment of this application, the public module provides a log writing interface, a unified database call interface, and a process verification interface in the form of public functions;

[0153] The unified database call interface encapsulates connection and execution logic for various databases, selects the corresponding connection and execution methods according to the type of the target database, supports the mixed scheduling of Shell scripts and database stored procedures, and adapts to various heterogeneous databases.

[0154] In one embodiment of this application, the execution program is further configured to: retry the job according to the retry policy configured for the job when the job fails to execute, the retry policy including the number of retries and the retry interval; and update the status of the job to failure when the number of retries is exhausted.

[0155] The job status includes pending execution, executing, successful, and failed, and the job status transitions are persistently recorded in the database.

[0156] In one embodiment of this application, it further includes: a dependency module, used for

[0157] According to the job dependencies, the start program is triggered first before the batch jobs are executed as a whole, and the start program performs the initialization process of the whole batch.

[0158] After the batch job is completed, a termination procedure is triggered, which performs final processing such as overall batch result verification, temporary file deletion, log cleanup, and accounting date update.

[0159] In one embodiment of this application, the monitoring and alarm module 360 ​​is further configured to:

[0160] The monitoring and alarm program periodically scans the execution status of jobs in the database and determines whether a job needs an alarm based on the alarm policy configuration. The alarm policy configuration includes alarm recipients, alarm levels, and alarm convergence policies. The alarm convergence policy is used to merge or suppress duplicate alarms for the same job.

[0161] When the alarm conditions are met, the interface is invoked to send an alarm message to the alarm recipient.

[0162] In one embodiment of this application, it further includes: a visualization module, used for

[0163] The visualization module enables configuration and management of job definition information, job dependencies, and parameter configuration information in the database, and displays the execution status of the jobs.

[0164] In response to a manual rerun command or a specified date rollback command initiated through the visualization module, the status of the target job in the database is adjusted, and the execution program is invoked to re-execute the target job; wherein, the specified date rollback command carries a specified business date, which is used to re-execute the target job with the specified business date as the job parameter.

[0165] It is understood that the batch job scheduling system based on Shell scripts and databases described above can implement all the steps of the batch job scheduling method based on Shell scripts and databases provided in the foregoing embodiments. The relevant explanations of the batch job scheduling method based on Shell scripts and databases are applicable to the batch job scheduling system based on Shell scripts and databases, and will not be repeated here.

[0166] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 4 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0167] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0168] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0169] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a batch job scheduling system based on shell scripts and databases at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0170] The control program is triggered by the timed triggering module according to a preset cycle. The control program, execution program and monitoring and alarm program are all implemented by Shell scripts. The database is used to store job metadata and job dependencies to decouple the scheduling engine from the business logic and eliminate the need to install a scheduling client on the target server.

[0171] The control program calls a common module to perform a process protection check: it detects the process status of the control program based on the process identifier (PID) file, and restarts the control program when the process of the control program is detected to have died, so that the control program can be automatically restarted by the timed trigger module in the next trigger cycle after the control program is unexpectedly killed.

[0172] The control program reads job definition information, job dependencies, and parameter configuration information from the database, filters executable jobs that meet the execution conditions from the jobs to be executed according to the job dependencies, and controls the number of jobs to be executed concurrently in a single batch according to the parameter configuration information.

[0173] The control program asynchronously calls the execution program and passes job parameters to the execution program. The execution program updates the status of the corresponding job to "in execution" and calls the corresponding execution unit to execute the job according to the job type. The job types include Shell script type, database stored procedure type and file check type.

[0174] After the job is completed, the execution program writes the execution result of the job back to the database and records the execution log.

[0175] The monitoring and alarm program determines alarms based on the execution status of jobs in the database and sends alarm messages when alarm conditions are met.

[0176] The above is as stated in this application. Figure 2 The batch job scheduling system execution method based on Shell scripts and databases disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0177] The electronic device can also perform Figure 2 This paper describes a method for executing a batch job scheduling system based on Shell scripts and a database, and implements the system in [the context of the paper]. Figure 2 The functions of the embodiments shown are not described in detail here.

[0178] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 2 The illustrated embodiment describes a method for executing a batch job scheduling system based on Shell scripts and a database, specifically used for the following:

[0179] The control program is triggered by the timed triggering module according to a preset cycle. The control program, execution program and monitoring and alarm program are all implemented by Shell scripts. The database is used to store job metadata and job dependencies to decouple the scheduling engine from the business logic and eliminate the need to install a scheduling client on the target server.

[0180] The control program calls a common module to perform a process protection check: it detects the process status of the control program based on the process identifier (PID) file, and restarts the control program when the process of the control program is detected to have died, so that the control program can be automatically restarted by the timed trigger module in the next trigger cycle after the control program is unexpectedly killed.

[0181] The control program reads job definition information, job dependencies, and parameter configuration information from the database, filters executable jobs that meet the execution conditions from the jobs to be executed according to the job dependencies, and controls the number of jobs to be executed concurrently in a single batch according to the parameter configuration information.

[0182] The control program asynchronously calls the execution program and passes job parameters to the execution program. The execution program updates the status of the corresponding job to "in execution" and calls the corresponding execution unit to execute the job according to the job type. The job types include Shell script type, database stored procedure type and file check type.

[0183] After the job is completed, the execution program writes the execution result of the job back to the database and records the execution log.

[0184] The monitoring and alarm program determines alarms based on the execution status of jobs in the database and sends alarm messages when alarm conditions are met.

[0185] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0187] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0188] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0189] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0190] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0191] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0192] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0193] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0194] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.< / pid> < / pid>

Claims

1. A batch job scheduling method based on Shell scripts and databases, characterized in that, include: The control program is triggered by the timed triggering module according to a preset cycle. The control program, execution program and monitoring and alarm program are all implemented by Shell scripts. The database is used to store job metadata and job dependencies to decouple the scheduling engine from the business logic and eliminate the need to install a scheduling client on the target server. The control program calls a common module to perform a process protection check: it detects the process status of the control program based on the process identifier (PID) file, and restarts the control program when the process of the control program is detected to have died, so that the control program can be automatically restarted by the timed trigger module in the next trigger cycle after the control program is unexpectedly killed. The control program reads job definition information, job dependencies, and parameter configuration information from the database, filters executable jobs that meet the execution conditions from the jobs to be executed according to the job dependencies, and controls the number of jobs to be executed concurrently in a single batch according to the parameter configuration information. The control program asynchronously calls the execution program and passes job parameters to the execution program. The execution program updates the status of the corresponding job to "in execution" and calls the corresponding execution unit to execute the job according to the job type. The job types include Shell script type, database stored procedure type and file check type. After the job is completed, the execution program writes the execution result of the job back to the database and records the execution log. The monitoring and alarm program determines alarms based on the execution status of jobs in the database and sends alarm messages when alarm conditions are met.

2. The method according to claim 1, characterized in that, The process status detection of the control program based on the process identifier (PID) file includes: Read the process identifier recorded in the PID file, check whether the process corresponding to the process identifier exists, and whether the process name corresponding to the process matches the program name of the control program; When the process does not exist or the process name does not match, it is determined that the control program has been terminated, the invalid PID file is cleaned up, the control program is restarted and a new PID file is written. When the process exists and the process name matches, the control program is determined to be running normally, and the program exits immediately upon triggering to avoid repeated execution.

3. The method according to claim 1, characterized in that, The step of controlling the number of jobs executed concurrently in a single batch according to the parameter configuration information includes: The configuration value of the maximum number of tasks to be processed in a single batch is read from the parameter configuration table of the database. The configuration value supports online dynamic adjustment and takes effect without restarting the scheduling process after adjustment. When the number of executable jobs that meet the execution conditions exceeds the configured value, jobs are dispatched according to the configured value. Undispatched jobs are left to participate in the screening and dispatch in the next trigger cycle, so as to achieve macro-serial and micro-concurrency controlled resource management.

4. The method according to claim 1, characterized in that, The execution conditions include dependency conditions and time window conditions: The dependency condition includes: all the preceding dependent jobs of the job have been successfully executed within the current batch cycle; The time window condition includes: the current time is within the daily execution time window configured for the job; For jobs configured with prerequisite file checks, the execution conditions also include: the prerequisite files required for the job are detected by the job with the file check type.

5. The method according to claim 1, characterized in that, The public module provides a log writing interface, a unified database call interface, and a process verification interface in the form of public functions. The unified database call interface encapsulates connection and execution logic for various databases, selects the corresponding connection and execution methods according to the type of the target database, supports the mixed scheduling of Shell scripts and database stored procedures, and adapts to various heterogeneous databases.

6. The method according to claim 1, characterized in that, The execution program is further configured to: retry the job according to the retry strategy configured for the job when the job fails to execute, wherein the retry strategy includes the number of retries and the retry interval; and update the status of the job to failure after the number of retries has been exhausted. The job status includes pending execution, executing, successful, and failed, and the job status transitions are persistently recorded in the database.

7. The method according to claim 1, characterized in that, The method further includes: According to the job dependencies, the start program is triggered first before the batch jobs are executed as a whole, and the start program performs the initialization process of the whole batch. After the batch job is completed, a termination procedure is triggered, which performs final processing such as overall batch result verification, temporary file deletion, log cleanup, and accounting date update.

8. The method according to claim 1, characterized in that, The monitoring and alarm program determines alarms based on the execution status of jobs in the database and sends alarm messages when alarm conditions are met, including: The monitoring and alarm program periodically scans the execution status of jobs in the database and determines whether a job needs an alarm based on the alarm policy configuration. The alarm policy configuration includes alarm recipients, alarm levels, and alarm convergence policies. The alarm convergence policy is used to merge or suppress duplicate alarms for the same job. When the alarm conditions are met, the interface is invoked to send an alarm message to the alarm recipient.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: The visualization module enables configuration and management of job definition information, job dependencies, and parameter configuration information in the database, and displays the execution status of the jobs. In response to a manual rerun command or a specified date rollback command initiated through the visualization module, the status of the target job in the database is adjusted, and the execution program is invoked to re-execute the target job; wherein, the specified date rollback command carries a specified business date, which is used to re-execute the target job with the specified business date as the job parameter.

10. A batch job scheduling system based on Shell scripts and a database, characterized in that, It includes a timed trigger module, a control module, a common module, a first execution module, a second execution module, and a monitoring and alarm module; The timed triggering module is used to trigger the control program according to a preset period. The control program, execution program and monitoring alarm program are all implemented by Shell scripts. The database is used to store job metadata and job dependencies to decouple the scheduling engine from the business logic and eliminate the need to install a scheduling client on the target server. The control module is used by the control program to call the common module to perform process protection checks on the control program: detect the process status of the control program based on the process identifier (PID) file, and restart the control program when the process of the control program is detected to have died, so that the control program can be automatically restarted by the timed trigger module in the next trigger cycle after the control program is unexpectedly killed; The common module is used by the control program to read job definition information, job dependencies and parameter configuration information from the database, filter executable jobs that meet the execution conditions from the jobs to be executed according to the job dependencies, and control the number of jobs to be executed concurrently in a single batch according to the parameter configuration information. The first execution module is used by the control program to asynchronously call the execution program and pass job parameters to the execution program. The execution program updates the status of the corresponding job to be executing and calls the corresponding execution unit to execute the job according to the job type. The job types include Shell script type, database stored procedure type and file check type. The second execution module is used by the execution program to write back the execution result of the job to the database and record the execution log after the job is completed; The monitoring and alarm module is used to monitor and alarm programs to make alarm judgments based on the execution status of jobs in the database, and send alarm messages when alarm conditions are met.

Images