Distributed batch processing system, method, computer device, and storage medium

Abstract

The present invention relates to the technical field of distributed batch platforms, and discloses a distributed batch processing system, method, computer device, and storage medium, including a batch scheduling center for initializing polling information of different polling mechanisms, executing the polling mechanism corresponding to the trigger condition when the trigger condition is satisfied, polling the execution status of batch jobs, obtaining task information of unprocessed jobs to be assigned in the batch jobs, and transmitting the task information of the batch jobs to a message queue; a batch operation center for taking out the task information from the message queue, performing multiple subcontracting processing on the jobs to be assigned according to the task information, and putting a number of sub-packages back into the message queue. By initializing and setting multiple different polling mechanisms to poll the execution status of batch jobs, technicians can timely master the task progress. Dynamically allocating batch services to multiple machines for parallel processing improves the processing efficiency, reliability, and scalability of batch services.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed batch platforms, and particularly to a distributed batch processing system, method, computer device, and storage medium. Background Art

[0002] Batch processing is a classic software architecture style, namely the data flow style. Components are a series of computing units in a fixed order, and components only interact through data transfer; each processing step is an independent program, and each step must start after the previous step ends; the data must be complete and is transferred as a whole. With the popularization of IBM mainframes in the banking industry, batch processing tasks have been used in mainframes for decades, and many large and medium-sized enterprises are still using them today.

[0003] Mainframes can process gigabyte-scale massive data from high-speed storage devices, apply batch processing programs, and produce valuable results; batch data can be automatically processed in mainframes without user intervention, submitted in the computer, read in and processed in batches, and then produce result outputs. Considering the limitations of a single mainframe, it is particularly important to build a self-built distributed batch platform. Currently, during the operation of batch jobs in traditional stand-alone systems, technicians do not know the amount of data that has been completed and cannot grasp the progress. When an exception occurs in the job, it cannot be processed in a timely manner. Summary of the Invention

[0004] Based on this, in view of the problem that technicians cannot grasp the job progress during the operation of batch jobs in current traditional stand-alone systems, it is necessary to provide a distributed batch processing system, method, computer device, and storage medium.

[0005] A distributed batch processing system includes a batch scheduling center, which is used to initialize the polling information of different polling mechanisms, and is also used to execute the polling mechanism corresponding to the trigger condition when the trigger condition is met, poll the execution status of batch jobs, obtain the task information of the unprocessed jobs to be assigned in the batch jobs, and transmit the task information to the message queue; a batch operation center, which is used to take out the task information from the message queue, perform multiple subcontracting processing on the jobs to be assigned according to the task information to obtain several sub-packages, and put the several sub-packages back into the message queue. The multiple subcontracting processing refers to dividing the batch jobs according to multiple dimensions to obtain several sub-packages.

[0006] In the above-mentioned distributed batch processing system, the batch scheduling center initializes and sets multiple different polling mechanisms. When the triggering condition is met, the corresponding polling mechanism is started to poll the execution status of the batch jobs in real time, enabling technicians to timely grasp the task progress and obtain the task information of the unprocessed and to-be-allocated jobs in the batch jobs, and then transmitting the task information to the message queue. The batch execution center obtains the task information uploaded by the batch scheduling center from the message queue, and executes the subcontracting tasks for the batch jobs according to the task information, supporting flexible multiple subcontracting of the batch jobs to meet the execution requirements for large amounts of data.

[0007] In one embodiment, the distributed batch processing system includes multiple batch scheduling centers and multiple batch operation centers. When the multiple batch scheduling centers are running simultaneously, they are registered in the batch plan task configuration table through the quartz framework, and only one of the batch scheduling centers meets the triggering condition at the same time; the multiple batch operation centers simultaneously compete to obtain the task information from the queue, and the batch operation center that has obtained the task information executes the subcontracting tasks.

[0008] A distributed batch processing method includes initializing the polling information of different polling mechanisms; when the triggering condition is met, executing the polling mechanism corresponding to the triggering condition to poll the execution status of the batch jobs, and obtaining the task information of the unprocessed and to-be-allocated jobs in the batch jobs; transmitting the task information of the batch jobs to the message queue; taking out the task information from the message queue; performing multiple subcontracting processing on the to-be-allocated jobs according to the task information to obtain several sub-packages, where the multiple subcontracting processing refers to dividing the to-be-allocated jobs according to multiple dimensions to obtain several sub-packages; and putting the several sub-packages back into the message queue.

[0009] In one embodiment, initializing the polling information of different polling mechanisms includes initializing different polling mechanisms according to the polling information in the batch system parameter configuration table, and controlling the time interval between the polling mechanisms according to the cron expression in the batch plan task configuration table.

[0010] In one embodiment, the polling mechanisms include timed polling, database polling, file polling, and job merging polling.

[0011] In one embodiment, when the trigger condition is satisfied, the corresponding polling mechanism is executed to poll the execution status of the batch job. Polling the execution status of the batch job includes setting a timed polling task using the Quartz framework. If the current time conforms to the cron expression in the batch plan task configuration table, it is determined that the trigger condition for the timed polling is satisfied, and the execution status of the batch job is polled at regular intervals; executing the structured query statement processed when the trigger condition for the database polling was last satisfied. If the query result is not empty, it is determined that the trigger condition for the database polling is satisfied, and the execution status of the batch job is polled in the database; determining whether the configuration file processed when the trigger condition for the database polling was last satisfied exists. If it exists, it is determined that the trigger condition for the file polling is satisfied, and the execution status of the batch job is polled in the file; when all sub-packages of the batch job have been executed, it is determined whether there are sub-packages that have been completed but not yet merged according to the status of the batch platform library table. If there are, it is determined that the trigger condition for the job merging polling is satisfied, and the execution status of the batch job is polled for job merging.

[0012] In one embodiment, the method further includes periodically querying the version numbers of the batch system parameter configuration table and the batch plan task configuration table; if the version number of the batch system parameter configuration table changes, reloading the polling information of different polling mechanisms; if the version number of the batch plan task configuration table changes, reloading the timed polling task.

[0013] In one embodiment, the sub-packaging process of the batch job according to the task information includes traversing the splitting method in the job parameter configuration and then using multi-threading to sub-package the batch task in sequence.

[0014] A computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of the above embodiments are implemented.

[0015] A computer-readable storage medium stores a computer program thereon, characterized in that when the computer program is executed by a processor, the steps of the method described in any one of the above embodiments are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings described below are only some embodiments recorded in the present specification. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 Block diagram of a distributed batch processing system according to one embodiment of the present invention;

[0018] Figure 2 Method flow schematic diagram of a distributed batch processing method according to one embodiment of the present invention;

[0019] Figure 3 Method flow schematic diagram of polling the execution status of batch jobs according to one embodiment of the present invention;

[0020] Figure 4 Method flow schematic diagram of periodically checking the version number of the configuration table according to one embodiment of the present invention;

[0021] Figure 5 Block diagram of a distributed batch processing device or system S00 according to one embodiment of the present invention. Detailed implementation manners

[0022] To facilitate the understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. The preferred embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present invention more thorough and comprehensive.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0024] The distributed batch platform is a basic platform for providing services externally, mainly to provide high-efficiency and high-performance batch processing services for the business. Batch processing is a classic software architecture style, namely the data flow style. The components are a series of computing units in a fixed order, and the components only interact through data transfer; each processing step is an independent program, and each step must start after the previous step ends; the data must be complete and is transferred as a whole. With the increase in business volume and the accumulation of business data, the originally fast-running batch processing jobs have now become very slow. In view of the problems of "many, chaotic, and miscellaneous" in the background jobs of the current bank production system, the situations of difficult monitoring, difficult operation, and difficult management have emerged.

[0025] This disclosure is based on the decoupling of the concept and business architecture by adopting a distributed approach. By slicing large volumes of business data and performing parallel processing on multiple nodes, a distributed batch platform that supports configuration management is provided.

[0026] Figure 1 FIG. is a structural block diagram of a distributed batch processing system according to an embodiment of the present invention. In one embodiment, the distributed batch processing system may include a batch scheduling center 100 and a batch operation center 200. The batch scheduling center 100 is used to initialize the polling information of different polling mechanisms, and each different polling mechanism corresponds to a corresponding trigger condition. The batch scheduling center 100 is also used to execute the polling mechanism corresponding to the trigger condition when the trigger condition is met, and poll the execution status of the batch job to obtain the task information of the unprocessed to-be-allocated jobs in the batch job. Polling is a way for the CPU to decide how to provide services to peripheral devices, also known as "Programmed I / O".

[0027] The batch scheduling center 100 can also transmit the task information to the message queue MQ (Message Queue). The batch operation center 200 is used to retrieve the task information from the message queue MQ. Perform multiple subcontracting processing on the to-be-allocated job according to the task information to obtain a number of sub-packages. Multiple subcontracting processing refers to dividing the to-be-allocated job according to multiple dimensions to obtain a number of sub-packages. After the subcontracting is completed, the batch operation center 200 can also put the obtained number of sub-packages back into the message queue MQ for waiting for the batch execution cluster to consume the tasks in the sub-packages.

[0028] In one embodiment, the batch execution center 200 uses the SpringBatch framework to execute batch tasks, supports multiple subcontracting of the to-be-allocated job, and supports configuring multiple subcontracting in a custom manner to meet the execution requirements of large amounts of data. The SpringBatch framework is a data processing framework provided by Spring, a lightweight and comprehensive batch processing framework, which can be used to develop powerful enterprise-level batch processing applications for daily operations. Many applications in the enterprise domain require batch processing to perform business operations in a mission-critical environment.

[0029] Multiple subcontracting refers to dividing batch operations according to multiple dimensions. When the number of data details corresponding to a batch of transactions is too large, and the details of each sub-package are still huge after subcontracting according to a single dimension, the dimension can be further increased for subcontracting. Multiple subcontracting can control the size of the sub-packages after subcontracting to meet performance requirements. For example, subcontracting according to a single dimension of institution type may result in too much data under a certain institution, affecting the overall transaction execution time. By subcontracting according to multiple different dimensions such as institution type, time type, task type, etc., it can be ensured that the details of the sub-packages are not too huge, greatly improving the processing efficiency of the distributed batch processing system for batch operations and making the system highly reliable.

[0030] In the above-mentioned distributed batch processing system, the batch scheduling center 100 initializes and sets various different polling mechanisms. When the trigger condition is met, the corresponding polling mechanism is started to poll the execution status of the batch operation in real time. Thus, technicians can timely grasp the task progress and obtain the task information of the unprocessed and to-be-allocated jobs in the batch operation, and transmit the task information to the message queue. The batch execution center 200 obtains the task information uploaded by the batch scheduling center 100 from the message queue, and performs the multiple subcontracting task of the batch operation according to the task information, supporting the flexible subcontracting of the batch operation to meet the execution requirements of large data volumes.

[0031] In one of the embodiments, the distributed batch system may further include a batch task management center and a centralized monitoring platform. The distributed batch system can integrate the end-of-day batch and the online batch together and uniformly manage them in the batch task management center. By integrating the end-of-day batch and the online batch together, it is beneficial for unified management. When it is necessary to immediately stop the batch operation for some reason, the system can promptly respond to the stop instruction. The management end function of the centralized monitoring platform is more comprehensive and can be used to monitor job operation, platform operation, and scheduling plan, and is also used to alarm batch errors. The batch processing service function interface of the centralized monitoring platform can filter tasks in various states, making the operation more convenient and improving the user-friendliness of the interface.

[0032] In the present disclosure, the distributed batch system provides services in the form of a basic platform, supporting centralized deployment and distributed deployment. The construction goal of the distributed batch system is to decouple from the business architecture, providing dynamic resource allocation, multi-dimensional sharding strategies, flexible task plans, and convenient visual management to efficiently execute batch processing tasks for ultra-large data sets. Multiple polling mechanisms are set in the batch scheduling center, so that the task processing progress can be timely grasped. The functions of the batch task management center are enriched, and the functions of file and configuration-based report generation are added, making the development of batch operations more convenient and efficient.

[0033] In one of the embodiments, the distributed batch processing system includes multiple batch scheduling centers 100 and multiple batch execution centers 200. The multiple batch scheduling centers 100 and the multiple batch execution centers 200 implement a contention mechanism. The multiple batch scheduling centers 100 implement the contention mechanism by triggering batch tasks through timed contention; the multiple batch execution centers implement the contention mechanism by contending for batch tasks in the contention queue. Any batch center can obtain and process batch tasks without affecting the operation of other centers.

[0034] The batch scheduling center 100 is registered in the batch plan task configuration table scheduler through the Quartz framework. When multiple batch scheduling centers 100 are running simultaneously, it is ensured that only one batch scheduling center 100 meets the trigger condition at the same time. The batch scheduling center 100 that meets the trigger condition is considered to have successfully contended. Multiple batch execution centers 200 simultaneously contend for task information from the queue, and the batch execution center that contends for the task information executes the subcontracting task. The batch execution center 200 that obtains the task information is considered to have successfully contended.

[0035] The Quartz framework is an open-source project in the field of Job scheduling. Using the Quartz framework, one or more timed tasks can be developed, and the execution time of each timed task can be specified individually, such as executing once every hour, executing once at 10 am on the first day of each month, executing once at 5 pm on the last day of each month, etc.

[0036] By implementing the contention mechanism, it can be ensured that after multiple batch scheduling centers 100 and multiple batch execution centers 200 are started simultaneously, the same tasks are not repeatedly executed. When any one of the batch scheduling centers 100 or batch execution centers 200 has a problem, it will not affect the task execution of other batch centers. At the same time, by implementing the contention mechanism, the concurrency efficiency of multiple batch scheduling centers 100 and multiple batch execution centers 200 can also be ensured, achieving the effect of load balancing.

[0037] In some disclosed embodiments, both the batch scheduling center 100 and the batch execution center 200 support containerized deployment. By containerizing the batch scheduling center 100 and the batch execution center 200, resources can be flexibly allocated. The contention mechanism of the two major batch centers can also perfectly adapt to the convenient horizontal expansion and graceful shutdown functions after containerized deployment. The graceful shutdown function means that when simultaneously shutting down the two major batch centers, namely the batch scheduling center 100 and the batch execution center 200, the batch scheduling center 100 and the batch execution center 200 will first stop receiving batch tasks and only truly shut down after completing the currently processing batch tasks, achieving the effect of graceful shutdown.

[0038] The batch scheduling center 100 and the batch execution center 200 achieve horizontal expansion of the centralized monitoring platform by implementing a contention mechanism and supporting containerized deployment. The distributed batch platform can be flexibly scaled up and down according to the number of services and service requirements, and continuously expanded horizontally to improve batch processing performance and cope with the possible gradually increasing data volume processing. The distributed batch platform is a batch service processing platform built using distributed parallel computing technology. Through a cluster with powerful computing capabilities composed of standardized infrastructure (x86 hosts, virtual machines, containers), the batch services are intelligently and dynamically allocated to multiple machines for parallel processing, greatly improving the efficiency of batch service processing and having extremely high reliability and scalability.

[0039] In one embodiment, the batch operation center 200 also has the functions of batch processing files and generating reports in a configured manner, which can make the development of batch services more convenient and efficient.

[0040] The present disclosure also provides a distributed batch processing method. Figure 2 It is a schematic flowchart of the distributed batch processing method according to an embodiment of the present invention. In one embodiment, the distributed batch processing method includes the following steps S100 to S600.

[0041] Step S100: Initialize the polling information of different polling mechanisms.

[0042] The batch scheduling center 100 supports multiple different triggering methods. The polling information is stored in the relevant library tables, and the polling information of each polling mechanism is initialized according to the polling information stored in the relevant library tables. Among them, after the user modifies the polling information in the relevant library tables, the stored relevant polling information is automatically updated.

[0043] Step S200: When the triggering condition is met, execute the polling mechanism corresponding to the triggering condition, poll the execution status of the batch jobs, and obtain the task information of the unprocessed jobs to be allocated in the batch jobs.

[0044] Different trigger conditions correspond to different polling mechanisms. Therefore, different trigger conditions will trigger different polling mechanisms. After the trigger condition is met, the batch scheduling center 100 will execute the polling mechanism corresponding to the trigger condition. For example, when the time meets the trigger condition for timed polling, the batch scheduling center 100 will execute timed polling. By designing multiple polling mechanisms, during the operation of batch jobs, technicians can timely grasp the completed data and the task progress. When it is found through polling that there are unassigned jobs to be processed in the batch job, the task information of the unassigned job is obtained, and the batch operation center 200 can process the unassigned job according to the task information. In some other embodiments of the present disclosure, when an exception occurs in the job, exception information can also be obtained, so as to process the exception situation in a timely manner according to the exception information.

[0045] Step S300: Transmit the task information to the message queue.

[0046] The batch scheduling center 100 can also transmit the task information of the unassigned job to be processed to the message queue MQ (Message Queue) to wait for the batch operation center 200 to consume the task information.

[0047] Step S400: Retrieve the task information from the message queue.

[0048] The batch operation center 200 can retrieve the task information from the message queue MQ.

[0049] Step S500: Perform multiple subcontracting processing on the unassigned job according to the task information to obtain a number of sub-packages. Multiple subcontracting processing means dividing the unassigned job according to multiple dimensions to obtain a number of sub-packages.

[0050] The batch operation center 200 can perform multiple subcontracting processing on the unassigned job according to the task information. Multiple subcontracting means dividing according to multiple dimensions. When the number of data details corresponding to a batch of transactions is too large and the details of each sub-package are still huge after subcontracting according to a single dimension, the dimension can be further increased for subcontracting. Multiple subcontracting can control the size of the sub-packages after subcontracting to meet the performance requirements.

[0051] Step S600: Re-put the number of sub-packages into the message queue.

[0052] After the multiple subcontracting is completed, the batch operation center 200 can also re-put the obtained sub-packages into the message queue MQ for waiting for the batch execution cluster to consume the tasks in the sub-packages.

[0053] The distributed batch system in this disclosure provides services externally in the form of a basic platform, supporting centralized deployment and distributed deployment. The construction goal of the distributed batch system is to decouple from the business architecture, providing dynamic resource allocation, multi-dimensional sharding strategies, flexible task scheduling, and convenient visual management to efficiently execute batch processing tasks for ultra-large datasets. Multiple polling mechanisms are set up in the batch scheduling center, so that the task processing progress can be grasped in a timely manner. The functions of the batch task management center are enriched, and the functions of file and configuration-based report generation are added, making the development of batch services more convenient and efficient.

[0054] In one embodiment, initializing the polling information of different polling mechanisms includes initializing different polling mechanisms according to the polling information in the batch system parameter configuration table, and controlling the interval time between each polling mechanism according to the cron expression in the batch plan task configuration table. When the batch platform starts, four polling mechanisms are initialized according to the information in the batch system parameter configuration table, and the polling time is controlled by the cron expression. The cron expression is a string separated by 5 or 6 spaces, divided into 6 or 7 fields, and each field represents a meaning. Corn from left to right (separated by spaces): seconds, minutes, hours, day of the month, month, day of the week, year. The interval time between each polling mechanism can be modified through the content of each string in the Cron expression.

[0055] The batch plan task configuration table records multiple data source information of the batch platform, such as database, file, polling rules for scheduled tasks, relevant configuration parameters of the thread pool, default file templates, and teller parameters. During operation, the platform-related parameters stored in the batch plan task configuration table can be dynamically modified on the web page. For example, 0 0 12 * *? means it is triggered at 12:00 noon every day.

[0056] In one embodiment, the polling mechanisms include timed polling, database polling, file polling, and job merging polling. By setting multiple polling mechanisms in the scheduling center, the distributed batch system in this disclosure can ensure that during the operation of batch jobs, the job execution status can be obtained in multiple ways, preventing the problem of possible undetected inspections that may exist in a single polling mechanism. Technical personnel can timely know the amount of data that has been completed and keep track of the job progress at any time. At the same time, when an abnormality is found during polling, it can be processed in a timely manner.

[0057] Figure 3 This is a schematic flowchart of the method for polling the execution status of a batch job in one embodiment of the present invention. In one embodiment, when the trigger condition is met, the corresponding polling mechanism is executed, and polling the execution status of the batch job includes the following steps S210 to S240.

[0058] Step S210: Set up a timed polling task using the quartz framework. If the current time conforms to the corn expression in the batch plan task configuration table, it is determined that the trigger condition for timed polling is met, and the execution status of the batch job is polled at regular intervals.

[0059] In some embodiments of the present disclosure, the purpose of polling the execution status of the job is achieved by setting up a timed task in real time through the quartz framework. If the current time conforms to the corn expression rule in the plan configuration table, it is determined that the trigger condition for timed polling is met. By polling the execution status of the batch job at regular intervals, the task progress can be grasped in a timely manner, making the development of batch services more convenient and reliable.

[0060] Step S220: Execute the structured query statement processed when the trigger condition for database polling was last met. If the query result is not empty, it is determined that the trigger condition for database polling is met, and the execution status of the batch job is polled in the database.

[0061] In some embodiments of the present disclosure, database polling polls the data in the database that meets the trigger content. Execute the SQL query statement that met the trigger condition for database polling last time, and determine whether the trigger condition for this database polling is met according to the query result. SQL (Structured Query Language) generally refers to the structured query language, which is a database query and programming language used to access data and query, update, and manage relational database systems.

[0062] If the query result of the SQL query statement for the previous trigger condition is not empty, it is determined that the trigger condition for this database polling is met. Poll the database status of the batch job.

[0063] Step S230: Determine whether the configuration file processed when the trigger condition for database polling was last met exists. If it exists, it is determined that the trigger condition for file polling is met, and the execution status of the batch job is polled for files.

[0064] In some embodiments of the present disclosure, file polling refers to polling the corresponding files in a specified path. If the corresponding files are detected in the specified path, the batch task is executed. That is, determine whether the configuration file processed when the trigger condition for database polling was last met still exists in the specified path. If the configuration file still exists in the specified path, it is determined that the trigger condition for file polling is met, and the execution status of the batch job is polled for files.

[0065] Step S240: After all sub-packages of the batch job have been executed, determine whether there are sub-packages that have been executed but not yet merged based on the status of the batch platform library table. If so, determine whether the trigger condition for job merging polling is met, and perform job merging polling on the execution status of the batch job.

[0066] In some embodiments of the present disclosure, job polling refers to performing job merging polling on the execution status of a batch job after all sub-packages of the batch job have been executed. Find the sub-packages that have been executed but not yet merged according to the status of the batch platform library table, and perform job merging operations on the above sub-packages.

[0067] The distributed batch system in the present disclosure adds a job merging thread, and sets up a timed task through the quartz framework to poll the job execution status in real time. By setting up multiple polling mechanisms in the scheduling center, it can be ensured that during the operation of the batch job, the job execution status can be obtained in multiple ways, preventing the limited inspection effect of a single polling mechanism from affecting the reliability of the system. Technical personnel can timely know the amount of data that has been completed and keep track of the job progress at any time.

[0068] In one embodiment, subcontracting the batch job according to the task information includes traversing the splitting method in the job parameter configuration, and then using multi-threading to subcontract the batch tasks in order. The batch platform execution center retrieves the batch tasks from the message queue and performs preprocessing on the batch tasks. After completing the preprocessing of the batch tasks, subcontracting is performed on them.

[0069] When subcontracting the batch tasks, by traversing the splitting method in the job parameter configuration, each batch task is subcontracted in order using multi-threading according to the corresponding splitting method of each batch task, so as to achieve sharding of data details through multiple splitting methods. After subcontracting, the sub-packages are put back into the message queue again, waiting for the batch execution cluster to consume the sub-package tasks cached in the message queue. Through the processing method in the above embodiments, the distributed batch processing system can flexibly expand and contract according to the business volume and business requirements, continuously expand horizontally to improve the batch processing performance, and cope with the possible gradually increasing data volume processing.

[0070] Figure 4 It is a schematic flowchart of the method for periodically checking the version number of the configuration table in one embodiment of the present invention. In one embodiment, the method further includes the following steps S700 to S900.

[0071] Step S700: Periodically query the version number of the batch system parameter configuration table and the version number of the batch plan task configuration table.

[0072] In some embodiments of the present disclosure, the centralized monitoring platform will periodically query the version numbers of the batch system parameter configuration table and the batch plan task configuration table, and determine whether the version numbers of the batch system parameter configuration table and the batch plan task configuration table have changed. According to whether the version numbers of the batch system parameter configuration table and the batch plan task configuration table have changed, it can be determined whether the data in the batch system parameter configuration table and the batch plan task configuration table has been modified.

[0073] Step S800: If the version number of the batch system parameter configuration table changes, re-load the polling information of different polling mechanisms.

[0074] When the centralized monitoring platform queries and finds that the version number of the batch system parameter configuration table has changed compared with the version number at the previous query, it can be determined that the data in the batch system parameter configuration table has been modified. The centralized monitoring platform records the new version number of the batch system parameter configuration table at this query, which can be used for comparison with the version number at the next query. In addition, the centralized monitoring platform also re-loads the four types of polling information to ensure the timeliness and accuracy of the polling information in the four polling mechanisms.

[0075] Step S900: If the version number of the batch plan task configuration table changes, re-load the timed polling task.

[0076] Similarly, when the centralized monitoring platform queries and finds that the version number of the batch plan task configuration table has changed compared with the version number at the previous query, it can be determined that the data in the batch plan task configuration table has been modified. The centralized monitoring platform records the new version number of the batch plan task configuration table at this query, which can be used for comparison with the version number at the next query. In addition, the centralized monitoring platform also re-loads the timed polling task to ensure the timeliness and accuracy of the execution actions of the four polling mechanisms.

[0077] In some embodiments of the present disclosure, in order to improve the execution efficiency, the batch system parameter configuration table and the batch plan task configuration table are stored using a local cache to ensure the timeliness of the batch system parameter configuration table and the batch plan task configuration table.

[0078] It should be understood that although Figures 2 - 4 the steps in the flowchart of Figures 2 - 4At least some of the steps may include multiple steps or multiple stages, which do not necessarily need to be executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages does not necessarily need to be sequential, but can be executed alternately or in turn with at least some of the steps or stages in other steps or other steps.

[0079] Based on the description of the distributed batch processing method embodiments described above, the present disclosure also provides a distributed batch processing device. The device may include a system (including a distributed system), software (application), module, component, etc. that uses the method described in the embodiments of this specification and combines the necessary implementation hardware. Based on the same inventive concept, the devices in one or more embodiments provided by the embodiments of the present disclosure are as described in the following embodiments. Since the implementation solutions for the device to solve problems are similar to the method, the implementation of the specific device in the embodiments of this specification can refer to the implementation of the foregoing method, and the repeated parts will not be elaborated. As used hereinafter, the term "unit" or "module" may be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0080] Regarding the device in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0081] It can be understood that the various embodiments of the method, device, etc. in this specification are all described in a progressive manner. The same / similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For the relevant parts, refer to the description of other method embodiments.

[0082] Figure 5 is a block diagram of a distributed batch processing device or system S00 shown according to an exemplary embodiment. Referring to Figure 5 , the distributed batch processing device or system S00 includes a processing component S20, which further includes one or more processors, and memory resources represented by a memory S22 for storing instructions executable by the processing component S20, such as application programs. The application programs stored in the memory S22 may include one or more modules each corresponding to a set of instructions. In addition, the processing component S20 is configured to execute instructions to perform the above method.

[0083] The distributed batch processing device or system S00 may further include: a power supply component S24 configured to perform power management of the distributed batch processing device or system S00, a wired or wireless network interface S26 configured to connect the distributed batch processing device or system S00 to a network, and an input / output (I / O) interface S28. The distributed batch processing device or system S00 may operate based on an operating system stored in the memory S22, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0084] In an exemplary embodiment, there is also provided a computer-readable storage medium including instructions, such as the memory S22 including instructions, and the above instructions can be executed by a processor of the database query device or system S00 to complete the above method. The storage medium may be a computer-readable storage medium. For example, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0085] In an exemplary embodiment, there is also provided a computer program product including instructions, and the above instructions can be executed by a processor of the distributed batch processing device or system S00 to complete the above method.

[0086] In the description of this specification, the descriptions referring to terms such as "some embodiments", "other embodiments", "ideal embodiments", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example.

[0087] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0088] The above-described embodiments only represent several implementation manners of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the invention patent should be subject to the appended claims.

Claims

1. A distributed batch processing system, characterized in that, including: A batch scheduling center for initializing polling information of different polling mechanisms, including controlling the time interval between each polling mechanism according to the cron expression in the batch plan task configuration table. The polling mechanisms include timed polling, database polling, file polling, and job merging polling. It is also used to execute the corresponding polling mechanism when the trigger condition is met, poll the execution status of the batch job, obtain the task information of the unassigned jobs in the batch job, and transmit the task information to the message queue; A batch running center for taking out the task information from the message queue, performing multiple subcontracting processes on the unassigned jobs according to the task information to obtain several sub-packages, and putting the several sub-packages back into the message queue. The multiple subcontracting process refers to dividing the unassigned jobs according to multiple dimensions to obtain several sub-packages; Among them, the execution of the corresponding polling mechanism when the trigger condition is met and polling the execution status of the batch job includes: Setting a timed polling task using the quartz framework. If the current time conforms to the corn expression in the batch plan task configuration table, it is determined that the trigger condition for the timed polling is met, and the execution status of the batch job is polled at regular intervals; Executing the structured query statement processed when the trigger condition for the database polling was last met. If the query result is not empty, it is determined that the trigger condition for the database polling is met, and the execution status of the batch job is polled for the database; Judging whether the configuration file processed when the trigger condition for the database polling was last met exists. If it exists, it is determined that the trigger condition for the file polling is met, and the execution status of the batch job is polled for the file; When all sub-packages of the batch job are completed, judging whether there are sub-packages that have been completed but not yet merged according to the status of the batch platform library table. If there are, it is determined that the trigger condition for the job merging polling is met, and the execution status of the batch job is polled for job merging; The batch scheduling center is also used for: Regularly querying the version number of the batch system parameter configuration table and the version number of the batch plan task configuration table; If the version number of the batch system parameter configuration table changes, reloading the polling information of different polling mechanisms; If the version number of the batch plan task configuration table changes, reloading the timed polling task.

2. The distributed batch processing system according to claim 1, characterized in that, The distributed batch processing system includes multiple batch scheduling centers and multiple batch running centers, When multiple batch scheduling centers run simultaneously, they are registered in the batch plan task configuration table through the quartz framework. Only one batch scheduling center meets the trigger condition at the same time; Multiple batch running centers simultaneously compete to obtain the task information from the queue, and the batch running center that grabs the task information executes the subcontracting task.

3. The distributed batch processing system according to claim 1, characterized in that, The distributed batch processing system includes a batch task management center and a centralized monitoring platform; The batch task management center is used to integrate the end-of-day batch and the online batch for unified management; The centralized monitoring platform is used to monitor job operation, platform operation, and scheduling plans, and is also used to alarm for batch errors, and is also used to screen tasks in various states through the batch processing service function interface.

4. A distributed batch processing method, characterized in that, It includes: Initializing polling information for different polling mechanisms, including: controlling the interval time between each polling mechanism according to the cron expression in the batch plan task configuration table, and the polling mechanisms include timed polling, database polling, file polling, and job merging polling; When the trigger condition is met, execute the polling mechanism corresponding to the trigger condition, poll the execution status of the batch job, and obtain the task information of the unprocessed and to-be-allocated jobs in the batch job; Transmit the task information to the message queue; Take out the task information from the message queue; Perform multiple subcontracting processing on the to-be-allocated job according to the task information to obtain several sub-packages. The multiple subcontracting processing refers to dividing the to-be-allocated job according to multiple dimensions to obtain several sub-packages; Put the several sub-packages back into the message queue; Among them, the step of when the trigger condition is met, execute the corresponding polling mechanism and poll the execution status of the batch job includes: Use the quartz framework to set a timed polling task. If the current time conforms to the corn expression in the batch plan task configuration table, it is determined that the trigger condition for the timed polling is met, and the execution status of the batch job is polled regularly; Execute the structured query statement processed when the trigger condition for the database polling was met last time. If the query result is not empty, it is determined that the trigger for the database polling is met, and the execution status of the batch job is polled through the database; Judge whether the configuration file processed when the trigger condition for the database polling was met last time exists. If it exists, it is determined that the trigger condition for the file polling is met, and the execution status of the batch job is polled through the file; When all sub-packages of the batch job are completed, judge whether there are sub-packages that have been completed but not yet merged according to the status of the batch platform library table. If there are, it is determined that the trigger condition for the job merging polling is met, and the execution status of the batch job is polled through job merging; The method further includes: Regularly query the version numbers of the batch system parameter configuration table and the batch plan task configuration table; If the version number of the batch system parameter configuration table changes, reload the polling information of different polling mechanisms; If the version number of the batch plan task configuration table changes, reload the timed polling task.

5. The distributed batch processing method according to claim 4, wherein The initialization of the polling information for different polling mechanisms includes: Initializing different polling mechanisms according to the polling information in the batch system parameter configuration table.

6. The distributed batch processing method according to claim 4, wherein The subcontracting processing of the batch job according to the task information includes: After traversing the splitting methods in the job parameter configuration, use multi-threading to subcontract the batch tasks in an orderly manner.

7. The distributed batch processing method according to claim 4, wherein The method further includes: When an exception occurs in the job, obtain the exception information; Process the abnormal situation according to the exception information.

8. The distributed batch processing method according to claim 4, wherein The cron expression is a string; controlling the time interval between each polling mechanism according to the cron expression in the batch scheduling task configuration table includes: Modifying the time interval between each polling mechanism according to the content of each string in the cron expression.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, wherein When the processor executes the computer program, it implements the steps of the method described in any one of claims 4 to 8.

10. A computer-readable storage medium, having a computer program stored thereon, wherein When the computer program is executed by the processor, it implements the steps of the method described in any one of claims 4 to 8.

Images