Batch processing device and method based on distributed architecture

Abstract

The invention provides a batch processing device and method based on a distributed architecture, and the device comprises a task triggering module, a scheduling module, and a task cluster comprising aplurality of task execution nodes. The task triggering module is used for initiating a batch running task execution request to the scheduling module according to a pre-configured batch running task execution strategy, the scheduling module is used for determining a task execution node for executing the batch running task execution request from a plurality of task execution nodes according to thebatch running task execution request, and sending a batch running task execution instruction to the determined task execution node, and the plurality of task execution nodes are used for executing thebatch running task according to the batch running task execution instruction sent by the scheduling module. According to the technical scheme, the batch processing reliability, the system performanceand the maintainability are improved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a device and method for running batch processing based on a distributed architecture. Background technique [0002] Batch running is an important business of the bank. The main business is to generate the general ledger and check the total score; the third is to perform large-scale transactions, such as interest settlement, accrual and collection and payment, report generation, and flow data export. The banking system involves huge and complex batch operations. At present, batch processing is mainly deployed on a single machine. With the development of business and the increase of data scale, there are many problems in the running batch solution deployed on a single machine: [0003] 1. Performance issues: A large number of high concurrency causes excessive computer performance consumption and affects task execution efficiency. [0004] 2. Robustness issues, tha...

AI Technical Summary

Problems solved by technology

[0003] 1. Performance problems: a large number of high concurrency, resulting in excessive computer performance consumption, affecting the efficiency of task execution

[0004] 2. Robustness issues, that is, reliability issues: In a stand-alone system, because each task is fixed and only deployed under one task node, there is a coupling between the task and the node. If a node fails, it will affect the performance of all tasks under the node. Execute, all running batch tasks deployed on it will fail

[0005] 3. Poor maintainability: When the system expands, all tasks need to be split and redeployed to ensure that the performance loss of each task node is balanced; and in the later stage of operation and maintenance, it is also necessary to check the tasks on it according to the performance loss of the node. The number of deployments is adjusted

[0007] For the above problems, no effective solution has been proposed

Images