Batch Execution Method Based on Distributed System

Abstract

The invention relates to a batch processing execution method based on a distributed system. The batch processing execution method comprises the steps that S1, a dimension job scheduling component issues a batch processing instruction and dimension data to each task node; S2, the task node acquires source data of a data source, puts the source data into a source data set NS, and outputs the sourcedata to a node job processing component; S3, the node job processing component processes the received source data and outputs the source data to a node algorithm component; S4, the node algorithm component calculates the landing position of the source data, and outputs a set of corresponding relationships between the data sources and the source data to a pre-compiled batch processing component; s5, pre-compiling the source data of the batch processing component to realize persistence of the source data; S6, after all the task nodes are executed, the distributed batch processing task is ended.The method has the functions of task scheduling, monitoring and the like of an existing open source framework; meanwhile, a standard and efficient large-scale data batch processing mode is formed, andthe problems of performance and the like of batch processing programs due to the ability problem of developers are effectively solved.

Description

technical field [0001] The invention relates to a data batch processing method for big data, in particular to a distributed system-based batch processing execution method. Background technique [0002] With the widespread promotion and application of distributed micro-service systems, especially in the context of rapid growth in data volume, many banking systems currently adopt the strategy of sub-database and sub-table to store large-scale day-to-day transaction data. Based on the needs of some special businesses, how to centrally and efficiently process large-scale day transactions within a specified time is a difficult problem faced by many distributed systems. For example, it is necessary to process the daytime transactions of T day on T+1 day, check the daytime transactions of T day according to certain business logic, and provide the downstream system with the end-of-day reconciliation details of T day after verification and correction. The downstream system calculate...

AI Technical Summary

Problems solved by technology

[0007] The invention provides a batch processing execution method based on a distributed system, which not only has the functions of task scheduling and monitoring of the current open source framework, but also forms a standard and efficient batch processing method for large-scale data to solve the problem of batch processing Performance and other issues caused by the ability of developers

Images