Batch processing task triggering method and system, and computer-readable storage medium

Abstract

The invention discloses a batch processing task triggering method and system, and a computer-readable storage medium. The data processing system is provided with a data processing architecture, and the data processing architecture comprises: a real-time processing layer and a batch processing layer; the real-time processing layer is used for receiving first to-be-processed data in real time, copying the first to-be-processed data to obtain second to-be-processed data, determining whether all the second to-be-processed data corresponding to a current batch processing task are received or not, and if all the second to-be-processed data corresponding to the current batch processing task are received, generating an event notification message; the batch processing layer is connected with the real-time processing layer and is used for processing all the received second to-be-processed data after receiving the event notification message so as to execute the current batch processing task. In this way, the delay time of batch processing tasks can be shortened.

Description

technical field [0001] The present application relates to the technical field of big data, and in particular to a batch processing task triggering method, system and computer-readable storage medium. Background technique [0002] At present, big data technology has become more and more mature, and the framework of real-time computing and offline computing has also tended to be mature. The result of real-time calculation is inaccurate; offline calculation is just the opposite, the timeliness is very low, and some calculations are even delayed by 1 day, 1 week or a month, and some problems such as out-of-order or repeated sending of data can be processed within this buffer time. The accuracy is high, so stream-batch integration has become the development direction, but the current stream-batch integration has the problems of isolation between real-time calculation and offline calculation and long delay of offline calculation. Contents of the invention [0003] The present a...

AI Technical Summary

Problems solved by technology

[0002] At present, big data technology has become more and more mature, and the framework of real-time computing and offline computing has also tended to be mature. The result of real-time calculation is inaccurate; offline calculation is just the opposite, the timeliness is very low, and some calculations are even delayed by 1 day, 1 week or a month, and some problems such as out-of-order or repeated sending of data can be processed within this buffer time. The accuracy is high, so stream-batch integration has become the development direction, but the current stream-batch integration has the problems of isolation between real-time calculation and offline calculation and long delay of offline calculation

Images