Interoperable data processing system and method based on streaming computing and batch computing

Abstract

The invention discloses an interoperable data processing system and method based on stream computing and batch computing. The system includes: a stream computing module, which includes several Topology computing tasks; a batch computing module, The batch calculation module includes several Job calculation tasks; an interoperability module, the interoperation module is connected with the flow calculation module and the batch calculation module respectively, the interoperation module includes a registration subunit and a trigger subunit, and the registration subunit It is used to collect the Topology information on the stream computing module and the Job information on the batch computing module, and the triggering subunit is used to receive the interoperability instructions of the stream computing module or the batch computing module, and call the computing task of the other party. The present invention realizes interoperability between different computing modules at the system level, and completes the mutual calling of logic units on different computing model systems with higher efficiency.

Description

technical field [0001] The present invention relates to the technical field of data processing of stream computing and batch computing, in particular to an interoperable data processing system and method based on stream computing and batch computing. Background technique [0002] In recent years, with the development of Internet technology, modern society has entered the era of big data. The amount of data generated in society is increasing, resulting in an increasing amount of data that computer systems need to process. Therefore, the efficient processing of big data has become a common concern. Generally speaking, for big data, divide and conquer is a more effective solution. This algorithm distributes a single computing task to different computers. Typical representatives of this type of algorithm include batch computing models, streaming computing models, and so on. [0003] A typical representative of the batch computing model is Hadoop, a distributed system infrastr...

AI Technical Summary

Problems solved by technology

The shortcomings of this type of MapReduce-based streaming processing have three aspects: a) The input data is divided into fixed-size fragments, and then processed by the MapReduce platform. The processing delay is proportional to the length of the data fragment and the overhead of initial processing tasks

Although Summingbird uses a unified programming interface to integrate the platform, it has good versatility and strong scalability, but it still has the disadvantage that the execution efficiency in actual operation is not ideal.

Images