Real-time data processing system and method supporting multiple engines

Abstract

The invention discloses a real-time data processing system and method supporting multiple engines, service processing is disassembled, a plug-in for realizing a certain specific function of a service is generated as required, and the plug-in and the engines have no direct relationship; analyzing service requirements to perform processing flow arrangement, dragging a corresponding plug-in to confirm a data processing method and a data flow direction, and obtaining flow arrangement information to construct a data execution link diagram; and obtaining execution engine information according to a data execution link diagram generated in the process arrangement, obtaining actual processing logic through an execution bridge, and calling and executing the actual processing logic. According to the method, plug-in type flow arrangement is supported based on multiple engines, each processing link is allowed to be defined, the flexibility is high, and the use is convenient.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a real-time data processing system and method supporting multiple engines. Background technique [0002] With the development of science and technology, massive data are generated in daily life and economic construction, and enterprises have higher and higher requirements for data timeliness. Making good use of these data in the first place can more effectively improve the competitiveness of enterprises. At present, the real-time data processing technology is becoming more and more mature, and the mainstream processing engines include Flink, Spark Stream, Storm, etc. Usually, the logic code of the real-time processing module is integrated with the processing engine, and minor adjustments have a wide impact on changes to the engine or business logic. With the development of business and the continuous improvement of new technologies, it is possible to use new engines ...

AI Technical Summary

Problems solved by technology

With the development of business and the continuous improvement of new technologies, it is possible to use new engines in the enterprise system platform. However, due to the limitations of the architecture, it is usually impossible to quickly integrate with the original modules based on the status quo of the system, and it takes a lot of manpower to refactor the code.

From the perspective of a single application, in real-time data processing, there are generally situations such as diverse data and complex business scenarios, which usually need to be processed according to the characteristics of the current data and business needs, with high customization; and real-time applications are usually provided by the corresponding personnel. There are problems such as decentralized operation and maintenance and multi-node management for task startup and shutdown.

[0003] The prior art has the following problems: (1) The choice of processing engine is single, and only one processing engine is supported for a certain real-time data processing business, which cannot be freely selected and has low flexibility

(2) The use of multiple engines is limited, and a certain type of engine is limited to a special processing category or a specific field. The use range is narrow, and it is inconvenient to update and iterate

(3) It is difficult to add a new engine. When a new engine needs to be introduced, the existing architecture needs to be greatly modified, and it is difficult to adapt the old process to the new engine.

(4) High customization of real-time processing flow, low code reusability, large maintenance, and poor versatility

(5) Real-time task management lacks support for multiple engines, task processing is scattered, and there is no unified task management module

Images