Multi-source Data Induction Method with High Reliability and Low Resource Overhead

Abstract

The present invention provides a multi-source data connection method with high reliability and low resource overhead, which includes: classifying at least one data source according to a preset connection rule, changing its data source information into configuration information, and Class data source assigns a connection service; use the connection service to receive the data sent by the corresponding data source, and send the data in a standardized format to the topic corresponding to the distributed message queue corresponding to this type of data source; respond to one of the applications The read information sent sends the data in the corresponding topic to the application. The present invention uses container technology, distributed message queue technology, and distributed coordination technology to enable the entire data connection process to have the capabilities of disaster recovery, load balancing, and dynamic capacity expansion; State, multiple replicas can be started at the same time to achieve load balancing and hot start.

Description

technical field [0001] The invention relates to multi-source data connection, and belongs to a multi-source data connection method with high reliability and low resource overhead. Background technique [0002] The difficulty of many source data connection technologies lies in the large number of data sources. Each data source has a corresponding data processing program, but there is a time interval for the data source to send data. There may be a large number of data sources without data in the same period of time. idle state. At the same time, the timeliness requirements for data processing are very high, and some processing logic may be more complicated, which needs to rely on the previously processed data, that is, the data connection service is stateful. [0003] There are currently two main methods for multi-source data connection. One is a complex connection program, which has many threads running, and each thread is responsible for leading data from one data source. ...

AI Technical Summary

Problems solved by technology

[0005] However, this solution also has some problems, mainly when a certain data source receives data, the solution needs to restart a new processing program, that is, the connection processing program has a cold start problem

In this way, when the logic of the ingestion handler is more complex, the startup time may be slower, thereby increasing the delay of incision data

Another problem is that the data source is connected to the upstream of the connection program, and the application is connected to the downstream. Once a problem occurs in the middle, the data will be lost and the reliability will be low.

Images