Data stream processing method and system

Abstract

The invention discloses a data stream processing method and system, and belongs to the field of big data processing. The method comprises the following steps: determining one of a plurality of Masternodes as a main node through a Zookeeper cluster; the main node provides an external interface to receive an online request of a service and distributes a task to the service; the main node generatesconfiguration information of the task according to the current state information reported by the Worker nodes and writes the configuration information into a ZooKeeper cluster, and the configuration information comprises information indicating the Worker nodes scheduled to execute the task; and if the Worker node monitors that the task scheduled to the Worker node exists in the ZooKeeper cluster,it starts Flume service to execute. According to the invention, the high availability of the Master node and the Worker node can be realized, the availability of Flume service is improved, and the problems of uneven resource use and waste are avoided; moreover, the online and offline operations of the services can be greatly simplified, and the mutual influence between the services is reduced.

Description

technical field [0001] The invention relates to the field of big data processing, in particular to a data flow processing method and system. Background technique [0002] In the prior art, the Flume service is usually started on each node of the cluster to transfer data from the data source to the sink. [0003] In the process of realizing the present invention, the inventors found that: when a node device has a regular failure (such as deadlock, abnormal consumption, etc.), the system has no perception and requires manual processing, which affects the timeliness of failure handling; in addition, due to the same cluster Each node adopts the same configuration, but the amount of business data of each node is uneven, which may easily lead to a large idle ratio of Flume collection threads; in addition, because new business operations are frequently launched, and business configuration files need to be manually modified, a restart is required when modifying The entire cluster a...

AI Technical Summary

Problems solved by technology

[0003] In the process of realizing the present invention, the inventors found that: when a node device has a regular failure (such as deadlock, abnormal consumption, etc.), the system has no perception and requires manual processing, which affects the timeliness of failure handling; in addition, due to the same cluster Each node adopts the same configuration, but the amount of business data of each node is uneven, which may easily lead to a large idle ratio of Flume collection threads; in addition, because new business operations are frequently launched, and business configuration files need to be manually modified, a restart is required when modifying The entire cluster, thus affecting the normal execution of other businesses in the same cluster

Images