Kafka cluster load balancing method, system and device and medium

Abstract

The invention discloses a kafka cluster load balancing method. The kafka cluster load balancing method comprises the following steps of obtaining a current network load of each node and a current read-write load, a current capacity load and a current theme partition load of a data disk under each node; based on one or more of the current network load, the current read-write load, the current capacity load and the current theme partition load, judging whether to trigger and adjust a current theme creation configuration strategy or not; and in response to triggering adjustment of the current theme creation configuration strategy, adjusting the current theme configuration strategy by utilizing the theme configuration optimization rule so as to carry out load balancing on the theme. The invention further discloses a system, computer equipment and a readable storage medium. According to the method, a load state of the cluster is obtained in real time, so the Kafka theme configuration optimization rule is dynamically adjusted, the possibility of Kafka cluster data inclination, overhigh single-point load and network congestion is reduced, and resource balance configuration of the Kafka cluster is realized.

Description

technical field [0001] The invention relates to the field of kafka clusters, in particular to a kafka cluster load balancing method, system, equipment and storage medium. Background technique [0002] As a message queue service, Kafka often contains multiple topics and carries a large amount of data in actual business. However, Kafka itself cannot perceive the read and write load, network pressure, topic partition, and disk capacity information in the cluster. When creating a topic, it will not Topic rules are optimized based on the above information. Unbalanced host rule configurations often lead to problems such as data skew, high single-point load, and network congestion. Contents of the invention [0003] In view of this, in order to overcome at least one aspect of the above problems, an embodiment of the present invention proposes a kafka cluster load balancing method, including the following steps: [0004] Obtain the current network load of each node and the curren...

AI Technical Summary

Problems solved by technology

[0002] As a message queue service, Kafka often contains multiple topics and carries a large amount of data in actual business. However, Kafka itself cannot perceive the read and write load, network pressure, topic partition, and disk capacity information in the cluster. When creating a topic, it will not Optimize topic rules based on the above information. Unbalanced host rule configurations often lead to problems such as data skew, high single-point load, and network congestion.

Images