A kafka cluster load balancing method, system, device and medium

Abstract

The invention discloses a Kafka cluster load balancing method, comprising the following steps: obtaining the current network load of each node and the current read and write load, current capacity load and current topic partition load of data disks under each node; One or more of the current network load, current read and write load, current capacity load, and current topic partition load determine whether to trigger adjustment of the current topic creation configuration policy; in response to triggering adjustment of the current topic creation configuration policy, use Topic configuration optimization rules adjust the current topic configuration policy to load balance topics. The invention also discloses a system, a computer device and a readable storage medium. The solution proposed by the present invention dynamically adjusts the Kafka topic configuration optimization rules by acquiring the cluster load status in real time, reduces the possibility of Kafka cluster data skew, excessive single-point load, and network congestion, and realizes the resource balanced configuration of the Kafka cluster.

Description

technical field [0001] The invention relates to the field of kafka clusters, in particular to a method, system, equipment and storage medium for load balancing of kafka clusters. Background technique [0002] As a message queue service, Kafka often contains multiple topics and carries a large amount of data in actual business. However, Kafka itself cannot perceive the read and write load, network pressure, topic partition, and disk capacity information in the cluster. Optimize topic rules based on the above information. Unbalanced host rule configuration often leads to data skew, high single-point load, and network congestion. SUMMARY OF THE INVENTION [0003] In view of this, in order to overcome at least one aspect of the above problem, an embodiment of the present invention proposes a Kafka cluster load balancing method, including the following steps: [0004] Obtain the current network load of each node and the current read and write load, current capacity load and cu...

AI Technical Summary

Problems solved by technology

[0002] As a message queue service, Kafka often contains multiple topics and carries a large amount of data in actual business. However, Kafka itself cannot perceive the read and write load, network pressure, topic partition, and disk capacity information in the cluster. When creating a topic, it will not Optimize topic rules based on the above information. Unbalanced host rule configurations often lead to problems such as data skew, high single-point load, and network congestion.

Images