Session parameter optimization method for distributed cluster, electronic device, and storage medium

Abstract

The application provides a session parameter optimization method of a distributed cluster, an electronic device and a storage medium. The distributed cluster comprises a zookeeper cluster node and a cache agent Broker node. The Broker node is connected with the zookeeper cluster node through a heartbeat session. The method comprises the following steps: detecting a network packet loss rate between the Broker node and the zookeeper cluster node; if the network packet loss rate is less than a preset threshold, determining a session timeout time corresponding to the Broker node according to the network packet loss rate based on a preset parameter mapping relationship; and performing parameter configuration on the Broker node through a dynamic configuration interface of the distributed cluster by using the session timeout time corresponding to the Broker node, so as to complete optimization of the session parameter. By using the above method, the corresponding session timeout time can be adaptively determined according to the network condition, so as to optimize the session parameter of the Broker node, and then the stability and production and consumption performance of the distributed cluster can be improved.

Description

Technical Field

[0001] This application belongs to the field of computer science, and in particular relates to a method for optimizing session parameters of a distributed cluster, an electronic device, and a storage medium. Background Technology

[0002] With the increasing prevalence of distributed cluster deployments across data centers, network packet loss across wide area networks has become commonplace. Network jitter / packet loss can easily lead to the loss of heartbeat messages between Broker nodes and ZooKeeper nodes in a distributed cluster. ZooKeeper nodes often use a fixed session timeout to monitor Broker nodes. Under this mechanism, heartbeat loss may cause unnecessary shrinkage of the synchronized replica set (ISR), session timeouts, and frequent elections between the controller node and the partition leader node.

[0003] If the rigidity of the session monitoring mechanism between Broker nodes and ZooKeeper nodes in a distributed cluster cannot be effectively resolved, it will seriously affect the stability and production / consumption performance of the distributed cluster. Summary of the Invention

[0004] In view of the above, it is necessary to provide a method, electronic device and storage medium for optimizing session parameters in a distributed cluster, which can solve the technical problem of the rigid session monitoring mechanism between Broker nodes and ZooKeeper nodes in a distributed cluster.

[0005] On one hand, this application provides a method for optimizing session parameters in a distributed cluster. The distributed cluster includes ZooKeeper cluster nodes and Broker nodes. The Broker nodes connect to the ZooKeeper cluster nodes via heartbeat sessions. The method includes: detecting the network packet loss rate between the Broker nodes and the ZooKeeper cluster nodes; if the network packet loss rate is less than a preset threshold; determining the session timeout time corresponding to the Broker node based on a preset parameter mapping relationship and the network packet loss rate; and using the session timeout time corresponding to the Broker node, configuring the parameters of the Broker node through the dynamic configuration interface of the distributed cluster to complete the optimization of session parameters.

[0006] In some embodiments of this application, the parameter mapping relationship includes a one-to-one correspondence between multiple network packet loss rates and multiple session timeout times, wherein the session timeout time corresponding to a larger network packet loss rate is greater than the session timeout time corresponding to a smaller network packet loss rate, and the session timeout time corresponding to the Broker node is used by the ZooKeeper cluster node to determine whether the Broker node is offline.

[0007] In some embodiments of this application, determining the session timeout time corresponding to the Broker node based on the network packet loss rate according to the preset parameter mapping relationship includes: if a session timeout time corresponding to the network packet loss rate is found based on the parameter mapping relationship, the session timeout time is determined as the session timeout time corresponding to the Broker node; if a session timeout time corresponding to the network packet loss rate cannot be found, the network packet loss rate falls into the network packet loss rate interval in the parameter mapping relationship; a time range is determined according to the session timeout times corresponding to the endpoints of the network packet loss rate interval; and interpolation is performed on the time range according to the network packet loss rate to obtain the session timeout time corresponding to the Broker node.

[0008] In some embodiments of this application, the step of interpolating the time range based on the network packet loss rate to obtain the session timeout time corresponding to the Broker node includes: determining the relative position ratio of the network packet loss rate in the network packet loss rate interval, determining the time difference between the upper and lower endpoints of the time range, and determining the product of the time difference and the relative position ratio, and determining the session timeout time corresponding to the Broker node based on the sum of the lower endpoint of the time range and the product.

[0009] In some embodiments of this application, if the network packet loss rate is greater than or equal to the preset threshold, the method further includes: outputting an alarm prompt, and / or, using a preset session timeout time, configuring parameters of the Broker node through the dynamic configuration interface, wherein the preset session timeout time is greater than or equal to the maximum session timeout time in the parameter mapping relationship.

[0010] In some embodiments of this application, the method further includes: performing network packet loss simulation on the distributed cluster using firewall tools based on preset multiple network packet loss rates and preset multiple session timeout times; determining the session timeout time corresponding to each network packet loss rate from the multiple session timeout times based on the performance index scores of the distributed cluster under the same network packet loss rate and corresponding different session timeout times; constructing a correspondence between each network packet loss rate and the corresponding session timeout time; and forming the parameter mapping relationship.

[0011] In some embodiments of this application, the ZooKeeper cluster nodes are used to elect a master node, which is used to elect a leader node for a partition and manage a synchronized replica set (ISR). The ISR includes broker nodes that maintain data synchronization with the leader node. The leader node is used to receive data sent from producer nodes and send the received data to consumer nodes. The method for determining the performance index score includes: under the same network packet loss rate, based on the performance index data of the distributed cluster before and after configuration using any session timeout, determining the performance index score of the distributed cluster corresponding to the performance of the any session timeout. The performance index data includes one or more of the following: the number of master node switching, the number of session timeouts between the ZooKeeper cluster nodes and the broker nodes, the number of ISR shrinking, the latency of the producer node writing data, and the latency of the consumer node reading data. The number of ISR shrinking is obtained by counting the number of times the leader node removes the lagging broker nodes from the ISR.

[0012] In some embodiments of this application, the method for determining the performance index score includes: determining the master node switching improvement rate based on the number of switching times of the master node before and after parameter configuration; determining the session timeout improvement rate based on the number of session timeouts between the ZooKeeper cluster node and the Broker node before and after parameter configuration; determining the ISR shrinkage improvement rate based on the number of ISR shrinkages before and after parameter configuration; determining the production performance improvement rate based on the data write latency of the producer node before and after parameter configuration; determining the consumption performance improvement rate based on the data read latency of the consumer node before and after parameter configuration; and obtaining the performance index score by weighted summation of one or more of the master node switching improvement rate, the session timeout improvement rate, the ISR shrinkage improvement rate, the production performance improvement rate, and the consumption performance improvement rate. On the other hand, this application provides an electronic device, the electronic device including: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the session parameter optimization method for the distributed cluster.

[0013] On the other hand, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor in an electronic device, implements the session parameter optimization method for the distributed cluster. In the distributed cluster session parameter optimization scheme provided in this application embodiment, the comparison result of the network packet loss rate and the preset threshold can reflect the severity of the communication link quality between the Broker node and the ZooKeeper cluster node. If the network packet loss rate is less than the preset threshold, it indicates that the communication link quality between the Broker node and the ZooKeeper cluster node is within an acceptable range. At this time, based on the network packet loss rate and using the parameter mapping relationship, the session timeout time corresponding to the Broker node can be quickly determined, and it can be ensured that the determined session timeout time is adapted to the current network conditions. Through the dynamic configuration interface of the distributed cluster, the parameters of the Broker node can be configured online, realizing hot updates of session parameters. Since there is no need to restart the Broker node, the efficiency of parameter configuration can be improved. In this way, the corresponding session timeout time is adaptively determined according to the network conditions, thereby optimizing the session parameters of the Broker node, effectively solving the technical problems of rigid session monitoring mechanisms that cannot adapt to changes in network conditions, thereby improving the stability and production / consumption performance of the distributed cluster. Attached Figure Description

[0014] Figure 1 This is a flowchart of a distributed cluster session parameter optimization method provided in an embodiment of this application.

[0015] Figure 2 This is a flowchart of a method for determining the session timeout time corresponding to a Broker node according to an embodiment of this application.

[0016] Figure 3 This is a flowchart of a method for constructing parameter mapping relationships provided in an embodiment of this application.

[0017] Figure 4 This is a schematic diagram of a method for evaluating the effect of parameter configuration provided in an embodiment of this application.

[0018] Figure 5 This is a schematic diagram of a network packet loss simulation test provided in an embodiment of this application.

[0019] Figure 6 This is a system architecture diagram of a distributed cluster session parameter optimization method provided in an embodiment of this application.

[0020] Figure 7 This is a functional block diagram of a session parameter optimization device provided in an embodiment of this application.

[0021] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0023] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Unless otherwise specified, the following embodiments and features described herein can be combined with each other.

[0024] With the increasing prevalence of distributed cluster deployments across data centers, network packet loss across wide area networks has become commonplace. Network jitter / packet loss can easily lead to the loss of heartbeat messages between Broker nodes and ZooKeeper nodes in a distributed cluster. ZooKeeper nodes often use a fixed session timeout to monitor Broker nodes. Under this mechanism, heartbeat loss may cause unnecessary shrinkage of the synchronized replica set (ISR), session timeouts, and frequent elections between the controller node and the partition leader node.

[0025] If the rigidity of the session monitoring mechanism between Broker nodes and ZooKeeper nodes in a distributed cluster cannot be effectively resolved, it will seriously affect the stability and production / consumption performance of the distributed cluster.

[0026] To address the aforementioned technical issues, this application provides a method for optimizing session parameters in a distributed cluster. This method adaptively determines the corresponding session timeout based on network conditions, thereby optimizing the session parameters of the Broker node. It effectively solves the technical problem of a rigid session monitoring mechanism that cannot adapt to changes in network conditions, thus improving the stability and production / consumption performance of the distributed cluster.

[0027] The session parameter optimization method for distributed clusters provided in this application embodiment can be applied to one or more electronic devices that are connected to the distributed cluster for communication. The electronic devices can monitor, manage and maintain the distributed cluster.

[0028] For example, electronic devices can be computers, mobile phones, laptops, servers, embedded devices, etc., where the server can be a cloud server or a server cluster. The above examples of electronic devices are merely illustrative, and practical applications are not limited to these.

[0029] In some embodiments of this application, the distributed cluster can be a message queue and stream processing platform used to implement scenarios such as log aggregation, message queues, and real-time data pipelines. For example, the distributed cluster can be an Apache Kafka cluster or an Apache Pulsar cluster. This application does not limit the specific type or number of distributed clusters.

[0030] In some embodiments of this application, the distributed cluster includes ZooKeeper cluster nodes and Broker nodes, with the Broker nodes connecting to the ZooKeeper cluster nodes via heartbeat sessions. For example, the Broker nodes can communicate with the ZooKeeper cluster nodes via a port (such as port 5157).

[0031] ZooKeeper cluster nodes can provide distributed coordination services for tasks such as metadata management and controller election. For example, a ZooKeeper cluster node can be a cluster of servers providing distributed coordination services.

[0032] There can be multiple Broker nodes, used for data storage, message forwarding, and read / write request processing, and they coordinate and synchronize the state of the distributed cluster based on metadata maintained by the ZooKeeper cluster nodes. For example, a Broker node can be a server.

[0033] like Figure 1 The diagram shown is a flowchart of a distributed cluster session parameter optimization method provided in an embodiment of this application. Depending on different needs, the order of the steps in this flowchart can be adjusted according to actual requirements, and some steps can be omitted. The method is applied to electronic devices, such as… Figure 8 The method for the electronic device 10 shown includes the following steps: S11 detects the network packet loss rate between Broker nodes and ZooKeeper cluster nodes in the distributed cluster.

[0034] In some embodiments of this application, there may be multiple Broker nodes in the distributed cluster, and electronic devices may detect the network packet loss rate between each Broker node and the ZooKeeper cluster nodes.

[0035] Among them, network packet loss rate can refer to the proportion of data packets lost during network transmission out of the total number of packets sent, and can be expressed as a percentage.

[0036] In some embodiments of this application, the electronic device can utilize network diagnostic tools to detect the network packet loss rate between the Broker node and the ZooKeeper cluster nodes. For example, the network diagnostic tool can be a command-line tool, including but not limited to: the ping command and the iperf command.

[0037] S12, determine whether the network packet loss rate is less than the preset threshold.

[0038] In some embodiments of this application, the preset threshold can be customized, and this application does not limit this. When the network packet loss rate is expressed as a percentage, the preset threshold can also be expressed as a percentage. For example, the preset threshold could be 80%.

[0039] In some embodiments of this application, for each Broker node in the distributed cluster, the electronic device can sequentially determine whether the network packet loss rate between each Broker node and the ZooKeeper cluster node is less than a preset threshold.

[0040] In other embodiments of this application, when the distributed cluster is large in scale and the number of Broker nodes is large, the electronic device can select the largest network packet loss rate from the network packet loss rates between all Broker nodes and ZooKeeper cluster nodes, and determine whether the largest network packet loss rate is less than a preset threshold.

[0041] In some embodiments of this application, if the network packet loss rate is less than a preset threshold, steps S13 to S14 are executed; if the network packet loss rate is greater than or equal to the preset threshold, step S15 is executed.

[0042] In this embodiment, the comparison result between the network packet loss rate and the preset threshold can reflect the severity of the communication link quality between the Broker node and the ZooKeeper cluster node.

[0043] S13, based on the preset parameter mapping relationship, determines the session timeout time corresponding to the Broker node according to the network packet loss rate.

[0044] In some embodiments of this application, the parameter mapping relationship can be obtained by simulating packet loss tests on a distributed cluster, or it can be set by the user. This application does not limit this.

[0045] In some embodiments of this application, the parameter mapping relationship includes a one-to-one correspondence between multiple network packet loss rates and multiple session timeout times, wherein the session timeout time corresponding to a larger network packet loss rate is greater than the session timeout time corresponding to a smaller network packet loss rate. The session timeout time corresponding to a Broker node can be used as a threshold for ZooKeeper cluster nodes to determine whether a Broker node is offline. For example, when any Broker node fails to send heartbeat information to a ZooKeeper cluster node within the corresponding session timeout time, the ZooKeeper cluster node can determine that the Broker node is offline.

[0046] For example, Table 1 shows an example of the parameter mapping relationship provided in an embodiment of this application. In the parameter mapping relationship shown in Table 1, the session timeout time corresponding to a larger network packet loss rate is greater than the session timeout time corresponding to a smaller network packet loss rate. Specifically, the session timeout time corresponding to a network packet loss rate of 0% is 6000ms, the session timeout time corresponding to a network packet loss rate of 10% is 8000ms, the session timeout time corresponding to a network packet loss rate of 20% is 10000ms, the session timeout time corresponding to a network packet loss rate of 30% is 14000ms, the session timeout time corresponding to a network packet loss rate of 40% is 18000ms, the session timeout time corresponding to a network packet loss rate of 50% is 22000ms, the session timeout time corresponding to a network packet loss rate of 60% is 26000ms, the session timeout time corresponding to a network packet loss rate of 70% is 30000ms, and when the network packet loss rate is greater than 80%, the corresponding session timeout time is 30000ms.

[0047] Table 1 In some embodiments of this application, determining the session timeout time corresponding to the Broker node based on the network packet loss rate according to the preset parameter mapping relationship includes: if a session timeout time corresponding to the network packet loss rate is found based on the parameter mapping relationship, the session timeout time is determined as the session timeout time corresponding to the Broker node; if a session timeout time corresponding to the network packet loss rate cannot be found, the network packet loss rate range in the parameter mapping relationship is determined, the time range is determined according to the session timeout times corresponding to the endpoints of the network packet loss rate range, and the time range is interpolated according to the network packet loss rate to obtain the session timeout time corresponding to the Broker node.

[0048] For example, following the parameter mapping relationship shown in Table 1, if the network packet loss rate between any Broker node and a ZooKeeper cluster node is 20%, according to the parameter mapping relationship shown in Table 1, the session timeout time corresponding to this Broker node can be determined to be 10000ms. If the network packet loss rate between any Broker node and a ZooKeeper cluster node is 15%, according to the parameter mapping relationship shown in Table 1, it can be determined that it falls within the network packet loss rate range of 10%~20%. Since the session timeout time corresponding to a network packet loss rate of 10% is 8000ms, and the session timeout time corresponding to a network packet loss rate of 20% is 10000ms, it can be determined that the time range corresponding to a network packet loss rate of 15% is 8000ms~10000ms. By interpolating the time range of 8000ms~10000ms based on the network packet loss rate of 15%, the session timeout time corresponding to this Broker node can be obtained.

[0049] In some embodiments of this application, for each Broker node in the distributed cluster, the electronic device can determine the session timeout time corresponding to each Broker node based on the network packet loss rate between each Broker node and the ZooKeeper cluster nodes using parameter mapping relationships.

[0050] In other embodiments of this application, when the distributed cluster is large in scale and the number of Broker nodes is large, the electronic device can select the largest network packet loss rate from the network packet loss rates between all Broker nodes and ZooKeeper cluster nodes, and determine the session timeout time common to all Broker nodes in the distributed cluster based on the largest network packet loss rate and the parameter mapping relationship.

[0051] In some embodiments of this application, the electronic device can collect the historical network packet loss rate and the reference session timeout time corresponding to the historical network packet loss rate between the Broker node and the ZooKeeper cluster node as training samples, use the training samples to train the machine learning model, and use the trained machine learning model to optimize the parameter mapping relationship.

[0052] In other embodiments of this application, the electronic device can collect the network packet loss rate between the Broker node and the ZooKeeper cluster node and input it into the trained machine learning model to obtain the session timeout time corresponding to the Broker node output by the machine learning model.

[0053] In other embodiments of this application, the method for determining the corresponding session timeout based on the network packet loss rate described above is only an example. In practical applications, electronic devices can also determine the corresponding session timeout based on more data, such as network latency and bandwidth utilization.

[0054] In this embodiment, if the network packet loss rate is less than a preset threshold, it means that the communication link quality between the Broker node and the ZooKeeper cluster node is within an acceptable range. At this time, based on the network packet loss rate, the session timeout time corresponding to the Broker node can be quickly determined using parameter mapping relationships, and the determined session timeout time can be ensured to be adapted to the current network conditions.

[0055] S14 utilizes the session timeout time corresponding to the Broker node and configures the parameters of the Broker node through the dynamic configuration interface of the distributed cluster.

[0056] In some embodiments of this application, the dynamic configuration interface can be implemented using a command-line tool. For example, the dynamic configuration interface for a Kafka cluster can be the command-line tool kafka-configs.shx.

[0057] In some embodiments of this application, the electronic device can use a dynamic configuration interface to modify the original session timeout of the Broker node to a determined session timeout, thereby achieving parameter configuration.

[0058] In this embodiment, the Broker node parameters can be configured online via the dynamic configuration interface of the distributed cluster, enabling hot updates of session parameters. Since there is no need to restart the Broker node, parameter configuration efficiency is improved.

[0059] In some embodiments of this application, after configuration is completed, the electronic device can determine a performance index score based on the performance index data of the distributed cluster before and after parameter configuration, and then evaluate the effect of parameter configuration based on the performance index score. The method for determining the performance index score based on the performance index data of the distributed cluster before and after parameter configuration can be found in the descriptions of steps S21-S23 and S31-S33 below.

[0060] S15, output alarm prompts, and / or, use the preset session timeout time to configure parameters of the Broker node through the dynamic configuration interface.

[0061] In some embodiments of this application, alarm prompts can be used to prompt users to optimize the network layer and provide corresponding optimization suggestions. In some embodiments of this application, the preset session timeout is greater than or equal to the maximum session timeout in the parameter mapping relationship. For example, the preset session timeout can be 30000ms.

[0062] In this embodiment, if the network packet loss rate is greater than or equal to a preset threshold, it indicates that the communication link between the Broker node and the ZooKeeper cluster node has been severely degraded. At this time, an alarm prompt is output and the parameters of the Broker node are configured using a larger preset session timeout, so that users can troubleshoot network faults in a timely manner. It can also effectively reduce the frequent misjudgment of the Broker node as offline due to network packet loss / network jitter.

[0063] In the distributed cluster session parameter optimization scheme provided in this application embodiment, the comparison result of the network packet loss rate and the preset threshold can reflect the severity of the communication link quality between the Broker node and the ZooKeeper cluster node. If the network packet loss rate is less than the preset threshold, it indicates that the communication link quality between the Broker node and the ZooKeeper cluster node is within an acceptable range. At this time, based on the network packet loss rate and using the parameter mapping relationship, the session timeout time corresponding to the Broker node can be quickly determined, and it can be ensured that the determined session timeout time is adapted to the current network conditions. Through the dynamic configuration interface of the distributed cluster, the parameters of the Broker node can be configured online, realizing hot updates of session parameters. Since there is no need to restart the Broker node, the parameter configuration efficiency can be improved. If the network packet loss rate is greater than or equal to the preset threshold, it indicates that the communication link between the Broker node and the ZooKeeper cluster node has been severely degraded. At this time, an alarm prompt is output and the parameters of the Broker node are configured using a larger preset session timeout time, so that users can troubleshoot network faults in a timely manner, and it can also effectively reduce the frequent misjudgment of the Broker node as offline due to network packet loss / network jitter. In this way, the corresponding session timeout time is adaptively determined according to the network conditions, thereby optimizing the session parameters of the Broker node. This effectively solves the technical problem of the rigid session monitoring mechanism being unable to adapt to changes in network conditions, thus improving the stability and production / consumption performance of the distributed cluster.

[0064] In some embodiments of this application, the electronic device interpolates a time range based on the network packet loss rate to obtain the session timeout time corresponding to the Broker node. For example... Figure 2 The diagram shown is a flowchart of a method for determining the session timeout time corresponding to a Broker node according to an embodiment of this application, including the following steps: S21, determine the relative position of the network packet loss rate within the network packet loss rate range.

[0065] In some embodiments of this application, the electronic device can determine a first packet loss rate difference based on the difference between the network packet loss rate and the lower endpoint of the network packet loss rate interval, and determine a second packet loss rate difference based on the difference between the upper and lower endpoints of the network packet loss rate interval. The ratio of the first packet loss rate difference to the second packet loss rate difference is determined as a relative position ratio. The upper endpoint of the network packet loss rate interval is greater than the lower endpoint.

[0066] For example, the method for calculating the relative position ratio can refer to the following formula (1): (1) in, Indicates relative positional proportions. This represents the network packet loss rate between the Broker node and the ZooKeeper cluster nodes. This represents the lower endpoint of the network packet loss rate range. This indicates the upper limit of the network packet loss rate range.

[0067] S22, determine the time difference between the upper and lower endpoints of the time range, and determine the product of the time difference and the relative position ratio.

[0068] In some embodiments of this application, the upper endpoint of the time range is greater than the lower endpoint.

[0069] S23. Determine the session timeout time corresponding to the Broker node based on the sum of the product of the lower endpoint of the time range and the product. For example, the method for calculating the session timeout for the Broker node can refer to the following formula (2): Target_timeout =timeout1 + (timeout2 - timeout1) × ratio; (2) Here, Target_timeout can represent the session timeout time corresponding to the Broker node, timeout1 represents the lower end of the time range, timeout2 represents the upper end of the time range, and ratio represents the relative position ratio.

[0070] In some embodiments of this application, electronic devices can determine parameter mapping relationships by performing simulated network packet loss tests on a distributed cluster. The simulated network packet loss test can be conducted using the controlled variable method, which involves keeping the packet loss rate constant and obtaining the performance of the distributed cluster at different session timeout times, thereby selecting the session timeout time corresponding to that packet loss rate to establish a parameter mapping relationship.

[0071] To more clearly illustrate the method for constructing parameter mapping relationships, the embodiments of this application are described as follows: ZooKeeper cluster nodes can be used to elect a master node. The master node can be used to elect the leader node of a partition and manage the In-Sync Replicas (ISR). The leader node is used to receive data sent from the producer nodes and send the received data to the consumer nodes. The ISR includes broker nodes that keep data synchronized with the leader node. The master node can also be called a controller. For a description of the election process for the master node and leader node, please refer to the descriptions in related technologies.

[0072] like Figure 3 The diagram shown is a flowchart of a method for constructing parameter mapping relationships according to an embodiment of this application, including the following steps: S31 uses firewall tools to simulate network packet loss on a distributed cluster based on multiple preset network packet loss rates and multiple preset session timeouts.

[0073] In some embodiments of this application, before performing network simulation packet loss tests, the electronic device can perform a rebalancing of the Leader nodes in the distributed cluster to ensure that the load of each Broker node is balanced.

[0074] In some embodiments of this application, the plurality of network packet loss rates and the plurality of session timeout times can be customized, and this application does not impose any limitations on this. For example, the plurality of network packet loss rates and the plurality of session timeout times can be referred to Table 1.

[0075] In some embodiments of this application, the firewall tool can be iptables. Utilizing iptables' rule configuration capabilities, the loss of heartbeat packets sent from the Broker node to the ZooKeeper cluster node in the OUTPUT direction and the loss of response packets returned from the ZooKeeper cluster node to the Broker node in the INPUT direction can be simulated, thereby imposing different network packet loss rates on the communication link between the Broker node and the ZooKeeper cluster node. Based on this, by keeping the network packet loss rate constant and configuring the Broker node with different session timeouts, it is possible to test whether the ZooKeeper cluster node is offline if it does not receive a heartbeat packet from the Broker node within different session timeout periods.

[0076] When simulating heartbeat and response packet loss, the ports on which iptables rules apply can be configured to precisely pinpoint the communication traffic between the Broker node and the ZooKeeper cluster nodes. For example, if the ZooKeeper cluster nodes and the Broker node communicate via port 5157, iptables rules can be configured on the Broker node to set network packet loss rate rules for the destination port (--dport 5157) in the OUTPUT direction and the source port (--sport 5157) in the INPUT direction. This ensures that packet loss simulation only affects the communication link between the Broker node and the ZooKeeper cluster nodes, avoiding impact on other network traffic.

[0077] In this embodiment of the application, the above-mentioned simulation operation will not have a substantial impact on the nodes themselves, the stored data, and the business logic of the distributed cluster.

[0078] S32, based on the performance index scores of the distributed cluster under the same network packet loss rate and corresponding different session timeout times, determine the session timeout time corresponding to each network packet loss rate from multiple session timeout times.

[0079] In some embodiments of this application, the method for determining the performance index score includes: under the same network packet loss rate, determining the performance index score of the distributed cluster corresponding to the timeout performance of the given session based on the performance index data of the distributed cluster before and after configuration using any session timeout time.

[0080] In some embodiments, performance metrics data may include one or more of the following: the number of master node switches, the number of session timeouts between ZooKeeper cluster nodes and Broker nodes, the number of ISR shrinks, the latency of producer nodes writing data, and the latency of consumer nodes reading data.

[0081] The number of times the master node switches can also be called the controller switch count, which can be used to characterize the frequency of master node changes. For example, when the master node fails or there is a network partition, the ZooKeeper cluster nodes will trigger a new round of master node election, and each master node change is counted as a master node switch.

[0082] Session timeout counts can be used to characterize the connection stability between ZooKeeper cluster nodes and Broker nodes. For example, if a Broker node does not send a heartbeat message to a ZooKeeper cluster node within the session timeout period, it is counted as a session timeout.

[0083] The number of ISR contractions is obtained by counting the number of times the Leader node removes a Broker node that is lagging behind in synchronization from the ISR.

[0084] The delay time for a producer node to write data, also known as the production delay time, can be used to characterize the average time required for a producer node to send data to the leader node and receive an acknowledgment response.

[0085] The latency of a consumer node reading data, also known as the consumption completion time, can be used to characterize the average time required for a consumer node to pull or read data from the Leader node and complete the reception.

[0086] For example, the number of controller switching times can be collected using the following command line: grep -i "elected as the controller" controller.log | wc -l.

[0087] The number of session timeouts can be collected using the following command line: grep -i "Session expired"kafkaServer.log | wc -l; The number of times an ISR (Internal Server Registry) shrinks can be collected using the following command line: `grep -i isr server.log | grepShrinking | wc -l`.

[0088] Production delays can be collected using the following command line: kafka-producer-perf-test.sh.

[0089] Consumption latency can be collected using the following command line: kafka-consumer-perf-test.sh.

[0090] For example, an electronic device can determine the master node switching improvement rate based on the number of switching times of the master node before and after parameter configuration, the session timeout improvement rate based on the number of session timeouts between ZooKeeper cluster nodes and Broker nodes before and after parameter configuration, the ISR shrinkage improvement rate based on the number of ISR shrinkages before and after parameter configuration, the production performance improvement rate based on the data write latency of producer nodes before and after parameter configuration, and the consumption performance improvement rate based on the data read latency of consumer nodes before and after parameter configuration. A performance index score is obtained by weighted summing one or more of the master node switching improvement rate, session timeout improvement rate, ISR shrinkage improvement rate, production performance improvement rate, and consumption performance improvement rate.

[0091] For example, the calculation method for the master node switching improvement rate can refer to the following formula: Master node switching improvement rate = (number of switching before parameter configuration - number of switching after parameter configuration) / max(number of switching before parameter configuration, 1). Where max() represents the maximum value operation.

[0092] The calculation methods for session timeout improvement rate, ISR contraction improvement rate, production performance improvement rate and consumption performance improvement rate can be found in the explanation of formula (3).

[0093] In some embodiments, the electronic device can determine the production and consumption performance improvement rate based on the average or weighted average of the production performance improvement rate and the consumption performance improvement rate, and obtain a performance index score by weighted summation of the master control node switching improvement rate, session timeout improvement rate, ISR contraction improvement rate, and production and consumption performance improvement rate.

[0094] The weighted weights for the master node switching improvement rate, session timeout improvement rate, ISR contraction improvement rate, and production / consumption performance improvement rate can be customized, and this application embodiment does not impose any restrictions on this.

[0095] For example, the weighted weight for the master node switching improvement rate can be 0.35, the weighted weight for the session timeout improvement rate can be 0.25, the weighted weight for the ISR contraction improvement rate can be 0.25, and the weighted weight for the production and consumption performance improvement rate can be 0.15.

[0096] In some embodiments, for each network packet loss rate, the electronic device can obtain performance index scores for multiple session timeout times corresponding to the distributed cluster, and select the session timeout time corresponding to the highest performance index score as the session timeout time corresponding to that network packet loss rate.

[0097] In some embodiments, the electronic device can evaluate the corresponding parameter configuration results based on the evaluation level corresponding to the preset score range in which each performance indicator score falls, so as to implement differentiated adjustment strategies for the aforementioned multiple session timeout times and / or multiple network packet loss rates. For example, the preset score range and the corresponding evaluation level can be customized, and this embodiment does not limit this. For example, the preset score range may include 0~0.3, 0.3~0.7, and 0.7~1, where the evaluation level corresponding to 0~0.3 may be poor, corresponding to a strategy that requires comprehensive parameter adjustment; the evaluation level corresponding to 0.3~0.7 may be medium, corresponding to a strategy that requires adjustment of some parameters; and the evaluation level corresponding to 0.7~1 may be excellent, corresponding to a strategy that does not require parameter adjustment.

[0098] In this embodiment, by comprehensively calculating performance index scores based on performance metrics such as master node switching improvement rate, session timeout improvement rate, ISR contraction improvement rate, and production / consumption performance improvement rate, the overall performance of the distributed cluster can be comprehensively evaluated, thereby ensuring the reliability and accuracy of the performance index scores. Under the same network packet loss rate, by selecting the session timeout corresponding to the highest performance index score from multiple session timeout performance index scores corresponding to the distributed cluster as the session timeout corresponding to that network packet loss rate, the reliability and effectiveness of the session timeout corresponding to each network packet loss rate can be ensured.

[0099] S33, construct the correspondence between each network packet loss rate and the corresponding session timeout, forming a parameter mapping relationship.

[0100] In some embodiments of this application, by constructing a correspondence between each network packet loss rate and the corresponding session timeout, each network packet loss rate can be associated with the corresponding session timeout.

[0101] like Figure 4 The diagram shown is a schematic representation of a parameter configuration effect evaluation method provided in an embodiment of this application. Figure 4 The steps shown can be referenced. Figure 3 Explanation.

[0102] like Figure 5 The diagram shown is a schematic diagram of a network packet loss simulation test provided in an embodiment of this application. Figure 5 The steps shown can be referenced. Figure 3 Explanation.

[0103] In this embodiment, network packet loss simulation is performed to test the performance of the distributed cluster under the same network packet loss rate and different session timeout times. Based on this, session timeout times corresponding one-to-one with multiple network packet loss rates are determined to establish parameter mapping relationships. In this way, the optimal session timeout configuration under different network environments can be obtained safely and efficiently without compromising the normal operation of the distributed cluster, thereby improving the reliability and effectiveness of the parameter mapping relationship and providing data support for subsequent rapid parameter configuration under different network conditions.

[0104] like Figure 6 The diagram shown is a system architecture diagram of a distributed cluster session parameter optimization method provided in an embodiment of this application. Figure 6 The system architecture shown includes a monitoring and data acquisition layer, a processing and computation layer, a Kafka cluster, and a verification tool layer. (For...) Figure 6 Please refer to the explanations of each level. Figures 1 to 5 Explanation.

[0105] The session parameter optimization method provided in this application can improve the stability and production / consumption performance of a distributed cluster. Table 2 shows an example of the improvement effect provided by an embodiment of this application. By modifying the session timeout of the Broker node from 6000ms to 18000ms, this application can reduce the number of controller switches, session timeouts, ISR shrinkages, average production latency, and consumption completion time of the distributed cluster.

[0106] Table 2 like Figure 7 The diagram shown is a functional block diagram of a session parameter optimization device provided in an embodiment of this application. The session parameter optimization device 11 includes a detection unit 111, a determination unit 112, and a configuration unit 113. The module / unit referred to in this application refers to a module / unit that can be... Figure 8 The processor 103 in the middle acquires a series of computer-readable instruction segments that are capable of performing a fixed function, and these segments are stored in Figure 8 The memory 102 is used for this purpose. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0107] The detection unit 111 is used to detect the network packet loss rate between the Broker node and the ZooKeeper cluster node in the distributed cluster.

[0108] The determining unit 112 is used to determine the session timeout time corresponding to the Broker node based on the preset parameter mapping relationship if the network packet loss rate is less than a preset threshold.

[0109] Configuration unit 113 is used to optimize session parameters by configuring the Broker node through the dynamic configuration interface of the distributed cluster, using the session timeout time corresponding to the Broker node.

[0110] like Figure 8 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Figure 8 The electronic device 10 shown can be a computer, mobile phone, tablet computer, laptop computer, server, etc. The embodiments of this application do not limit the specific type of electronic device 10.

[0111] exist Figure 8 The electronic device 10 may include a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is coupled to the communication module 101, the memory 102, and the input / output interface 104 via the bus 105.

[0112] Communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB) and Controller Area Network (CAN). The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication networks, frequency modulation (FM), near field communication (NFC), and infrared (IR).

[0113] Memory 102 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 103, and can be used to store executable programs (e.g., machine instructions) of other running programs, as well as user and application data. The RAM may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0114] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 103. Non-volatile memory can include disk storage devices and flash memory.

[0115] Memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by processor 103. The one or more computer programs include multiple instructions that, when executed by processor 103, enable a session parameter optimization method for a distributed cluster executed on electronic device 10.

[0116] In other embodiments, such as Figure 8 The electronic device 10 shown also includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10.

[0117] Processor 103 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0118] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute computer programs stored in the memory 102 to implement the session parameter optimization method of the distributed cluster described above.

[0119] The input / output interface 104 is used to provide a channel for user input or output. For example, the input / output interface 104 can be used to connect various input / output devices, such as a mouse, keyboard, touch device, display screen, etc., so that users can enter information or visualize information.

[0120] Bus 105 is used at least to provide a channel for communication between communication modules 101, memory 102, processor 103, and input / output interface 104 in electronic device 10.

[0121] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0122] This application also provides a computer-readable storage medium storing a computer program, which includes program instructions. When the program instructions are executed, the method implemented can refer to the methods in the above embodiments of this application.

[0123] The computer-readable storage medium can be the internal memory of the electronic device described in the above embodiments, such as the hard disk or memory of the electronic device. Alternatively, the computer-readable storage medium can be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., installed on the electronic device.

[0124] In some embodiments, a computer-readable storage medium may include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of the electronic device, etc.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0126] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0128] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0129] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this application may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method for optimizing session parameters in a distributed cluster, characterized in that, The distributed cluster includes ZooKeeper cluster nodes and Broker cache proxy nodes. The Broker nodes connect to the ZooKeeper cluster nodes via heartbeat sessions. The method includes: Detect the network packet loss rate between the Broker node and the ZooKeeper cluster node; If the network packet loss rate is less than a preset threshold, the session timeout time corresponding to the Broker node is determined based on the preset parameter mapping relationship and the network packet loss rate. By utilizing the session timeout time corresponding to the Broker node, the parameters of the Broker node are configured through the dynamic configuration interface of the distributed cluster to optimize the session parameters.

2. The session parameter optimization method as described in claim 1, characterized in that, The parameter mapping relationship includes a one-to-one correspondence between multiple network packet loss rates and multiple session timeout times, wherein the session timeout time corresponding to a larger network packet loss rate is greater than the session timeout time corresponding to a smaller network packet loss rate. The session timeout time corresponding to the Broker node is used by the ZooKeeper cluster node to determine whether the Broker node is offline.

3. The session parameter optimization method as described in claim 1, characterized in that, The determination of the session timeout time corresponding to the Broker node based on the preset parameter mapping relationship and the network packet loss rate includes: Based on the parameter mapping relationship, if the session timeout time corresponding to the network packet loss rate is found, the session timeout time is determined as the session timeout time corresponding to the Broker node. If the session timeout time corresponding to the network packet loss rate cannot be found, determine the network packet loss rate range that the network packet loss rate falls into in the parameter mapping relationship, determine the time range according to the session timeout time corresponding to the endpoints of the network packet loss rate range, and perform interpolation operation on the time range according to the network packet loss rate to obtain the session timeout time corresponding to the Broker node.

4. The session parameter optimization method as described in claim 3, characterized in that, The step of interpolating the time range based on the network packet loss rate to obtain the session timeout time corresponding to the Broker node includes: Determine the relative position of the network packet loss rate within the network packet loss rate range; Determine the time difference between the upper and lower endpoints of the time range, and determine the product of the time difference and the relative position ratio; The session timeout time corresponding to the Broker node is determined based on the sum of the product of the lower endpoint of the time range and the product.

5. The session parameter optimization method as described in claim 1, characterized in that, If the network packet loss rate is greater than or equal to the preset threshold, the method further includes: Output alarm prompts; and / or Using a preset session timeout, the Broker node is configured with parameters through the dynamic configuration interface. The preset session timeout is greater than or equal to the maximum session timeout in the parameter mapping relationship.

6. The session parameter optimization method according to any one of claims 1 to 5, characterized in that, The method further includes: Based on multiple preset network packet loss rates and multiple preset session timeouts, network packet loss simulation is performed on the distributed cluster using firewall tools; Based on the performance index scores of the distributed cluster under the same network packet loss rate and corresponding different session timeout times, the session timeout time corresponding to each network packet loss rate is determined from the multiple session timeout times. Establish the correspondence between each network packet loss rate and the corresponding session timeout, forming the parameter mapping relationship.

7. The session parameter optimization method as described in claim 6, characterized in that, The ZooKeeper cluster nodes are used to elect a master node. The master node is used to elect a leader node for each partition and manage the Integral Replica Set (ISR). The ISR includes broker nodes that maintain data synchronization with the leader node. The leader node is used to receive data sent from producer nodes and send the received data to consumer nodes. The method for determining the performance metric score includes: Under the same network packet loss rate, based on the performance index data of the distributed cluster before and after configuration using any session timeout time, the performance index score of the distributed cluster corresponding to the performance of any session timeout is determined. The performance index data includes one or more of the following: the number of switching of the master node, the number of session timeouts between the ZooKeeper cluster node and the Broker node, the number of ISR shrinkages, the latency of the producer node writing data, and the latency of the consumer node reading data. The number of ISR shrinkages is obtained by counting the number of times the Leader node removes the Broker node that is lagging behind in synchronization from the ISR.

8. The session parameter optimization method as described in claim 7, characterized in that, The method for determining the performance index score includes: The switching improvement rate of the master control node is determined based on the number of switching times of the master control node before and after parameter configuration. The session timeout improvement rate is determined based on the number of session timeouts between the ZooKeeper cluster nodes and the Broker nodes before and after parameter configuration. The ISR contraction improvement rate is determined based on the number of contractions before and after parameter configuration. The production performance improvement rate is determined based on the data write latency of the producer node before and after parameter configuration. The consumer performance improvement rate is determined based on the data read latency of the consumer node before and after parameter configuration. The performance index score is obtained by weighted summing of one or more of the master control node switching improvement rate, the session timeout improvement rate, the ISR contraction improvement rate, the production performance improvement rate, and the consumption performance improvement rate.

9. An electronic device, characterized in that, The electronic device includes: Memory; A processor and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the session parameter optimization method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor in an electronic device, implements the session parameter optimization method as described in any one of claims 1 to 8.

Images