Method, device and electronic equipment for optimizing configuration of distributed system

Abstract

The application discloses a configuration optimization method and device of a distributed system and electronic equipment. The method comprises the following steps: determining the running state of the distributed system; in the case that the running state does not meet the preset performance index, adjusting the number of agents and the number of partitions of the distributed system through a configuration optimization model to obtain a target number of agents and a target number of partitions, wherein the configuration optimization model comprises a global agent optimization model and a local partition optimization model, the global agent optimization model is used for optimizing the system performance by adjusting the number of agents, and the local partition optimization model is used for optimizing the system performance by adjusting the number of partitions; and configuring the distributed system according to the target number of agents and the target number of partitions. The application solves the technical problems of excessive data delay, processing bottleneck difficulty and resource waste in the high-concurrency real-time data stream processing of the Kafka configuration in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed systems, in particular to a configuration optimization method and device for a distributed system and an electronic device. BACKGROUND

[0002] With the rapid development of information technology and the continuous emergence of large-scale data, distributed data processing platforms play a crucial role in enterprise operation and maintenance, cloud computing, and big data analysis. Among them, Kafka, as a widely used distributed data stream processing platform, can support high-throughput real-time data exchange. Its application in the producer-consumer mode can provide strong technical support for the digital transformation of operation and maintenance technology. For example, in the process of automated deployment, the system needs to process a large amount of real-time logs, alarm information, and device status data. Using Kafka to transmit these data in real time can ensure that operation and maintenance personnel can obtain system status in the first time and improve the response speed to sudden failures and abnormal events. In addition, in the scenarios of intelligent monitoring and operation and maintenance automation, Kafka can serve as the core hub of data flow, helping to quickly distribute and process monitoring data to support fault detection, alarm, and self-processing functions.

[0003] However, the performance of Kafka is highly dependent on its internal configuration, especially the settings of partitions and brokers. The number of partitions determines the parallel processing capability of data, while the number of brokers affects data storage and system fault tolerance. Therefore, although Kafka itself provides a certain degree of fault tolerance and high concurrency support, its default configuration often cannot meet the needs of specific scenarios, especially in complex network environments with multiple nodes distributed, the throughput, delay, and system availability of data processing become key problems to be solved. If Kafka is not properly configured, it may lead to decreased data processing efficiency, increased data delay, and even processing bottlenecks, affecting the real-time response capability and stability of the entire operation and maintenance system.

[0004] Currently, there is no effective solution to the above problems. SUMMARY

[0005] The embodiments of the present application provide a configuration optimization method and device for a distributed system and an electronic device to at least solve the technical problems of excessive data delay, processing bottleneck difficulty, and resource waste in high-concurrency real-time data stream processing caused by the Kafka configuration in the related art.

[0006] According to an aspect of the embodiments of the present application, a method for optimizing configuration of a distributed system is provided, comprising: determining a running state of the distributed system, wherein the running state comprises a load condition and a resource usage condition of each instance in the distributed system; in a case where the running state does not satisfy a preset performance index, adjusting a number of agents and a number of partitions of the distributed system by a configuration optimization model to obtain a target number of agents and a target number of partitions, wherein the configuration optimization model comprises a global agent optimization model and a local partition optimization model, the global agent optimization model is used to optimize system performance by adjusting the number of agents, and the local partition optimization model is used to optimize system performance by adjusting the number of partitions; and configuring the distributed system according to the target number of agents and the target number of partitions.

[0007] Optionally, the configuration optimization model comprises at least a first target function, a second target function and a third target function, wherein the first target function is used to measure a global delay of the distributed system, the second target function is used to measure a resource cost of the distributed system under the target number of agents, and the third target function is used to measure a resource utilization rate and a performance index of the distributed system under the target number of partitions.

[0008] Optionally, in a case where the running state does not satisfy the preset performance index, adjusting the number of agents and the number of partitions of the distributed system by the configuration optimization model to obtain the target number of agents and the target number of partitions comprises: in a case where the load condition exceeds a first preset threshold, determining the target number of agents of the distributed system by the global agent optimization model, wherein the target number of agents is used to represent a number of agents required by the distributed system under the load condition; and in a case where the resource usage condition exceeds a second preset threshold, determining the target number of partitions of the distributed system by the local partition optimization model, wherein the target number of partitions is used to represent a number of partitions required by the distributed system under the resource usage condition.

[0009] Optionally, the target number of agents of the distributed system is determined by the global agent optimization model, comprising: determining the target number of agents by a fourth objective function in the global agent optimization model, wherein the fourth objective function is determined by: determining an initial number of partitions of the distributed system; determining a delay constraint and an availability constraint under a load condition, wherein the delay constraint is used to keep a relationship between the number of agents, the initial number of partitions, a replication factor and a replication delay of the distributed system satisfying a preset delay of the system, the availability constraint is used to keep an agent unavailable time of the distributed system satisfying a preset unavailable time of the system, wherein the replication factor is used to represent a number of replicas of a partition in the distributed system, the replication delay is used to represent a replication time of an agent in the distributed system, the preset delay is used to represent a maximum delay time that can be accepted by the system, the agent unavailable time is used to represent a service suspension time of an agent in the distributed system, and the preset unavailable time is used to represent a maximum agent unavailable time that can be accepted by the system; and determining the fourth objective function according to the initial number of partitions, the delay constraint and the availability constraint.

[0010] Optionally, the target number of partitions of the distributed system is determined by the local partition optimization model, comprising: determining the target number of partitions by a fifth objective function in the local partition optimization model, wherein the fifth objective function is determined by: determining a throughput constraint of the distributed system, wherein the throughput constraint is used to keep a number of partitions of the distributed system satisfying a throughput demand of the system; and determining the fifth objective function according to the target number of agents, the delay constraint and the throughput constraint.

[0011] Optionally, the method further comprises: determining a target running state of the distributed system cluster within a preset time period, wherein the target running state is a running state of the distributed system after the target number of agents and the target number of partitions are configured; adjusting the target number of agents and the target number of partitions by the configuration optimization model in a case where the target running state does not satisfy a preset performance index; and keeping the configuration of the distributed system in a case where the target running state satisfies the preset performance index.

[0012] Optionally, the preset performance index at least comprises: a global delay of the distributed system, a delay of each partition in the distributed system and a maximum load of each agent in the distributed system.

[0013] According to another aspect of the embodiments of the present application, a configuration optimization apparatus of a distributed system is also provided, comprising: a determination module configured to determine a running state of the distributed system, wherein the running state comprises a load condition and a resource usage condition of each instance in the distributed system; an adjustment module configured to, in a case where the running state does not satisfy a preset performance indicator, adjust a number of agents and a number of partitions of the distributed system by using a configuration optimization model to obtain a target number of agents and a target number of partitions, wherein the configuration optimization model comprises a global agent optimization model and a local partition optimization model, the global agent optimization model is configured to optimize system performance by adjusting the number of agents, and the local partition optimization model is configured to optimize system performance by adjusting the number of partitions; and a configuration module configured to configure the distributed system according to the target number of agents and the target number of partitions.

[0014] According to still another aspect of the embodiments of the present application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory is configured to store program instructions, and the processor is connected with the memory and is configured to execute the configuration optimization method of the distributed system.

[0015] According to yet another aspect of the embodiments of the present application, a non-volatile storage medium is also provided, comprising a stored computer program, wherein a device in which the non-volatile storage medium is located executes the configuration optimization method of the distributed system by running the computer program.

[0016] According to still another aspect of the embodiments of the present application, a computer program product is also provided, comprising computer instructions, which, when executed by a processor, implement the configuration optimization method of the distributed system.

[0017] In the embodiments of the present application, the running state of the distributed system is determined, wherein the running state comprises a load condition and a resource usage condition of each instance in the distributed system; in a case where the running state does not satisfy a preset performance indicator, the number of agents and the number of partitions of the distributed system are adjusted by using a configuration optimization model to obtain a target number of agents and a target number of partitions, wherein the configuration optimization model comprises a global agent optimization model and a local partition optimization model, the global agent optimization model is configured to optimize system performance by adjusting the number of agents, and the local partition optimization model is configured to optimize system performance by adjusting the number of partitions; and the distributed system is configured according to the target number of agents and the target number of partitions, thereby achieving the purpose of adaptively adjusting the number of agents and the number of partitions to adapt to different load and resource conditions, and thereby realizing the technical effects of reducing data delay and improving throughput in a high-concurrency scenario, and optimizing resource utilization and reducing system overhead in a resource-limited scenario, and thereby solving the technical problems of excessive data delay, difficult processing bottleneck and resource waste in real-time data stream processing in a high-concurrency scenario due to the Kafka configuration in the related art. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0019] Figure 1 Fig. 1 is a hardware structure diagram of a computer terminal for implementing a configuration optimization method of a distributed system according to an embodiment of the application;

[0020] Figure 2 Fig. 2 is a flowchart of a configuration optimization method of a distributed system according to an embodiment of the application;

[0021] Figure 3 Fig. 3 is a schematic diagram of a configuration optimization process of a distributed system according to an embodiment of the application;

[0022] Figure 4 Fig. 4 is a structure diagram of a configuration optimization device of a distributed system according to an embodiment of the application. DETAILED DESCRIPTION

[0023] In order to make the personnel in the art better understand the scheme of the application, the technical scheme in the embodiments of the application will be clearly and completely described below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the application.

[0024] It should be noted that the terms "first", "second", and the like in the specification and claims of the application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] First, some nouns or terms appearing in the process of explaining the embodiments of the application are applicable to the following explanations:

[0026] Kafka: A high-performance, distributed, and publish-subscribe model-based message queue service designed for handling large-scale real-time data streams. Kafka can provide high throughput, low latency, and persistent data storage, suitable for various scenarios such as log aggregation, stream processing, and event processing.

[0027] Broker: The broker in Kafka is the basic node for data storage and processing, responsible for receiving data sent by producers and storing it, while also handling data reading requests from consumers. A Kafka cluster is composed of multiple brokers, and each broker can store data partitions of multiple topics.

[0028] Partition: Partition is a subset of a Kafka topic, used to achieve parallel storage and processing of data. Each topic can have multiple partitions, and data is evenly distributed among these partitions when written to improve processing capacity and fault tolerance.

[0029] Producer: In Kafka, the producer is a client responsible for sending data to topics. They can be applications, services, or other data sources that participate in Kafka's data stream processing process by writing data to brokers.

[0030] Consumer: The consumer is a client that reads data from Kafka topics, subscribing to topics and reading data from brokers. Consumers can be applications, services, used to process data, analyze data, or forward them to other systems.

[0031] Throughput: Throughput refers to the amount of data a system can process in a unit of time, which is an important indicator of Kafka's data processing capacity. High throughput means that Kafka can quickly process large amounts of data streams, suitable for real-time data processing scenarios.

[0032] Latency: Latency refers to the time interval between data being sent by the producer and being received by the consumer. In real-time data stream processing, low latency is crucial as it directly affects the system's response speed and real-time performance.

[0033] To solve the problem that the Kafka configuration in the related art cannot adapt to high-concurrency real-time data stream processing, the embodiment of the present application provides a configuration optimization method of a distributed system, which can be run in Figure 1 The computer terminal is described as follows.

[0034] The configuration optimization method of the distributed system provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal, or a similar computing device.Figure 1 A hardware structure block diagram of a computer terminal for implementing a configuration optimization method of a distributed system is shown. As shown, Figure 1 The computer terminal 10 can include one or more processors (processors can include but not limited to microprocessor MCU or programmable logic device FPGA processing device, etc.), a memory 104 for storing data, and a transmission module 106 for communication function through wired and / or wireless network connection. In addition, it can also include a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the I / O interface), a network interface, a BUS bus. Those skilled in the art can understand that, Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can include more or less components than Figure 1 shown, or have a different configuration than Figure 1 shown.

[0035] It should be noted that the one or more processors and / or other data processing circuits described above can be referred to herein as "data processing circuits" in general. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any other combination. In addition, the data processing circuit can be a single independent processing module, or all or part of any one of the other elements combined into the computer terminal 10. As referred to in the embodiments of the present application, the data processing circuit is a processor control (for example, the selection of the variable resistance terminal path connected with the interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the configuration optimization method of the distributed system in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the configuration optimization method of the distributed system described above. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 can further include a memory remotely disposed with respect to the processor, which can be connected to the computer terminal 10 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission module 106 is configured to receive or send data via a network. The network can include, for example, a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module that is configured to communicate with the Internet via wireless communication.

[0038] The display can be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with the user interface of the computer terminal 10.

[0039] It is noted that in some alternative embodiments, the above-described Figure 1 The computer terminal can include hardware elements (including circuitry), software elements (including computer code stored on a computer readable medium), or a combination of both hardware and software elements. It should be noted that in some embodiments, the functions of the computer terminal described above can be provided by one or more of the computer terminals 10. Figure 1 is merely one example of a particular implementation and is intended to illustrate the types of components that can be present in the computer terminal described above.

[0040] In the above-described operating environment, the embodiments of the present disclosure provide a configuration optimization method for a distributed system. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0041] Figure 2 is a flowchart of a configuration optimization method for a distributed system according to an embodiment of the present disclosure, as Figure 2 shown, the method includes the following steps:

[0042] In step S202, the running state of the distributed system is determined, wherein the running state includes the load condition and resource usage of each instance in the distributed system.

[0043] In step S202, the main task is to monitor the current running state of the distributed system (taking Kafka as an example), including but not limited to the load condition, resource usage rate and performance indicators of each instance in Kafka. Among them, the Kafka instance refers to a single server node running the Kafka service, and one or more Kafka brokers are running on each instance, responsible for storing data, processing data streams and ensuring data transmission between producers and consumers.

[0044] This step provides the necessary data basis for subsequent optimization decisions, for example, by monitoring the CPU usage, memory occupation, network I / O, etc. of the agent, the real-time load of the system can be evaluated; at the same time, through the monitoring of data throughput, delay and other performance parameters, it can be known whether the system reaches the preset performance indicators.

[0045] Step S204, in the case where the running state does not meet the preset performance indicators, the number of agents and the number of partitions of the distributed system are adjusted by configuring an optimization model to obtain the target number of agents and the target number of partitions, wherein the configuration optimization model includes a global agent optimization model and a local partition optimization model, the global agent optimization model is used to optimize the system performance by adjusting the number of agents, and the local partition optimization model is used to optimize the system performance by adjusting the number of partitions.

[0046] In the above step S204, when the monitored running state does not meet the preset performance indicators, that is, the system has performance bottleneck or resource waste, the configuration optimization model (a multi-objective optimization model) will be started to adjust the number of agents and the number of partitions under the current running state of Kafka to achieve the target number of agents and the target number of partitions, so as to optimize the system performance. The preset performance indicators include but are not limited to minimum throughput, maximum delay, resource utilization threshold, etc., which can be set according to different business needs and scenarios.

[0047] Among them, the configuration optimization model is divided into two sub-models: global agent optimization model and local partition optimization model.

[0048] Specifically:

[0049] The global agent optimization model mainly focuses on the adjustment of the number of agents, aiming to optimize the performance of the entire system by increasing or reducing the number of agent nodes. Specifically, when the system load is high or the delay increases, the global agent optimization model can increase the number of agents through the K-Max algorithm to disperse the load, improve data throughput and fault tolerance, so as to ensure that the system can maintain high efficiency and stability in high concurrency and large-scale data stream transmission.

[0050] The local partition optimization model focuses on the optimization of the number of partitions, which reduces the system delay, improves resource utilization and ensures the load balancing among partitions by adjusting the number of partitions. Specifically, in the case of limited resources or uneven load, the local partition optimization model can reduce the number of agents and optimize resource allocation through the K-Min algorithm, while ensuring the load balancing of each partition and reducing the delay, so as to improve the running efficiency of the entire system.

[0051] Step S206, according to the target number of agents and the target number of partitions, the distributed system is configured.

[0052] In step S206, after obtaining the target number of brokers and the target number of partitions, the actual configuration adjustment of Kafka can be performed, such as adding or deleting broker nodes, and re-partitioning the partitions of the topic.

[0053] Through steps S202 to S206, the purpose of adaptively adjusting the number of brokers and partitions to adapt to different load and resource conditions is achieved, thereby achieving the technical effects of reducing data delay, improving throughput in a high-concurrency scenario, and optimizing resource utilization and reducing system overhead in a resource-limited scenario, thereby solving the technical problems of excessive data delay, processing bottleneck difficulty, and resource waste in the related art Kafka configuration in high-concurrency real-time data stream processing. The following will be described in detail.

[0054] In step S204, the configuration optimization model includes at least a first objective function, a second objective function, and a third objective function, wherein the first objective function is used to measure the global delay of the distributed system, the second objective function is used to measure the resource cost of the distributed system under the target number of brokers, and the third objective function is used to measure the resource utilization and performance indicators of the distributed system under the target number of partitions.

[0055] In the embodiments of the present application, the configuration optimization model aims to optimize the performance of real-time data stream processing by dynamically adjusting the number of brokers (Brokers) and the number of partitions (Partitions) of Kafka. The configuration optimization model includes a first objective function, a second objective function, and a third objective function, and the specific expressions are as follows:

[0056] min(L global (b,P),C(b),R(P))

[0057] Wherein, L global (b,P) represents the first objective function, which depends on the number of brokers b and the number of partitions P, and is used to measure the global delay of Kafka; C(b) represents the second objective function, which is used to measure the resource cost of Kafka under the target number of brokers; R(P) represents the third objective function, which is used to measure the resource utilization and performance indicators of Kafka under the target number of partitions.

[0058] By combining the above three objective functions, the configuration optimization model can achieve global delay minimization, resource cost minimization, and resource utilization maximization, thereby ensuring that Kafka can provide stable, efficient, and cost-effective real-time data stream processing services in different operation and maintenance scenarios. This multi-objective optimization idea makes the optimization strategy flexible to adapt to different business demands and resource constraints, providing solid technical support for digital transformation of operation and maintenance.

[0059] Optionally, in the case that the running state does not meet the preset performance indicators, the number of agents and the number of partitions of the distributed system are adjusted through the configuration optimization model to obtain a target number of agents and a target number of partitions, including: in the case that the load condition exceeds a first preset threshold, determining the target number of agents of the distributed system through a global agent optimization model, wherein the target number of agents represents the number of agents required by the distributed system under the load condition; in the case that the resource usage condition exceeds a second preset threshold, determining the target number of partitions of the distributed system through a local partition optimization model, wherein the target number of partitions represents the number of partitions required by the distributed system under the resource usage condition.

[0060] In the embodiments of the present application, when the running state of Kafka does not meet the preset performance indicators, i.e., the system performance decreases or the resource utilization is improper, the number of agents and the number of partitions can be dynamically adjusted through the configuration optimization model to ensure that the system can recover to the best running state. Specifically:

[0061] When the monitoring shows that the load condition of Kafka (such as data throughput, processing delay, etc.) exceeds the first preset threshold, it indicates that the processing capacity of the system may have reached or approached the limit, and the data processing speed and system response time may not meet the business requirements. At this time, the number of agents can be increased through the global agent optimization model to distribute the load, so as to improve the parallel processing capacity and fault tolerance of the system. Among them, the K-Max algorithm is the core of global agent optimization, which considers multiple performance indicators, including system delay, agent load and resource cost, to ensure that the number of agents is increased while also considering the system delay and cost control, to achieve the best performance and resource utilization balance.

[0062] When the resource usage condition of Kafka (such as storage space, CPU and memory usage, etc.) exceeds the second preset threshold, it indicates that the system resources may have approached saturation, and continuing to increase data processing or maintaining the existing configuration may cause resource bottlenecks, affecting system stability and processing efficiency. At this time, the number of partitions can be adjusted through the local partition optimization model to optimize resource utilization and reduce system delay. Among them, the K-Min algorithm is the core of local partition optimization, which minimizes system resource consumption by reducing the number of agents and optimizing resource allocation strategies, while ensuring partition load balancing and system performance.

[0063] In the embodiments of the present application, by configuring the optimization model, an adaptive system optimization strategy is provided, which dynamically adjusts the number of agents and the number of partitions of the distributed system according to different performance indicators and resource usage. The overall process is as shown in Figure 3 The figure shows that starting from monitoring the real-time load and resource availability of the system, the applicable optimization algorithm (such as K-Max or K-Min) is selected according to the current operating conditions. Among them, the K-Max algorithm is used to increase the number of agents to improve the system throughput and reduce the system delay when the system load is high, while the K-Min algorithm is used to minimize resource consumption and maintain partition load balancing by reducing the number of agents and optimizing the partition configuration in the case of resource constraints. Subsequently, the system performance is evaluated, and if the performance is optimal, the current configuration is maintained; if the performance is not good, the adjustment is re-performed. The whole process is a closed-loop adaptive optimization mechanism, which continuously evaluates the system performance, dynamically adjusts the configuration according to the evaluation results, until the performance is optimal, ensuring that the system can maintain efficient and stable data processing capability under various conditions.

[0064] Optionally, the target number of agents of the distributed system is determined by the global agent optimization model, comprising: determining the target number of agents by a fourth objective function in the global agent optimization model, wherein the fourth objective function is determined by: determining the initial number of partitions of the distributed system; determining the delay constraint and the availability constraint under the load condition, wherein the delay constraint is used to maintain the relationship between the number of agents, the initial number of partitions, the replication factor and the replication delay of the distributed system to satisfy the preset delay of the system, the availability constraint is used to maintain the agent unavailable time of the distributed system to satisfy the preset unavailable time of the system, wherein the replication factor is used to represent the number of copies of the partition in the distributed system, the replication delay is used to represent the replication time of the agent in the distributed system, the preset delay is used to represent the maximum delay time that the system can accept, the agent unavailable time is used to represent the service suspension time of the agent in the distributed system, and the preset unavailable time is used to represent the maximum agent unavailable time that the system can accept; determining the fourth objective function according to the initial number of partitions, the delay constraint and the availability constraint.

[0065] In the embodiments of the present application, the global agent optimization model mainly relies on the K-Max algorithm, which dynamically adjusts the number of agents to cope with high-concurrency loads. Specifically, by monitoring the load of the system in real time, the number of agents is intelligently increased by using the calculation results of K-Max combined with the fourth objective function, to ensure that the system delay remains within an acceptable range, while improving the fault tolerance of the system, so that the system can still operate stably even in the event of agent failure. The specific expression is as follows:

[0066] maxb subject to P·r·lr ≤ b · L max and P · u ≤ b · U

[0067] where b denotes the number of brokers, P denotes the number of partitions (here, the initial number of partitions in Kafka), r denotes the replication factor, l r denotes the replication delay, L max denotes the preset delay, u denotes the unavailability time of a broker, and U denotes the preset unavailability time of a broker.

[0068] As can be seen from the above expressions, the fourth objective function is determined based on the initial number of partitions, the delay constraint, and the availability constraint, and the purpose is to maximize the number of brokers to cope with high-concurrency data traffic.

[0069] Specifically, the delay constraint focuses on the adjustment of the number of brokers not causing the data processing delay to exceed the maximum value allowed by the system. In Kafka, the relationship between the number of brokers b, the number of partitions P (here, the initial number of partitions in Kafka), the replication factor r, and the replication delay l r is involved to ensure that the transmission time of data from the producer to the consumer satisfies the preset delay, i.e., the relationship between the number of partitions P and the number of brokers b and the replication factor r in the system must satisfy the delay limit L max . Among them, the replication factor determines the number of partition replicas, which affects the fault tolerance of the system; the replication delay is the time required for data replication between brokers, and too long replication delay will increase the system delay.

[0070] The availability constraint focuses on the unavailability time u of a broker to ensure that the adjustment of the number of brokers b can maintain the high availability of the system, i.e., the unavailability time u of a broker does not exceed the preset unavailability time U of the system. It relates to the fault tolerance and continuity of data processing of the system, and by increasing the number of brokers, other brokers can take over the tasks of a single broker when it fails, ensuring the continuous operation of the system.

[0071] Further, by introducing Lagrange multipliers (such as λ1 and λ2), the above global broker optimization problem can be solved to find the optimal broker configuration scheme. In the solving process, the number of brokers b is dynamically adjusted, and the specific formula is as follows:

[0072] L(b, P, λ1, λ2) = -b + λ1(P · r · l r -b · L max ) + λ2(P · u - b · U)

[0073] By taking the partial derivative of b and P and setting it to 0, we can get:

[0074]

[0075] Solve the optimal value of b, get the target agent number of the system under different loads.

[0076] Optionally, the target partition number of the distributed system is determined through the local partition optimization model, comprising: determining the target partition number through a fifth objective function in the local partition optimization model, wherein the fifth objective function is determined by: determining the throughput constraint of the distributed system, wherein the throughput constraint is used to keep the partition number of the distributed system meet the throughput demand of the system; determining the fifth objective function according to the target agent number, the delay constraint and the throughput constraint.

[0077] In the embodiments of the present application, the local partition optimization model mainly relies on the K-Min algorithm, which dynamically adjusts the agent number and the partition number to cope with data stream processing in resource-constrained scenarios. Specifically, by monitoring the resource availability of the system in real time, the agent number and the partition number are intelligently adjusted by using the calculation results of the K-Min combined with the fifth objective function, which can reduce the data transmission delay between partitions and ensure the load balance between partitions. The specific expression is as follows:

[0078]

[0079] Wherein, T prod and T cons are the throughput demands of the producer and the consumer, T p and T c are the throughput of a single producer and consumer.

[0080] From the above expression, the fifth function is determined according to the agent number (here, the target agent number calculated above), the delay constraint and the throughput demand, its purpose is to further reduce the agent number and optimize the partition number on the basis of the above target agent number, delay constraint and throughput constraint, so as to determine the optimal partition number that can meet the system throughput demand, that is, the above target partition number. The delay constraint is consistent with the delay constraint in the above K-Max algorithm, which will not be described here. The throughput constraint focuses on the throughput demand of the producer and the consumer in Kafka. In the cloud network operation scene, the throughput demand may vary due to different business scenarios, such as automated deployment, intelligent monitoring and fault self-processing.

[0081] Optionally, the method further comprises: determining a target running state of the distributed system cluster within a preset time period, wherein the target running state is a running state of the distributed system after the target number of agents and the target number of partitions are configured; in a case where the target running state does not meet the preset performance index, adjusting the target number of agents and the target number of partitions by configuring an optimization model; and in a case where the target running state meets the preset performance index, maintaining the configuration of the distributed system.

[0082] The preset performance index at least includes: a global delay of the distributed system, a delay of each partition in the distributed system, and a maximum load of each agent in the distributed system.

[0083] In the embodiments of the present application, not only the dynamic adjustment of the number of agents and the number of partitions based on the real-time load condition is included, but also a continuous optimization and monitoring cycle is involved to ensure that the system configuration can continuously meet the preset performance index. The specific process can be as follows:

[0084] Firstly, the system monitors its running state within a preset time period (such as every 10 minutes, every hour, etc.), and evaluates the performance under the current configuration (the number of agents and the number of partitions). The target running state refers to the running state expected to be reached when the system configuration is a specific target number of agents and a target number of partitions within the time period, including but not limited to throughput, delay, resource utilization, and other key performance indicators.

[0085] The performance index is a quantitative performance of the system running state, including system throughput, delay, load of agents and partitions, etc. The index can be set according to specific business requirements and system design targets, for example, the system may require that the data delay does not exceed 500 milliseconds in 99% of cases, and the throughput can handle 100,000 messages per second. After the target running state is determined, the system compares this state with the preset performance index to evaluate whether the current configuration can meet the business requirements. The specific expression can be as follows:

[0086]

[0087] L global represents the global delay of Kafka, L i represents the delay of each partition in Kafka, H max represents the maximum load capacity of each agent in Kafka.

[0088] When the target running state does not meet the preset performance indicators, it indicates that the current number of agents or the number of partitions may not be ideal, or the system load has changed, resulting in that the original configuration is no longer applicable. At this time, the configuration optimization model is activated again to adjust the target number of agents and the target number of partitions. This adjustment process still uses the K-Max algorithm (for high-load scenarios) or the K-Min algorithm (for resource-constrained scenarios) to achieve a new performance optimization state.

[0089] If the target running state meets the preset performance indicators within the preset time period, that is, the system can efficiently and stably run under the current target number of agents and target number of partitions, the system will maintain the current configuration to reduce unnecessary resource waste and potential risks caused by system adjustment.

[0090] This closed-loop optimization mechanism ensures that the Kafka cluster can continuously adapt to changing operation and maintenance environments and business demands by periodically evaluating system performance and dynamically adjusting the configuration. This not only improves the system's response speed and data processing efficiency, but also optimizes resource utilization and reduces operation and maintenance costs.

[0091] In the embodiments of the present application, the multi-objective optimization theory is combined with the Kafka distributed data stream processing platform to achieve real-time data stream processing optimization in high-concurrency and resource-constrained scenarios by dynamically adjusting the number of agents and the number of partitions. The K-Max and K-Min algorithms are used to adaptively configure the Kafka cluster, which not only improves the system's throughput and reduces the delay, but also maintains high availability and optimal resource utilization. This mechanism can intelligently respond to changing load conditions, avoiding the complexity and uncertainty of manual intervention, achieving automatic optimization of system performance, especially in key businesses such as automated deployment, intelligent monitoring, and fault self-processing. It can quickly respond to large-scale data streams, reduce processing delays, enhance system fault tolerance, avoid resource waste, greatly improve operation and maintenance efficiency, and reduce costs, effectively promoting the digital transformation of cloud network operation and maintenance technology and meeting the higher demands of future business development.

[0092] According to the embodiments of the present application, a configuration optimization device for a distributed system is provided. It should be noted that the configuration optimization device for a distributed system in the embodiments of the present application can be used to execute the configuration optimization method for a distributed system provided by the embodiments of the present application. The configuration optimization device for a distributed system provided by the embodiments of the present application is introduced as follows.

[0093] Figure 4 is a structural diagram of a configuration optimization device for a distributed system according to the embodiments of the present application. As shown in Figure 4 The device includes:

[0094] The determining module 40 is configured to determine a running state of the distributed system, wherein the running state comprises a load condition and a resource usage condition of each instance in the distributed system.

[0095] The adjusting module 42 is configured to, when the running state does not satisfy the preset performance index, adjust the number of agents and the number of partitions of the distributed system by configuring an optimization model to obtain a target number of agents and a target number of partitions, wherein the optimization model comprises a global agent optimization model and a local partition optimization model, the global agent optimization model is configured to optimize the system performance by adjusting the number of agents, and the local partition optimization model is configured to optimize the system performance by adjusting the number of partitions.

[0096] The configuring module 44 is configured to configure the distributed system according to the target number of agents and the target number of partitions.

[0097] By the determining module 40, the adjusting module 42 and the configuring module 44 in the configuration optimization device of the distributed system, the number of agents and the number of partitions are adaptively adjusted to adapt to different load and resource conditions, so as to achieve the technical effects of reducing data delay, improving throughput in a high-concurrency scenario, and optimizing resource utilization and reducing system overhead in a resource-limited scenario, thereby solving the technical problems of excessive data delay, processing bottleneck difficulty and resource waste in the Kafka configuration in the related art in high-concurrency real-time data stream processing.

[0098] In the configuration optimization device of the distributed system provided in the embodiments of the present application, the adjusting module is further configured to, when the load condition exceeds a first preset threshold, determine the target number of agents of the distributed system by the global agent optimization model, wherein the target number of agents is used to represent the number of agents required by the distributed system under the load condition; and when the resource usage condition exceeds a second preset threshold, determine the target number of partitions of the distributed system by the local partition optimization model, wherein the target number of partitions is used to represent the number of partitions required by the distributed system under the resource usage condition.

[0099] In the distributed system configuration optimization apparatus provided in the embodiments of the present application, the adjusting module is further configured to determine the target number of agents by a fourth objective function in the global agent optimization model, wherein the fourth objective function is determined by the following manner: determining an initial number of partitions of the distributed system; determining a delay constraint and an availability constraint under a load condition, wherein the delay constraint is used to keep a relationship between the number of agents, the initial number of partitions, a replication factor and a replication delay of the distributed system satisfying a preset delay of the system, the availability constraint is used to keep an agent unavailability time of the distributed system satisfying a preset unavailability time of the system, wherein the replication factor is used to represent a number of replicas of a partition in the distributed system, the replication delay is used to represent a replication time of an agent in the distributed system, the preset delay is used to represent a maximum delay time that can be accepted by the system, the agent unavailability time is used to represent a service suspension time of an agent in the distributed system, and the preset unavailability time is used to represent a maximum agent unavailability time that can be accepted by the system; and determining the fourth objective function according to the initial number of partitions, the delay constraint and the availability constraint.

[0100] In the distributed system configuration optimization apparatus provided in the embodiments of the present application, the adjusting module is further configured to determine a throughput constraint of the distributed system, wherein the throughput constraint is used to keep a number of partitions of the distributed system satisfying a throughput requirement of the system; and determine a fifth objective function according to the target number of agents, the delay constraint and the throughput constraint.

[0101] In the distributed system configuration optimization apparatus provided in the embodiments of the present application, the configuring module is further configured to determine a target running state of the distributed system cluster in a preset time period, wherein the target running state is a running state of the distributed system after the target number of agents and the target number of partitions are configured; in a case where the target running state does not satisfy a preset performance index, adjusting the target number of agents and the target number of partitions by the configuration optimization model; and in a case where the target running state satisfies the preset performance index, keeping the configuration of the distributed system.

[0102] The embodiments of the present application further provide an electronic device, including a memory and a processor, wherein the memory is configured to store program instructions; and the processor is connected with the memory and is configured to execute the distributed system configuration optimization method.

[0103] It should be noted that the electronic device is configured to execute the distributed system configuration optimization method shown in the above description, and thus the related explanations in the distributed system configuration optimization method are also applicable to the electronic device, which will not be repeated here. Figure 2

[0104] ​The embodiment of the present application further provides a nonvolatile storage medium, which comprises a stored computer program, wherein a device in which the nonvolatile storage medium is located executes the configuration optimization method of the distributed system by running the computer program.

[0105] It should be noted that the nonvolatile storage medium is used for executing the configuration optimization method of the distributed system as shown in the method for optimizing configuration of distributed system, and therefore the related explanations in the configuration optimization method of the distributed system are also applicable to the nonvolatile storage medium, which will not be repeated here. Figure 2

[0106] The embodiment of the present application further provides a computer program product, which comprises computer instructions, and the computer instructions are executed by a processor to implement the configuration optimization method of the distributed system.

[0107] It should be noted that the computer program product is used for executing the configuration optimization method of the distributed system as shown in the method for optimizing configuration of distributed system, and therefore the related explanations in the configuration optimization method of the distributed system are also applicable to the computer program product, which will not be repeated here. Figure 2

[0108] The serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0109] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0110] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the unit described as the division is only a logical function division, and there can be another division manner in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0111] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0112] ​​In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0113] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes various media that can store program codes, such as a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, etc.

[0114] The above is only the preferred embodiment of the present application, and it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A method for configuration optimization of a distributed system, characterized in that, The method comprises the following steps: determining the running state of the distributed system, wherein the running state comprises the load condition and the resource usage of each instance in the distributed system; in the case that the running state does not meet the preset performance index, adjusting the number of agents and the number of partitions of the distributed system through a configuration optimization model to obtain a target number of agents and a target number of partitions, wherein the configuration optimization model comprises a global agent optimization model and a local partition optimization model, the global agent optimization model is used to optimize the system performance by adjusting the number of agents, and the local partition optimization model is used to optimize the system performance by adjusting the number of partitions; the configuration optimization model at least comprises a first objective function, a second objective function and a third objective function, wherein the first objective function is used to measure the global delay of the distributed system, the second objective function is used to measure the resource cost of the distributed system under the target number of agents, and the third objective function is used to measure the resource utilization and performance index of the distributed system under the target number of partitions; configuring the distributed system according to the target number of agents and the target number of partitions; in the case that the running state does not meet the preset performance index, adjusting the number of agents and the number of partitions of the distributed system through a configuration optimization model to obtain a target number of agents and a target number of partitions, comprising: in the case that the load condition exceeds a first preset threshold, determining the target number of agents of the distributed system through the global agent optimization model, wherein the target number of agents is used to represent the number of agents required by the distributed system under the load condition; in the case that the resource usage exceeds a second preset threshold, determining the target number of partitions of the distributed system through the local partition optimization model, wherein the target number of partitions is used to represent the number of partitions required by the distributed system under the resource usage.

2. The method of claim 1, wherein, determining the target number of agents of the distributed system through the global agent optimization model, comprising: determining the target number of agents through a fourth objective function in the global agent optimization model, wherein the fourth objective function is determined by the following way: determining the initial number of partitions of the distributed system; determining the delay constraint and the availability constraint under the load condition, wherein the delay constraint is used to keep the relationship between the number of agents, the initial number of partitions, the replication factor and the replication delay of the distributed system meet the preset delay, and the availability constraint is used to keep the agent unavailability time of the distributed system meet the preset unavailability time, wherein the replication factor is used to represent the number of copies of the partition in the distributed system, the replication delay is used to represent the replication time of the agent in the distributed system, the preset delay is used to represent the maximum delay time that the distributed system can accept, the agent unavailability time is used to represent the service suspension time of the agent in the distributed system, and the preset unavailability time is used to represent the maximum agent unavailability time that the distributed system can accept. determine the fourth objective function according to the initial partition quantity, the delay constraint and the availability constraint.

3. The method of claim 2, wherein, determine the target partition quantity of the distributed system through a local partition optimization model, including: determining the target partition quantity through a fifth objective function in the local partition optimization model, wherein the fifth objective function is determined by the following way: determine a throughput constraint of the distributed system, wherein the throughput constraint is used to keep the partition quantity of the distributed system to meet the throughput demand of the distributed system; determine the fifth objective function according to the target agent quantity, the delay constraint and the throughput constraint.

4. The method of claim 1, wherein, The method further includes: determine a target running state of the distributed system cluster within a preset time period, wherein the target running state is a running state after the distributed system is configured with the target agent quantity and the target partition quantity; in the case that the target running state does not meet the preset performance index, adjust the target agent quantity and the target partition quantity through the configuration optimization model; in the case that the target running state meets the preset performance index, keep the configuration of the distributed system.

5. The method of claim 4, wherein, The preset performance index at least includes: global delay of the distributed system, delay of each partition in the distributed system and maximum load of each agent in the distributed system.

6. An apparatus for configuration optimization of a distributed system, characterized by include: a determination module configured to determine a running state of a distributed system, wherein the running state includes load conditions and resource usage conditions of each instance in the distributed system; The adjusting module is configured to adjust the number of agents and the number of partitions of the distributed system by a configuration optimization model to obtain a target number of agents and a target number of partitions when the running state does not meet the preset performance index, wherein the configuration optimization model comprises a global agent optimization model and a local partition optimization model, the global agent optimization model is configured to optimize system performance by adjusting the number of agents, and the local partition optimization model is configured to optimize system performance by adjusting the number of partitions; the configuration optimization model at least comprises a first target function, a second target function and a third target function, wherein the first target function is configured to measure the global delay of the distributed system, the second target function is configured to measure the resource cost of the distributed system under the target number of agents, and the third target function is configured to measure the resource utilization and performance index of the distributed system under the target number of partitions; the adjusting of the number of agents and the number of partitions of the distributed system by the configuration optimization model to obtain the target number of agents and the target number of partitions when the running state does not meet the preset performance index comprises: determining the target number of agents of the distributed system by the global agent optimization model when the load condition exceeds a first preset threshold, wherein the target number of agents is configured to represent the number of agents required by the distributed system under the load condition; and determining the target number of partitions of the distributed system by the local partition optimization model when the resource usage condition exceeds a second preset threshold, wherein the target number of partitions is configured to represent the number of partitions required by the distributed system under the resource usage condition. The configuration module is configured to configure the distributed system according to the target number of agents and the target number of partitions.

7. An electronic device, comprising: Comprise: a memory and a processor, wherein the memory is configured to store program instructions; the processor is connected with the memory and is configured to execute the configuration optimization method of the distributed system according to any one of claims 1 to 5.

8. A non-volatile storage medium, characterized by The non-volatile storage medium comprises a stored computer program, wherein the device where the non-volatile storage medium is located executes the configuration optimization method of the distributed system according to any one of claims 1 to 5 by running the computer program.

9. A computer program product comprising computer instructions, characterized in that, The computer instructions are executed by the processor to implement the configuration optimization method of the distributed system according to any one of claims 1 to 5.

Images