Configuration optimization method and device of distributed system and electronic equipment

Abstract

The invention discloses a configuration optimization method and device of a distributed system and electronic equipment. The method comprises the following steps: determining an operation state of the distributed system; under the condition that the running state does not meet the preset performance index, the proxy number and the partition number of the distributed system are adjusted through a configuration optimization model, the target proxy number and the target partition number are obtained, the configuration optimization model comprises a global proxy optimization model and a local partition optimization model, and the local partition optimization model comprises a global proxy optimization model and a local partition optimization model; the global agent optimization model is used for optimizing the system performance by adjusting the number of agents, and the local partition optimization model is used for optimizing the system performance by adjusting the number of partitions; and configuring the distributed system according to the number of the target agents and the number of the target partitions. According to the method and the device, the technical problems of overlarge data delay, difficulty in processing bottleneck and resource waste due to the fact that Kafka is configured in high-concurrency real-time data stream processing in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the technical field of distributed systems, and in particular to a configuration optimization method, device and electronic equipment for a distributed system. Background Art

[0002] With the rapid development of information technology and the continuous emergence of large-scale data, distributed data processing platforms play a vital role in enterprise operation and maintenance, cloud computing and big data analysis. Among them, Kafka, as a widely used distributed data stream processing platform, can support high-throughput real-time data exchange. Its application in the producer-consumer model can provide strong technical support for the digital transformation of operation and maintenance technology. For example, during the automated deployment process, the system needs to process a large amount of real-time logs, alarm information and equipment status data. Using Kafka to transmit this data in real time can ensure that the operation and maintenance personnel can obtain the system status in the first time and improve the response speed to sudden failures and abnormal events. In addition, in intelligent monitoring and operation and maintenance automation scenarios, Kafka can serve as the core hub of the data stream to help quickly distribute and process monitoring data to support fault detection, alarm and self-processing functions.

[0003] However, Kafka's performance is highly dependent on its internal configuration, especially the settings of partitions and brokers. The number of partitions determines the parallel processing capability of data, while the number of brokers affects data storage and system fault tolerance. Therefore, although Kafka itself provides a certain degree of fault tolerance and high concurrency support, its default configuration often cannot meet the needs of specific scenarios, especially in a complex network environment with multi-node distribution. Data processing throughput, latency and system availability have become key issues that need to be solved urgently. If Kafka is not configured properly, it may lead to decreased data processing efficiency, increased data latency, and even processing bottlenecks, affecting the real-time response capability and stability of the entire operation and maintenance system.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a configuration optimization method, device and electronic device for a distributed system, so as to at least solve the technical problems of excessive data delay, processing bottleneck difficulties and resource waste caused by the Kafka configuration in the related technology in high-concurrency real-time data stream processing.

[0006] According to one aspect of an embodiment of the present application, a configuration optimization method for a distributed system is provided, comprising: determining an operating state of a distributed system, wherein the operating state includes the load condition and resource usage condition of each instance in the distributed system; when the operating state does not meet preset performance indicators, adjusting the number of proxies and the number of partitions of the distributed system through a configuration optimization model to obtain a target number of proxies and a target number of partitions, wherein the configuration optimization model includes a global proxy optimization model and a local partition optimization model, the global proxy optimization model is used to optimize system performance by adjusting the number of proxies, and the local partition optimization model is used to optimize system performance by adjusting the number of partitions; configuring the distributed system according to the target number of proxies and the target number of partitions.

[0007] Optionally, the configuration optimization model includes at least: a first objective function, a second objective function and a third objective function, wherein the first objective function is used to measure the global delay of the distributed system, the second objective function is used to measure the resource cost of the distributed system under the target number of agents, and the third objective function is used to measure the resource utilization and performance indicators of the distributed system under the target number of partitions.

[0008] Optionally, when the operating status does not meet the preset performance indicators, the number of proxies and the number of partitions of the distributed system are adjusted by configuring the optimization model to obtain the target number of proxies and the target number of partitions, including: when the load exceeds a first preset threshold, the target number of proxies of the distributed system is determined by the global proxy optimization model, wherein the target number of proxies is used to represent the number of proxies required by the distributed system under load conditions; when the resource usage exceeds a second preset threshold, the target number of partitions of the distributed system is determined by the local partition optimization model, wherein the target number of partitions is used to represent the number of partitions required by the distributed system under resource usage conditions.

[0009] Optionally, determining the target number of proxies in a distributed system through a global proxy optimization model includes: determining the target number of proxies through a fourth objective function in the global proxy optimization model, wherein the fourth objective function is determined in the following manner: determining the initial number of partitions of the distributed system; determining a delay constraint and an availability constraint under load conditions, wherein the delay constraint is used to keep the relationship between the number of proxies, the initial number of partitions, the replication factor, and the replication delay of the distributed system satisfying a preset delay of the system, and the availability constraint is used to keep the proxy unavailability time of the distributed system satisfying a preset unavailability time of the system, wherein the replication factor is used to represent the number of copies of a partition in the distributed system, the replication delay is used to represent the replication time of a proxy in the distributed system, the preset delay is used to represent the maximum delay time that the system can accept, the proxy unavailability time is used to represent the service suspension time of a proxy in the distributed system, and the preset unavailability time is used to represent the maximum proxy unavailability time that the system can accept; determining the fourth objective function based on the initial number of partitions, the delay constraint, and the availability constraint.

[0010] Optionally, determining the target number of partitions of the distributed system through a local partition optimization model includes: determining the target number of partitions through a fifth objective function in the local partition optimization model, wherein the fifth objective function is determined in the following manner: determining a throughput constraint of the distributed system, wherein the throughput constraint is used to maintain the number of partitions of the distributed system to meet the throughput requirements of the system; determining the fifth objective function based on the target number of proxies, delay constraints and throughput constraints.

[0011] Optionally, the method also includes: determining a target operating state of the distributed system cluster within a preset time period, wherein the target operating state is the operating state after the distributed system is configured with a target number of proxies and a target number of partitions; if the target operating state does not meet preset performance indicators, adjusting the target number of proxies and the target number of partitions by configuring an optimization model; and if the target operating state meets preset performance indicators, maintaining the configuration of the distributed system.

[0012] Optionally, the preset performance indicators include at least: a global delay of the distributed system, a delay of each partition in the distributed system, and a maximum load of each proxy in the distributed system.

[0013] According to another aspect of an embodiment of the present application, a configuration optimization device for a distributed system is also provided, including: a determination module, used to determine the operating status of the distributed system, wherein the operating status includes the load status and resource usage of each instance in the distributed system; an adjustment module, used to adjust the number of proxies and the number of partitions of the distributed system through a configuration optimization model when the operating status does not meet the preset performance indicators, so as to obtain a target number of proxies and a target number of partitions, wherein the configuration optimization model includes a global proxy optimization model and a local partition optimization model, the global proxy optimization model is used to optimize system performance by adjusting the number of proxies, and the local partition optimization model is used to optimize system performance by adjusting the number of partitions; a configuration module is used to configure the distributed system according to the target number of proxies and the target number of partitions.

[0014] According to another aspect of the embodiment of the present application, there is also provided an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the configuration optimization method for implementing the above-mentioned distributed system.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned configuration optimization method of the distributed system by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned configuration optimization method of the distributed system when executed by a processor.

[0017] In an embodiment of the present application, by determining the running state of a distributed system, wherein the running state includes the load and resource usage of each instance in the distributed system; when the running state does not meet the preset performance indicators, the number of proxies and the number of partitions of the distributed system are adjusted by configuring an optimization model to obtain a target number of proxies and a target number of partitions, wherein the configuration optimization model includes a global proxy optimization model and a local partition optimization model, the global proxy optimization model is used to optimize system performance by adjusting the number of proxies, and the local partition optimization model is used to optimize system performance by adjusting the number of partitions; the distributed system is configured according to the target number of proxies and the target number of partitions, so as to achieve the purpose of adaptively adjusting the number of proxies and partitions to adapt to different load and resource conditions, thereby achieving the technical effect of reducing data delay and improving throughput in high concurrency scenarios, and optimizing resource utilization and reducing system overhead in resource-constrained scenarios, thereby solving the technical problems of excessive data delay, processing bottleneck difficulties and resource waste in high-concurrency real-time data stream processing due to the configuration of Kafka in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a hardware structure diagram of a computer terminal for implementing a configuration optimization method for a distributed system according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of a configuration optimization method for a distributed system according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a configuration optimization process of a distributed system according to an embodiment of the present application;

[0022] Figure 4 It is a structural diagram of a configuration optimization device for a distributed system according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] First, some nouns or terms that appear in the process of explaining the embodiments of the present application are subject to the following explanations:

[0026] Kafka: A high-performance, distributed, publish-subscribe-based message queue service designed to process large amounts of real-time data streams. Kafka can provide high throughput, low latency, and persistent data storage, and is suitable for a variety of scenarios such as log aggregation, stream processing, and event processing.

[0027] Broker: The broker in Kafka is the basic node for data storage and processing. It is responsible for receiving and storing data sent by producers, and also processes data reading requests from consumers. A Kafka cluster consists of multiple brokers, each of which can store data partitions for multiple topics.

[0028] Partition: A partition is a subset of a Kafka topic that is used to implement parallel storage and processing of data. Each topic can have multiple partitions, and data will be evenly distributed to these partitions when written to improve processing power and fault tolerance.

[0029] Producer: In Kafka, producers are clients responsible for sending data to topics. They can be applications, services, or other data sources that participate in Kafka's data stream processing by writing data to the agent.

[0030] Consumer: Consumers are clients that read data from Kafka topics. They subscribe to topics and read data from brokers. Consumers can be applications or services that process data, analyze it, or forward it to other systems.

[0031] Throughput: Throughput refers to the amount of data that the system can process per unit time. It is an important indicator to measure Kafka's data processing capabilities. High throughput means that Kafka can quickly process large amounts of data streams, which is suitable for real-time data processing scenarios.

[0032] Latency: Latency refers to the time interval between data being sent from the producer to being received by the consumer. In real-time data stream processing, low latency is crucial because it directly affects the responsiveness and real-time performance of the system.

[0033] In order to solve the problem that Kafka configuration in the related art cannot adapt to high-concurrency real-time data stream processing, the embodiment of the present application provides a configuration optimization method for a distributed system, which can be run on Figure 1 In the computer terminal shown, the computer terminal is described below.

[0034] The configuration optimization method embodiment of the distributed system provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a configuration optimization method for a distributed system. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0035] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the configuration optimization method of the distributed system in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the configuration optimization method of the distributed system mentioned above. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0038] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0039] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0040] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a configuration optimization method for a distributed system. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] Figure 2 is a flow chart of a configuration optimization method for a distributed system according to an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0042] Step S202: determining the running status of the distributed system, wherein the running status includes the load status and resource usage status of each instance in the distributed system.

[0043] In the above step S202, the main task is to monitor the current running status of the distributed system (taking Kafka as an example), including but not limited to the load of each instance in Kafka, resource utilization and performance indicators of the entire system. Among them, Kafka instance refers to a single server node running the Kafka service, and each instance runs one or more Kafka brokers (Brokers), which are responsible for storing data, processing data streams, and ensuring data transmission between producers and consumers.

[0044] This step provides the necessary data basis for subsequent optimization decisions. For example, by monitoring the proxy's CPU usage, memory usage, network I / O and other indicators, the real-time load of the system can be evaluated. At the same time, by monitoring performance parameters such as data throughput and latency, it can be understood whether the system has reached the preset performance indicators.

[0045] Step S204, when the operating status does not meet the preset performance indicators, the number of proxies and the number of partitions of the distributed system are adjusted by configuring the optimization model to obtain the target number of proxies and the target number of partitions, wherein the configuration optimization model includes a global proxy optimization model and a local partition optimization model, the global proxy optimization model is used to optimize the system performance by adjusting the number of proxies, and the local partition optimization model is used to optimize the system performance by adjusting the number of partitions.

[0046] In the above step S204, when the monitored operating state does not meet the preset performance indicators, that is, the system has a performance bottleneck or resource waste, the configuration optimization model (a multi-objective optimization model) will be started to adjust the number of brokers and partitions in the current operating state of Kafka to achieve the target number of brokers and the target number of partitions, thereby optimizing system performance. The preset performance indicators include but are not limited to minimum throughput, maximum latency, resource utilization threshold, etc., and the preset performance indicators can be set according to different business needs and scenarios.

[0047] Among them, the configuration optimization model is divided into two sub-models: the global agent optimization model and the local partition optimization model.

[0048] Specifically:

[0049] The global proxy optimization model mainly focuses on adjusting the number of proxies, aiming to optimize the performance of the entire system by increasing or decreasing proxy nodes. Specifically, when the system load is high or the latency increases, the global proxy optimization model can increase the number of proxies through the K-Max algorithm to disperse the load, improve data throughput and fault tolerance, and ensure that the system can remain efficient and stable during high concurrency and large-scale data stream transmission.

[0050] The local partition optimization model focuses on optimizing the number of partitions, reducing system latency, improving resource utilization, and ensuring load balancing between partitions by adjusting the number of partitions. Specifically, in the case of limited resources or uneven load, the local partition optimization model can reduce the number of proxies and optimize resource allocation through the K-Min algorithm, while ensuring load balancing for each partition and reducing latency, thereby improving the operating efficiency of the entire system.

[0051] Step S206, configuring the distributed system according to the target number of agents and the target number of partitions.

[0052] In the above step S206, after the configuration optimization model obtains the target number of brokers and the target number of partitions, actual configuration adjustments can be made to Kafka, such as adding or deleting broker nodes and re-dividing the partitions of the topic.

[0053] Through the above steps S202 to S206, the purpose of adaptively adjusting the number of proxies and partitions to adapt to different loads and resource conditions is achieved, thereby achieving the technical effects of reducing data delay and improving throughput in high-concurrency scenarios, and optimizing resource utilization and reducing system overhead in resource-constrained scenarios, thereby solving the technical problems of excessive data delay, processing bottleneck difficulties and resource waste in high-concurrency real-time data stream processing due to the configuration of Kafka in the related technology. The following is a detailed description.

[0054] In the above step S204, the configuration optimization model includes at least: a first objective function, a second objective function and a third objective function, wherein the first objective function is used to measure the global delay of the distributed system, the second objective function is used to measure the resource cost of the distributed system under the target number of agents, and the third objective function is used to measure the resource utilization and performance indicators of the distributed system under the target number of partitions.

[0055] In the embodiment of the present application, the configuration optimization model aims to optimize the performance of real-time data stream processing by dynamically adjusting the number of brokers and partitions of Kafka. The configuration optimization model includes a first objective function, a second objective function and a third objective function, and the specific expression is as follows:

[0056] min(L global (b,P),C(b),R(P))

[0057] Among them, L global (b,P) represents the first objective function, which depends on the number of proxies b and the number of partitions P, and is used to measure the global latency of Kafka. C(b) represents the second objective function, which is used to measure the resource cost of Kafka at the target number of proxies. R(P) represents the third objective function, which is used to measure the resource utilization and performance indicators of Kafka at the target number of partitions.

[0058] Combining the above three objective functions, the configuration optimization model can achieve global latency minimization, resource cost minimization, and resource utilization maximization, thereby ensuring that Kafka can provide stable, efficient, and cost-effective real-time data stream processing services in different operation and maintenance scenarios. This multi-objective optimization idea enables the optimization strategy to flexibly adapt to different business needs and resource constraints, providing solid technical support for the digital transformation of operation and maintenance.

[0059] Optionally, when the operating status does not meet the preset performance indicators, the number of proxies and the number of partitions of the distributed system are adjusted by configuring the optimization model to obtain the target number of proxies and the target number of partitions, including: when the load exceeds a first preset threshold, the target number of proxies of the distributed system is determined by the global proxy optimization model, wherein the target number of proxies is used to represent the number of proxies required by the distributed system under load conditions; when the resource usage exceeds a second preset threshold, the target number of partitions of the distributed system is determined by the local partition optimization model, wherein the target number of partitions is used to represent the number of partitions required by the distributed system under resource usage conditions.

[0060] In the embodiment of the present application, when the operating status of Kafka does not meet the preset performance indicators, that is, the system performance is degraded or the resource utilization is inappropriate, the number of proxies and partitions can be dynamically adjusted through the configuration optimization model to ensure that the system can be restored to the optimal operating state. Specifically:

[0061] When monitoring shows that Kafka's load (such as data throughput, processing delay, etc.) exceeds the first preset threshold, it indicates that the system's processing capacity may have reached or is close to the limit, and the data processing speed and system response time may not meet business needs. At this time, the global proxy optimization model can be used to increase the number of proxies to disperse the load and improve the system's parallel processing capabilities and fault tolerance. Among them, the K-Max algorithm is the core of global proxy optimization, which takes into account multiple performance indicators, including system latency, proxy load, and resource cost, to ensure that while increasing the number of proxies, the system's latency and cost control are also taken into account to achieve the best balance between performance and resource utilization.

[0062] When Kafka's resource usage (such as storage space, CPU and memory usage, etc.) exceeds the second preset threshold, it indicates that the system resources may be close to saturation. Continuing to increase the amount of data processing or maintaining the existing configuration may lead to resource bottlenecks, affecting system stability and processing efficiency. At this time, the number of partitions can be adjusted through the local partition optimization model to optimize resource utilization and reduce system latency. Among them, the K-Min algorithm is the core of local partition optimization. It minimizes system resource consumption by reducing the number of agents and optimizing resource allocation strategies, while ensuring partition load balancing and system performance.

[0063] In the embodiment of the present application, an adaptive system optimization strategy is provided by configuring the optimization model to dynamically adjust the number of proxies and partitions in the distributed system according to different performance indicators and resource usage. Figure 3 As shown in the figure, it starts with monitoring the real-time system load and resource availability, and selecting an appropriate optimization algorithm (such as K-Max or K-Min) according to the current operating status. Among them, the K-Max algorithm is used to increase the number of proxies to improve system throughput and reduce system latency when the system load is high, while the K-Min algorithm is used to minimize resource consumption and maintain partition load balance by reducing the number of proxies and optimizing partition configuration when resources are limited. Subsequently, the system performance is evaluated. If the performance is optimal, the current configuration is maintained; if the performance is poor, it is readjusted. The entire process is a closed-loop adaptive optimization mechanism that continuously evaluates system performance and dynamically adjusts the configuration according to the evaluation results until the optimal performance is achieved, ensuring that the system can maintain efficient and stable data processing capabilities under various conditions.

[0064] Optionally, determining the target number of proxies in a distributed system through a global proxy optimization model includes: determining the target number of proxies through a fourth objective function in the global proxy optimization model, wherein the fourth objective function is determined in the following manner: determining the initial number of partitions of the distributed system; determining a delay constraint and an availability constraint under load conditions, wherein the delay constraint is used to keep the relationship between the number of proxies, the initial number of partitions, the replication factor, and the replication delay of the distributed system satisfying a preset delay of the system, and the availability constraint is used to keep the proxy unavailability time of the distributed system satisfying a preset unavailability time of the system, wherein the replication factor is used to represent the number of copies of a partition in the distributed system, the replication delay is used to represent the replication time of a proxy in the distributed system, the preset delay is used to represent the maximum delay time that the system can accept, the proxy unavailability time is used to represent the service suspension time of a proxy in the distributed system, and the preset unavailability time is used to represent the maximum proxy unavailability time that the system can accept; determining the fourth objective function based on the initial number of partitions, the delay constraint, and the availability constraint.

[0065] In the embodiment of the present application, the global proxy optimization model mainly relies on the K-Max algorithm, which dynamically adjusts the number of proxies to cope with high concurrent loads. Specifically, by monitoring the load of the system in real time, the number of proxies is intelligently increased by combining the calculation results of K-Max and the fourth objective function to ensure that the system delay remains within an acceptable range and improve the fault tolerance of the system. Even when the proxy fails, the system can still maintain stable operation. The specific expression is as follows:

[0066] maxb subject to P·r·lr ≤ b·L max and P·u≤b·U

[0067] Where b is the number of brokers, P is the number of partitions (here is the initial number of partitions of Kafka), r is the replication factor, and l is the number of partitions. r Represents replication delay, L max represents the preset delay, u represents the proxy unavailable time, and U represents the preset proxy unavailable time.

[0068] It can be seen from the above expression that the fourth objective function is determined based on the initial number of partitions, delay constraints and availability constraints, and its purpose is to maximize the number of proxies to cope with high concurrent data traffic.

[0069] Specifically, the delay constraint focuses on adjusting the number of proxies so that the data processing delay does not exceed the maximum value allowed by the system. In Kafka, this involves the number of proxies b, the number of partitions P (here is the initial number of partitions of Kafka), the replication factor r, and the replication delay l. r To ensure that the transmission time of data from producer to consumer meets the preset delay, that is, the relationship between the number of partitions P in the system and the number of proxies b and the replication factor r must meet the delay limit L max Among them, the replication factor determines the number of partition copies and affects the fault tolerance of the system; the replication delay is the time required for data replication between agents. Too long replication delay will increase system latency.

[0070] The availability constraint focuses on the unavailable time u of the proxy, ensuring that the adjustment of the number of proxies b can maintain the high availability of the system, that is, the unavailable time u of the proxy does not exceed the preset unavailable time U of the system. It is related to the fault tolerance of the system and the continuity of data processing. By increasing the number of proxies, when a single proxy fails, other proxies can take over its tasks to ensure the continuous operation of the system.

[0071] Furthermore, by introducing Lagrange multipliers (such as λ1 and λ2), the above global proxy optimization problem can be solved to find the optimal proxy number configuration solution. In the solution process, the number of agents b is dynamically adjusted, and the specific formula is as follows:

[0072] L(b,P,λ1,λ2)=-b+λ1(P·r·l r -b·L max )+λ2(P·ub·U)

[0073] By taking partial derivatives of b and P and setting them to 0, we can obtain:

[0074]

[0075] Solve for the optimal value of b and obtain the target number of agents for the system under different loads.

[0076] Optionally, determining the target number of partitions of the distributed system through a local partition optimization model includes: determining the target number of partitions through a fifth objective function in the local partition optimization model, wherein the fifth objective function is determined in the following manner: determining a throughput constraint of the distributed system, wherein the throughput constraint is used to maintain the number of partitions of the distributed system to meet the throughput requirements of the system; determining the fifth objective function based on the target number of proxies, delay constraints and throughput constraints.

[0077] In the embodiment of the present application, the local partition optimization model mainly relies on the K-Min algorithm, which dynamically adjusts the number of proxies and partitions to cope with data flow processing in resource-constrained scenarios. Specifically, by real-time monitoring of the system's resource availability, using K-Min and combining the calculation results of the fifth objective function, intelligently adjusting the number of proxies and partitions can reduce data transmission delays between partitions and ensure load balancing between partitions. The specific expression is as follows:

[0078]

[0079] Among them, T prod and T cons are the throughput requirements of producers and consumers respectively, T p and T c The throughput of a single producer and consumer respectively.

[0080] It can be seen from the above expression that the fifth function is determined based on the number of proxies (here is the target number of proxies calculated above), delay constraints and throughput requirements. Its purpose is to reduce the number of proxies again on the basis of the above target number of proxies, delay constraints and throughput constraints, and optimize the number of partitions at the same time, so as to determine the optimal number of partitions that can meet the system throughput requirements, that is, the above target number of partitions. Among them, the delay constraint is consistent with the delay constraint in the above K-Max algorithm, which will not be repeated here. The throughput constraint focuses on the requirements of producers and consumers in Kafka for system throughput. In cloud network operation and maintenance scenarios, throughput requirements may vary due to different business scenarios, such as automated deployment, intelligent monitoring, and self-handling of faults.

[0081] Optionally, the above method also includes: determining the target operating state of the distributed system cluster within a preset time period, wherein the target operating state is the operating state after the distributed system is configured with a target number of proxies and a target number of partitions; when the target operating state does not meet the preset performance indicators, adjusting the target number of proxies and the target number of partitions by configuring an optimization model; when the target operating state meets the preset performance indicators, maintaining the configuration of the distributed system.

[0082] The preset performance indicators include at least: the global delay of the distributed system, the delay of each partition in the distributed system, and the maximum load of each proxy in the distributed system.

[0083] In the embodiment of the present application, not only the number of proxies and partitions are dynamically adjusted based on real-time load conditions, but also a continuous optimization and monitoring cycle is involved to ensure that the system configuration can continue to meet the preset performance indicators. The specific process can be as follows:

[0084] First, the system monitors its operating status within a preset time period (such as every 10 minutes, every hour, etc.) to evaluate the performance under the current configuration (number of proxies and number of partitions). The target operating status refers to the expected operating status when the system is configured with a specific target number of proxies and target number of partitions within the time period, including but not limited to key performance indicators such as throughput, latency, and resource utilization.

[0085] Among them, performance indicators are quantitative expressions of the system's operating status, including system throughput, latency, proxy and partition load, etc. The indicators can be set according to specific business needs and system design goals. For example, the system may require that in 99% of cases, the data delay does not exceed 500 milliseconds and the throughput can handle 100,000 messages per second. After determining the target operating state, the system will compare this state with the preset performance indicators to evaluate whether the current configuration can meet business needs. The specific expression can be as follows:

[0086]

[0087] Among them, L global Represents the global delay of Kafka, L i represents the latency of each partition in Kafka, H max Indicates the maximum load capacity of each Kafka broker.

[0088] When the target running state does not reach the preset performance index, it means that the current number of agents or partitions may not be ideal, or the system load has changed, causing the original configuration to no longer apply. At this time, the configuration optimization model will be activated again to adjust the target number of agents and the target number of partitions. This adjustment process still uses the K-Max algorithm (for high-load scenarios) or the K-Min algorithm (for resource-constrained scenarios) to achieve a new optimal performance state.

[0089] If within the preset time period, the target operating status meets the preset performance indicators, that is, the system can operate efficiently and stably under the current target number of agents and target number of partitions, then the system will maintain the current configuration unchanged to reduce unnecessary waste of resources and potential risks brought by system adjustments.

[0090] This closed-loop optimization mechanism ensures that the Kafka cluster can continuously adapt to the ever-changing operation and maintenance environment and business needs by regularly evaluating system performance and dynamically adjusting configuration based on the evaluation results. It not only improves the system's response speed and data processing efficiency, but also optimizes resource utilization and reduces operation and maintenance costs.

[0091] In the embodiment of the present application, the multi-objective optimization theory is combined with the Kafka distributed data stream processing platform, and the real-time data stream processing optimization in high concurrency and resource-constrained scenarios is realized by dynamically adjusting the number of agents and the number of partitions. Among them, the Kafka cluster is adaptively configured using the K-Max and K-Min algorithms, which not only improves the throughput of the system and reduces latency, but also maintains high availability and optimal utilization of resources. This mechanism can intelligently respond to changing load conditions, avoid the complexity and uncertainty of manual intervention, and realize automatic optimization of system performance, especially in key businesses such as automated deployment, intelligent monitoring and self-processing of faults. It can quickly respond to large-scale data streams, reduce processing delays, enhance system fault tolerance, avoid waste of resources, greatly improve operation and maintenance efficiency, reduce costs, and effectively promote the digital transformation of cloud network operation and maintenance technology, and can adapt to the higher demands of future business development.

[0092] According to an embodiment of the present application, a configuration optimization device for a distributed system is provided. It should be noted that the configuration optimization device for a distributed system in the embodiment of the present application can be used to execute the configuration optimization method for a distributed system provided in the embodiment of the present application. The configuration optimization device for a distributed system provided in the embodiment of the present application is introduced below.

[0093] Figure 4 1 is a structural diagram of a configuration optimization device for a distributed system provided according to an embodiment of the present application. Figure 4 As shown, the device comprises:

[0094] A determination module 40 is used to determine the operating status of the distributed system, wherein the operating status includes the load status and resource usage status of each instance in the distributed system;

[0095] An adjustment module 42 is used to adjust the number of proxies and the number of partitions of the distributed system by configuring an optimization model to obtain a target number of proxies and a target number of partitions when the running state does not meet the preset performance index, wherein the configuration optimization model includes a global proxy optimization model and a local partition optimization model, the global proxy optimization model is used to optimize the system performance by adjusting the number of proxies, and the local partition optimization model is used to optimize the system performance by adjusting the number of partitions;

[0096] The configuration module 44 is used to configure the distributed system according to the target number of agents and the target number of partitions.

[0097] Through the determination module 40, adjustment module 42 and configuration module 44 in the configuration optimization device of the above-mentioned distributed system, the purpose of adaptively adjusting the number of agents and partitions to adapt to different loads and resource conditions is achieved, thereby achieving the technical effects of reducing data delay and improving throughput in high-concurrency scenarios, and optimizing resource utilization and reducing system overhead in resource-constrained scenarios, thereby solving the technical problems of excessive data delay, processing bottleneck difficulties and resource waste in high-concurrency real-time data stream processing caused by Kafka configuration in related technologies.

[0098] In the configuration optimization device of a distributed system provided in an embodiment of the present application, the adjustment module is also used to determine the target number of proxies of the distributed system through a global proxy optimization model when the load exceeds a first preset threshold, wherein the target number of proxies is used to represent the number of proxies required by the distributed system under the load condition; and to determine the target number of partitions of the distributed system through a local partition optimization model when the resource usage exceeds a second preset threshold, wherein the target number of partitions is used to represent the number of partitions required by the distributed system under the resource usage condition.

[0099] In the configuration optimization device of a distributed system provided in an embodiment of the present application, the adjustment module is also used to determine the target number of proxies through the fourth objective function in the global proxy optimization model, wherein the fourth objective function is determined in the following manner: determining the initial number of partitions of the distributed system; determining the delay constraint and the availability constraint under load conditions, wherein the delay constraint is used to keep the relationship between the number of proxies, the initial number of partitions, the replication factor and the replication delay of the distributed system to meet the preset delay of the system, and the availability constraint is used to keep the proxy unavailability time of the distributed system to meet the preset unavailability time of the system, wherein the replication factor is used to represent the number of copies of the partition in the distributed system, the replication delay is used to represent the replication time of the proxy in the distributed system, the preset delay is used to represent the maximum delay time that the system can accept, the proxy unavailability time is used to represent the suspension time of the proxy in the distributed system, and the preset unavailability time is used to represent the maximum proxy unavailability time that the system can accept; the fourth objective function is determined based on the initial number of partitions, the delay constraint and the availability constraint.

[0100] In the configuration optimization device for a distributed system provided in an embodiment of the present application, the adjustment module is also used to determine the throughput constraint of the distributed system, wherein the throughput constraint is used to maintain the number of partitions of the distributed system to meet the throughput requirements of the system; and the fifth objective function is determined based on the target number of agents, the delay constraint and the throughput constraint.

[0101] In the configuration optimization device of a distributed system provided in an embodiment of the present application, the configuration module is also used to determine the target operating state of the distributed system cluster within a preset time period, wherein the target operating state is the operating state of the distributed system after the target number of proxies and the target number of partitions are configured; when the target operating state does not meet the preset performance indicators, the target number of proxies and the target number of partitions are adjusted through the configuration optimization model; when the target operating state meets the preset performance indicators, the configuration of the distributed system is maintained.

[0102] An embodiment of the present application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the configuration optimization method for implementing the above-mentioned distributed system.

[0103] It should be noted that the above electronic equipment is used to execute Figure 2 The configuration optimization method of the distributed system shown in the figure, therefore the relevant explanations and instructions in the above-mentioned configuration optimization method of the distributed system are also applicable to the electronic device, and will not be repeated here.

[0104] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned configuration optimization method of the distributed system by running the computer program.

[0105] It should be noted that the above non-volatile storage medium is used to execute Figure 2 The configuration optimization method of the distributed system shown in the figure, therefore the relevant explanations in the above-mentioned configuration optimization method of the distributed system are also applicable to the non-volatile storage medium, and will not be repeated here.

[0106] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the configuration optimization method of the above-mentioned distributed system when executed by a processor.

[0107] It should be noted that the above-mentioned computer program product is used to execute Figure 2 The configuration optimization method of the distributed system shown in the figure, therefore the relevant explanations in the above-mentioned configuration optimization method of the distributed system are also applicable to the computer program product, and will not be repeated here.

[0108] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0109] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0111] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0112] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0113] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0114] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A configuration optimization method for a distributed system, characterized in that: include: Determine the operating status of the distributed system, wherein the operating status includes the load status and resource usage status of each instance in the distributed system; When the running state does not meet the preset performance index, the number of proxies and the number of partitions of the distributed system are adjusted by configuring an optimization model to obtain a target number of proxies and a target number of partitions, wherein the configuration optimization model includes a global proxy optimization model and a local partition optimization model, the global proxy optimization model is used to optimize the system performance by adjusting the number of proxies, and the local partition optimization model is used to optimize the system performance by adjusting the number of partitions; The distributed system is configured according to the target number of agents and the target number of partitions.

2. The method according to claim 1, characterized in that The configuration optimization model includes at least: a first objective function, a second objective function and a third objective function, wherein the first objective function is used to measure the global delay of the distributed system, the second objective function is used to measure the resource cost of the distributed system under the target number of agents, and the third objective function is used to measure the resource utilization and performance indicators of the distributed system under the target number of partitions.

3. The method according to claim 1, characterized in that When the running state does not meet the preset performance index, the number of proxies and the number of partitions of the distributed system are adjusted by configuring the optimization model to obtain the target number of proxies and the target number of partitions, including: When the load condition exceeds a first preset threshold, determining a target number of proxies of the distributed system through a global proxy optimization model, wherein the target number of proxies is used to represent the number of proxies required by the distributed system under the load condition; When the resource usage exceeds a second preset threshold, a target number of partitions of the distributed system is determined by a local partition optimization model, wherein the target number of partitions is used to represent the number of partitions required by the distributed system under the resource usage.

4. The method according to claim 3, characterized in that Determining the target number of agents of the distributed system by a global agent optimization model includes: determining the target number of agents by a fourth objective function in the global agent optimization model, wherein the fourth objective function is determined by: Determining an initial number of partitions of the distributed system; Determine a delay constraint and an availability constraint under the load condition, wherein the delay constraint is used to keep the relationship between the number of proxies, the initial number of partitions, the replication factor and the replication delay of the distributed system to satisfy a preset delay of the system, and the availability constraint is used to keep the proxy unavailability time of the distributed system to satisfy a preset unavailability time of the system, wherein the replication factor is used to represent the number of copies of a partition in the distributed system, the replication delay is used to represent the replication time of a proxy in the distributed system, the preset delay is used to represent a maximum delay time that the system can accept, the proxy unavailability time is used to represent a service suspension time of a proxy in the distributed system, and the preset unavailability time is used to represent a maximum proxy unavailability time that the system can accept; The fourth objective function is determined according to the initial number of partitions, the delay constraint, and the availability constraint.

5. The method according to claim 4, characterized in that Determining the target number of partitions of the distributed system by using a local partition optimization model includes: determining the target number of partitions by using a fifth objective function in the local partition optimization model, wherein the fifth objective function is determined by: Determining a throughput constraint of the distributed system, wherein the throughput constraint is used to maintain a number of partitions of the distributed system that meets a throughput requirement of the system; The fifth objective function is determined according to the target number of agents, the delay constraint, and the throughput constraint.

6. The method according to claim 1, characterized in that The method further comprises: Determining a target operating state of the distributed system cluster within a preset time period, wherein the target operating state is the operating state of the distributed system after the target number of proxies and the target number of partitions are configured; When the target operating state does not meet the preset performance index, adjusting the target number of proxies and the target number of partitions through the configuration optimization model; When the target operating state meets the preset performance indicator, the configuration of the distributed system is maintained.

7. The method according to claim 6, characterized in that The preset performance indicators include at least: the global delay of the distributed system, the delay of each partition in the distributed system, and the maximum load of each proxy in the distributed system.

8. A configuration optimization device for a distributed system, characterized in that: include: A determination module, configured to determine an operating state of a distributed system, wherein the operating state includes a load condition and a resource usage condition of each instance in the distributed system; An adjustment module, used for adjusting the number of proxies and the number of partitions of the distributed system by configuring an optimization model to obtain a target number of proxies and a target number of partitions when the operating state does not meet the preset performance index, wherein the configuration optimization model includes a global proxy optimization model and a local partition optimization model, the global proxy optimization model is used to optimize the system performance by adjusting the number of proxies, and the local partition optimization model is used to optimize the system performance by adjusting the number of partitions; A configuration module is used to configure the distributed system according to the target number of agents and the target number of partitions.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the configuration optimization method of the distributed system described in any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the configuration optimization method of the distributed system described in any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the configuration optimization method of a distributed system described in any one of claims 1 to 7 is implemented.

Images