Method and device for adjusting Kafka cluster configuration parameters, equipment and medium

Abstract

The invention relates to the technical field of computer processing, in particular to a Kafka cluster configuration parameter adjusting method and device, equipment and a medium, and is used for relieving problems caused by a fixed change means of code modification. The method comprises the following steps: according to related historical data of a Kafka cluster, predicting a first predicted load index in a future time period through a first time sequence model, and predicting a second predicted load index in the future time period through a second time sequence model; weighting the first predicted load indicator; weighting the second predicted load index; determining a predicted load index according to the weighted first predicted load index and the weighted second predicted load index; determining a plurality of first candidate adjustment actions according to the predicted load index; selecting a first candidate adjustment action from the plurality of first candidate adjustment actions as a first adjustment action; and adjusting a partition allocation index and a copy allocation index of the Kafka cluster configuration parameters by using the first adjustment action.

Description

Technical Field

[0001] This application relates to the field of computer processing technology, specifically to a method, apparatus, device, and medium for adjusting Kafka cluster configuration parameters. Background Technology

[0002] In daily business operations, Kafka clusters play a unified role in data caching and distribution, ensuring reliable message delivery and efficient processing. Properly configured parameters are a key prerequisite for the stable and efficient operation of a Kafka cluster, as various configuration parameters affect the overall operational status of the Kafka cluster through different mechanisms.

[0003] In existing technologies, application-layer code is modified to adjust configuration parameters to adapt to load requirements. However, this fixed approach to modification cannot effectively adapt to real-time changes in business needs. Furthermore, because it requires improvements to the server-side source code, further modifications are necessary if the changes no longer meet current requirements, resulting in a significant workload for development and testing. Summary of the Invention

[0004] This application provides a method, apparatus, device, and medium for adjusting Kafka cluster configuration parameters. It can make predictions through a predictive model. Compared with fixed change methods such as code modification, it can sense business changes in real time, improve prediction accuracy, and alleviate the problems caused by code modification.

[0005] Firstly, this application provides a method for adjusting Kafka cluster configuration parameters, including: Based on the relevant historical data of the Kafka cluster, the first predicted load indicator for the future time period is predicted using the first time series model, and the second predicted load indicator for the future time period is predicted using the second time series model. The first predicted load index is weighted using the weights corresponding to the first time series model; the second predicted load index is weighted using the weights corresponding to the second time series model; and the predicted load index is determined based on the weighted first predicted load index and the weighted second predicted load index. Based on the predicted load index, a plurality of first candidate adjustment actions are determined; from the plurality of first candidate adjustment actions, one first candidate adjustment action is selected as the first adjustment action; Using the first adjustment action, adjust the partition allocation metric and replica allocation metric of the Kafka cluster configuration parameters.

[0006] The above method can use relevant historical data of the Kafka cluster to predict two forecast results through two time series forecast models, and determine the forecast load index based on the two forecast results. Based on the forecast load index, the partition allocation index and replica allocation index of the Kafka cluster configuration parameters are then adjusted. In this embodiment of the invention, the use of models for forecasting, compared with the fixed change method of code modification, can sense business changes in real time, improve the forecast accuracy, and alleviate the problems caused by code modification.

[0007] In one possible embodiment, the weights corresponding to the first time series model and the weights corresponding to the second time series model are determined in the following manner: Based on the relevant test data of the Kafka cluster, the first test load metric is predicted using the first time series model, and the second test load metric is predicted using the second time series model. Calculate the error of the first test load index and the error of the second test load index; Based on the first negative correlation, the weights of the first time series model under the error of the first test load index are determined, and based on the second negative correlation, the weights of the second time series model under the error of the second test load index are determined.

[0008] In one possible embodiment, selecting one candidate adjustment action as the target adjustment action from a plurality of candidate adjustment actions includes: Generate any random number; If the random number is less than the exploration probability, then a first candidate adjustment action is randomly selected from multiple first candidate adjustment actions; If the random number is not less than the exploration probability, then determine the action Q value corresponding to each first candidate adjustment action, refer to the action Q values ​​corresponding to multiple first candidate adjustment actions, and select the candidate adjustment action corresponding to the maximum value of the action Q value from multiple first candidate adjustment actions; After adjusting the partition allocation metrics and replica allocation metrics of the Kafka cluster using the selected first candidate adjustment action, if the security verification of the Kafka cluster passes, the selected first candidate adjustment action will be used as the first adjustment action.

[0009] In one possible embodiment, determining the action Q value corresponding to each first candidate adjustment action, and selecting the candidate adjustment action corresponding to the maximum action Q value from the multiple first candidate adjustment actions by referring to the action Q values ​​corresponding to multiple first candidate adjustment actions, including: The predicted load index is input into the evaluation network, where the action Q value corresponding to each first candidate adjustment action is determined. The action Q value corresponding to multiple first candidate adjustment actions is referenced, and the candidate adjustment action corresponding to the maximum action Q value is selected from the multiple first candidate adjustment actions. The evaluation network outputs the candidate adjustment action corresponding to the maximum action Q value. The evaluation network is obtained by training the base network multiple times. In each training process, training samples are extracted from the training set, the evaluation network outputs the training results corresponding to the training samples, the target network outputs the target action Q value corresponding to the training results, and the parameters of the evaluation network are updated using the target action Q value; and when the cumulative number of training sessions reaches a preset value, the parameters of the evaluation network are copied into the target network.

[0010] In one possible embodiment, the method further includes: Each decision tree determines the corresponding residual prediction performance metrics based on the relevant historical data in the Kafka cluster. The performance metrics of the Kafka cluster in the future time period are obtained by performing a weighted summation operation on the residual prediction performance metrics corresponding to multiple decision trees. Based on the performance metrics of the Kafka cluster within the future time period, multiple second candidate adjustment actions are determined, and one of these is selected as the second adjustment action. If there is a second candidate adjustment action with a large adjustment range among the multiple second candidate adjustment actions, then the second adjustment action is the second candidate adjustment action with the large adjustment range; if all of the multiple second candidate adjustment actions are second candidate adjustment actions with small adjustment ranges, then the second adjustment action is the second candidate adjustment action with the small adjustment range. The second adjustment action is used to adjust the configuration parameters of the producer and the consumer.

[0011] In one possible embodiment, based on the performance metrics of the Kafka cluster over a future time period, multiple second candidate adjustment actions are determined, and one second candidate adjustment action is selected from these multiple second candidate adjustment actions as the second adjustment action, including: The performance metrics of the Kafka cluster in the future time period are input into the prediction algorithm. The prediction algorithm determines multiple second candidate adjustment actions based on the performance metrics of the Kafka cluster in the future time period, selects one second candidate adjustment action from the multiple second candidate adjustment actions as the second adjustment action, and obtains the second adjustment action output by the prediction algorithm. After adjusting the configuration parameters of the producer and the consumer using the second adjustment action, the method further includes: Collect the adjusted performance parameters of the Kafka cluster; Based on the throughput and latency in the adjusted performance parameters of the Kafka cluster, a first comprehensive performance score for the Kafka cluster is determined; and based on the throughput and latency in the unadjusted performance parameters of the Kafka cluster, a second comprehensive performance score for the Kafka cluster is determined. If the first comprehensive performance score is less than the second comprehensive performance score, then the parameters of the prediction algorithm are adjusted.

[0012] Secondly, this application provides a device for adjusting Kafka cluster configuration parameters, including: The prediction module is used to predict the first predicted load indicator in the future time period based on the relevant historical data of the Kafka cluster using a first time series model, and to predict the second predicted load indicator in the future time period using a second time series model. The weighting module is used to weight the first predicted load index using the weights corresponding to the first time series model; to weight the second predicted load index using the weights corresponding to the second time series model; and to determine the predicted load index based on the weighted first predicted load index and the weighted second predicted load index. The selection module is used to determine multiple first candidate adjustment actions based on the predicted load index; and to select one first candidate adjustment action from the multiple first candidate adjustment actions as the first adjustment action. The adjustment module is used to adjust the partition allocation index and replica allocation index of the Kafka cluster configuration parameters using the first adjustment action.

[0013] In one possible embodiment, the weighting module is further configured to: Based on the relevant test data of the Kafka cluster, the first test load metric is predicted using the first time series model, and the second test load metric is predicted using the second time series model. Calculate the error of the first test load index and the error of the second test load index; Based on the first negative correlation, the weights of the first time series model under the error of the first test load index are determined, and based on the second negative correlation, the weights of the second time series model under the error of the second test load index are determined.

[0014] In one possible embodiment, the selection module is specifically used for: Generate any random number; If the random number is less than the exploration probability, then a first candidate adjustment action is randomly selected from multiple first candidate adjustment actions; If the random number is not less than the exploration probability, then determine the action Q value corresponding to each first candidate adjustment action, refer to the action Q values ​​corresponding to multiple first candidate adjustment actions, and select the candidate adjustment action corresponding to the maximum value of the action Q value from multiple first candidate adjustment actions; After adjusting the partition allocation metrics and replica allocation metrics of the Kafka cluster using the selected first candidate adjustment action, if the security verification of the Kafka cluster passes, the selected first candidate adjustment action will be used as the first adjustment action.

[0015] In one possible embodiment, the selection module is further configured to: The predicted load index is input into the evaluation network, where the action Q value corresponding to each first candidate adjustment action is determined. The action Q value corresponding to multiple first candidate adjustment actions is referenced, and the candidate adjustment action corresponding to the maximum action Q value is selected from the multiple first candidate adjustment actions. The evaluation network outputs the candidate adjustment action corresponding to the maximum action Q value. The evaluation network is obtained by training the base network multiple times. In each training process, training samples are extracted from the training set, the evaluation network outputs the training results corresponding to the training samples, the target network outputs the target action Q value corresponding to the training results, and the parameters of the evaluation network are updated using the target action Q value; and when the cumulative number of training sessions reaches a preset value, the parameters of the evaluation network are copied into the target network.

[0016] In one possible embodiment, the device further includes: The performance prediction module is used to determine the corresponding residual prediction performance metrics based on the relevant historical data of the Kafka cluster, using each decision tree. The performance metrics of the Kafka cluster in the future time period are obtained by performing a weighted summation operation on the residual prediction performance metrics corresponding to multiple decision trees. The selection module is also used to determine multiple second candidate adjustment actions based on the performance indicators of the Kafka cluster in the future time period, and select one second candidate adjustment action from the multiple second candidate adjustment actions as the second adjustment action; wherein, if there is a second candidate adjustment action with a large adjustment range among the multiple second candidate adjustment actions, then the second adjustment action is the second candidate adjustment action with the large adjustment range; if all the multiple second candidate adjustment actions are second candidate adjustment actions with small adjustment ranges, then the second adjustment action is the second candidate adjustment action with the small adjustment range. The adjustment module is also used to adjust the configuration parameters of the producer and the configuration parameters of the consumer using the second adjustment action.

[0017] In one possible embodiment, the selection module is further configured to: The performance metrics of the Kafka cluster in the future time period are input into the prediction algorithm. The prediction algorithm determines multiple second candidate adjustment actions based on the performance metrics of the Kafka cluster in the future time period, selects one second candidate adjustment action from the multiple second candidate adjustment actions as the second adjustment action, and obtains the second adjustment action output by the prediction algorithm. Collect the adjusted performance parameters of the Kafka cluster; Based on the throughput and latency in the adjusted performance parameters of the Kafka cluster, a first comprehensive performance score for the Kafka cluster is determined; and based on the throughput and latency in the unadjusted performance parameters of the Kafka cluster, a second comprehensive performance score for the Kafka cluster is determined. If the first comprehensive performance score is less than the second comprehensive performance score, then the parameters of the prediction algorithm are adjusted.

[0018] Thirdly, this application provides an electronic device, comprising: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method described in any one of the first aspects according to the obtained program instructions.

[0019] Fourthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in any one of the first aspects.

[0020] Fifthly, this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects. Attached Figure Description

[0021] Figure 1 A flowchart illustrating a method for adjusting Kafka cluster configuration parameters provided in this application embodiment; Figure 2 A flowchart illustrating a training model method provided in an embodiment of this application; Figure 3 A flowchart illustrating a first adjustment action selection method provided in an embodiment of this application; Figure 4 A flowchart illustrating an adjustment method provided in an embodiment of this application; Figure 5 A flowchart illustrating another adjustment method provided in this application embodiment; Figure 6 A flowchart illustrating a method for adjusting configuration parameters for producers and consumers provided in an embodiment of this application; Figure 7 A structural diagram of a device for adjusting Kafka cluster configuration parameters provided in an embodiment of this application; Figure 8 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0023] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, and / or more; this application does not impose limitations on its embodiments.

[0024] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of this application, and do not imply that the applicant has already and / or necessarily used such solutions.

[0025] The acquisition, dissemination, storage, and use of data in this application all comply with relevant national laws and regulations.

[0026] Before introducing the method for adjusting Kafka cluster configuration parameters provided in the embodiments of this application, the technical background of the embodiments of this application will be described in detail below for ease of understanding.

[0027] Kafka is an open-source stream processing platform used to handle high-throughput distributed message queue systems. Taking the consumption of a ride-hailing QR code as an example, the following is a typical Kafka cluster deduction message processing flow: Message production: When a user uses a ride code to pay for their ride, the system generates a payment deduction message. The producer sends this message to a specific Topic in the Kafka cluster. Message storage: After receiving a message, the Kafka cluster stores it in the corresponding partition. The Kafka cluster ensures high availability and durability of messages through partitioning and replication mechanisms. Message consumption: Consumers pull deduction messages from the Kafka cluster and process them; Payment processing: Consumers continuously listen to the message queue, process payment messages according to business logic, and deduct the payment. Message confirmation: The consumer submits the offset to confirm that the message has been successfully processed; Monitoring, alarms, and logs: Monitor the message queue's running status and message processing, set up an alarm mechanism, and record key status changes throughout the message processing process in the log for backup.

[0028] During the entire message processing process, as the business volume increases, the following problems become increasingly serious: First, slow nodes, i.e., nodes with slower processing speeds, cause uneven distribution of disk space and IO (Input / Output) across the entire cluster due to read / write operations; Second, uneven load distribution across the Kafka cluster and different business traffic patterns between different topics reduce cluster stability; Third, competition for machine resources between different partitions on the same node leads to processing delays on a single node.

[0029] In a Kafka cluster, a partition is the smallest unit of parallel processing within a topic. A topic can be divided into multiple partitions, each of which is essentially an ordered, immutable message log file. Messages are appended to the end of the partition in the order they were sent. Producers can send messages to different partitions in parallel, and multiple consumer instances can consume messages from different partitions in parallel. This is the core reason why Kafka clusters support high throughput. For example, a topic with 8 partitions can theoretically support 8 consumers consuming simultaneously, significantly improving consumption efficiency.

[0030] Replication is a data backup mechanism provided by Kafka clusters for partitions. Each partition can be configured with multiple replicas (including one leader replica and several follower replicas). Replicas are distributed across different nodes to prevent data loss due to single-node failure.

[0031] In existing technologies, partitions and replicas are allocated using common partitioning algorithms such as RangeAssignor, RoundRobinAssignor, or StickyAssignor. These algorithms are then translated into code, and the code is executed to optimize message distribution. For example, RangeAssignor sorts the partitions and consumers for each Topic and then tries to distribute partitions evenly among consumers. However, this algorithm is only suitable for certain scenarios, namely when the number of partitions and consumers is relatively balanced. When a consumer subscribes to multiple Topics, it may lead to uneven distribution. Furthermore, changing this approach requires modifying the corresponding code, making this fixed allocation algorithm unsuitable for real-time changes in production environments. Additionally, because modifications to the server-side source code are necessary, changes are required if the solution becomes unsuitable, resulting in a significant workload for development and testing.

[0032] This application proposes a core technical concept: using a predictive model for prediction, compared to the conventional change method of code modification, it can sense business changes in real time, improve prediction accuracy, and alleviate the problems caused by the conventional change method of code modification.

[0033] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0034] Combination Figure 1 The present invention provides a method for adjusting Kafka cluster configuration parameters, including: S110: Based on the relevant historical data of the Kafka cluster, predict the first predicted load indicator for the future time period using the first time series model, and predict the second predicted load indicator for the future time period using the second time series model.

[0035] In detail, feature extraction is performed on relevant historical data from the Kafka cluster. The extracted features include timestamps, load patterns, historical performance, network bandwidth utilization, node health status, partition load balancer health values, and disk read / write speeds. A first time-series model uses these extracted features to predict the load metric, resulting in the first predicted load metric. A second time-series model uses these extracted features to predict the load metric, resulting in the second predicted load metric.

[0036] Load metrics include latency, CPU (Central Processing Unit) load, network utilization, disk I / O, and Y; Y can be a business load metric or an underlying resource load metric for the Kafka cluster.

[0037] The first time series model is LSTM (Long Short-Term Memory); the second time series model is ARIMA (AutoRegressive Integrated Moving Average Model).

[0038] Because ARIMA is good at capturing linear trends and seasonality, but has a weak ability to fit nonlinear patterns, while LSTM is good at capturing nonlinear and long-term dependencies, but its modeling of linear trends is not as stable as ARIMA. Since the present invention proposes that the prediction results need to have both linear trends and nonlinear fluctuations as well as seasonal disturbances in business, the present invention proposes to combine the two using a dynamic weighted average method, as shown in step 120.

[0039] S120: Use the weights corresponding to the first time series model to weight the first predicted load index; use the weights corresponding to the second time series model to weight the second predicted load index; determine the predicted load index based on the weighted first predicted load index and the weighted second predicted load index.

[0040] In detail, the weights corresponding to the first time series model and the second time series model are determined in the following way: Based on the relevant test data of the Kafka cluster, the first test load metric is predicted using the first time series model, and the second test load metric is predicted using the second time series model. Calculate the error of the first test load index and the error of the second test load index; Based on the first negative correlation, the weights of the first time series model under the error of the first test load index are determined, and based on the second negative correlation, the weights of the second time series model under the error of the second test load index are determined.

[0041] For example, on the test set, a first test load metric is predicted using LSTM, and a second test load metric is predicted using ARIMA. The error of the test load metric is determined using the mean absolute error. MAE = (1 / N) Σ|actual value - predicted value|; Using the above formula, since N is the number of load indicators, the difference between each first test load indicator predicted by the first time series model and its corresponding actual value is calculated. The sum of these differences is divided by the number of load indicators N, and the resulting value is taken as the error of the first test load indicator. Similarly, the difference between each second test load indicator predicted by the second time series model and its corresponding actual value is calculated. The sum of these differences is divided by the number of load indicators N, and the resulting value is taken as the error of the second test load indicator.

[0042] Where MAE_LSTM is the error of the first test load metric; MAE_ARIMA is the error of the second test load metric.

[0043] The weights are calculated using the following formula: W_ARIMA =1 / (MAE_ARIMA + epsilon) (1); W_ LSTM= 1 / (MAE_ LSTM + epsilon) (2); Where W_ARIMA represents the weights of the unnormalized second time series model, and W_LSTM represents the weights of the unnormalized first time series model. Epsilon is a constant that can be set by the user.

[0044] The following formula is used to normalize the weights and eliminate the dimensional differences between different indicators.

[0045] SUM_W= W_ARIMA+ W_ LSTM (3); W_ARIMA'= W_ARIMA / SUM_W (4); W_ LSTM' = W_ LSTM / SUM_W (5); Where W_LSTM' represents the weights corresponding to the first time series model; W_ARIMA' represents the weights corresponding to the second time series model. The first negative correlation includes formulas (2), (3), and (5). The second negative correlation includes formulas (1), (3), and (4).

[0046] The first predictive load index output by the first time series model is Y_LSTM; the second predictive load index output by the second time series model is Y_ARIMA.

[0047] The predicted load index Y is determined using the following formula: Y = W_ARIMA' Y_ARIMA+ W_ LSTM' Y_LSTM.

[0048] Based on the above, this embodiment of the invention provides a combined model including LSTM-ARIMA, which can implement steps 110 and 120.

[0049] Optionally, embodiments of the present invention propose a method for training a combined model. Specifically: First, obtain a sample set. Within this set, historical data can be preprocessed and cleaned. Preprocessing includes handling missing values, using the Python Pandas library to fill in missing values. Python is an interpreted scripting language, and Pandas is a data analysis package within Python.

[0050] Preprocessing methods include using sliding windows to truncate data. Sliding windows are a key data preprocessing technique in time series modeling that adapts to the time series characteristics of LSTM or ARIMA. The core idea is to divide continuous time series data into fixed-length "input-output" sample pairs, allowing LSTM or ARIMA to learn the time series dependency patterns, while solving the efficiency and memory capacity problems of LSTM or ARIMA in processing long sequences.

[0051] The sliding window is an adaptive sliding window that monitors performance metrics data in real time, sets peak and off-peak traffic thresholds, sets different groups based on different thresholds, and sets the sliding window size by group.

[0052] Then the combined model is trained multiple times.

[0053] For each training session, specifically, combined Figure 2 As shown, the training process includes: S210: Based on the samples in the training set, predict the first training result using the first time series model, and adjust the parameters of the first time series model based on the first training result and the actual result; S220: Based on the samples in the training set, predict the second training result using the second time series model, and adjust the parameters of the second time series model based on the second training result and the actual result; S230: After adjusting the first time series model and the second time series model respectively, test the first time series model and the second time series model using the test set; S240: Determine the weights corresponding to the first time series model and the weights corresponding to the second time series model using the test set.

[0054] Specifically, multiple samples are divided into training set and test set; the training set is used when training the first time series model and the second time series model, and the test set is used to test the trained first time series model and the second time series model. At the same time, the test set can also be used to determine the weights corresponding to the first time series model and the second time series model.

[0055] S130: Based on the predicted load indicators, determine multiple first candidate adjustment actions; select one first candidate adjustment action from the multiple first candidate adjustment actions as the first adjustment action.

[0056] For example, based on latency, CPU (Central Processing Unit) load, network utilization, disk I / O, and Y within a future time period, multiple first-candidate adjustment actions are determined. The first-candidate adjustment actions may include some or all of the following: reducing or increasing the number of partitions corresponding to which topics, removing or adding partitions to each node in the Kafka cluster, increasing or decreasing the replicas corresponding to partitions, and deleting or adding replicas corresponding to partitions of nodes in the Kafka cluster.

[0057] In detail, combined Figure 3 As shown, an embodiment of the present invention proposes a selection method, including: S310: Generate any random number.

[0058] S320: If the random number is less than the exploration probability, then randomly select one first candidate adjustment action from multiple first candidate adjustment actions.

[0059] S330: If the random number is not less than the exploration probability, then determine the action Q value corresponding to each first candidate adjustment action, refer to the action Q values ​​corresponding to multiple first candidate adjustment actions, and select the first candidate adjustment action corresponding to the maximum action Q value from multiple first candidate adjustment actions.

[0060] For example, embodiments of the present invention can provide - Greedy strategy. Through... A greedy strategy chooses between exploration and exploitation. Specifically, it generates a random number rand between 0 and 1. To explore the probability, if rand < This indicates that exploration is chosen, so regardless of the Q-value (action value function value) of each first candidate adjustment action, a first candidate adjustment action is randomly selected. If rand This indicates that the selection can be made by choosing the first candidate adjustment action corresponding to the maximum value in the action Q value.

[0061] In step 230, the action Q value corresponding to each first candidate adjustment action is determined. Referring to the action Q values ​​corresponding to multiple first candidate adjustment actions, the implementation scheme for selecting the first candidate adjustment action corresponding to the maximum action Q value from among the multiple first candidate adjustment actions is as follows: The predicted load metric is input into the evaluation network, where the action Q value corresponding to each first candidate adjustment action is determined. The action Q value corresponding to multiple first candidate adjustment actions is referenced, and the candidate adjustment action corresponding to the maximum action Q value is selected from the multiple first candidate adjustment actions. The evaluation network outputs the first candidate adjustment action corresponding to the maximum action Q value.

[0062] The evaluation network is obtained by training the training set multiple times.

[0063] If only the evaluation network is trained during the training process, the evaluation network will also generate the target action Q value. Therefore, updating the parameters of the evaluation network will also cause the target action Q value to change synchronously, which can easily lead to training oscillations or divergence.

[0064] To address this, a Deep Q-Network dual-network structure is used for model training. This embodiment of the invention proposes that, in each training process, training samples are extracted from the training set, the evaluation network outputs the training results corresponding to the training samples, the target network outputs the target action Q-value corresponding to the training results, and the parameters of the evaluation network are updated using the target action Q-value; and when the cumulative number of training iterations reaches a preset value, the parameters of the evaluation network are copied to the target network.

[0065] Since the target action Q-value is determined by the target network, and the parameters of the target network are updated once every certain period of time, the target action Q-value is more stable, thus improving training convergence.

[0066] For example, this embodiment of the invention provides a training method, including: extracting training samples from a training set; inputting the training samples into an evaluation network; calculating action Q-values ​​corresponding to multiple first candidate adjustment actions; selecting the first candidate adjustment action corresponding to the action Q-value with the largest action Q-value; adjusting partition allocation indicators and replica allocation indicators according to the first candidate adjustment actions to obtain the adjusted load indicators of the Kafka cluster; inputting the adjusted load indicators of the Kafka cluster into a target network; the target network outputs a target action Q-value; updating the parameters of the evaluation network using the target action Q-value; determining whether the number of updates exceeds a preset value; if so, copying the parameters of the evaluation network into the target network; if not, not updating the parameters of the target network, continuing to obtain another training sample, and continuing to update the evaluation network until training ends.

[0067] S340: After adjusting the partition allocation metrics and replica allocation metrics of the Kafka cluster using the selected first candidate adjustment action, if the security verification of the Kafka cluster passes, the selected first candidate adjustment action will be used as the first adjustment action.

[0068] The partition allocation metrics include the number of partitions and their distribution; the replica allocation metrics include the number of replicas and their location.

[0069] The security checks for the Kafka cluster include verifying whether there is sufficient disk space. If the disk space for the Kafka cluster is sufficient, the security check passes; if the disk space is insufficient, the security check fails.

[0070] S140: Using the first adjustment action, adjust the partition allocation index and replica allocation index of the Kafka cluster configuration parameters.

[0071] In detail, since the first adjustment action specifies which topics correspond to a decrease or increase in the number of partitions, the removal or addition of partitions on each node in the Kafka cluster, the increase or decrease of replicas corresponding to partitions, and the deletion or addition of replicas corresponding to partitions on nodes in the Kafka cluster, the partition allocation and replica allocation metrics in the Kafka cluster configuration parameters can be adjusted according to the first adjustment action. For example, adding partition 1 to the first adjustment action means adding partition 1 to the first node in the Kafka cluster, excluding the node that originally contained partition 2.

[0072] The solution provided by this invention can predict load indicators for future time periods using relevant historical data. Based on these load indicators, the partition allocation and replica allocation parameters of the Kafka cluster configuration can be adjusted. This ensures that the performance of the Kafka cluster in the future is more consistent with actual operating conditions. Furthermore, by using models for prediction, compared to fixed-method changes such as code modification, it can sense business changes in real time, predict indicators more accurately, and alleviate the problems caused by code modification.

[0073] Based on the above, in practical applications, embodiments of the present invention provide an adjustment method, combined with... Figure 4 As shown, it includes: S410: Determine the predicted load metrics based on relevant historical data from the Kafka cluster; The specific scheme for step 410 is as follows: Figure 1 Steps 110 and 120.

[0074] S420: Generate any random number; S430: If the random number is not less than the exploration probability, the predicted load index is input into the evaluation network. The evaluation network determines the action Q value corresponding to each candidate adjustment action. By referring to the action Q values ​​corresponding to multiple candidate adjustment actions, a first candidate adjustment action is selected from multiple candidate adjustment actions in order of size. The evaluation network outputs the selected first candidate adjustment action. S440: Determine whether the security check of the Kafka cluster has passed. If yes, execute S450; otherwise, execute S430. S450: Select the first candidate adjustment action as the first adjustment action, use the first adjustment action to adjust the partition allocation index and replica allocation index of the Kafka cluster configuration parameters, calculate the reward r, and adjust the evaluation network based on the reward r.

[0075] In detail, with the random number not less than the exploration probability, the predicted load metric S is used, i.e., the state space: S = {latency, CPU load, network utilization, disk I / O, Y}. When evaluating the network for the first time to select the first candidate adjustment action, the network can choose the action with the largest Q value. The evaluation network outputs the first candidate adjustment action, for example, action space: A = {add partition, remove partition, migrate leader, adjust replicas}. Then, it checks if the Kafka cluster's security check passes. If it fails, the second largest Q value is selected as the first candidate adjustment action, and the Kafka cluster's security check is checked again until it passes. When it passes, the selected first candidate adjustment action is used to adjust the partition allocation and replica allocation metrics of the Kafka cluster configuration parameters. Prometheus is used to collect metrics, collecting the performance parameters of the adjusted Kafka cluster. The reward r is calculated using the adjusted performance parameters, and the parameters of the evaluation network are adjusted using the reward r.

[0076] The reward r is determined using the following formula: The reward function is: R(s,a) = α (1-latency_rank) + β (1- cpu_utilization_rank)+ γ throughput_score; Where: R(s,a) is the reward r; They are coefficients, respectively. latency_rank represents the current latency ranking relative to historical levels; cpu_utilization_rank indicates the current ranking of network utilization; throughput_score represents the system load score.

[0077] When the random number is less than the exploration probability, this embodiment of the invention provides another adjustment method, combined with Figure 5 As shown, it includes: S510: Determine the predicted load metrics based on relevant historical data from the Kafka cluster; The specific scheme for step 510 is as follows: Figure 1 Steps 110 and 120.

[0078] S520: Generate any random number; S530: If the random number is less than the exploration probability, randomly select one first candidate adjustment action from multiple first candidate adjustment actions; S540: Determine whether the security check of the Kafka cluster has passed. If yes, execute S550; otherwise, execute S530. S550: Select the first candidate adjustment action as the first adjustment action, use the first adjustment action to adjust the partition allocation index and replica allocation index of the Kafka cluster configuration parameters, calculate the reward r, and adjust the evaluation network based on the reward r.

[0079] In detail, after determining that the security verification of the Kafka cluster fails, you can randomly select another first-candidate adjustment action from multiple first-candidate adjustment actions and verify whether the verification passes again, until it passes. In addition, the first-candidate adjustment actions that have already been selected are excluded when making the selection.

[0080] In addition to adjusting the partition allocation and replica allocation metrics of the Kafka cluster configuration parameters, this embodiment of the invention can also adjust the producer and consumer configuration parameters of the Kafka cluster configuration parameters. The specific scheme for adjusting the producer and consumer configuration parameters of the Kafka cluster configuration parameters is as follows: In some embodiments, combined with Figure 6 As shown in the figure, this embodiment of the invention also provides a method for adjusting configuration parameters, including: S610: Determine the corresponding residual prediction performance metrics based on the relevant historical data of the Kafka cluster for each decision tree.

[0081] S620: Perform a weighted summation operation based on the residual prediction performance metrics corresponding to multiple decision trees to obtain the performance metrics of the Kafka cluster in the future time period.

[0082] The performance metrics of a Kafka cluster include network traffic, message size, latency, and throughput.

[0083] Steps 610 and 620 use the Gradient Boosting Tree (GBDT) model to predict performance metrics for future time periods.

[0084] For example, relevant historical data is input into the gradient boosting tree model. Features, such as message size, network latency, and historical traffic patterns, are extracted within the model. This allows for the determination of the corresponding residual prediction performance metric for the Kafka cluster based on the relevant historical data, using each decision tree. A weighted summation of the residual prediction performance metrics from multiple decision trees is then performed to obtain the performance metric for the Kafka cluster in the future time period. The gradient boosting tree model outputs the performance metric for the Kafka cluster in the future time period. That is, the predicted performance metric is Y = [network_traffic, message_size, latency, throughput]

[0085] The gradient boosting tree model is obtained by training the training set multiple times.

[0086] In each training process, training samples are extracted from the training set, input into the gradient boosting tree model, and the model's prediction performance is evaluated through the objective function Yi′.

[0087] The objective function is Yi′=α Throughput i +(1-α) (1 / Latency i ); In this embodiment of the invention, in a Kafka cluster with network traffic, message size, latency, and throughput as performance indicators, latency and throughput are chosen as the core variables of the objective function. This is because these two indicators directly reflect the core value of the Kafka cluster: efficiently and timely completing data transmission / processing tasks. Network traffic and message size are more of a constraint or influencing factor.

[0088] According to the updated formula F m (X)=F m-1 (X)+η h m (X) where η is the learning rate and hm(X) is the base learner in the m-th iteration. The model is continuously iterated and optimized until the prediction error of the model for the performance parameter Y converges to the preset threshold, thus obtaining a stable performance prediction model.

[0089] S630: Based on the performance metrics of the Kafka cluster in the future time period, determine multiple second candidate adjustment actions, and select one second candidate adjustment action from the multiple second candidate adjustment actions as the second adjustment action; If among multiple second candidate adjustment actions, there is one with a large adjustment range, then the second adjustment action is the second candidate adjustment action with a large adjustment range; if among multiple second candidate adjustment actions, all are second candidate adjustment actions with small adjustment ranges, then the second adjustment action is the second candidate adjustment action with a small adjustment range.

[0090] The producer's configuration parameters include: batch.size (batch size), linger.ms (wait time), compression.type (compression type), and acks (acknowledgment mechanism). The consumer's configuration parameters include: fetch.min.bytes (minimum number of bytes to fetch), fetch.max.wait.ms (maximum wait time), and max.partition.fetch.bytes (maximum number of bytes to fetch per partition).

[0091] The adjustment range refers to the magnitude of the adjustment to the producer's configuration parameters and the magnitude of the adjustment to the consumer's configuration parameters. For example, adjusting batch.size from 64 to 2048 is a large adjustment range; adjusting batch.size from 32 to 48 is a small adjustment range.

[0092] Optionally, the performance metrics of the Kafka cluster are input into the prediction algorithm. Based on the performance metrics of the Kafka cluster, the prediction algorithm determines multiple second candidate adjustment actions, selects one of the multiple second candidate adjustment actions as the second adjustment action, and obtains the second adjustment action output by the prediction algorithm.

[0093] The prediction algorithm is either PPO (Proximal Policy Optimization) or A3C (Asynchronous Advantage Actor-Critic).

[0094] The core of A3C is asynchronous multi-threaded exploration and no strict policy update constraints. Therefore, among the multiple second candidate adjustment actions identified, there are second candidate adjustment actions with large adjustment magnitudes. A3C selects the second adjustment action with the large adjustment magnitude.

[0095] The core of PPO is near-end policy optimization and pruning constraints, which is a robust update. Therefore, among the multiple second candidate adjustment actions identified, all are second candidate adjustment actions with small adjustment magnitudes. The second adjustment action selected by PPO is the second candidate adjustment action with a small adjustment magnitude.

[0096] In this embodiment of the invention, either PPO or A3C can be used to select the second adjustment action.

[0097] The prediction algorithm is obtained by training the training set multiple times.

[0098] In each training process, training samples are extracted from the training set, input into the prediction algorithm, the prediction results are determined, and the prediction algorithm is adjusted using the prediction results.

[0099] S640: Using the second adjustment action, adjust the configuration parameters of the producer and the consumer.

[0100] The configuration parameter is X=[batch.size,linger.ms,compression.type,acks,fetch.min.bytes,fetch.max.wait.ms,max.partition.fetch.bytes]; The second adjustment action automatically adjusts parameters such as batch.size, linger.ms, and compression.type.

[0101] The second adjustment action automatically adjusts parameters such as fetch.min.bytes, fetch.max.wait.ms, and max.partition.fetch.bytes.

[0102] In detail, the second adjustment action is the action of adjusting the configuration parameters of the producer and the configuration parameters of the consumer.

[0103] In some embodiments, after step 640, the method further includes: Collect the adjusted performance parameters of the Kafka cluster; Based on the throughput and latency in the adjusted performance parameters of the Kafka cluster, determine the first comprehensive performance score of the Kafka cluster; and based on the throughput and latency in the unadjusted performance parameters of the Kafka cluster, determine the second comprehensive performance score of the Kafka cluster. If the first comprehensive performance score is less than the second comprehensive performance score, then the parameters of the prediction algorithm are adjusted.

[0104] The overall performance score can be calculated using an objective function, which is Yi′=α. Throughput i +(1-α) (1 / Latency i ); Substituting the adjusted throughput and latency parameters of the Kafka cluster into the objective function yields the first comprehensive performance score. Substituting the unadjusted throughput and latency parameters of the Kafka cluster into the objective function yields the second comprehensive performance score. If the first comprehensive performance score is greater than the second comprehensive performance score, the performance has significantly improved, and the adjusted producer and consumer configuration parameters can continue to operate. If the first comprehensive performance score is less than the second comprehensive performance score, it indicates a significant performance decline, requiring adjustment of the prediction algorithm parameters to make the prediction algorithm more accurate.

[0105] Of course, in this embodiment of the invention, if the first comprehensive performance score is less than the second comprehensive performance score, in addition to adjusting the parameters of the prediction algorithm, the parameters of the gradient boosting tree model are also adjusted.

[0106] Based on the same inventive concept, this application provides a device for adjusting Kafka cluster configuration parameters. Please refer to... Figure 7 As shown, the device includes: The prediction module 710 is used to predict a first predicted load indicator in the future time period based on the relevant historical data of the Kafka cluster using a first time series model, and to predict a second predicted load indicator in the future time period using a second time series model. The weighting module 720 is used to weight the first predicted load index using the weights corresponding to the first time series model; to weight the second predicted load index using the weights corresponding to the second time series model; and to determine the predicted load index based on the weighted first predicted load index and the weighted second predicted load index. Selection module 730 is used to determine a plurality of first candidate adjustment actions based on the predicted load index; and select one first candidate adjustment action from the plurality of first candidate adjustment actions as the first adjustment action; The adjustment module 740 is used to adjust the partition allocation index and replica allocation index of the Kafka cluster configuration parameters using the first adjustment action.

[0107] Optionally, the weighting module 720 is further configured to: Based on the relevant test data of the Kafka cluster, the first test load metric is predicted using the first time series model, and the second test load metric is predicted using the second time series model. Calculate the error of the first test load index and the error of the second test load index; Based on the first negative correlation, the weights of the first time series model under the error of the first test load index are determined, and based on the second negative correlation, the weights of the second time series model under the error of the second test load index are determined.

[0108] Optionally, the selection module 730 is specifically used for: Generate any random number; If the random number is less than the exploration probability, then a first candidate adjustment action is randomly selected from multiple first candidate adjustment actions; If the random number is not less than the exploration probability, then determine the action Q value corresponding to each first candidate adjustment action, refer to the action Q values ​​corresponding to multiple first candidate adjustment actions, and select the candidate adjustment action corresponding to the maximum value of the action Q value from multiple first candidate adjustment actions; After adjusting the partition allocation metrics and replica allocation metrics of the Kafka cluster using the selected first candidate adjustment action, if the security verification of the Kafka cluster passes, the selected first candidate adjustment action will be used as the first adjustment action.

[0109] Optionally, the selection module 730 is further configured to: The predicted load index is input into the evaluation network, where the action Q value corresponding to each first candidate adjustment action is determined. The action Q value corresponding to multiple first candidate adjustment actions is referenced, and the candidate adjustment action corresponding to the maximum action Q value is selected from the multiple first candidate adjustment actions. The evaluation network outputs the candidate adjustment action corresponding to the maximum action Q value. The evaluation network is obtained by training the base network multiple times. In each training process, training samples are extracted from the training set, the evaluation network outputs the training results corresponding to the training samples, the target network outputs the target action Q value corresponding to the training results, and the parameters of the evaluation network are updated using the target action Q value; and when the cumulative number of training sessions reaches a preset value, the parameters of the evaluation network are copied into the target network.

[0110] Optionally, the device further includes: The performance prediction module is used to determine the corresponding residual prediction performance metrics based on the relevant historical data of the Kafka cluster, using each decision tree. The performance metrics of the Kafka cluster in the future time period are obtained by performing a weighted summation operation on the residual prediction performance metrics corresponding to multiple decision trees. The selection module 730 is further configured to determine multiple second candidate adjustment actions based on the performance indicators of the Kafka cluster within a future time period, and select one second candidate adjustment action from the multiple second candidate adjustment actions as the second adjustment action; wherein, if there is a second candidate adjustment action with a large adjustment range among the multiple second candidate adjustment actions, then the second adjustment action is the second candidate adjustment action with the large adjustment range; if all the multiple second candidate adjustment actions are second candidate adjustment actions with small adjustment ranges, then the second adjustment action is the second candidate adjustment action with the small adjustment range. The adjustment module 740 is also used to adjust the configuration parameters of the producer and the configuration parameters of the consumer using the second adjustment action.

[0111] Optionally, the selection module 730 is further configured to: The performance metrics of the Kafka cluster in the future time period are input into the prediction algorithm. The prediction algorithm determines multiple second candidate adjustment actions based on the performance metrics of the Kafka cluster in the future time period, selects one second candidate adjustment action from the multiple second candidate adjustment actions as the second adjustment action, and obtains the second adjustment action output by the prediction algorithm. Collect the adjusted performance parameters of the Kafka cluster; Based on the throughput and latency in the adjusted performance parameters of the Kafka cluster, a first comprehensive performance score for the Kafka cluster is determined; and based on the throughput and latency in the unadjusted performance parameters of the Kafka cluster, a second comprehensive performance score for the Kafka cluster is determined. If the first comprehensive performance score is less than the second comprehensive performance score, then the parameters of the prediction algorithm are adjusted.

[0112] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be implemented entirely in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element and / or software instructions.

[0113] Based on the same inventive concept, embodiments of this application provide an electronic device. Figure 8 The diagram illustrates a schematic representation of an electronic device structure provided in some embodiments, such as... Figure 8As shown, the electronic device includes: a processor 810, a memory 820, a communication interface 830, and a communication bus 840, wherein the processor 810, the communication interface 830, and the memory 820 communicate with each other through the communication bus 840. The memory 820 stores a computer program that, when executed by the processor 810, causes the processor 810 to perform the method for adjusting Kafka cluster configuration parameters as described in any one of the claims.

[0114] The processor 810 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) and / or other programmable logic devices, discrete gate and / or transistor logic devices, or discrete hardware components.

[0115] The memory 820 is connected to the processor 810 via the communication bus 840 and completes communication between them. The memory 820 is used to store computer program instructions.

[0116] The communication bus 840 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 840 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The transceiver is used to enable communication between the database access device and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0117] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium, the computer program product including: computer program code, which, when run on a computer, causes the computer to execute the method for adjusting Kafka cluster configuration parameters as described in any one of the claims.

[0118] Based on the same inventive concept, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the Kafka cluster configuration parameter adjustment methods discussed above. Since the principle of the above computer program product in solving the problem is similar to that of the data analysis report construction method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be described again.

[0119] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of user-operated steps to be executed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for adjusting Kafka cluster configuration parameters, characterized in that, include: Based on the relevant historical data of the Kafka cluster, the first predicted load indicator for the future time period is predicted using the first time series model, and the second predicted load indicator for the future time period is predicted using the second time series model. The first predicted load index is weighted using the weights corresponding to the first time series model; the second predicted load index is weighted using the weights corresponding to the second time series model; and the predicted load index is determined based on the weighted first predicted load index and the weighted second predicted load index. Based on the predicted load index, a plurality of first candidate adjustment actions are determined; from the plurality of first candidate adjustment actions, one first candidate adjustment action is selected as the first adjustment action; Using the first adjustment action, adjust the partition allocation metric and replica allocation metric of the Kafka cluster configuration parameters.

2. The method according to claim 1, characterized in that, The weights corresponding to the first time series model and the weights corresponding to the second time series model are determined in the following manner: Based on the relevant test data of the Kafka cluster, the first test load metric is predicted using the first time series model, and the second test load metric is predicted using the second time series model. Calculate the error of the first test load index and the error of the second test load index; Based on the first negative correlation, the weights of the first time series model under the error of the first test load index are determined, and based on the second negative correlation, the weights of the second time series model under the error of the second test load index are determined.

3. The method according to claim 1, characterized in that, From multiple candidate adjustment actions, select one candidate adjustment action as the target adjustment action, including: Generate any random number; If the random number is less than the exploration probability, then a first candidate adjustment action is randomly selected from multiple first candidate adjustment actions; If the random number is not less than the exploration probability, then determine the action Q value corresponding to each first candidate adjustment action, refer to the action Q values ​​corresponding to multiple first candidate adjustment actions, and select the candidate adjustment action corresponding to the maximum value of the action Q value from multiple first candidate adjustment actions; After adjusting the partition allocation metrics and replica allocation metrics of the Kafka cluster using the selected first candidate adjustment action, if the security verification of the Kafka cluster passes, the selected first candidate adjustment action will be used as the first adjustment action.

4. The method according to claim 3, characterized in that, Determine the action Q value corresponding to each first candidate adjustment action. Referring to the action Q values ​​corresponding to multiple first candidate adjustment actions, select the candidate adjustment action corresponding to the maximum action Q value from among the multiple first candidate adjustment actions, including: The predicted load index is input into the evaluation network, where the action Q value corresponding to each first candidate adjustment action is determined. The action Q value corresponding to multiple first candidate adjustment actions is referenced, and the candidate adjustment action corresponding to the maximum action Q value is selected from the multiple first candidate adjustment actions. The evaluation network outputs the candidate adjustment action corresponding to the maximum action Q value. The evaluation network is obtained by training the base network multiple times. In each training process, training samples are extracted from the training set, the evaluation network outputs the training results corresponding to the training samples, the target network outputs the target action Q value corresponding to the training results, and the parameters of the evaluation network are updated using the target action Q value; and when the cumulative number of training sessions reaches a preset value, the parameters of the evaluation network are copied into the target network.

5. The method according to claim 1, characterized in that, The method further includes: Each decision tree determines the corresponding residual prediction performance metrics based on the relevant historical data in the Kafka cluster. The performance metrics of the Kafka cluster in the future time period are obtained by performing a weighted summation operation on the residual prediction performance metrics corresponding to multiple decision trees. Based on the performance metrics of the Kafka cluster within the future time period, multiple second candidate adjustment actions are determined, and one of these is selected as the second adjustment action. If there is a second candidate adjustment action with a large adjustment range among the multiple second candidate adjustment actions, then the second adjustment action is the second candidate adjustment action with the large adjustment range; if all of the multiple second candidate adjustment actions are second candidate adjustment actions with small adjustment ranges, then the second adjustment action is the second candidate adjustment action with the small adjustment range. The second adjustment action is used to adjust the configuration parameters of the producer and the consumer.

6. The method according to claim 5, characterized in that, Based on the performance metrics of the Kafka cluster over a future time period, several second candidate adjustment actions are identified. One of these second candidate adjustment actions is then selected as the second adjustment action, including: The performance metrics of the Kafka cluster in the future time period are input into the prediction algorithm. The prediction algorithm determines multiple second candidate adjustment actions based on the performance metrics of the Kafka cluster in the future time period, selects one second candidate adjustment action from the multiple second candidate adjustment actions as the second adjustment action, and obtains the second adjustment action output by the prediction algorithm. After adjusting the configuration parameters of the producer and the consumer using the second adjustment action, the method further includes: Collect the adjusted performance parameters of the Kafka cluster; Based on the throughput and latency in the adjusted performance parameters of the Kafka cluster, a first comprehensive performance score for the Kafka cluster is determined; and based on the throughput and latency in the unadjusted performance parameters of the Kafka cluster, a second comprehensive performance score for the Kafka cluster is determined. If the first comprehensive performance score is less than the second comprehensive performance score, then the parameters of the prediction algorithm are adjusted.

7. A device for adjusting Kafka cluster configuration parameters, characterized in that, include: The prediction module is used to predict the first predicted load indicator in the future time period based on the relevant historical data of the Kafka cluster using a first time series model, and to predict the second predicted load indicator in the future time period using a second time series model. The weighting module is used to weight the first predicted load index using the weights corresponding to the first time series model; to weight the second predicted load index using the weights corresponding to the second time series model; and to determine the predicted load index based on the weighted first predicted load index and the weighted second predicted load index. The selection module is used to determine multiple first candidate adjustment actions based on the predicted load index; and to select one first candidate adjustment action from the multiple first candidate adjustment actions as the first adjustment action. The adjustment module is used to adjust the partition allocation index and replica allocation index of the Kafka cluster configuration parameters using the first adjustment action.

8. The apparatus according to claim 7, characterized in that, The weighting module is also used for: Based on the relevant test data of the Kafka cluster, the first test load metric is predicted using the first time series model, and the second test load metric is predicted using the second time series model. Calculate the error of the first test load index and the error of the second test load index; Based on the first negative correlation, the weights of the first time series model under the error of the first test load index are determined, and based on the second negative correlation, the weights of the second time series model under the error of the second test load index are determined.

9. The apparatus according to claim 7, characterized in that, The selection module is specifically used for: Generate any random number; If the random number is less than the exploration probability, then a first candidate adjustment action is randomly selected from multiple first candidate adjustment actions; If the random number is not less than the exploration probability, then determine the action Q value corresponding to each first candidate adjustment action, refer to the action Q values ​​corresponding to multiple first candidate adjustment actions, and select the candidate adjustment action corresponding to the maximum value of the action Q value from multiple first candidate adjustment actions; After adjusting the partition allocation metrics and replica allocation metrics of the Kafka cluster using the selected first candidate adjustment action, if the security verification of the Kafka cluster passes, the selected first candidate adjustment action will be used as the first adjustment action.

10. The apparatus according to claim 9, characterized in that, The selection module is also used for: The predicted load index is input into the evaluation network, where the action Q value corresponding to each first candidate adjustment action is determined. The action Q value corresponding to multiple first candidate adjustment actions is referenced, and the candidate adjustment action corresponding to the maximum action Q value is selected from the multiple first candidate adjustment actions. The evaluation network outputs the candidate adjustment action corresponding to the maximum action Q value. The evaluation network is obtained by training the base network multiple times. In each training process, training samples are extracted from the training set, the evaluation network outputs the training results corresponding to the training samples, the target network outputs the target action Q value corresponding to the training results, and the parameters of the evaluation network are updated using the target action Q value; and when the cumulative number of training sessions reaches a preset value, the parameters of the evaluation network are copied into the target network.

11. The apparatus according to claim 7, characterized in that, The device further includes: The performance prediction module is used to determine the corresponding residual prediction performance metrics based on the relevant historical data of the Kafka cluster, using each decision tree. The performance metrics of the Kafka cluster in the future time period are obtained by performing a weighted summation operation on the residual prediction performance metrics corresponding to multiple decision trees. The selection module is also used to determine multiple second candidate adjustment actions based on the performance indicators of the Kafka cluster in the future time period, and select one second candidate adjustment action from the multiple second candidate adjustment actions as the second adjustment action; wherein, if there is a second candidate adjustment action with a large adjustment range among the multiple second candidate adjustment actions, then the second adjustment action is the second candidate adjustment action with the large adjustment range; if all the multiple second candidate adjustment actions are second candidate adjustment actions with small adjustment ranges, then the second adjustment action is the second candidate adjustment action with the small adjustment range. The adjustment module is also used to adjust the configuration parameters of the producer and the configuration parameters of the consumer using the second adjustment action.

12. The apparatus according to claim 11, characterized in that, The selection module is further used for: The performance metrics of the Kafka cluster in the future time period are input into the prediction algorithm. The prediction algorithm determines multiple second candidate adjustment actions based on the performance metrics of the Kafka cluster in the future time period, selects one second candidate adjustment action from the multiple second candidate adjustment actions as the second adjustment action, and obtains the second adjustment action output by the prediction algorithm. Collect the adjusted performance parameters of the Kafka cluster; Based on the throughput and latency in the adjusted performance parameters of the Kafka cluster, determine the first comprehensive performance score of the Kafka cluster; And based on the throughput and latency in the performance parameters of the Kafka cluster before adjustment, a second comprehensive performance score for the Kafka cluster is determined; If the first comprehensive performance score is less than the second comprehensive performance score, then the parameters of the prediction algorithm are adjusted.

13. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method according to any one of claims 1-6 in accordance with the obtained program instructions.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-6.

15. A computer program product, characterized in that, The computer program product includes: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of claims 1-6.

Images