Method and device for creating kafka theme partition, electronic equipment, medium and program product

Abstract

The invention discloses a kafka theme partition creation method and device, electronic equipment, a medium and a program product, and belongs to the technical field of message middleware. The method comprises the following steps: acquiring index data of at least one node in a kafka cluster on at least one dimension; inputting the index data of at least one node into a trained Kafka cluster node performance comparison model to obtain an output result output by the Kafka cluster node performance comparison model, and determining a node of which the performance is higher than a preset value in the at least one node as a target node according to a performance sorting result of the at least one node in the output result; and creating a kafka theme partition on the target node. Therefore, the node of which the performance is higher than the preset value is selected as the target node for creating the kafka theme partition, so that the accuracy of partition distribution can be improved, and the occurrence of overhigh load of a single node and network congestion can be avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of message middleware, and in particular to a method and device for creating a kafka topic partition, an electronic device, a medium and a program product. BACKGROUND

[0002] In a distributed system, Kafka message middleware is known for its high throughput, low latency and high fault tolerance, and is widely used in asynchronous communication, system decoupling and load balancing scenarios. Messages in Kafka are classified by topics, and a topic usually contains multiple partitions that can be processed in parallel. Topic partitions are stored on designated nodes, and the performance of cluster nodes directly affects the processing capacity of partitions. Therefore, when creating topic partitions, the performance of the nodes should be considered, and the node with the best performance should be selected first. A reasonable topic partition allocation strategy can improve the utilization rate of node performance, improve cluster performance and save server resources.

[0003] In related technologies, a random and polling allocation mechanism is usually used to distribute topic partitions relatively evenly to each node in the kafka cluster. This scheme does not take into account the actual load bearing capacity of each node, and is prone to problems such as high single-node load and network congestion. In addition to this, the commonly used method also includes allocating topic partitions to nodes based on factors such as the actual demand situation of the topic partitions, the actual load bearing capacity of each node and the network rate. This scheme highly depends on the personal experience of the operation and maintenance personnel, and requires the operation and maintenance personnel to weigh the weights of multiple indicators to set rules, which has limited accuracy and may cause problems such as high single-node load and network congestion. SUMMARY

[0004] The present application provides a method and device for creating a kafka topic partition, an electronic device, a medium and a program product, which can at least solve the problem of high single-node load and network congestion caused by the partition allocation method in related technologies.

[0005] To solve the above technical problems, the present application is implemented as follows: In a first aspect, an embodiment of the present application provides a method for creating a kafka topic partition, the method comprising: obtaining index data of at least one node in a kafka cluster in at least one dimension; inputting the index data of the at least one node into a trained kafka cluster node performance comparison model to obtain an output result output by the kafka cluster node performance comparison model, wherein the kafka cluster node performance comparison model is configured to sort the performance of the at least one node, and the output result is configured to represent a performance sorting result of the at least one node; determining, according to the output result, a node whose performance is higher than a preset value among the at least one node as a target node; and creating a kafka topic partition on the target node.

[0006] In a second aspect, an embodiment of the present application provides a device for creating a kafka topic partition, the device comprising: a first obtaining module configured to obtain index data of at least one node in a kafka cluster in at least one dimension; a second obtaining module configured to input the index data of the at least one node into a trained kafka cluster node performance comparison model to obtain an output result output by the kafka cluster node performance comparison model, wherein the kafka cluster node performance comparison model is configured to sort the performance of the at least one node, and the output result is configured to represent a performance sorting result of the at least one node; a determining module configured to determine, according to the output result, a node whose performance is higher than a preset value among the at least one node as a target node; and a creating module configured to create a kafka topic partition on the target node.

[0007] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the method according to the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to implement the steps of the method according to the first aspect.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, and the program instructions are executed by a computer to cause the computer to perform the steps of the method according to the first aspect.

[0010] The technical solution provided by the present application can have the following beneficial effects: In the embodiment of the present application, the index data of at least one node in the Kafka cluster in at least one dimension can be obtained; then the index data of the at least one node is input into the trained Kafka cluster node performance comparison model to obtain the output result output by the Kafka cluster node performance comparison model, and according to the performance ranking result of the at least one node in the output result, the node with performance higher than a preset value in the at least one node is determined as a target node; finally, the Kafka topic partition is created on the target node. By selecting the node with performance higher than the preset value as the target node for creating the Kafka topic partition, the accuracy of the Kafka topic partition allocation can be improved, and the problem of high single node load and network congestion can be avoided.

[0011] In the embodiments of the present application, it should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0013] Figure 1 A flowchart of a method for creating a Kafka topic partition is shown; Figure 2 A schematic diagram of a Kafka cluster experimental environment architecture is shown; Figure 3 A schematic diagram of a Kafka running index data collection architecture is shown; Figure 4 A schematic diagram of a DBSCAN algorithm noise point identification is shown; Figure 5 A flowchart of a method for training a Kafka node performance comparison model is shown; Figure 6 A schematic diagram of another method for creating a Kafka topic partition is shown; Figure 7 A schematic diagram of the results before and after optimization is shown; Figure 8 A structural schematic diagram of a device for creating a Kafka topic partition is shown; Figure 9 A structural schematic diagram of an electronic device is shown; Figure 10A structural schematic diagram of another electronic device provided in an example embodiment of the present application is shown. DETAILED DESCRIPTION

[0014] The example embodiments will be described in detail herein with reference to the drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0015] In the related art, the Kafka official provides a default topic partition creation strategy: Kafka adopts a random and polling allocation mechanism to distribute partitions relatively uniformly to nodes in the cluster, without considering the actual load of the cluster nodes. This random and polling allocation mechanism can ensure the relative balance of the number of partitions of each node. However, it does not take into account the actual load of the nodes, and in actual applications, due to unbalanced producer load, unbalanced consumer load, or different message volumes and message rates of different business topics, problems such as single-point load being too high and network congestion occur.

[0016] In addition, a commonly used method includes a method of manually formulating a Kafka topic partition allocation rule according to real-time network load, data disk read-write load, capacity load, and topic partition load. This optimization method based on predefined rules highly depends on the personal experience of the operation and maintenance personnel, and requires the operation and maintenance personnel to set rules by weighing the weights of multiple indicators according to experience, which is difficult to achieve high accuracy.

[0017] To solve at least one of the above technical problems, an example embodiment of the present application provides a creation scheme for a Kafka topic partition.

[0018] Figure 1 A flowchart of a method for creating a Kafka topic partition is shown, which can be executed by an electronic device. The electronic device can be a mobile phone, a computer, or the like. As shown in Figure 1 The method mainly includes the following steps: S101: Obtain index data of at least one node in a Kafka cluster in at least one dimension.

[0019] In the example embodiment of the present application, the index data of at least one node in the Kafka cluster in at least one dimension can be obtained. In this way, it is helpful to understand the specific situation of the at least one node in the Kafka cluster, and facilitate the analysis of the at least one node.

[0020] S102: input the index data of at least one node into the trained Kafka cluster node performance comparison model to obtain an output result output by the Kafka cluster node performance comparison model.

[0021] The Kafka cluster node performance comparison model is configured to sort the performance of at least one node, and the output result is configured to represent the performance sorting result of at least one node.

[0022] In the embodiments of the present application, the index data of at least one node can be input into the trained Kafka cluster node performance comparison model to obtain an output result output by the Kafka cluster node performance comparison model, which represents the performance sorting result of at least one node. In this way, the performance of at least one node in the Kafka cluster can be more comprehensively compared and sorted by sorting the performance of at least one node by using the Kafka cluster node performance comparison model. Compared with the method of manually formulating the Kafka topic partition allocation rule in the related art, the performance of at least one node is sorted by using the Kafka cluster node performance comparison model in the embodiments of the present application, which does not need human intervention and can improve the accuracy.

[0023] S103: determining a node whose performance is higher than a preset value in at least one node as a target node according to the output result.

[0024] In the embodiments of the present application, a node whose performance is higher than a preset value in at least one node can be determined as a target node according to the performance sorting result of at least one node in the output result, so that the selected target node is more accurate and has better performance.

[0025] S104: creating a Kafka topic partition on the target node.

[0026] In the embodiments of the present application, a node whose performance is higher than a preset value is selected from at least one node as a target node for partition creation, which can improve the utilization rate of node performance and avoid problems such as excessively high single-point load and network congestion.

[0027] In the embodiments of the present application, the index data of at least one node in the Kafka cluster in at least one dimension can be acquired; then the index data of the at least one node is input into the trained Kafka cluster node performance comparison model to obtain an output result output by the Kafka cluster node performance comparison model, and according to the performance ranking result of the at least one node in the output result, a node with performance higher than a preset value in the at least one node is determined as a target node; finally, a Kafka topic partition is created on the target node. By selecting the node with performance higher than the preset value as the target node for creating the Kafka topic partition, the accuracy of Kafka topic partition allocation can be improved, and thus the problem of excessively high single-node load and network congestion can be avoided.

[0028] In an optional implementation, the index data includes: the number of topics, the number of producers, the number of consumers, the number of active threads, the number of partitions, disk I / O performance, network traffic, memory usage, and CPU usage.

[0029] In actual application, by analyzing the characteristics of the Kafka cluster node, it can be determined that the core indicators affecting the performance of the node include: the number of topics, the number of producers, the number of consumers, the number of active threads, the number of partitions, disk I / O performance, network traffic, memory usage, and CPU usage. Therefore, the node performance can be evaluated by the above-mentioned index data. Moreover, the deficiency of the related method of only focusing on the physical indicators of the server can be made up.

[0030] In an optional implementation, the training data is acquired by the following manner: A sample Kafka cluster is built, wherein the sample Kafka cluster includes at least one sample node; Sample index data of the at least one sample node in at least one dimension is acquired; According to the result of two-by-two combination of the sample index data of the sample nodes in at least one dimension and the true value of the performance comparison result of any two sample nodes in the sample nodes, the training data is acquired.

[0031] In the embodiments of the present application, an experimental environment identical to the hardware environment of the online environment can be built. As shown in Figure 2 A Kafka cluster including five nodes can be built, and a three-node Zookeeper cluster is used as a configuration and coordination center. Then, a data collector (kafka_exporter) can be deployed on each Kafka node to store the collected data into a time series database Prometheus, and the specific architecture is as shown in Figure 3The performance of Kafka is shown in the figure. In practical applications, Kafka can be stress tested, and in order to meet the diversity of the data set, the number of topics, the number of producers, and the number of consumers need to be adjusted during testing. The collected indicators include the number of topics, the number of producers, the number of consumers, the number of active threads, the number of partitions, disk I / O performance, network traffic, memory usage, CPU usage, and throughput. In the embodiments of the present application, the message throughput TP (unit: MB / s) can be used to measure the performance of Kafka. The parameter description of the collected sample indicator data is shown in Table 1.

[0032] Table 1

[0033] In practical applications, sample indicator data can be exported to a text format (for example, zai) by using a Prometheus query statement PromQL, and then the sample indicator data can be obtained.

[0034]

[0035] In practical applications, data denoising can be performed on sample indicator data, which can improve model accuracy, reduce errors, improve data quality, ensure more reliable analysis results, and provide a good foundation for subsequent data mining and machine learning. In practical applications, the popular DBSCAN algorithm can be selected, which can automatically identify noise and has high model robustness.

[0036] The data is processed using the libraries numpy (numerical calculation), pandas (data processing), and scikit-learn (including DBSCAN algorithm) in Python. Figure 4 The algorithm identifies noise points (hollow circles are noise points) according to part of the sample data. Noise data will be removed from the sample set and will not participate in subsequent training.

[0037] In the embodiments of the present application, training data can be obtained according to the result of the pairwise combination of sample indicator data of sample nodes in at least one dimension and the true value of the performance comparison result of any two sample nodes in the sample nodes, and then the model is trained through the above training data.

[0038] In an optional implementation, the true value is obtained by the following method: Based on the Kafka performance evaluation function, the performance of the sample indicator data of any two sample nodes in at least one dimension is compared, and the true value of the performance comparison result of any two sample nodes in the sample nodes is obtained; The formula corresponding to the Kafka performance evaluation function is as follows:

[0039] wherein f represents the Kafka performance evaluation function, N1 and N2 represent two sample nodes, P represents the sample index data of the sample node, and TP represents the throughput of the sample node.

[0040] In the embodiments of the present application, a Kafka performance evaluation function can be constructed, and an example formula is as follows:

[0041] wherein f represents the Kafka performance evaluation function, N1 and N2 represent two sample nodes, P represents the sample index data of the sample node, and TP represents the throughput of the sample node. The function can compare two cluster nodes according to the throughput rates of the Kafka cluster nodes at different times. If the throughput of the Kafka cluster node corresponding to N1 is lower than the throughput rate of the Kafka cluster node corresponding to N2, 0 is output, and if the throughput rate of the Kafka cluster node corresponding to N1 is lower than the throughput rate of the Kafka cluster node corresponding to N2, 1 is output.

[0042] After obtaining the performance evaluation function, the sample data rows can be combined two by two, so that the original samples are converted into training data with a number of The row in the training data is represented by , wherein and represent two different cluster node index data, represents the value of the performance evaluation function. The converted training data set is as follows:

[0043] In this way, it can fully utilize the data samples to train a performance comparison model with better optimization effect under the condition of a small number of samples.

[0044] In related technologies, the mainstream supervised learning algorithms include decision trees, random forests, XGBoost (Extreme Gradient Boosting). The decision tree is prone to overfitting, sensitive to data changes, and poor at processing complex relationships. The random forest has high calculation cost, poor interpretability, and complex processing of high-dimensional data. The advantage of XGBoost lies in high accuracy, effective bias-variance trade-off, strong flexibility, providing feature importance ranking, and high model robustness.

[0045] Therefore, in practical applications, the XGBoost algorithm can be used, and the core objective of the XGBoost algorithm is to minimize an objective function composed of a loss function and a regularization term. For example, the simplified formula can be represented as:

[0046] where Obj represents the total objective that needs to be minimized, N represents the number of training samples, represents the true value of the i-th sample, represents the predicted value of the i-th sample. represents the target loss function, which measures the difference between the predicted value and the true value, T represents the number of trees in the model (the number of iterations), represents the regularization term of the t-th tree, which is used to control the complexity of the model.

[0047] In an optional implementation, the target loss function in the Kafka cluster node performance comparison model is constructed based on the weight of at least one key indicator data, the predicted value of the performance comparison result of any two sample nodes in the sample nodes, and the true value of the performance comparison result of any two sample nodes in the sample nodes. The key indicator data is used to indicate the key indicator data in the sample indicator data corresponding to the sample nodes, and the sample nodes are sample nodes in training data used for training of the Kafka cluster node performance comparison model.

[0048] For example, the formula of the target loss function is as follows:

[0049]

[0050] where, represents the weight of the i-th feature attribute, n represents the number of indicators, and here n = 3, which means that the first three indicators are used to participate in the weight calculation.

[0051] In order to better cope with high-load and high-concurrency scenarios, in practical applications, a target loss function that is positively correlated with the number of topics, the number of producers, and the number of consumers can be used. Different from the conventional Mean Squared Error (MSE) or Mean Absolute Error (MAE) function, the weighted loss function emphasizes the importance of the number of topics, the number of producers, and the number of consumers.

[0052] In an optional implementation, the performance ranking result includes a throughput ranking result of the node.

[0053] The determining, according to the output result, of a node with performance higher than a preset value as a target node can include: The determining, according to the output result, of a node with the maximum throughput as a target node.

[0054] In actual application, a node with the maximum throughput can be determined as a target node, and a topic partition is created on the target node, so that the utilization of node performance can be improved, and problems such as too high single-point load and network congestion can be avoided.

[0055] Figure 5 A flowchart of a Kafka node performance comparison model training method provided by an example embodiment of the application is shown, as shown in Figure 5 The method includes the following steps: Step 501: Model training starts; Step 502: Kafka cluster experimental environment is built: a test environment same as the online hardware environment is built; Step 503: Kafka running index data collection: the number of topics, the number of producers, the number of consumers, the number of active threads, the number of partitions, disk I / O performance, network traffic, memory usage, CPU usage, and throughput.

[0056] Step 504: Data denoising: a popular DBSCAN algorithm is selected to identify and delete noise points.

[0057] Step 505: Training of the Kafka node performance comparison model. It can include the following steps: 1, construction of a Kafka performance comparison function; 2, construction of algorithm training data; 3, custom algorithm loss function; 4, algorithm code based on Python.

[0058] Step 506: Model output.

[0059] In actual application, the algorithm code implementation is as follows: the libraries numpy (numerical calculation), pandas (data processing), xgboost (training and prediction model), and scikit-learn (division of data set and evaluation of model performance) in Python can be used to complete the training of data. The parameter values used are as follows: # Set parameters params = { 'objective':'reg:squarederror',# Regression task 'max_depth': 6, 'eta': 0.1, 'seed': 42} objective: set to reg:squarederror, indicating a regression task. max_depth: set the maximum depth of the tree. eta: learning rate. seed: random seed, used to ensure the reproducibility of the results.

[0060] After the training is completed, model export can be performed. For example, the save_model method of XGBoost can be used to save the model as a file kafka_xgboost_model.json.

[0061] Figure 6 Another method for creating a Kafka topic partition provided by an example embodiment of the present application is shown in the schematic diagram as shown in Figure 6 The method includes the following steps: Step 601: Load the XGBoot model (kafka_xgboost_model.json) Step 602: Model initialization.

[0062] Step 603: Obtain real-time indicator data of all Kafka nodes.

[0063] Step 604: PromQL query.

[0064] Step 605: Return. In this step, the indicator data obtained by the query can be returned.

[0065] Step 606: Use the indicator data of the Kafka nodes as input parameters to predict the node throughput.

[0066] Step 607: Predict the throughput.

[0067] Step 608: Return. In this step, the sorting result of the throughput of each node predicted by the Kafka node performance comparison model can be returned.

[0068] Step 609: Select the node with the largest throughput as the Kafka partition allocation node.

[0069] Step 610: Create a partition.

[0070] In actual applications, a custom Kafka partition allocation strategy can be used. First, custom Kafka partition allocation is performed: by writing custom partition allocation Java code, the Kafka partition allocation logic is optimized. The code mainly includes the following three parts: (1) Obtain real-time indicator data of all Kafka nodes from Prometheus (2) Load the XGBoost algorithm model file kafka_xgboost_model.json, and use the index data of the Kafka node as the input parameter to predict the node performance ranking (3) Select the node with the largest throughput as the Kafka partition allocation node According to the architecture provided in the embodiments of the present application, a new Kafka cluster is deployed using the optimized partition allocation code. Comparative experiments can be performed on the throughputs of the two clusters, and the experimental sampling results are shown in Table 3. Table 3: Comparison table before and after optimization

[0071] The comparison chart before and after the experiment is shown in Figure 7 As can be seen from the experimental results, the performance of the optimized Kafka cluster has been improved by 1% to 6%. It is worth noting that as the number of node partitions increases, the optimization algorithm improves the cluster more obviously. When the number of partitions reaches 27 and 30, the efficiency is improved by 7.42% and 6.91%. In a real environment, the necessity of optimizing a high-load Kafka cluster is higher, and the effect will be more obvious.

[0072] The execution subject of the method for creating a kafka topic partition provided in the embodiments of the present application can be a kafka topic partition creation device. In the embodiments of the present application, the method for creating a kafka topic partition is executed by the kafka topic partition creation device, and the kafka topic partition creation device provided in the embodiments of the present application is described.

[0073] Figure 8 The structure of the kafka topic partition creation device provided in an example embodiment of the present application is shown. The kafka topic partition creation device can implement all or part of the embodiments shown in Figure 1 The kafka topic partition creation device includes a first obtaining module 801, a second obtaining module 802, a determining module 803, and a creating module 804.

[0074] In the embodiments of the present application, the first obtaining module 801 is configured to obtain index data of at least one node in a Kafka cluster in at least one dimension; the second obtaining module 802 is configured to input the index data of the at least one node into a trained Kafka cluster node performance comparison model to obtain an output result output by the Kafka cluster node performance comparison model, wherein the Kafka cluster node performance comparison model is configured to sort the performance of the at least one node, and the output result is configured to represent a performance sorting result of the at least one node; the determining module 803 is configured to determine a node whose performance is higher than a preset value in the at least one node as a target node according to the output result; and the creating module 804 is configured to create a Kafka topic partition on the target node.

[0075] In an optional implementation, a target loss function in the Kafka cluster node performance comparison model is constructed based on a weight of at least one key index data, a predicted value of a performance comparison result of any two sample nodes in sample nodes, and a true value of the performance comparison result of any two sample nodes in the sample nodes, the key index data is used to indicate key index data in sample index data corresponding to the sample nodes, the sample nodes are sample nodes in training data, and the training data is used to train the Kafka cluster node performance comparison model.

[0076] In an optional implementation, the index data includes a number of topics, a number of producers, a number of consumers, a number of active threads, a number of partitions, disk I / O performance, network traffic, memory usage, and CPU usage.

[0077] In an optional implementation, the training data is obtained by the following method: A sample Kafka cluster is built, wherein the sample Kafka cluster includes at least one sample node; Sample index data of the at least one sample node in at least one dimension is obtained; The training data is obtained according to a result of two-by-two combination of the sample index data of the sample nodes in at least one dimension and a true value of a performance comparison result of any two sample nodes in the sample nodes.

[0078] In an optional implementation, the true value is obtained by the following method: Performance of the sample index data of any two sample nodes in at least one dimension is compared based on a Kafka performance evaluation function to obtain the true value of the performance comparison result of any two sample nodes in the sample nodes; The formula corresponding to the Kafka performance evaluation function is as follows:

[0079] Wherein, f represents the Kafka performance evaluation function, N1 and N2 represent two sample nodes, P represents the sample index data of the sample node, and TP represents the throughput of the sample node.

[0080] In an optional implementation, the performance ranking result includes a throughput ranking result of the node.

[0081] The determining module 803 is configured to determine, according to the output result, a node with performance higher than a preset value in the at least one node as a target node, and the determining includes: According to the output result, determine a node with the maximum throughput in the at least one node as a target node.

[0082] The kafka topic partition creation apparatus in the embodiments of the present applicationapplicationbe an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic deviceapplicationbe a terminal or other devices than a terminal. For example, the electronic deviceapplicationbe a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), andapplicationbe a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the embodiments of the present application do not have a specific limitation.

[0083] The kafka topic partition creation apparatus in the embodiments of the present applicationapplicationbe a device with an operating system. The operating systemapplicationbe an Android operating system, an ios operating system, or other possible operating systems, and the embodiments of the present application do not have a specific limitation.

[0084] The kafka topic partition creation apparatus provided in the embodiments of the present applicationapplicationbe capable of implementing the method embodiments, and each process of the method embodiments is not repeated here to avoid repetition. Figure 1 The kafka topic partition creation apparatus provided in the embodiments of the present applicationapplicationbe capable of implementing the method embodiments, and each process of the method embodiments is not repeated here to avoid repetition.

[0085] Optionally, as shown in Figure 9 The electronic device 900 is provided in the embodiments of the present application, and includes a processor 901 and a memory 902. The memory 902 stores programs or instructions executable on the processor 901. When the programs or instructions are executed by the processor 901, the above-mentioned Figure 1 The steps of the method for creating a kafka topic partition are shown in FIG. 9, and the same technical effects can be achieved. To avoid repetition, the steps will not be described here.

[0086] It should be noted that the electronic device in the embodiments of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0087] Figure 10 A structural block diagram of another electronic device 1000 is shown in an example embodiment of the present application. The electronic device 1000 can be implemented as a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart watch, a television, etc. The electronic device 1000 can also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, etc.

[0088] Generally, the electronic device 1000 includes a processor 1001 and a memory 1002.

[0089] The processor 1001 can include one or more processing cores, such as an 8-core processor, a 10-core processor, etc. The processor 1001 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 1001 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1001 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content to be displayed on a display screen. In some embodiments, the processor 1001 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0090] The memory 1002 can include one or more computer-readable storage media. The memory 1002 can also include high-speed random access memory and non-volatile, computer-readable storage media such as one or more magnetic disk storage devices, flash memory devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one instruction for being executed by the processor 1001 to implement all or part of the steps in the method of creating a kafka topic partition shown in the method embodiments of the present application.

[0091] In some embodiments, the electronic device 1000 can also optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002 and the peripheral device interface 1003 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1003 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007 and a power supply 1008.

[0092] In some embodiments, the electronic device 1000 further includes one or more sensors 1009. The one or more sensors 1009 include, but are not limited to, an acceleration sensor 1010, a gyroscope sensor 1011, a pressure sensor 1012, an optical sensor 1013 and a proximity sensor 1014.

[0093] Those skilled in the art can understand that, Figure 10 The structure shown in the above embodiments does not constitute a limitation on the electronic device 1000, and can include more or fewer components than those shown in the figure, or combine certain components, or use different component arrangements.

[0094] The embodiments of the present application also provide a computer readable storage medium, which stores a program or instructions, and the program or instructions are executed by a processor to implement the above-mentioned processes of the method of creating a kafka topic partition and achieve the same technical effects. To avoid repetition, details are not described here.

[0095] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0096] The embodiment of the present application further provides a chip, which comprises a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running programs or instructions, realizes each process of the method for creating the kafka topic partition, and can achieve the same technical effects, to avoid repetition, here will not repeat.

[0097] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system chip, a system chip, a chip system or a system on chip, etc.

[0098] The embodiment of the present application further provides a computer program product, which comprises a computer program stored on a non-transitory computer readable storage medium, the computer program comprises program instructions, when the program instructions are executed by a computer, the steps of the method for creating the kafka topic partition are realized, and the same technical effects can be achieved, to avoid repetition, here will not repeat.

[0099] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The application is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the application are indicated by the appended claims.

[0100] It should be understood that the present application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A method for creating Kafka topic partitions, characterized in that, include: Obtain metric data for at least one node in a Kafka cluster across at least one dimension. The indicator data of at least one of the nodes is input into a trained Kafka cluster node performance comparison model to obtain the output result of the Kafka cluster node performance comparison model. The Kafka cluster node performance comparison model is used to rank the performance of at least one of the nodes, and the output result is used to characterize the performance ranking result of at least one of the nodes. Based on the output results, at least one node with performance higher than a preset value is identified as the target node. Create a Kafka topic partition on the target node.

2. The method according to claim 1, characterized in that, The objective loss function in the Kafka cluster node performance comparison model is constructed based on the weight of at least one key indicator data, the predicted value of the performance comparison result of any two sample nodes in the sample nodes, and the true value of the performance comparison result of any two sample nodes in the sample nodes. The key indicator data is used to indicate the key indicator data in the sample indicator data corresponding to the sample node. The sample node is the sample node in the training data, and the training data is used to train the Kafka cluster node performance comparison model.

3. The method according to claim 1, characterized in that, The metrics include: number of topics, number of producers, number of consumers, number of active threads, number of partitions, disk I / O performance, network traffic, memory usage, and CPU usage.

4. The method according to claim 2, characterized in that, The training data was obtained in the following ways: Build a sample Kafka cluster, wherein the sample Kafka cluster includes at least one sample node; Obtain sample indicator data for at least one of the sample nodes in at least one dimension; The training data is obtained based on the pairwise combination of the sample index data of the sample nodes in at least one dimension and the true value of the performance comparison result of any two sample nodes.

5. The method according to claim 4, characterized in that, The actual value is obtained in the following way: Based on the Kafka performance evaluation function, the performance of any two sample nodes is compared in at least one dimension to obtain the true value of the performance comparison result of any two sample nodes. The formula corresponding to the Kafka performance evaluation function is as follows: Where f represents the Kafka performance evaluation function, N1 and N2 represent the two sample nodes, P represents the sample metric data of the sample node, and TP represents the throughput of the sample node.

6. The method according to claim 1, characterized in that, The performance ranking results include the throughput ranking results of the nodes; The step of determining at least one node with performance higher than a preset value as a target node based on the output result includes: Based on the output results, at least one node with the highest throughput among the nodes is determined as the target node.

7. A device for creating Kafka topic partitions, characterized in that, include: The first acquisition module is used to acquire metric data of at least one node in the Kafka cluster in at least one dimension. The second acquisition module is used to input the indicator data of at least one of the nodes into a trained Kafka cluster node performance comparison model to obtain the output result of the Kafka cluster node performance comparison model. The Kafka cluster node performance comparison model is used to rank the performance of at least one of the nodes, and the output result is used to characterize the performance ranking result of at least one of the nodes. The determination module is used to determine, based on the output results, at least one node among the nodes whose performance is higher than a preset value as a target node; Create a module to create Kafka topic partitions on the target node.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the steps of the method as described in any one of claims 1 to 6.

Images