Kafka cluster deployment method, device, storage medium and electronic device

Abstract

The present application discloses a deployment method, storage medium and electronic device for a Kafka cluster, and relates to the field of intelligent configuration technology. The deployment method for the Kafka cluster includes: obtaining deployment configuration information of the Kafka cluster input by a user; determining the connection configuration selection result of the Zookeeper cluster; based on the connection configuration selection result, obtaining deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster; generating a corresponding template file based on the deployment configuration information, and executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on a preset configuration management tool. The method provided by the present application can effectively improve the deployment efficiency of the Kafka cluster, save labor costs, reduce error rates, and thus enhance the user experience.

Description

Technical Field

[0001] The present application relates to the field of intelligent configuration technology, and in particular to a deployment method, device, storage medium and electronic device for a Kafka cluster. Background Art

[0002] Kafka is a high-throughput distributed publish-subscribe messaging system. A Kafka cluster consists of one or more server nodes, known as brokers. Each message queue consists of multiple partitions, and these partitions and their replicas are distributed across multiple brokers. When a broker stops, the leader of its partitions transfers leadership to another broker. Deploying a Kafka cluster requires individual configuration for each broker, and often involves a large number of brokers, requiring an external Zookeeper cluster. However, because Kafka employs a distributed architecture, it can be clustered by one or more servers and managed by a Zookeeper cluster, which requires at least three nodes. Therefore, deploying a Kafka cluster requires configuring both the Zookeeper cluster and the Kafka cluster. Because each server has a different role and requires different commands and operations, manually deploying the cluster server by server is inefficient, laborious, and error-prone.

[0003] Therefore, how to provide a more effective Kafka cluster deployment solution to improve the deployment and configuration efficiency of the Kafka cluster has become a difficult problem that needs to be solved urgently. Summary of the Invention

[0004] The present application provides a Kafka cluster deployment method to address the defects of low efficiency and high error rate in the deployment and configuration process of the Kafka cluster in the prior art.

[0005] This application provides a Kafka cluster deployment method, including:

[0006] Get the deployment configuration information of the Kafka cluster entered by the user;

[0007] Determine a connection configuration selection result of the Zookeeper cluster; based on the connection configuration selection result, obtain deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster;

[0008] A corresponding template file is generated based on the deployment configuration information, and deployment operations of the Kafka cluster and the Zookeeper cluster are performed according to the deployment tasks in the template file based on a preset configuration management tool.

[0009] Furthermore, the step of obtaining the deployment configuration information of the Kafka cluster input by the user specifically includes:

[0010] Based on the interactive query method, obtain the deployment configuration information of the Kafka cluster entered by the user based on the first query configuration item;

[0011] The method of obtaining the deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster based on the connection configuration selection result specifically includes: if the connection configuration selection result is to select to deploy and use a new Zookeeper cluster, then based on an interactive query method, obtaining the deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item.

[0012] Furthermore, the interactive inquiry method is used to obtain the deployment configuration information of the Kafka cluster input by the user based on the first query configuration item, specifically including:

[0013] Based on the interactive inquiry method, the number of nodes of the Kafka cluster input by the user based on the first node number query configuration item, the deployment location of the Kafka cluster input by the user based on the first deployment location query configuration item, the data storage location of the Kafka cluster input by the user based on the first data storage location query configuration item, and the listening port number of the Kafka cluster input by the user based on the first port number query configuration item are obtained;

[0014] The interactive query method is used to obtain the deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item, specifically including:

[0015] Based on the interactive query method, the number of nodes in the Zookeeper cluster input by the user based on the second node number query configuration item is obtained, the deployment location of the Zookeeper cluster input by the user based on the second deployment location query configuration item is obtained, the data storage location of the Zookeeper cluster input by the user based on the second data storage location query configuration item is obtained, and the listening port number of the Zookeeper cluster input by the user based on the second port number query configuration item is obtained.

[0016] Furthermore, generating a corresponding template file based on the deployment configuration information specifically includes generating a corresponding Playbook template file based on the deployment configuration information of the Kafka cluster, the deployment configuration information of the Zookeeper cluster and a preset standard template format.

[0017] Furthermore, the preset configuration management tool executes the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file, specifically including: executing the deployment operations of the Kafka cluster and the Zookeeper cluster respectively according to the deployment tasks in the Playbook template file based on the Ansible tool.

[0018] Furthermore, before executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on the preset configuration management tool, it also includes: establishing authentication connections between the server nodes of the Kafka cluster and the server nodes of the Zookeeper cluster based on asymmetric encryption keys.

[0019] Furthermore, after executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on the preset configuration management tool, it also includes: real-time monitoring of the deployment operation process of the Kafka cluster and the Zookeeper cluster, sending the deployment operation process and its monitoring results to the display terminal, and judging whether there is any abnormality in the deployment operation process based on the monitoring results. If so, determining the corresponding abnormal execution steps.

[0020] This application also provides a Kafka cluster deployment device, including:

[0021] A first deployment configuration information acquisition unit is used to acquire the deployment configuration information of the Kafka cluster input by the user;

[0022] A second deployment configuration information acquisition unit is configured to determine a connection configuration selection result of the Zookeeper cluster; based on the connection configuration selection result, acquire deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster;

[0023] A deployment management unit is used to generate a corresponding template file based on the deployment configuration information, and execute the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on a preset configuration management tool.

[0024] Furthermore, the first deployment configuration information obtaining unit is specifically configured to:

[0025] Based on the interactive query method, obtain the deployment configuration information of the Kafka cluster entered by the user based on the first query configuration item;

[0026] The second deployment configuration information acquiring unit is specifically configured to:

[0027] If the connection configuration selection result is to select to deploy and use a new Zookeeper cluster, then based on the interactive inquiry method, the deployment configuration information of the Zookeeper cluster input by the user based on the second inquiry configuration item is obtained.

[0028] Furthermore, the interactive inquiry method is used to obtain the deployment configuration information of the Kafka cluster input by the user based on the first query configuration item, specifically including:

[0029] Based on the interactive inquiry method, the number of nodes of the Kafka cluster input by the user based on the first node number query configuration item, the deployment location of the Kafka cluster input by the user based on the first deployment location query configuration item, the data storage location of the Kafka cluster input by the user based on the first data storage location query configuration item, and the listening port number of the Kafka cluster input by the user based on the first port number query configuration item are obtained;

[0030] The interactive query method is used to obtain the deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item, specifically including:

[0031] Based on the interactive query method, the number of nodes in the Zookeeper cluster input by the user based on the second node number query configuration item is obtained, the deployment location of the Zookeeper cluster input by the user based on the second deployment location query configuration item is obtained, the data storage location of the Zookeeper cluster input by the user based on the second data storage location query configuration item is obtained, and the listening port number of the Zookeeper cluster input by the user based on the second port number query configuration item is obtained.

[0032] Furthermore, the deployment management unit is specifically used to generate a corresponding Playbook template file based on the deployment configuration information of the Kafka cluster, the deployment configuration information of the Zookeeper cluster and a preset standard template format.

[0033] Furthermore, the deployment management unit is specifically used to: execute the deployment operations of the Kafka cluster and the Zookeeper cluster respectively according to the deployment tasks in the Playbook template file based on the Ansible tool.

[0034] Furthermore, before executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on the preset configuration management tool, it also includes: an authentication connection unit, which is used to establish an authentication connection between the server node of the Kafka cluster and the server node of the Zookeeper cluster based on an asymmetric encryption key.

[0035] Furthermore, after executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on the preset configuration management tool, it also includes: a process display and exception determination unit, which is used to monitor the deployment operation process of the Kafka cluster and the Zookeeper cluster in real time, send the deployment operation process and its monitoring results to the display terminal, and judge whether there is an exception in the deployment operation process based on the monitoring results. If so, determine the corresponding exception execution steps.

[0036] The present application also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute any of the above-mentioned Kafka cluster deployment methods through the computer program.

[0037] The present application also provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, it executes and implements any of the above-mentioned Kafka cluster deployment methods.

[0038] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-described Kafka cluster deployment methods.

[0039] The Kafka cluster deployment method provided in this application first obtains the deployment configuration information of the Kafka cluster input by the user; determines the connection configuration selection result of the Zookeeper cluster, and based on the connection configuration selection result, obtains the deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster; then generates a corresponding template file based on the deployment configuration information, and executes the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on a preset configuration management tool. This can effectively improve the deployment efficiency of the Kafka cluster, save labor costs, reduce error rates, and thus improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 This is a flow chart of the Kafka cluster deployment method provided in this application;

[0043] Figure 2 This is a specific application diagram of the Kafka cluster deployment method provided by this application;

[0044] Figure 3 This is a schematic diagram of the structure of the deployment device of the Kafka cluster provided by this application;

[0045] Figure 4 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] The following describes in detail the implementation of the Kafka cluster deployment method described in this application. Figure 1As shown in FIG, it is a flow chart of the deployment method of the Kafka cluster provided by this application, and the specific implementation process includes the following steps:

[0049] Step 101: Obtain the deployment configuration information of the Kafka cluster input by the user.

[0050] Kafka is a distributed message queue based on a publish / subscribe model. First, Kafka partitions received messages, with each topic's messages having a different partition. This ensures that message storage is not limited by the storage space of a single server node, while also allowing message processing to be parallelized across multiple server nodes. Secondly, to ensure high availability, each partition has a certain number of replicas. This way, if some server nodes become unavailable, the server nodes with the replicas will take over, ensuring application continuity.

[0051] During the specific implementation of this step, the deployment configuration information of the Kafka cluster input by the user based on the first query configuration item can be obtained through an interactive inquiry method. Specifically, obtaining the deployment configuration information of the Kafka cluster input by the user based on the first query configuration item through an interactive inquiry method may include: based on the interactive inquiry method, obtaining the number of nodes of the Kafka cluster input by the user based on the first node number query configuration item, obtaining the deployment location of the Kafka cluster input by the user based on the first deployment location query configuration item, obtaining the data storage location of the Kafka cluster input by the user based on the first data storage location query configuration item, and obtaining the listening port number of the Kafka cluster input by the user based on the first port number query configuration item, etc.

[0052] In an embodiment of the present invention, the parameters involved in the query configuration items include but are not limited to the number of nodes of the Kafka cluster and Zookeeper cluster, the listening port number, the deployment location (IP address of the server node), the data storage location and other related parameters. In order to improve the speed of automated deployment, these parameters are configured by default. If you are not satisfied with some of the default configurations, you can also modify them according to actual needs according to the prompts.

[0053] Specifically, the parameter configuration process based on the query configuration item is as follows: The Kafka cluster deployment version selection process includes: the query configuration item corresponding to the deployment version can be "Please select the Kafka software package you want to deploy:", and the user enters the complete path of the installation package after the colon based on the query configuration item, such as: / home / kafka_2.13-3.2.1.tgz.

[0054] The Kafka cluster deployment configuration query selection process includes: the first software package query configuration item can be "Please select the Kafka software package you want to deploy:", the user enters the complete path of the installation package after the colon based on the first data storage location query configuration item, such as: / home / kafka_2.13-3.2.1.tgz; the first node quantity query configuration item can be "Number of Kafka cluster nodes:", the user can enter a specific number after the colon based on the query configuration item, such as: 3; the first IP address query configuration item can be "Please enter the IP address of Kafka node 1:", the user can enter the IP address based on the first IP address configuration item, such as: 192.168.0.1; the first port number query configuration item can be "Please enter the listening port of Kafka node 1 (default 9092):", which can be directly used based on the first port number query configuration item. Press Enter to use the default port or enter a custom port. The port number you enter must be greater than 6000 and less than 10000. If you enter a port outside this range, you will be prompted with the message "The port number you entered is out of range (6000-10000)." The first deployment location query configuration item can be "Please enter the Kafka installation path for Kafka node 1 (default: / apps / kafka-node01):." Based on this first deployment location query configuration item, you can directly press Enter to use the default directory or enter a custom directory as needed. The first data storage location query configuration item can be "Please enter the data log storage location for Kafka node 1 (default: / tmp / kafka-logs):." Based on this first data storage location query configuration item, you can directly press Enter to use the default directory or enter a custom directory as needed. Repeat this process until you have entered the deployment configuration information for all nodes.

[0055] Step 102: Determine the connection configuration selection result of the Zookeeper cluster; based on the connection configuration selection result, obtain the deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster.

[0056] ZooKeeper is a distributed, open source distributed application coordination service. It is a software that provides consistency services for distributed applications. Its functions include: configuration maintenance, domain name services, distributed synchronization, group services, etc.

[0057] In the embodiment of the present application, preparing multiple server nodes to build a Kafka cluster requires a Zookeeper cluster for distributed coordination, so you must first prepare the Zookeeper cluster. For example, you can refer to Building a Zookeeper Cluster on Linux.

[0058] During the specific implementation of this step, the deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item can be obtained through an interactive query method. Specifically, obtaining the deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item through an interactive query method may include: obtaining the number of nodes of the Zookeeper cluster input by the user based on the second node number query configuration item, obtaining the deployment location of the Zookeeper cluster input by the user based on the second deployment location query configuration item, obtaining the data storage location of the Zookeeper cluster input by the user based on the second data storage location query configuration item, and obtaining the listening port number of the Zookeeper cluster input by the user based on the second port number query configuration item, etc.

[0059] In an embodiment of the present invention, the parameters involved in the query configuration items include but are not limited to the number of nodes of the Kafka cluster and Zookeeper cluster, the listening port number, the deployment location (IP address of the server node), the data storage location and other related parameters. In order to improve the speed of automated deployment, these parameters are configured by default. If you are not satisfied with some of the default configurations, you can also modify them according to actual needs according to the prompts.

[0060] Specifically, the Zookeeper cluster connection configuration query process includes the following: The second cluster selection configuration item can be "Use an existing Zookeeper cluster (yes / no)?". If you choose to use an existing Zookeeper cluster, enter "yes." The second data storage location query configuration item can be "Please enter the Zookeeper cluster connection address." Based on this second data storage location query configuration item, the user simply enters an available Zookeeper cluster connection address after the colon. For example: Zookeeper01:2181, Zookeeper 02:2181, Zookeeper 03:2181. Multiple addresses are separated by commas. If you choose to deploy and use a new Zookeeper cluster, enter "no" in the first query. The Zookeeper cluster deployment configuration query will then proceed.

[0061] The Zookeeper cluster deployment configuration query process includes: the second node number query configuration item can be "Zookeeper cluster node number:", based on the second node number query configuration item, the user can enter a specific number after the colon, such as: 3; the second IP address query configuration item "Please enter the IP address of Zookeeper node 1:", the user enters the IP address; the second port number query configuration item can be "Please enter the listening port of Zookeeper node 1 (default 2181):", based on the second port number query configuration item, the user can directly enter the Enter key to use the default You can also enter a custom port number. The port number you enter must be greater than 2000 and less than 4000. If you enter a port outside this range, you will be prompted with the message "The port number you entered is out of range (2000-4000)." The second deployment location query configuration item can be "Please enter the installation path of Zookeeper node 1 (default: / apps / zookeeper-node01):." Based on this second deployment location query configuration item, the user can directly press Enter to use the default directory, or enter a custom directory as needed. Repeat this process until the deployment configuration information for all nodes is entered.

[0062] Other query configuration items are not listed here one by one, and can be configured according to actual needs. It should be noted that during the query configuration process, the configured parameters cannot be changed. If you want to change them, you can only exit the deployment tool (configuration management tool) and re-execute the deployment tool to configure them. Figure 2 As shown, after the above-mentioned query configuration task is completed, the deployment tool will generate an Ansible Playbook template file based on the input configuration information, and then use the Ansible deployment tool (i.e., the deployment tool) to remotely execute the installation and configuration commands according to the tasks in the Playbook template file to complete the deployment. Kafka and Zookeeper are usually installed using the official binary packages. After the installation is completed, the relevant parameters in the configuration file need to be modified according to the cluster configuration. Through the deployment tool described in this application, the relevant installation and configuration operations do not need to be manually performed on each server node. The deployment instructions will be executed through the preset steps of the Playbook template file in the Ansible tool, and you only need to wait for the set deployment to be completed. As a result, this application improves the efficiency of deploying replica sets in operating systems (such as CentOS). It should be noted that the configuration item settings of the deployment tool for Kafka and Zookeeper clusters, except for the configuration items defined in the query step, all use the pre-recommended default configurations. If the user has higher requirements for certain configurations, they can also be modified according to the above process.

[0063] This application explains the complex and tedious Kafka cluster deployment process by providing a few simple questions and answers to obtain cluster architecture and deployment configuration information. This information is then integrated into a Playbook template file, which is then deployed quickly and easily using the Ansible deployment tool. The Playbook template file in this application is a template that has been verified through multiple practical applications. The Playbook template file provides a unified layout of the cluster deployment process, helping users better understand the cluster deployment process and the writing standards of the Playbook template file.

[0064] Step 103: Generate a corresponding template file based on the deployment configuration information, and execute the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on a preset configuration management tool.

[0065] Wherein, a corresponding template file is generated based on the deployment configuration information, and the corresponding specific implementation process includes: generating a corresponding Playbook template file based on the deployment configuration information of the Kafka cluster, the deployment configuration information of the Zookeeper cluster and the preset standard template format. The preset configuration management tool executes the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file, and the corresponding specific implementation process includes: executing the deployment operations of the Kafka cluster and the Zookeeper cluster respectively according to the deployment tasks in the Playbook template file based on the Ansible tool. Wherein, Ansible is a newly emerged automated operation and maintenance tool, developed based on Python, which combines the advantages of many operation and maintenance tools (puppet, cfengine, chef, func, fabric), and can realize batch system configuration, batch program deployment, batch command execution and other functions. Playbook is Ansible's script for configuring, deploying, and managing nodes. Through the detailed description of Playbook, executing a series of tasks therein can make the remote host reach the expected state. Playbook is like a series of to-do-lists listed by the Ansible controller for the controlled node, and the controlled node must complete them. Ansible is an open source tool, and its specific working principle and working method are beyond the scope of this application. Among them, the deployment configuration information includes the deployment configuration information of the Kafka cluster and the deployment configuration information of the Zookeeper cluster used to provide coordination services for the Kafka cluster.

[0066] It should be noted that before executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on the preset configuration management tool, it is necessary to pre-establish authentication connections with the server nodes of the Kafka cluster and the server nodes of the Zookeeper cluster based on asymmetric encryption keys. For example, before executing the deployment tool, it is necessary to ensure that each deployed server node can be remotely connected through the management host using SSH keys. Each server node needs to have Python version 2.4 or higher installed. If it is lower than Python 2.5, pythonsimplejson needs to be installed; if selinux is enabled, libselinux-python needs to be installed.

[0067] In addition, after executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on the preset configuration management tool, in order to avoid the entire deployment process from being wasted due to anomalies in certain sub-steps during the deployment process, the technical solution disclosed in the present application also includes: real-time monitoring of the deployment operation process of the Kafka cluster and the Zookeeper cluster, sending the deployment operation process and its monitoring results to a display terminal, and judging whether the deployment operation process has an anomaly based on the monitoring results, if so, determining the corresponding abnormal execution step. By performing real-time monitoring during the execution process, the execution process and result records are displayed on the screen, judging whether the execution process of the replica set deployment operation has an anomaly based on the real-time monitoring results, if so, determining the corresponding abnormal execution step and the reason, for the user to troubleshoot and solve the error. When the cluster is successfully deployed, the current status of the cluster and the number of nodes will be printed on the screen for the user to check. If the deployment tool does not complete the automatic deployment of the cluster normally, the deployment tool needs to be re-executed regardless of which step the deployment tool exits from.

[0068] The purpose of this application is to query the configuration of the Kafka cluster and Zookeeper cluster, and finally combine the collected deployment configuration information with the Ansible deployment tool to execute it on each server node, ultimately achieving the purpose of automatic cluster deployment. It solves the tedious manual deployment process of installing and deploying the Kafka cluster in the Linux system, generates corresponding deployment instructions based on the Kafka version and deployment configuration information, and performs corresponding deployment in multiple CentOS operating systems, thereby achieving the effect of one-click cluster deployment, greatly improving work efficiency, saving labor costs, and reducing error rates. This application allows operation and maintenance personnel or beginners to flexibly and quickly deploy Kafka clusters, and is suitable for development, testing and online production environments.

[0069] The Kafka cluster deployment method provided in this application first obtains the deployment configuration information of the Kafka cluster input by the user; determines the connection configuration selection result of the Zookeeper cluster, and based on the connection configuration selection result, obtains the deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster; then generates a corresponding template file based on the deployment configuration information, and executes the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on a preset configuration management tool. This can effectively improve the deployment efficiency of the Kafka cluster, save labor costs, reduce error rates, and thus improve the user experience.

[0070] The deployment device of the Kafka cluster provided in this application is described below. The deployment device of the Kafka cluster described below and the deployment method of the Kafka cluster described above can refer to each other.

[0071] refer to Figure 3 As shown, it is a structural diagram of the deployment device of the Kafka cluster provided by this application.

[0072] The deployment device of the Kafka cluster described in this application specifically includes the following parts:

[0073] The first deployment configuration information acquisition unit 301 is used to acquire the deployment configuration information of the Kafka cluster input by the user;

[0074] The second deployment configuration information acquisition unit 302 is used to determine the connection configuration selection result of the Zookeeper cluster; based on the connection configuration selection result, acquire the deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster;

[0075] The deployment management unit 303 is configured to generate a corresponding template file based on the deployment configuration information, and execute deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on a preset configuration management tool.

[0076] Furthermore, the first deployment configuration information acquiring unit is specifically configured to:

[0077] Based on the interactive query method, obtain the deployment configuration information of the Kafka cluster entered by the user based on the first query configuration item;

[0078] The second deployment configuration information acquiring unit is specifically configured to:

[0079] If the connection configuration selection result is to select to deploy and use a new Zookeeper cluster, then based on the interactive inquiry method, the deployment configuration information of the Zookeeper cluster input by the user based on the second inquiry configuration item is obtained.

[0080] Furthermore, the interactive inquiry method is used to obtain the deployment configuration information of the Kafka cluster input by the user based on the first query configuration item, specifically including:

[0081] Based on the interactive inquiry method, the number of nodes of the Kafka cluster input by the user based on the first node number query configuration item, the deployment location of the Kafka cluster input by the user based on the first deployment location query configuration item, the data storage location of the Kafka cluster input by the user based on the first data storage location query configuration item, and the listening port number of the Kafka cluster input by the user based on the first port number query configuration item are obtained;

[0082] The interactive query method is used to obtain the deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item, specifically including:

[0083] Based on the interactive query method, the number of nodes in the Zookeeper cluster input by the user based on the second node number query configuration item is obtained, the deployment location of the Zookeeper cluster input by the user based on the second deployment location query configuration item is obtained, the data storage location of the Zookeeper cluster input by the user based on the second data storage location query configuration item is obtained, and the listening port number of the Zookeeper cluster input by the user based on the second port number query configuration item is obtained.

[0084] Furthermore, the deployment management unit is specifically used to generate a corresponding Playbook template file based on the deployment configuration information of the Kafka cluster, the deployment configuration information of the Zookeeper cluster and a preset standard template format.

[0085] Furthermore, the deployment management unit is specifically used to: execute the deployment operations of the Kafka cluster and the Zookeeper cluster respectively according to the deployment tasks in the Playbook template file based on the Ansible tool.

[0086] Furthermore, before executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on the preset configuration management tool, it also includes: an authentication connection unit, which is used to establish an authentication connection between the server node of the Kafka cluster and the server node of the Zookeeper cluster based on an asymmetric encryption key.

[0087] Furthermore, after executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on the preset configuration management tool, it also includes: a process display and exception determination unit, which is used to monitor the deployment operation process of the Kafka cluster and the Zookeeper cluster in real time, send the deployment operation process and its monitoring results to the display terminal, and judge whether there is an exception in the deployment operation process based on the monitoring results. If so, determine the corresponding exception execution steps.

[0088] The Kafka cluster deployment device provided in this application first obtains the deployment configuration information of the Kafka cluster input by the user; determines the connection configuration selection result of the Zookeeper cluster, and based on the connection configuration selection result, obtains the deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster; then generates a corresponding template file based on the deployment configuration information, and executes the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on a preset configuration management tool. This can effectively improve the deployment efficiency of the Kafka cluster, save labor costs, reduce error rates, and thus enhance the user experience.

[0089] Figure 4 The following is a schematic diagram of the physical structure of an electronic device. Figure 4 As shown, the electronic device may include: a processor 401, a communications interface 404, a memory 402, and a communication bus 403, wherein the processor 401, the communications interface 404, and the memory 402 communicate with each other via the communication bus 403. The processor 401 may call the logic instructions in the memory 402 to execute a Kafka cluster deployment method, which includes: obtaining deployment configuration information of the Kafka cluster input by a user; determining a connection configuration selection result of the Zookeeper cluster; based on the connection configuration selection result, obtaining deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster; generating a corresponding template file based on the deployment configuration information, and executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on a preset configuration management tool.

[0090] In addition, the logic instructions in the above-mentioned memory 402 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0091] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the Kafka cluster deployment method provided by the above methods, the method including: obtaining the deployment configuration information of the Kafka cluster input by the user; determining the connection configuration selection result of the Zookeeper cluster; based on the connection configuration selection result, obtaining the deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster; generating a corresponding template file based on the deployment configuration information, and executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on a preset configuration management tool.

[0092] On the other hand, the present application also provides a computer-readable storage medium, which includes a stored program, wherein when the program is run, the deployment method of the Kafka cluster provided by the above methods is executed, the method including: obtaining the deployment configuration information of the Kafka cluster input by the user; determining the connection configuration selection result of the Zookeeper cluster; based on the connection configuration selection result, obtaining the deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster; generating a corresponding template file based on the deployment configuration information, and executing the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on a preset configuration management tool.

[0093] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0094] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A Kafka cluster deployment method, characterized in that: include: Get the deployment configuration information of the Kafka cluster entered by the user; Determine the connection configuration selection result of the Zookeeper cluster; Based on the connection configuration selection result, obtaining deployment configuration information of a Zookeeper cluster input by a user for providing coordination services for the Kafka cluster; the connection configuration selection result includes using an existing Zookeeper cluster or using a new Zookeeper cluster; Generate a corresponding template file based on the deployment configuration information, and execute the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on a preset configuration management tool; The acquisition of the Kafka cluster deployment configuration information input by the user specifically includes: Based on the interactive query method, obtain the deployment configuration information of the Kafka cluster entered by the user based on the first query configuration item; The method of obtaining the deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster based on the connection configuration selection result specifically includes: if the connection configuration selection result is to select to deploy and use a new Zookeeper cluster, then based on an interactive query method, obtaining the deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item.

2. The Kafka cluster deployment method according to claim 1, characterized in that: The interactive inquiry method is used to obtain the deployment configuration information of the Kafka cluster input by the user based on the first query configuration item, specifically including: Based on the interactive inquiry method, the number of nodes of the Kafka cluster input by the user based on the first node number query configuration item, the deployment location of the Kafka cluster input by the user based on the first deployment location query configuration item, the data storage location of the Kafka cluster input by the user based on the first data storage location query configuration item, and the listening port number of the Kafka cluster input by the user based on the first port number query configuration item are obtained; The interactive query method is used to obtain the deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item, specifically including: Based on the interactive query method, the number of nodes in the Zookeeper cluster input by the user based on the second node number query configuration item is obtained, the deployment location of the Zookeeper cluster input by the user based on the second deployment location query configuration item is obtained, the data storage location of the Zookeeper cluster input by the user based on the second data storage location query configuration item is obtained, and the listening port number of the Zookeeper cluster input by the user based on the second port number query configuration item is obtained.

3. The deployment method of the Kafka cluster according to claim 1, characterized in that: Generating a corresponding template file based on the deployment configuration information specifically includes: Generate a corresponding Playbook template file based on the deployment configuration information of the Kafka cluster, the deployment configuration information of the Zookeeper cluster and a preset standard template format.

4. The deployment method of the Kafka cluster according to claim 3, characterized in that: The preset configuration management tool executes the deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file, specifically including: Based on the Ansible tool, the deployment operations of the Kafka cluster and the Zookeeper cluster are respectively performed according to the deployment tasks in the Playbook template file.

5. The Kafka cluster deployment method according to claim 1, characterized in that: Before executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on the preset configuration management tool, the method further includes: Based on the asymmetric encryption key, authentication connections are established with the server nodes of the Kafka cluster and the server nodes of the Zookeeper cluster respectively.

6. The method for deploying a Kafka cluster according to claim 1, wherein: After executing the deployment operation of the Kafka cluster and the Zookeeper cluster according to the deployment task in the template file based on the preset configuration management tool, the method further includes: The deployment operation process of the Kafka cluster and the Zookeeper cluster is monitored in real time, the deployment operation process and its monitoring results are sent to the display terminal, and it is determined whether there is an abnormality in the deployment operation process based on the monitoring results. If so, the corresponding abnormal execution steps are determined.

7. A Kafka cluster deployment device, characterized in that: include: A first deployment configuration information acquisition unit is used to acquire the deployment configuration information of the Kafka cluster input by the user; The second deployment configuration information acquisition unit is used to determine the connection configuration selection result of the Zookeeper cluster; Based on the connection configuration selection result, obtaining deployment configuration information of a Zookeeper cluster input by a user for providing coordination services for the Kafka cluster; the connection configuration selection result includes using an existing Zookeeper cluster or using a new Zookeeper cluster; A deployment management unit, configured to generate a corresponding template file based on the deployment configuration information, and execute deployment operations of the Kafka cluster and the Zookeeper cluster according to the deployment tasks in the template file based on a preset configuration management tool; The acquisition of the Kafka cluster deployment configuration information input by the user specifically includes: Based on the interactive query method, obtain the deployment configuration information of the Kafka cluster entered by the user based on the first query configuration item; The method of obtaining the deployment configuration information of the Zookeeper cluster input by the user for providing coordination services for the Kafka cluster based on the connection configuration selection result specifically includes: if the connection configuration selection result is to select to deploy and use a new Zookeeper cluster, then based on an interactive query method, obtaining the deployment configuration information of the Zookeeper cluster input by the user based on the second query configuration item.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the Kafka cluster deployment method according to any one of claims 1 to 6 when running.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the Kafka cluster deployment method according to any one of claims 1 to 6 through the computer program.

Images