ClickHouse cluster management methods and related equipment

Abstract

The present invention relates to the field of cluster management technology, and provides a ClickHouse cluster management method and related equipment. The ClickHouse cluster management method is applied to a ClickHouse cluster management platform, including: deploying a ClickHouse cluster including ClickHouse nodes and Zookeeper nodes to a container orchestration engine environment; collecting all node information in the container orchestration engine environment, synchronizing to the database of the ClickHouse cluster management platform, and binding the administrator account of the ClickHouse cluster to a query platform integrated in the ClickHouse cluster management platform; and linking the ClickHouse cluster with a big data platform to perform big data management on the ClickHouse cluster. The present invention combines the container orchestration engine Kubernetes and the ClickHouse cluster management platform to realize automatic deployment and expansion of the ClickHouse cluster, reduce operation and maintenance management costs, and connect the query platform and the big data platform to realize a fully functional online analytical processing OLAP system based on ClickHouse.

Description

Technical Field

[0001] The present invention relates to the technical field of cluster management, and in particular to a ClickHouse cluster management method and related equipment. Background Art

[0002] ClickHouse is a column-based database management system for online analytical processing (OLAP). Currently, ClickHouse has been widely used in various fields of OLAP, and cluster access requests are also increasing.

[0003] However, the use and operation of ClickHouse clusters have certain thresholds, and require a certain understanding of sharding, nodes, data stratification, etc. The deployment and operation of existing ClickHouse clusters are relatively complex, and the established OLAP platforms generally have problems such as poor real-time performance, inconsistent synchronization tools, and inconsistent data isolation permissions.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] In view of this, the present invention provides a ClickHouse cluster management method and related equipment, which can combine the container orchestration engine (Kubernetes) and the ClickHouse cluster management platform to solve the problems of automatic deployment and expansion of the ClickHouse cluster, as well as the difficulties in operation and maintenance management, and realize an OLAP system based on ClickHouse.

[0006] According to one aspect of the present invention, there is provided a ClickHouse cluster management method, which is applied to a ClickHouse cluster management platform, including: deploying a ClickHouse cluster including ClickHouse nodes and Zookeeper nodes into a container orchestration engine environment; collecting all node information in the container orchestration engine environment, synchronizing it to a database of the ClickHouse cluster management platform, and binding the administrator account of the ClickHouse cluster to a query platform integrated in the ClickHouse cluster management platform; and linking the ClickHouse cluster with a big data platform to perform big data management on the ClickHouse cluster.

[0007] In some embodiments, before deploying the ClickHouse cluster including ClickHouse nodes and Zookeeper nodes into the container orchestration engine environment, it also includes: performing node configuration on the host resources in the container orchestration engine environment to generate a ClickHouse host with a ClickHouse label and a Zookeeper host with a Zookeeper label; in the ClickHouse cluster, the ClickHouse node is deployed on the ClickHouse host, and the Zookeeper node is deployed on the Zookeeper host.

[0008] In some embodiments, deploying a ClickHouse cluster comprising ClickHouse nodes and Zookeeper nodes into a container orchestration engine environment includes: deploying a ClickHouse generator into the container orchestration engine environment, and deploying the ClickHouse cluster through the ClickHouse cluster management platform and the description file of the ClickHouse generator.

[0009] In some embodiments, the deployment of the ClickHouse cluster through the description file of the ClickHouse cluster management platform and the ClickHouse generator includes: in response to a cluster application request, binding the working domain of the ClickHouse cluster; generating configuration information and account information of the ClickHouse cluster; setting resource occupancy information of the ClickHouse cluster; adding the container orchestration engine domain name and service name of the Zookeeper node to the description file; integrating the settings of all description files to generate resource description files corresponding to the ClickHouse node, the Zookeeper node, and the service node of the Zookeeper node respectively; submitting the resource description file to the container orchestration engine environment, generating the ClickHouse node and the Zookeeper node, and linking the domain name of the ClickHouse node to the target service node.

[0010] In some embodiments, binding the working domain of the ClickHouse cluster includes: binding the working domain according to the application identifier and the group identifier carried in the cluster application request, and the working domain covers several ClickHouse hosts and several Zookeeper hosts.

[0011] In some embodiments, the configuration information includes: user account and password configuration information, user authority configuration information, query configuration information, write configuration information, cluster storage configuration information and log storage configuration information; the account information includes an administrator account with full permissions, a read-write account with read and write permissions, and a read-only account with read-only permissions.

[0012] In some embodiments, the setting of resource occupancy information of the ClickHouse cluster includes: setting the resource occupancy information according to shard information and replica information, the resource occupancy information including the number of shards, the number of replicas, central processing unit occupancy information, memory occupancy information, storage space occupancy information and synchronization resource occupancy information.

[0013] In some embodiments, linking the domain name of the ClickHouse node to the target service node includes: submitting the domain name of the ClickHouse node to a gateway service deployed in the container orchestration engine environment, generating a domain name access service, and linking the domain name of the ClickHouse node to the target service node.

[0014] In some embodiments, the big data platform includes: a data transmission platform, a data authority platform, a data query tool, a data reporting platform and a data real-time computing platform.

[0015] In some embodiments, the big data management of the ClickHouse cluster includes one or more of the following: configuring the table structure of the ClickHouse cluster through the big data platform, and the table structure is at the same level as the index of the data table of the big data platform; configuring the library and table permissions of the ClickHouse cluster through the data permission platform; importing offline data in the data source into the ClickHouse cluster through the data transmission platform, and importing data in the ClickHouse cluster into the data source; importing real-time data into the ClickHouse cluster through the data real-time computing platform; querying and exporting data on the ClickHouse cluster through the data query tool; configuring data reports of the ClickHouse cluster through the data reporting platform.

[0016] In some embodiments, the ClickHouse cluster management method also includes: performing operation and maintenance management on the ClickHouse cluster through the ClickHouse cluster management platform; wherein, performing operation and maintenance management on the ClickHouse cluster includes one or more of the following: displaying all ClickHouse clusters for search through the cluster list page of the ClickHouse cluster management platform; displaying all detailed information of a target ClickHouse cluster through the cluster details page of the ClickHouse cluster management platform; modifying the configuration of the target ClickHouse cluster in response to a cluster change request issued from the cluster details page; and performing operation and maintenance management on the target ClickHouse cluster according to the number of replicas and backups carried by the cluster expansion request in response to a cluster expansion request issued from the cluster details page. Expand the capacity of the ClickHouse cluster; perform permission management on the target ClickHouse cluster in response to the permission management request issued from the cluster details page; perform library table management on the target ClickHouse cluster in response to the library table management request issued from the cluster details page; receive cluster application requests through the application cluster page of the ClickHouse cluster management platform; display approved cluster application requests and unapproved cluster application requests through the approval list page of the ClickHouse cluster management platform; manage each ClickHouse version through the version management page of the ClickHouse cluster management platform; query, manage and export the ClickHouse cluster associated with the administrator account and its node information in response to the query request issued by the administrator account from the query platform.

[0017] According to one aspect of the present invention, a ClickHouse cluster management platform is provided for implementing the ClickHouse cluster management method as described in any of the above embodiments, including: a cluster deployment module for deploying a ClickHouse cluster including ClickHouse nodes and Zookeeper nodes into a container orchestration engine environment; a data management module for collecting all node information in the container orchestration engine environment, synchronizing it to the database of the ClickHouse cluster management platform, and binding the administrator account of the ClickHouse cluster to a query platform integrated in the ClickHouse cluster management platform; and a platform interface module for linking the ClickHouse cluster with a big data platform to perform big data management on the ClickHouse cluster.

[0018] According to one aspect of the present invention, an online analytical processing system is provided, comprising a ClickHouse cluster management platform as described in any of the above embodiments.

[0019] According to one aspect of the present invention, there is provided a computer device, comprising: a processor; and a memory, wherein the memory stores executable instructions; wherein when the executable instructions are executed by the processor, the ClickHouse cluster management method as described in any of the above embodiments is implemented.

[0020] According to one aspect of the present invention, a computer-readable storage medium is provided for storing a program, wherein when the program is executed by a processor, the ClickHouse cluster management method as described in any of the above embodiments is implemented.

[0021] The beneficial effects of the present invention compared with the prior art include at least:

[0022] The present invention combines the container orchestration engine Kubernetes and the ClickHouse cluster management platform to realize the automatic deployment and expansion of the ClickHouse cluster, reduce the operation and maintenance management costs, and realize the user self-service management of the ClickHouse life cycle; combines the flexibility of Kubernetes with the powerful OLAP performance of Clickhouse, and connects the query platform and the big data platform to realize a full-featured OLAP system based on ClickHouse.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0025] Figure 1 A schematic diagram showing the steps of a ClickHouse cluster management method in one embodiment of the present invention is shown;

[0026] Figure 2 A schematic diagram showing the steps of the ClickHouse cluster deployment process in one embodiment of the present invention;

[0027] Figure 3 A schematic diagram showing the process of ClickHouse cluster deployment and retrieval in one embodiment of the present invention is shown;

[0028] Figure 4 A schematic diagram showing the architecture of an online analytical processing system according to an embodiment of the present invention is shown;

[0029] Figure 5 A functional module diagram of the ClickHouse cluster management platform in one embodiment of the present invention is shown;

[0030] Figure 6 A schematic diagram showing a page of a cluster details page in one embodiment of the present invention;

[0031] Figure 7 A schematic diagram of a module of a ClickHouse cluster management platform in one embodiment of the present invention is shown;

[0032] Figure 8 A schematic diagram showing the structure of a computer device in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein. Instead, these embodiments are provided to make the present invention comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art.

[0034] The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] In addition, the processes shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps can be decomposed, some steps can be combined or partially combined, and the actual execution order may change according to the actual situation. It should be noted that the embodiments of the present invention and the features in different embodiments can be combined with each other without conflict.

[0036] The ClickHouse cluster management method of the present invention is applied to the ClickHouse cluster management platform. The ClickHouse cluster management platform (hereinafter referred to as the CH-PaaS platform) is a user-visualized self-service ClickHouse cluster operation and maintenance management tool developed by the present invention, which is used to manage the ClickHouse cluster, and has online analytical processing (OLAP) functions such as data source access, data source export, and data query, and is linked with the big data platform to achieve data synchronization from the big data platform to the OLAP platform based on ClickHouse.

[0037] Figure 1 Shows the main steps of ClickHouse cluster management method, refer to Figure 1 As shown, in one embodiment, the ClickHouse cluster management method includes:

[0038] Step S110, deploy the ClickHouse cluster including ClickHouse nodes and Zookeeper nodes into the container orchestration engine environment.

[0039] ClickHouse is a columnar database management system for OLAP, with the characteristics of large data write throughput, horizontal expansion of distributed systems, high data compression ratio and low storage cost, low query rate per second (QPS) and fast query speed. The container orchestration engine (Kubernetes) is an open source system for automatically deploying, expanding and managing containerized applications, with the following features: Portable: Supports public cloud, private cloud, hybrid cloud and multiple clouds; Extensible: Modular, plug-in, mountable, composable, and supports various forms of expansion; Automation: automatic deployment, automatic restart, automatic replication, automatic scaling / extension, and powerful self-healing capabilities through declarative syntax.

[0040] ZooKeeper (distributed application coordination service) nodes can provide configuration maintenance, domain name services, distributed synchronization, group services, etc. Because ClickHouse needs to use Zookeeper for replica synchronization, it is necessary to configure Zookeeper nodes in the ClickHouse cluster.

[0041] Step S120, collect all node information in the container orchestration engine environment, synchronize it to the database of the ClickHouse cluster management platform, and bind the administrator account of the ClickHouse cluster to the query platform integrated in the ClickHouse cluster management platform.

[0042] Through the collection program, the CH-PaaS platform can collect information about all ClickHouse nodes and Zookeeper nodes in the Kubernetes environment, synchronize node updates with its own database, including Zookeeper node information to the MySQL (relational database management system) database and Qconfig configuration center, feed back node information to users, and bind the corresponding cluster administrator's account and node to the query platform (SQLPad), so that users can use SQLPad to query relevant data of the ClickHouse cluster.

[0043] SQLPad is an open source ClickHouse workbench tool that connects to SQL (Structured Query Language). It uses HTTP (Hyper Text Transfer Protocol) to connect, can execute and output SQL, and provides functions such as result format export and visual charts.

[0044] Step S130, link the ClickHouse cluster with the big data platform, and perform big data management on the ClickHouse cluster.

[0045] The above-mentioned ClickHouse cluster management method, combined with Kubernetes and CH-PaaS platform, realizes the automatic deployment and expansion of ClickHouse cluster, reduces the operation and maintenance management costs, and realizes the self-service management of ClickHouse life cycle users; combining the flexibility of Kubernetes with the powerful OLAP performance of Clickhouse, and connecting the query platform and the big data platform, it can realize a full-featured OLAP system based on ClickHouse.

[0046] In one embodiment, before deploying a ClickHouse cluster including ClickHouse nodes and Zookeeper nodes into a container orchestration engine environment, it also includes: performing node configuration on host resources in the container orchestration engine environment to generate a ClickHouse host with a ClickHouse label and a Zookeeper host with a Zookeeper label; in the ClickHouse cluster, the ClickHouse nodes are deployed on the ClickHouse host, and the Zookeeper nodes are deployed on the Zookeeper host.

[0047] Before deploying ClickHouse, you need to add the host resources to the Kubernetes environment, and add taints and labels to set up dedicated hosts when configuring Nodes, so that subsequent cluster resources can be attached to the correct host. The label of the ClickHouse host is set to "ck-host", and the label of the Zookeeper host is set to "zookeeper". In this embodiment, 80 ClickHouse hosts and 14 Zookeeper hosts are deployed in the data center.

[0048] In one embodiment, a ClickHouse cluster including ClickHouse nodes and Zookeeper nodes is deployed to a container orchestration engine environment, including: deploying a ClickHouse generator to the container orchestration engine environment, and deploying the ClickHouse cluster through a ClickHouse cluster management platform and a description file of the ClickHouse generator.

[0049] The ClickHouse generator, ClickHouse-Operator, is a custom resource type (Custom Resource Definition, CRD) configuration management tool based on Kubernetes. Deploy ClickHouse-Operator in the Kubernetes environment and use the description file (ClickHouse Installation, CHI) to deploy the specific ClickHouse cluster. ClickHouse-Operator connects and calls the Core API (core interface) of Kubernetes to implement resource calling and deployment. CHI is used to manage the ClickHouse cluster, which stipulates many configuration parameters about ClickHouse. This embodiment selects a part of it and embeds the configuration generation process into the CH-PaaS platform.

[0050] Figure 2 Shows the main steps of the ClickHouse cluster deployment process, refer to Figure 2 As shown, in one embodiment, the deployment of the ClickHouse cluster is performed through the description file of the ClickHouse cluster management platform and the ClickHouse generator, including:

[0051] S210, in response to the cluster application request, bind the working domain of the ClickHouse cluster.

[0052] Among them, binding the work domain of the ClickHouse cluster includes: binding the work domain according to the application ID and group ID carried in the cluster application request, and the work domain covers several ClickHouse hosts and several Zookeeper hosts. After the CH-PaaS platform receives the cluster application request submitted by the user and approved by the super administrator, it first binds the IDC (Internet Data Center) work domain according to the App id (application ID) and Group id (group ID), distinguishes the specific physical cluster by IDC, and generates the cluster name in the form of App id + IDC.

[0053] S220, generate configuration information and account information of the ClickHouse cluster.

[0054] Among them, configuration information includes Config configuration and User configuration, specifically including user account and password configuration information, user permission configuration information, query configuration information, write configuration information, cluster storage configuration information and log storage configuration information; account information includes three main accounts: an administrator account with full permissions (DDL account), a read-write account with read and write permissions (JDBC account), and a read-only account with read-only permissions (Read account).

[0055] S230, set the resource usage information of the ClickHouse cluster.

[0056] Among them, setting the resource occupancy information of the ClickHouse cluster includes: setting the resource occupancy information according to the shard information and replica information (provided by the administrator on the CH-PaaS platform), and the resource occupancy information includes the number of shards, the number of replicas, the CPU occupancy information, the memory occupancy information, the storage space occupancy information and the synchronization resource occupancy information.

[0057] S240: Add the domain name and service name of the container orchestration engine of the Zookeeper node to the description file.

[0058] The Kubernetes domain name and service name of the Zookeeper node are generated according to the code rules and added to CHI to ensure subsequent communication between the two.

[0059] S250, integrate the settings of all description files and generate resource description files corresponding to ClickHouse nodes, Zookeeper nodes, and service nodes of Zookeeper nodes respectively.

[0060] After integrating all CHI settings, they are synthesized into three YAML (a format for expressing data serialization) resource description files of Kubernetes, corresponding to ClickHouse, Zookeeper, and Zookeeper-svr, for further processing by the CH-PaaS platform. Among them, the key configuration of the ClickHouse cluster includes Freemarker template settings (parameters injected by the CH-PaaS platform).

[0061] S260, submit the resource description file to the container orchestration engine environment, generate ClickHouse nodes and Zookeeper nodes, and link the domain name of the ClickHouse node to the target service node.

[0062] Among them, linking the domain name of the ClickHouse node to the target service node includes: submitting the domain name of the ClickHouse node to the gateway service deployed in the container orchestration engine environment, generating a domain name access service, and linking the domain name of the ClickHouse node to the target service node.

[0063] The CH-PaaS platform submits three YAML resource description files to Kubernetes, generates ClickHouse nodes and Zookeeper nodes, and uses the domain name generated by the ClickHouse node to submit to the gateway service (istio-Ingress service) deployed in the Kubernetes environment. The istio-Ingress service forms a domain name access service through the Destination Rule, Gateway, and Virtual Service, linking the domain name of the ClickHouse node to the final target service node.

[0064] The operation principle and process of horizontal expansion of ClickHouse cluster are the same as the above cluster deployment principle and process, so they will not be repeated here.

[0065] Figure 3 Shows the main process of ClickHouse cluster deployment and retrieval, including initial deployment and subsequent cluster status detection. Figure 3 As shown, in one embodiment, deploying and retrieving a ClickHouse cluster through the CH-PaaS platform includes:

[0066] S310, bind the IDC work domain according to the application identifier App id and the group identifier Group id. At this time, the domain name has been provided but is unreadable. In this step, if any parameter is abnormal, the process ends.

[0067] S320, run the description file CHI based on the CH-PaaS platform. Specifically includes: call the isito document → read the Big Data Interface → read the PVC resource object → run the description file CHI → write to the database MySQL → apply for deployment. If the description file CHI is not started normally, the process ends. If the operation is successful, continue with the subsequent process.

[0068] S330, based on the Zookeeper node information of the CH-PaaS platform. Specifically including: starting the task → detecting whether CHI has been started → if so, detecting whether the deployment point pod is running → if so, detecting whether the IP (Internet Protocol) of the working domain is completed. If it is, the node information is synchronized to the MySQL database, and the domain name is readable at this time. If the IP is not completed, return to restart the task (you can wait for a preset time). In addition, in the step of detecting whether CHI has been started, if CHI has not been started, and in the step of detecting whether the deployment point pod is running, if the pod is not running, it also returns to restart the task. If step S330 runs successfully, continue with the subsequent process. If it fails / is abnormal, end the process.

[0069] S340, synchronize IP information based on CH-PaaS platform. Specifically includes: start task → run MySQL node → configure node IP. If successful, the process ends and IP is writable. If failed, it returns to restart the task.

[0070] In one embodiment, the big data platform includes: a data transmission platform (DTC), which is used for task management and distribution of data processing; a data permission platform (DSC), which is used for querying data tables and applying for and approving development permissions; a data query tool (Adhoc), which is a query tool of the big data platform and mainly connects to the Hive data source; a data reporting platform (Artnova), which can generate reports based on different data sources; and a data real-time computing platform (Muise), which is used for forwarding, synchronization and preprocessing of data.

[0071] Through the collection program and development interface of the CH-PaaS platform, big data systems such as DTC, DSC, Adhoc, ArtNova, and Muise can be linked with the ClickHouse cluster managed by the CH-PaaS platform to achieve cluster Metadata management, data synchronization, data query, permission authentication and other functions, and realize an OLAP system based on ClickHouse.

[0072] Figure 4 Shows the main architecture of the online analytical processing system, refer to Figure 4As shown, in one embodiment, the OLAP system 400 includes:

[0073] The container orchestration engine Kubernetes 410 is configured with ClickHouse-Operator, ClickHouse nodes and Zookeeper nodes. The container orchestration engine 410 is linked with the stream processing platform Kafka420, and is linked with the Hive data source 430 and multiple big data platforms. The Hive data source 430 is linked with the original databases such as the MySQL database 440. The big data platform includes: data query tool Adhoc 450a, data transmission platform DTC 450b, data permission platform DSC 450c, metadata management platform Metadata 450d, data reporting platform Artnova 450e, visualization tool Kibana / Grafana 450f, etc. The container orchestration engine 410 is also linked with the CH-PaaS platform 460 to realize the OLAP system 400 based on ClickHouse.

[0074] In one embodiment, big data management is performed on a ClickHouse cluster, including one or more of the following:

[0075] Through the big data platform, the table structure of the ClickHouse cluster is configured. The table structure and the index of the data table of the big data platform are at the same level. By creating tables through the big data platform Metadata, users can configure the ClickHouse table structure on the big data platform, which is at the same level as the index of various tables in the big data, making management easier.

[0076] Configure the library and table permissions of the ClickHouse cluster through the data permission platform; users can apply for various ClickHouse library and table permissions on the DSC platform.

[0077] Through the data transmission platform, the offline data in the data source is imported into the ClickHouse cluster, and the data in the ClickHouse cluster is imported into the data source; after passing the authority authentication, the user can configure the Muise task through the DTC platform to import offline data such as Hive (that is, data that is not written in real time) into ClickHouse. At the same time, the data in ClickHouse can also be imported into data sources such as Hive to achieve connectivity with the big data platform.

[0078] Import real-time data into the ClickHouse cluster through the data real-time computing platform; after passing the authority authentication, users can configure Flink tasks through the Muise real-time computing platform, use Flink-jar tasks, or directly configure Flink-SQL to import data into ClickHouse.

[0079] Use the data query tool to query and export data on the ClickHouse cluster; after applying for query permissions, users can select existing clusters for SQL queries in the adhoc query interface of the big data platform, or export part of the data for analysis.

[0080] Configure data reports for the ClickHouse cluster through the data reporting platform; after applying for query permissions, users can configure data reports for the ClickHouse data source on the Artnova platform.

[0081] Figure 5 Shows the functional modules of the ClickHouse cluster management platform, refer to Figure 5 As shown, in one embodiment, the CH-PaaS platform 500 includes a cluster management module 510, a query platform SQL Pad 520, a metadata management module 530, a task management module 540, a version management module 550, and a user management module 560. The cluster management module 510 may include functional modules such as cluster deployment, cluster configuration, cluster monitoring, cluster offline, cluster change, and cluster metadata. The query platform SQL Pad 520 may include functional modules such as user registration / login, SQL / DDL search, and export data. The metadata management module 530 may include functional modules such as database management, data table management, and DDL management. The task management module 540 may include functional modules such as node detection, monitoring detection, query synchronization, and configuration detection. The version management module 550 may include functional modules such as CH (ClickHouse) version management and ZK (Zookeeper) version management. The user management module 560 may include functional modules such as CH user management and PaaS user management.

[0082] In one embodiment, the ClickHouse cluster management method further includes: performing operation and maintenance management on the ClickHouse cluster through the ClickHouse cluster management platform. The CH-PaaS platform is a user-visualized self-service ClickHouse cluster operation and maintenance management tool developed by the present invention, which includes functions such as cluster status query, cluster management, cluster audit and expansion, and integrates all deployment and operation and maintenance operations of the ClickHouse cluster.

[0083] Perform operations and maintenance on the ClickHouse cluster, including one or more of the following:

[0084] All ClickHouse clusters are displayed and searchable through the cluster list page of the ClickHouse cluster management platform. On the cluster list page, specific functions such as cluster name, domain name, user name, node IP, and database search are supported.

[0085] Through the cluster details page of the ClickHouse cluster management platform, all detailed information of a target ClickHouse cluster is displayed. Figure 6 The page diagram showing the cluster details page, refer to Figure 6 As shown, on the cluster details page 600, the basic information and detailed information of the target ClickHouse cluster are displayed, and also includes a cluster configuration bar, which specifically includes instance information items, cluster change items, permission management items, library and table information items, account and password display items, DDL management items, etc.

[0086] In response to a cluster change request from the cluster details page, modify the configuration of the target ClickHouse cluster. Specifically, users can modify the special configuration of their cluster in the instance information, including commonly used query, write, network and other settings.

[0087] In response to the cluster expansion request issued from the cluster details page, the target ClickHouse cluster is expanded according to the number of replicas and backups carried in the cluster expansion request. Users can apply for horizontal and vertical cluster expansion in the instance information and select the required number of shards and replicas.

[0088] In response to the permission management request issued from the cluster details page, the target ClickHouse cluster is managed for permissions. In the permission management, cluster applicants and super administrators can add and delete cluster administrators by themselves.

[0089] In response to the library and table management request issued from the cluster details page, the target ClickHouse cluster is managed. In the library and table information, the source data of the library and table can be managed and modified. The creation of the table covers the columns, partitions, TTL settings, etc. of the table structure.

[0090] Receive cluster application requests through the cluster application page of the ClickHouse cluster management platform. Through the cluster application page, users can apply to create a cluster by themselves according to their needs.

[0091] The approval list page of the ClickHouse cluster management platform displays approved cluster application requests and unapproved cluster application requests. The super administrator can adjust the corresponding resource configuration according to the application on the approval list page.

[0092] Manage each ClickHouse version through the version management page of the ClickHouse cluster management platform.

[0093] In response to the query request sent by the administrator account from the query platform, the ClickHouse cluster and its node information associated with the administrator account are queried, managed, and exported. By integrating the SQLPad query platform in the CH-PaaS platform, linking internal accounts, and synchronizing the permission management of users and accounts, users can manage their clusters and nodes with permissions in the CH-PaaS platform on the interface. Users can query and manage metadata on the SQLPad query platform, and support exporting data into files in three formats: csv, xlsx, and json.

[0094] In summary, the ClickHouse cluster management method of the present invention realizes the life cycle management of the ClickHouse cluster through the CH-PaaS platform, including cluster deployment, horizontal and vertical expansion cluster, cluster configuration modification, cluster offline, node status real-time detection, integrated query, partial abnormal automatic detection and other functions, and realizes the full range of monitoring of Kubernetes host, Pod, and cluster; the ClickHouse cluster is hosted to Kubernetes management, which reduces a large number of repeated configurations for physical machine environment and configuration, reduces the complexity of the system, and reduces the management cost of the system. The CH-PaaS platform can improve the ease of use of the ClickHouse cluster, liberate ordinary business and data development from the cluster setting of ClickHouse, and there is no need to pay attention to the cluster deployment and operation and maintenance details of ClickHouse, which greatly reduces the use threshold of the ClickHouse cluster, so that the cluster can be used out of the box. In addition, more users are also allowed to use ClickHouse, a powerful OLAP management system, to improve the OLAP processing performance problem of massive data, because the high compression ratio of ClickHouse storage data reduces the cost of storing data, and compared with other database management systems, the efficiency and stability of the original system are improved in terms of OLAP. The CH-PaaS platform of the present invention is fully linked with the big data platform to realize the full-cycle management of offline data and real-time data. The data of the two parties flow mutually, giving full play to the advantageous processing capabilities of each platform, and realizing the effect of integrating the CH-PaaS platform with the OLAP platform.

[0095] The embodiment of the present invention also provides a ClickHouse cluster management platform, namely the CH-PaaS platform, which can be used to implement the ClickHouse cluster management method described in any of the above embodiments. The features and principles of the ClickHouse cluster management method described in any of the above embodiments can be applied to the following CH-PaaS platform embodiment. In the following CH-PaaS platform embodiment, the features and principles of ClickHouse cluster management that have been explained will not be repeated.

[0096] Figure 7 Shows the main modules of the ClickHouse cluster management platform, refer to Figure 7 As shown, in one embodiment, the ClickHouse cluster management platform 700 includes: a cluster deployment module 710, which is used to deploy a ClickHouse cluster including ClickHouse nodes and Zookeeper nodes into a container orchestration engine environment; a data management module 720, which is used to collect all node information in the container orchestration engine environment, synchronize it to the database of the ClickHouse cluster management platform, and bind the administrator account of the ClickHouse cluster to the query platform integrated in the ClickHouse cluster management platform; and a platform interface module 730, which is used to link the ClickHouse cluster with the big data platform and perform big data management on the ClickHouse cluster.

[0097] Furthermore, the ClickHouse cluster management platform 700 may also include modules for implementing other process steps of the above-mentioned ClickHouse cluster management method embodiments, for example, Figure 5 The specific principles of each module can refer to the description of the above-mentioned ClickHouse cluster management method embodiments, which will not be repeated here.

[0098] In addition, an embodiment of the present invention further provides an online analytical processing system, namely, an OLAP system, including the CH-PaaS platform described in any of the above embodiments. Figure 4 As shown, the description will not be repeated here.

[0099] The CH-PaaS platform of the present invention can realize automatic deployment and expansion of ClickHouse clusters, reduce operation and maintenance management costs, and realize user self-service management of ClickHouse life cycle; combining the flexibility of Kubernetes with the powerful OLAP performance of Clickhouse, connecting major data platforms, and realizing a full-featured OLAP system based on ClickHouse.

[0100] An embodiment of the present invention also provides a computer device, including a processor and a memory, wherein executable instructions are stored in the memory, and when the executable instructions are executed by the processor, the ClickHouse cluster management method described in any of the above embodiments is implemented.

[0101] The computer device of the present invention can realize automatic deployment and expansion of ClickHouse cluster, reduce operation and maintenance management costs, and realize user self-service management of ClickHouse life cycle; combining the flexibility of Kubernetes with the powerful OLAP performance of Clickhouse, connecting major data platforms, and realizing a full-featured OLAP system based on ClickHouse.

[0102] Figure 8 is a schematic diagram of the structure of a computer device in an embodiment of the present invention. It should be understood that: Figure 8 The modules are only schematically shown, and these modules may be virtual software modules or actual hardware modules. The combination and splitting of these modules and the addition of other modules are all within the protection scope of the present invention.

[0103] like Figure 8 As shown, the computer device 800 is in the form of a general-purpose computing device. The components of the computer device 800 include, but are not limited to: at least one processing unit 810, at least one storage unit 820, a bus 830 connecting different platform components (including the storage unit 820 and the processing unit 810), a display unit 840, etc.

[0104] The storage unit 820 stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps of the ClickHouse cluster management method described in any of the above embodiments. For example, the processing unit 810 can perform the following steps: Figure 1 Steps shown.

[0105] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0106] The storage unit 820 may also include a program / utility 8204 having one or more program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0107] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0108] The computer device 800 can also communicate with one or more external devices, and the external device can be one or more of the devices such as keyboard, pointing device, Bluetooth device, etc. These external devices enable the user to communicate interactively with the computer device 800. The computer device 800 can also communicate with one or more other computing devices, and the computer device shown includes a router and a modem. This communication can be carried out through an input / output (I / O) interface 850. In addition, the computer device 800 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 860. The network adapter 860 can communicate with other modules of the computer device 800 through a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the computer device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0109] The embodiment of the present invention also provides a computer-readable storage medium for storing a program, which implements the ClickHouse cluster management method described in any of the above embodiments when the program is executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product runs on a terminal device, the program code is used to enable the terminal device to execute the ClickHouse cluster management method described in any of the above embodiments.

[0110] When the storage medium of the present invention is executed by the processor, it can realize the automatic deployment and expansion of the ClickHouse cluster, reduce the operation and maintenance management costs, and realize the user self-service management of the ClickHouse life cycle; combining the flexibility of Kubernetes with the powerful OLAP performance of Clickhouse, connecting major data platforms, and realizing a full-featured OLAP system based on ClickHouse.

[0111] The storage medium may be a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the storage medium of the present invention is not limited thereto, and may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0112] The storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media include, but are not limited to, an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0113] The readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. The data signal of this propagation may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable signal medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0114] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device, such as through the Internet using an Internet service provider.

[0115] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A ClickHouse cluster management method, characterized in that: Applied to ClickHouse cluster management platform, including: Deploy a ClickHouse cluster including ClickHouse nodes and Zookeeper nodes into a container orchestration engine environment, including: deploying a ClickHouse generator into the container orchestration engine environment, and deploying the ClickHouse cluster through the ClickHouse cluster management platform and the description file of the ClickHouse generator; Wherein, the deployment of the ClickHouse cluster through the description file of the ClickHouse cluster management platform and the ClickHouse generator includes: in response to a cluster application request, binding the working domain of the ClickHouse cluster; generating configuration information and account information of the ClickHouse cluster; setting resource occupancy information of the ClickHouse cluster; adding the container orchestration engine domain name and service name of the Zookeeper node to the description file; integrating the settings of all description files to generate resource description files corresponding to the ClickHouse node, the Zookeeper node and the service node of the Zookeeper node respectively; submitting the resource description file to the container orchestration engine environment, generating the ClickHouse node and the Zookeeper node, and linking the domain name of the ClickHouse node to the target service node; Collect all node information in the container orchestration engine environment, synchronize it to the database of the ClickHouse cluster management platform, and bind the administrator account of the ClickHouse cluster to the query platform integrated in the ClickHouse cluster management platform; and Link the ClickHouse cluster with the big data platform to perform big data management on the ClickHouse cluster.

2. The ClickHouse cluster management method according to claim 1, characterized in that Before deploying the ClickHouse cluster including ClickHouse nodes and Zookeeper nodes to the container orchestration engine environment, it also includes: Perform node configuration on the host resources in the container orchestration engine environment to generate a ClickHouse host with a ClickHouse label and a Zookeeper host with a Zookeeper label; In the ClickHouse cluster, the ClickHouse node is deployed on the ClickHouse host machine, and the Zookeeper node is deployed on the Zookeeper host machine.

3. The ClickHouse cluster management method according to claim 1, characterized in that: The working domain binding the ClickHouse cluster includes: According to the application identifier and group identifier carried in the cluster application request, the working domain is bound, and the working domain covers several ClickHouse hosts and several Zookeeper hosts.

4. The ClickHouse cluster management method according to claim 1, characterized in that: The configuration information includes: user account and password configuration information, user authority configuration information, query configuration information, write configuration information, cluster storage configuration information and log storage configuration information; The account information includes an administrator account with full permissions, a read-write account with read-write permissions, and a read-only account with read-only permissions.

5. The ClickHouse cluster management method according to claim 1, characterized in that: The setting of resource occupancy information of the ClickHouse cluster includes: The resource occupancy information is set according to the shard information and the replica information, and the resource occupancy information includes the number of shards, the number of replicas, the CPU occupancy information, the memory occupancy information, the storage space occupancy information and the synchronization resource occupancy information.

6. The ClickHouse cluster management method according to claim 1, characterized in that: Linking the domain name of the ClickHouse node to the target service node includes: Submit the domain name of the ClickHouse node to the gateway service deployed in the container orchestration engine environment, generate a domain name access service, and link the domain name of the ClickHouse node to the target service node.

7. The ClickHouse cluster management method according to claim 1, characterized in that: The big data platform includes: a data transmission platform, a data authority platform, a data query tool, a data reporting platform and a data real-time computing platform.

8. The ClickHouse cluster management method according to claim 7, characterized in that: The big data management of the ClickHouse cluster includes one or more of the following: Through the big data platform, configure the table structure of the ClickHouse cluster, where the table structure is at the same level as the index of the data table of the big data platform; Configure the library and table permissions of the ClickHouse cluster through the data permission platform; Through the data transmission platform, the offline data in the data source is imported into the ClickHouse cluster, and the data in the ClickHouse cluster is imported into the data source; Importing real-time data into the ClickHouse cluster through the data real-time computing platform; Through the data query tool, query and export data on the ClickHouse cluster; Through the data reporting platform, configure the data reporting of the ClickHouse cluster.

9. The ClickHouse cluster management method according to claim 1, characterized in that: Also includes: Perform operation and maintenance management on the ClickHouse cluster through the ClickHouse cluster management platform; The operation and maintenance management of the ClickHouse cluster includes one or more of the following: Through the cluster list page of the ClickHouse cluster management platform, all ClickHouse clusters are displayed and searchable; Display all detailed information of a target ClickHouse cluster through the cluster details page of the ClickHouse cluster management platform; In response to a cluster change request issued from the cluster details page, modify the configuration of the target ClickHouse cluster; In response to a cluster expansion request issued from the cluster details page, the target ClickHouse cluster is expanded according to the number of replicas and the number of backups carried in the cluster expansion request; In response to a permission management request issued from the cluster details page, performing permission management on the target ClickHouse cluster; In response to a library table management request issued from the cluster details page, perform library table management on the target ClickHouse cluster; Receive a cluster application request through the cluster application page of the ClickHouse cluster management platform; Through the approval list page of the ClickHouse cluster management platform, the approved cluster application requests and the unapproved cluster application requests are displayed; Manage each ClickHouse version through the version management page of the ClickHouse cluster management platform; In response to the query request issued by the administrator account from the query platform, the ClickHouse cluster associated with the administrator account and its node information are queried, managed and exported.

10. A ClickHouse cluster management platform, characterized in that: The method for implementing the ClickHouse cluster management method according to any one of claims 1 to 9 comprises: A cluster deployment module is used to deploy a ClickHouse cluster including ClickHouse nodes and Zookeeper nodes into a container orchestration engine environment, including: deploying a ClickHouse generator into the container orchestration engine environment, and deploying the ClickHouse cluster through the ClickHouse cluster management platform and the description file of the ClickHouse generator; Wherein, the deployment of the ClickHouse cluster through the description file of the ClickHouse cluster management platform and the ClickHouse generator includes: in response to a cluster application request, binding the working domain of the ClickHouse cluster; generating configuration information and account information of the ClickHouse cluster; setting resource occupancy information of the ClickHouse cluster; adding the container orchestration engine domain name and service name of the Zookeeper node to the description file; integrating the settings of all description files to generate resource description files corresponding to the ClickHouse node, the Zookeeper node and the service node of the Zookeeper node respectively; submitting the resource description file to the container orchestration engine environment, generating the ClickHouse node and the Zookeeper node, and linking the domain name of the ClickHouse node to the target service node; A data management module, which is used to collect all node information in the container orchestration engine environment, synchronize it to the database of the ClickHouse cluster management platform, and bind the administrator account of the ClickHouse cluster to the query platform integrated in the ClickHouse cluster management platform; and The platform interface module is used to link the ClickHouse cluster with the big data platform and perform big data management on the ClickHouse cluster.

11. An online analytical processing system, characterized in that: Including the ClickHouse cluster management platform as described in claim 10.

12. A computer device, characterized in that: include: A processor; a memory storing executable instructions; Wherein, when the executable instructions are executed by the processor, the ClickHouse cluster management method as described in any one of claims 1-9 is implemented.

13. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by the processor, the ClickHouse cluster management method as described in any one of claims 1 to 9 is implemented.

Images