A data management method of a ClickHouse cluster and a related device

Abstract

The application discloses a data management method and related device of a ClickHouse cluster, and relates to the technical field of data management. In the application, a sub-physical machine group meeting a preset same-machine condition of a shard replica is screened out from a physical machine group corresponding to the ClickHouse cluster, so that a physical machine deploying multiple replicas of a single shard can be found, and shard replica information of multiple cloud hosts is reconfigured according to a preset non-same-machine configuration rule of the shard replica, thereby effectively avoiding the problem of data loss of multiple replicas caused by failure of the physical machine deploying multiple replicas of a single shard. In addition, once it is determined that the initial shard replica information of the multiple cloud hosts is modified into respective corresponding target shard replica information, target replica data respectively associated with the multiple target shard replica information can be stored on the multiple cloud hosts, so that the efficiency of data management of the ClickHouse cluster is greatly improved.

Description

Technical Field

[0001] This application relates to the field of data management technology, and in particular to a data management method and related apparatus for a ClickHouse cluster. Background Technology

[0002] With the rapid development of virtualization technology, physical machines can be virtualized into multiple cloud hosts with independent operating systems to improve the resource utilization of physical machines. The data processing methods of each cloud host are similar to or the same as those of the physical machine.

[0003] However, if a physical machine fails and cannot be recovered, the business data stored on multiple virtualized cloud hosts will be lost and cannot be recovered. Therefore, in order to avoid such problems, a multi-replica approach is usually adopted to improve the high availability of cloud host functions.

[0004] Among them, see Figure 1 As shown, ClickHouse is based on a sharded + replica massively parallel processing (MMP) architecture and asynchronous multi-master (decentralized) replication technology. After data is written to any available replica in the ClickHouse cluster, it will distribute the data to other replicas in the background to ensure that the same data is maintained on different replicas.

[0005] Furthermore, regarding the issue of multiple replicas of a single shard in a ClickHouse cluster deployed on various cloud hosts on the same physical machine, if the physical machine fails, the data of multiple replicas of that shard will be lost, necessitating the recovery of this data. Related technologies typically involve calling the Infrastructure as a Service (IaaS) interface to launch a new cloud host, installing all the same software as the corresponding cloud host, and copying all the metadata of the cloud host. This allows the data of the multiple replicas of the shard to be automatically synchronized after restarting the ClickHouse cluster, until the data of the multiple replicas is consistent, thus completing the recovery of the multiple replica data.

[0006] However, the data management method of the ClickHouse cluster described above has the disadvantage that calling the IaaS interface can only be done serially and cannot be processed in parallel. In addition, data synchronization is affected by the amount of data, which means that it takes a lot of time to solve the problem of multiple replicas of a single shard being lost due to the failure of the physical machine that deployed multiple replicas. In other words, it cannot guarantee timely recovery of multiple replicas of a single shard.

[0007] Therefore, the above method is inefficient for managing data in the ClickHouse cluster. Summary of the Invention

[0008] This application provides a data management method and related apparatus for a ClickHouse cluster, which avoids the problem of multiple replicas being lost due to the failure of the physical machine deploying multiple replicas of a single shard, and improves the efficiency of data management for the ClickHouse cluster.

[0009] Firstly, embodiments of this application provide a data management method for a ClickHouse cluster. The method includes:

[0010] Obtain the initial shard replica information of each cloud host in the ClickHouse cluster, and select sub-physical clusters that meet the preset shard replica co-location condition from the physical clusters corresponding to the ClickHouse cluster; wherein, each cloud host has the same ClickHouse service capabilities, and each cloud host is obtained by virtualization of the corresponding physical machine;

[0011] Based on the initial shard replica information of multiple cloud hosts corresponding to the sub-physical cluster, and the preset shard replica non-same-machine configuration rules, the target shard replica information of multiple cloud hosts is obtained.

[0012] Based on the obtained target shard replica information, the initial shard replica information of multiple cloud hosts is modified;

[0013] Once it is confirmed that the initial shard replica information of multiple cloud hosts has been successfully modified to their respective target shard replica information, the target replica data associated with each of the multiple target shard replica information is stored on the multiple cloud hosts.

[0014] Secondly, embodiments of this application also provide a data management device for a ClickHouse cluster, the device comprising:

[0015] The acquisition module is used to obtain the initial shard replica information of each cloud host in the ClickHouse cluster, and to filter out the sub-physical clusters that meet the preset shard replica co-location conditions from the physical clusters corresponding to the ClickHouse cluster; wherein, each cloud host has the same ClickHouse service capabilities, and each cloud host is obtained by virtualization of the corresponding physical machine;

[0016] The configuration module is used to obtain target shard replica information of multiple cloud hosts based on the initial shard replica information of multiple cloud hosts corresponding to the sub-physical cluster, as well as the preset shard replica non-same-machine configuration rules.

[0017] The modification module is used to modify the initial shard replica information of multiple cloud hosts based on the obtained target shard replica information;

[0018] The storage module is used to store the target replica data associated with each of the multiple target shard replicas on the multiple cloud hosts when it is determined that the initial shard replica information of multiple cloud hosts has been successfully modified to their respective target shard replica information.

[0019] In an optional embodiment, when selecting sub-physical clusters from the physical clusters corresponding to the ClickHouse cluster that meet the preset condition of sharded replicas on the same machine, the acquisition module is specifically used for:

[0020] For each physical machine in the physical cluster, perform the following operations:

[0021] Based on the initial shard replica information of at least two cloud hosts corresponding to the first physical machine, obtain the shard type to which each of the at least two initial shard replica information belongs; wherein, the first physical machine is any physical machine among the various physical machines;

[0022] If at least two initial shard replicas contain initial shard replicas of the same type, then the first physical machine is determined to meet the condition of having the same shard replica, and the first physical machine is designated as one of the physical machines in the sub-physical machine cluster.

[0023] In an optional embodiment, when obtaining target shard replica information of multiple cloud hosts based on the initial shard replica information of multiple cloud hosts corresponding to the sub-physical cluster and the preset shard replica non-co-location configuration rules, the configuration module is specifically used for:

[0024] Based on the initial shard replica information of multiple cloud hosts, obtain various shards and their corresponding replica counts;

[0025] Perform the following operations for each type of shard:

[0026] Based on the number of replicas in a shard, select multiple physical machines from the sub-physical cluster that meet the required number of replicas;

[0027] Each initial shard replica information corresponding to a shard is used as the target shard replica information for each cloud host corresponding to a physical machine.

[0028] In an optional embodiment, when modifying the initial shard replica information of multiple cloud hosts based on the obtained multiple target shard replica information, the modification module is specifically used for:

[0029] For multiple target shard replicas, perform the following operations respectively:

[0030] Parse the target shard replica information to obtain the target host identifier of the target cloud host and the target shard replica identifier corresponding to the target cloud host; wherein, the target shard replica identifier is the initial shard replica identifier of the corresponding cloud host;

[0031] Based on the target host identifier and the target shard replica identifier, the initial shard replica information of the corresponding cloud host is modified.

[0032] In an optional embodiment, when modifying the initial shard replica information of the corresponding cloud host based on the target host identifier and the target shard replica identifier, the modification module is specifically used for:

[0033] Obtain the logical relationship of the shard replicas to which the initial shard replica information belongs; wherein, the logical relationship of the shard replicas includes: the physical machine to which the corresponding cloud host belongs, and the host identifier of each cloud host;

[0034] In the sharded replica logical relationship, modify the initial host identifier associated with the target sharded replica identifier to the target host identifier.

[0035] In an optional embodiment, after modifying the initial shard replica information of the corresponding cloud host based on the target host identifier and the target shard replica identifier, the modification module is further configured to:

[0036] The initial shard replica information of the target cloud host is parsed to obtain the initial shard replica identifier of the target cloud host;

[0037] Change the initial shard replica identifier corresponding to the target cloud host to the target shard replica identifier.

[0038] In an optional embodiment, before modifying the initial shard replica information of multiple cloud hosts based on the obtained multiple target shard replica information, the modification module is further configured to:

[0039] The initial replica data stored on each of the multiple cloud hosts is copied to obtain the target replica data corresponding to each of the multiple initial replica data.

[0040] Once multiple target replica data are successfully obtained, multiple initial replica data are cleared, and multiple cloud hosts are checked according to a preset data check cycle until multiple initial replica data are completely cleared.

[0041] Thirdly, an electronic device is proposed, comprising a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of the ClickHouse cluster data management method described in the first aspect.

[0042] Fourthly, a computer-readable storage medium is proposed, comprising program code that, when run on an electronic device, causes the electronic device to perform the steps of the ClickHouse cluster data management method described in the first aspect.

[0043] Fifthly, a computer program product is provided, which, when invoked by a computer, causes the computer to execute the data management method steps of the ClickHouse cluster as described in the first aspect.

[0044] The beneficial effects of this application are as follows:

[0045] In the ClickHouse cluster data management method provided in this application embodiment, the initial shard replica information of each cloud host in the ClickHouse cluster is obtained, and sub-physical clusters that meet the preset shard replica co-location conditions are selected from the physical clusters corresponding to the ClickHouse cluster; wherein, each cloud host has the same ClickHouse service capabilities, and each cloud host is obtained by virtualization of the corresponding physical machine; then, based on the initial shard replica information of multiple cloud hosts corresponding to the sub-physical clusters, and the preset shard replica non-co-location configuration rules, the target shard replica information of multiple cloud hosts is obtained; further, based on the obtained target shard replica information, the initial shard replica information of multiple cloud hosts is modified; finally, when it is determined that the initial shard replica information of multiple cloud hosts has been successfully modified to their respective target shard replica information, the target replica data associated with each of the multiple target shard replica information is stored on the multiple cloud hosts.

[0046] By employing this approach, sub-physical clusters that meet the preset co-location criteria for sharded replicas are selected from the physical clusters corresponding to the ClickHouse cluster. This allows for the identification of physical machines deploying multiple replicas of a single shard. Furthermore, based on preset non-co-location configuration rules for sharded replicas, the sharded replica information of multiple cloud hosts is reconfigured. This effectively avoids the problem of multiple replica data loss due to the failure of the physical machine deploying multiple replicas of a single shard. In addition, once the initial sharded replica information of multiple cloud hosts is modified to their respective target sharded replica information, the target replica data associated with each target sharded replica information can be stored on multiple cloud hosts. This also avoids the technical drawbacks of related technologies, where calling IaaS interfaces can only be performed serially and cannot be processed in parallel. Additionally, data synchronization is affected by the amount of data, resulting in a significant time consumption. Therefore, this greatly improves the data management efficiency of the ClickHouse cluster.

[0047] Furthermore, other features and advantages of this application will be set forth in the following description and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0049] Figure 1 An exemplary illustration shows an application scenario diagram of a load balancer with ClickHouse service provided in an embodiment of this application;

[0050] Figure 2 An exemplary schematic diagram of an optional system architecture applicable to embodiments of this application is shown;

[0051] Figure 3 An exemplary schematic diagram of a ClickHouse cluster provided in an embodiment of this application is shown;

[0052] Figure 4 An exemplary illustration shows a schematic diagram of the implementation process of data management for a ClickHouse cluster provided in an embodiment of this application;

[0053] Figure 5 An exemplary embodiment of this application provides a method based on... Figure 4 A logical diagram;

[0054] Figure 6 An exemplary diagram illustrates a type of copy data corresponding to a physical machine provided in an embodiment of this application;

[0055] Figure 7 An exemplary illustration shows a flowchart of a method for obtaining target fragment copy information provided in an embodiment of this application;

[0056] Figure 8 An exemplary schematic diagram of a data backup and clearing method provided in an embodiment of this application is shown;

[0057] Figure 9 An exemplary illustration shows a flowchart of a method for modifying initial fragment copy information provided in an embodiment of this application;

[0058] Figure 10An exemplary illustration shows a specific scenario diagram of modifying initial fragment replica information provided in an embodiment of this application;

[0059] Figure 11 An exemplary embodiment of this application provides a method based on... Figure 4 Specific scene illustrations;

[0060] Figure 12 An exemplary schematic diagram of the overall architecture of a server provided in an embodiment of this application is shown;

[0061] Figure 13 An exemplary embodiment of this application provides a method based on... Figure 12 A schematic diagram of the data processing flow;

[0062] Figure 14 An exemplary schematic diagram of a data management device for a ClickHouse cluster provided in an embodiment of this application is shown.

[0063] Figure 15 An exemplary schematic diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0065] It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A connected to B can represent: A and B directly connected, or A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0066] Furthermore, the data collection, dissemination, and use in the technical solution of this application all comply with the requirements of relevant national laws and regulations.

[0067] To facilitate understanding by those skilled in the art, some of the nouns and terms involved in the embodiments of this application will be briefly described and explained as follows:

[0068] (1) ClickHouse: is a columnar database management system (DBMS) for online analytical processing (OLAP).

[0069] (2) Sharding: Distribute massive data across multiple nodes. Each node only stores and processes a portion of the massive data (i.e., the overall data). In single-replica mode, one shard corresponds to one node, while in multi-replica mode, one shard corresponds to multiple nodes.

[0070] (3) Replica: In order to ensure data security and high availability of services in abnormal situations, data is redundantly stored on multiple nodes, and the data on multiple nodes are replicas of each other.

[0071] (4) Physical machine: This is the term for a physical computer relative to a virtual machine (cloud host), that is, a real entity, such as a physical server in a computer room. It can also provide a hardware environment for virtual machines and is sometimes called a "host" or "host".

[0072] (5) Cloud host: It is a virtual machine. It should be noted that one physical machine can be virtualized into multiple cloud hosts.

[0073] (6) ZooKeeper: is a distributed and open-source distributed application coordination service. It is software that provides consistency services for distributed applications. The functions provided include, but are not limited to, configuration maintenance, name service, distributed synchronization, and group service. It should be noted that in the embodiments of this application, multiple replicas under the same shard can synchronize data through the ZooKeeper distributed coordination service.

[0074] It should also be noted that Zookeeper's data nodes can be viewed as a tree structure or directory. Each node in the tree is called a Znode (i.e., Zookeeper node). Moreover, a Znode can have multiple child nodes. Therefore, it can be said that all the stored data in Zookeeper is composed of Znodes and stored in key / value format.

[0075] (7) Data Definition Language (DDL): Used to define database objects such as databases, tables, and columns. It is mainly used to operate the database and create data tables.

[0076] (8) Structured Query Language (SQL): It is a database language with multiple functions such as data manipulation and data definition. This language has interactive features and can provide great convenience to users. Database management systems should make full use of SQL to improve the working quality and efficiency of computer application systems.

[0077] (9) Rollback: refers to the behavior of restoring the program or data to the previous correct state when a program or data processing error occurs. Rollback includes program rollback and data rollback.

[0078] (10) Big Data Cloud: Relying on cloud computing technology, service capabilities can be provided to consumers in the cloud, thereby realizing the processing, storage and computing of big data.

[0079] Furthermore, based on the above explanations of terms and related terminology, the design concept of the embodiments of this application will be briefly introduced below:

[0080] With the popularization of virtualization technology, a single physical machine can be virtualized into multiple independent host components, i.e., cloud hosts. Therefore, each physical machine can achieve single-machine multi-user operation. Furthermore, each component (i.e., cloud host) can run its own operating system, and the management method is similar to that of the physical machine, which greatly improves the resource utilization of the physical machine.

[0081] However, if the physical machine fails and cannot be repaired, the business data stored on these virtualized cloud servers will be lost and unrecoverable. Therefore, to avoid this problem, most application solutions use multiple copies to ensure high availability.

[0082] Therefore, since ClickHouse can be used in multi-dimensional analysis scenarios of big data clouds, has the characteristics of efficient and flexible querying, and is also an MPP architecture, linear expansion of functions and high reliability can be achieved through a sharded + replica structure. In addition, since ClickHouse uses asynchronous multi-master replication technology, when data is written to any available replica, the system will distribute the data to other replicas in the background to ensure that the system maintains the same data on different replicas.

[0083] In most cases, not all replicas of a single shard will fail. Therefore, the failed replicas can automatically recover data after the fault is repaired, ensuring that business operations are not affected. However, in rare cases, all replicas of a single shard may fail, which means that the failed replicas cannot automatically recover data after the fault is repaired, and the impact on business operations is significant. In such cases, manual data recovery is necessary, which takes a lot of time (i.e., a long time cycle).

[0084] In related technologies, to address the issue that when multiple replicas of a single shard in a ClickHouse cluster are deployed on the same host cloud server, and the physical machine hosting the host fails, the data of that shard will be affected and requires manual repair, a method of replacing the virtual machine is used to ensure that the multiple replicas are not on the same physical machine. Specifically, this involves calling the IaaS layer interface to start a new cloud server, installing all software, copying metadata, and restarting ClickHouse to automatically synchronize the data of other replicas until the replica data is consistent.

[0085] It is not difficult to see that the above sampling method will take a lot of time to solve the problem of multiple replicas of a single shard being lost due to the failure of the physical machine that deployed multiple replicas of a single shard. This is because the IaaS interface can only be called serially and cannot be processed in parallel. In addition, data synchronization is affected by the amount of data. In other words, it cannot guarantee timely recovery of multiple replicas of a single shard.

[0086] In view of this, in order to solve the problem of multiple replicas on the same host and improve the efficiency of data management of the ClickHouse cluster, a data management method for the ClickHouse cluster that supports parallel data synchronization and can complete the switching of multiple replicas to different hosts within minutes, thereby ensuring high availability of replica data under sharding, is adopted. Specifically, it includes: obtaining the initial shard replica information of each cloud host in the ClickHouse cluster, and selecting sub-physical clusters that meet the preset conditions for shard replica co-hosting from the physical clusters corresponding to the ClickHouse cluster. Then, based on the initial shard replica information of multiple cloud hosts corresponding to the sub-physical clusters and the preset shard replica non-co-hosting configuration rules, the target shard replica information of multiple cloud hosts is obtained. Based on the obtained target shard replica information, the initial shard replica information of multiple cloud hosts is modified. Then, when it is determined that the initial shard replica information of multiple cloud hosts has been successfully modified to their respective target shard replica information, the target replica data associated with each of the multiple target shard replica information is stored on the multiple cloud hosts.

[0087] In particular, the preferred embodiments of this application will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

[0088] See Figure 2The diagram illustrates a system architecture applicable to an embodiment of this application. This system architecture includes a ClickHouse cluster 201 and a server 202. The ClickHouse cluster 201 and the server 202 can interact via a communication network, where the communication network can employ wireless or wired communication methods.

[0089] For example, ClickHouse cluster 201 can access the network and communicate with server 202 via cellular mobile communication technology, wherein the cellular mobile communication technology includes, for example, 5th generation mobile network (5G) technology.

[0090] Optionally, the ClickHouse cluster 201 can access the network and communicate with the server 202 via short-range wireless communication, such as Wireless Fidelity (Wi-Fi) technology.

[0091] This application embodiment does not impose any limitation on the number of communication devices involved in the above system architecture. For example, there may be more servers, or no servers, or other network devices may be included, such as... Figure 2 As shown, only ClickHouse cluster 201 and server 202 are described as examples. The following is a brief introduction to each of the above devices and their respective functions.

[0092] See Figure 3 As shown, the ClickHouse cluster 201 includes multiple physical machines 301 and multiple cloud hosts 302. Each physical machine 301 can be virtualized into or virtualize multiple cloud hosts 302, and each cloud host 302 deploys the same ClickHouse service, that is, each cloud host 302 has the same ClickHouse service capabilities. Furthermore, in this embodiment, each cloud host 302 can be used to store a shard of replica data. Therefore, each shard corresponds to a physical machine, and each shard of replica data corresponds to a cloud host 302.

[0093] Server 202 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0094] It is worth noting that, in this embodiment, server 202 is used to obtain the initial shard replica information of each cloud host in the ClickHouse cluster, and to select sub-physical clusters that meet the preset shard replica co-location conditions from the physical clusters corresponding to the ClickHouse cluster; then, based on the initial shard replica information of multiple cloud hosts corresponding to the sub-physical clusters and the preset shard replica non-co-location configuration rules, target shard replica information of multiple cloud hosts is obtained; further, based on the obtained target shard replica information, the initial shard replica information of multiple cloud hosts is modified; finally, when it is determined that the initial shard replica information of multiple cloud hosts has been successfully modified to their respective target shard replica information, the target replica data associated with each of the multiple target shard replica information is stored on the multiple cloud hosts.

[0095] The following describes the ClickHouse cluster data management method provided by the exemplary embodiments of this application in conjunction with the above system architecture and with reference to the accompanying drawings. It should be noted that the above system architecture is only shown for the purpose of understanding the spirit and principles of this application, and the embodiments of this application are not limited in any way.

[0096] See Figure 4 The diagram illustrates an implementation flow of data management for a ClickHouse cluster according to an embodiment of this application. Assuming the execution entity is a server, the specific implementation flow of this method is as follows:

[0097] S401: Obtain the initial shard replica information of each cloud host in the ClickHouse cluster, and select the sub-physical clusters that meet the preset shard replica co-location condition from the physical clusters corresponding to the ClickHouse cluster.

[0098] Each cloud host has the same ClickHouse service capabilities. Each cloud host is virtualized from a corresponding physical machine. Each initial shard replica information represents the shard replica allocation of the corresponding cloud host, that is, which shard and which replica data is stored on the corresponding cloud host.

[0099] Furthermore, the aforementioned preset condition for sharded replicas on the same machine can specifically be: all replica data of the same shard are stored on multiple cloud hosts corresponding to the same physical machine.

[0100] It should also be noted that ClickHouse's metadata includes metrika.xml and macros.xml. metrika.xml is a secondary configuration file used for ClickHouse cluster setup, primarily defining cluster information, shard information, and replica information, i.e., configurations for Zookeeper, Shards, and Replicas. macros.xml defines which shard and replica role each ClickHouse service deployed on a cloud host belongs to. Furthermore, multiple replicas on the same shard synchronize data through the ZooKeeper distributed coordination service. In this embodiment, the content in metrika.xml can be referred to as the shard replica logical relationship, and the content in macros.xml can be referred to as the shard replica identifier. Therefore, the shard replica identifier indicates which shard and replica role the replica data stored on the corresponding cloud host belongs to.

[0101] S402: Based on the initial shard replica information of multiple cloud hosts corresponding to the sub-physical cluster, and the preset shard replica non-same-machine configuration rules, obtain the target shard replica information of multiple cloud hosts.

[0102] Specifically, when executing step S402, after the server obtains a sub-physical cluster that meets the above-mentioned preset conditions for sharded replicas on the same machine, it can reassemble the initial sharded replica information of multiple cloud hosts corresponding to the sub-physical cluster, as well as the preset sharded replica non-same-machine configuration rules, to obtain the reassembled initial sharded replica information, that is, to obtain the target sharded replica information of multiple cloud hosts, so that the replica data stored by all cloud hosts corresponding to the same physical machine corresponds to different shards.

[0103] S403: Based on the obtained multiple target shard replica information, modify the initial shard replica information of multiple cloud hosts.

[0104] Specifically, when executing step S403, after the service obtains multiple target shard replica information, it can modify the initial shard replica information of multiple cloud hosts respectively, that is, modify the content of the initial shard replica information according to the content contained in the obtained target shard replica information.

[0105] S404: When it is confirmed that the initial shard replica information of multiple cloud hosts has been successfully modified to their respective target shard replica information, the target replica data associated with each of the multiple target shard replica information is stored on the multiple cloud hosts.

[0106] Obviously, based on the method steps S401 to S404 above, refer to Figure 5As shown, the server selects sub-physical clusters from the physical clusters corresponding to the ClickHouse cluster that meet the preset condition of sharded replica co-location. This allows it to find physical machines that deploy multiple replicas of a single shard. Furthermore, based on preset sharded replica non-co-location configuration rules, the server reconfigures the sharded replica information of multiple cloud hosts. This effectively avoids the problem of multiple replica data loss caused by the failure of the physical machine deploying multiple replicas of a single shard. In addition, once the initial sharded replica information of multiple cloud hosts is modified to their respective target sharded replica information, the target replica data associated with each target sharded replica information can be stored on multiple cloud hosts. This avoids the technical drawbacks of related technologies where IaaS interface calls can only be performed serially and cannot be processed in parallel, and data synchronization is affected by data volume, resulting in significant time consumption. Therefore, this greatly improves the data management efficiency of the ClickHouse cluster.

[0107] In one optional implementation, when executing step S401, the server can perform the following operations for each physical machine in the physical cluster: based on the initial shard replica information of at least two cloud hosts corresponding to the first physical machine, obtain the shard type to which each of the at least two initial shard replica information belongs, where the first physical machine is any physical machine among the various physical machines; if there are initial shard replica information of the same shard type among the at least two initial shard replica information, then it is determined that the first physical machine meets the shard replica co-hosting condition, and the first physical machine is included as a physical machine in the sub-physical cluster; conversely, if there are no initial shard replica information of the same shard type among the at least two initial shard replica information, then it is determined that the first physical machine does not meet the shard replica co-hosting condition, and the first physical machine is not included as a physical machine in the sub-physical cluster; in this way, the selection of physical machines with multiple replicas on the same host in the ClickHouse cluster can be completed, so as to carry out the subsequent ClickHouse cluster data management method and solve the problem of multiple replicas on the same host in the ClickHouse cluster.

[0108] It should be noted that for physical machines that meet the preset conditions for sharded replicas on the same machine, the corresponding multiple cloud hosts can store all replica data of one type of shard. After storing all replica data of one type of shard, if there are still other cloud hosts that have not yet stored replica data, they can then store replica data of another type of shard. Therefore, please refer to... Figure 6 As shown, for multiple cloud hosts corresponding to a single physical machine, one or more sharded replicas of data may be stored.

[0109] In one optional implementation, during step S402, the server can first obtain various shards and their corresponding replica counts based on the initial shard replica information of multiple cloud hosts. Then, for each shard, the server performs the following operations to obtain the target shard replica information for each of the multiple cloud hosts. (See [reference]). Figure 7 As shown, the specific implementation process is as follows:

[0110] S701: Based on the number of replicas in a sharding, select multiple physical machines from the sub-physical cluster that meet the required number of replicas.

[0111] Specifically, when executing step S701, after the server determines the various shards and their respective replica counts for all physical machines in the sub-physical cluster, it can select multiple physical machines from the sub-physical machine cluster that meet the replica count of the aforementioned shard, that is, select multiple shards with the replica count of the aforementioned shard, so as to enable non-same-machine configuration of shard replicas for the initial shard replica information.

[0112] For example, assuming that the number of replicas of the above-mentioned shard (e.g., Shard1) is 3, the above-mentioned sub-physical cluster contains 4 physical machines, and the number of the above 4 physical machines and their respective shards and replicas are shown in Table 1:

[0113] Table 1

[0114]

[0115] Therefore, based on the physical machines and their corresponding shards and replicas in the table above, the server can select multiple physical machines from the above 4 physical machines that have the same number of replicas as Shard1, namely Phy.Mach.1, Phy.Mach.3 and Phy.Mach.4.

[0116] It should also be noted that, based on the table above, it is not difficult to see that multiple cloud hosts corresponding to a physical machine may not only store replica data of one type of shard, such as physical machine Phy.Mach.1, but may also store replica data of multiple types of shards, such as physical machine Phy.Mach.4.

[0117] S702: Use any initial shard replica information corresponding to one type of shard as the target shard replica information of any cloud host corresponding to each of the multiple physical machines.

[0118] Specifically, when executing step S702, after the server selects multiple physical machines, it can use any initial shard copy information corresponding to a shard (i.e., the initial shard copy information of any cloud host among the multiple cloud hosts corresponding to a shard) as the target shard copy information of any cloud host corresponding to each of the multiple physical machines, so as to realize the non-same-machine configuration of shard copies of the initial shard copy information.

[0119] For example, taking the four physical machines in Table 1 above as an example, after the server selects the physical machines Phy.Mach.1, Phy.Mach.3, and Phy.Mach.4 that meet the required number of replicas, it can use any initial shard replica information corresponding to Shard1 (e.g., Ori.Sha-Rep.Me1.1, Ori.Sha-Rep.Me1.2, and Ori.Sha-Rep.Me1.3) as the target shard replica information for any cloud host corresponding to each of the physical machines. This ensures that among the multiple cloud hosts corresponding to each of the physical machines Phy.Mach.1, Phy.Mach.3, and Phy.Mach.4, at least one cloud host is configured with the target shard replica information. It should be noted that since the number of replicas for Shard1 is 3, the corresponding number of cloud hosts is also 3. Therefore, Shard1 corresponds to 3 initial shard replica information.

[0120] It should also be noted that since Shard1 itself corresponds to a physical machine, in the above process of selecting physical machines, the physical machine corresponding to Shard1 itself can be directly used as one of the multiple physical machines that meet the number of replicas, or you can choose to exclude the physical machine corresponding to Shard1 itself and select multiple physical machines that meet the number of replicas.

[0121] In one alternative implementation, see [link to relevant documentation]. Figure 8 As shown, before executing step S403, the server can first copy the initial replica data stored by each of the multiple cloud hosts to obtain the target replica data corresponding to each of the multiple initial replica data, so as to complete the backup of the replica data. Then, when it is confirmed that multiple target replica data have been successfully obtained, the multiple initial replica data are cleared, and the multiple cloud hosts are checked according to the preset data check cycle until the multiple initial replica data are completely cleared, so that the initial shard replica information of the multiple cloud hosts can be modified based on the obtained multiple target shard replica information.

[0122] Therefore, in one optional implementation, when executing step S403, after the server obtains multiple target shard replica information and clears multiple initial replica data, it can perform the following operations on the multiple target shard replica information respectively to modify the initial shard replica information of multiple cloud hosts. (See [link to relevant documentation]). Figure 9 As shown, the specific implementation process of this method is as follows:

[0123] S901: Parse the target shard replica information to obtain the target host identifier of the target cloud host and the target shard replica identifier corresponding to the target cloud host.

[0124] The target shard replica identifier is the initial shard replica identifier of the corresponding cloud host.

[0125] For example, when executing step S901, the server parses the above-mentioned target shard replica information and can obtain the target host identifier of the target cloud host from the target shard replica information. For example, the target host identifier is "n110nidm1ck0002", and the target shard replica identifier corresponding to the target cloud host is "S01-R02", which represents the second replica (data) of the first shard.

[0126] S902: Modify the initial shard replica information of the corresponding cloud host based on the target host identifier and the target shard replica identifier.

[0127] In one optional implementation, when executing step S902, after obtaining the target host identifier and the target shard replica identifier, the server can further obtain the shard replica logical relationship to which the initial shard replica information belongs. The shard replica logical relationship includes: the physical machine to which the corresponding cloud host belongs, the host identifier of each corresponding cloud host, and then modifying the initial host identifier associated with the target shard replica identifier in the shard replica logical relationship to the target host identifier.

[0128] Optionally, after modifying the initial shard replica information of the corresponding cloud host based on the target host identifier and the target shard replica identifier, the server can also parse the initial shard replica information of the target cloud host to obtain the initial shard replica identifier of the target cloud host, thereby modifying the initial shard replica identifier corresponding to the target cloud host to the target shard replica identifier.

[0129] For example, see Figure 10As shown, the server can first update the sharded replica logical relationship in the metrika.xml file (i.e., modify the host identifier), then change the macros.xml file on the virtual machine "n110nidm1ck0002" from "S01-R02" to "S02-R01" (i.e., modify the sharded replica identifier), such as the new_macros.xml text box; similarly, the macros.xml file on the virtual machine "n110nidm1ck0003" can be changed from "S02-R01" to "S01-R02". Finally, by restarting the ClickHouse service, the role swapping of sharded replicas and subsequent replica data exchange can be automatically completed.

[0130] It should be noted that, as Figure 10 As shown, each cloud server deploys the same ClickHouse service and uses macros.xml. <macros>< / macros> The tag defines the shard replica identifier (i.e., the unique identifier of the replica), and is used via metrika.xml. <shard>< / shard> Tag pairs define which shard and replica a cloud host belongs to, that is, the logical relationship of the shard and replica information of the corresponding cloud host.

[0131] Further, see Figure 11 As shown, based on the ClickHouse cluster data management method described above, the server can improve the problem of replicas falling on the same physical machine, effectively solving the problem of replicas on the same host machine, such as... Figure 11 As shown, there are a total of 4 physical machines in the ClickHouse cluster that have the above problem. Figure 11 Each filling method corresponds to one physical machine, where each physical machine is virtualized into two cloud hosts, and each cloud host deploys the same ClickHouse service.

[0132] Optional, see below Figure 12 As shown, this is a schematic diagram of the overall architecture of a server provided in an embodiment of this application. The overall architecture includes various processing modules for executing the data management method of the ClickHouse cluster described above, specifically including: a data backup module 1201, a DDL export module 1202, an invalid replica cleanup module 1203, a database deletion module 1204, a service start / stop module 1205, a shard replica modification module 1206, a physical deletion module 1207, and a DDL import module 1208. The functions of each processing module are as follows:

[0133] The data backup module 1201 is used to complete data backup to ensure that the data can be rolled back. It is optional and belongs to the non-essential operation of the quick device. In this embodiment, based on multiple rounds of verification and testing, once the database deletion and physical deletion modules are executed, the old shards and copies can be specified to execute the provided data management method to achieve data recovery. Of course, it can be omitted when the data disk space is limited.

[0134] The DDL export module 1202 is used to export the database creation statements, the data table creation statements for each database, and the statements for clearing invalid copies. It is a prerequisite step for the DDL import module 1208.

[0135] The invalid copy cleanup module 1203 is used to clean up invalid copies based on the invalid copy cleanup statement generated by the DDL export module 1202, but the relevant data cleanup operation can only be performed after the delete library module 1204 is executed.

[0136] The database deletion module 1204 is used to generate and execute DROP statements for all databases (used to delete existing tables, procedures, or views in the database, or to delete existing indexes in a table). Before deleting a database, it verifies the parameters "max_table_size_to_drop" and "max_partition_size_to_drop" to ensure that databases with large data storage (e.g., greater than 50G) can be deleted.

[0137] In addition, since the deletion operation in the ClickHouse cluster is asynchronous, the asynchronous deletion is continuously checked during the deletion process until it is completed before exiting the loop check logic. Therefore, when the asynchronous deletion ends, the invalid replica cleanup module 1203 is called to clear the invalid replicas, ensuring that there are no invalid Znode information in the Zookeeper cluster.

[0138] The service start / stop module 1205 is used to start or stop the ClickHouse service on each cloud host in the ClickHouse cluster.

[0139] Modify sharded replica module 1206 to modify the macros.xml file. <shard>Tags and <replica>The content of the tags was used to complete the swapping of the fragmented copies (characters).

[0140] The physical deletion module 1207 is used to perform operating system-level deletion operations on residual metadata and data files to ensure that the newly recovered data is brand new and complete.

[0141] The DDL import module 1208 is used to import the SQL file generated by the DDL export module 1202 into the ClickHouse service to complete the restoration of metadata.

[0142] Furthermore, based on the overall server architecture provided above, the rectification of the multi-replica issue on the same host machine of the entire ClickHouse cluster can be completed within minutes. (See [link / reference]). Figure 13 As shown, the specific data processing flow is as follows:

[0143] S1301: Modify the logical correspondence between shards and replicas in the metrika.xml file.

[0144] For example, when performing step S1301, the server can modify the metrika.xml file and distribute it to multiple corresponding cloud hosts, and call the service start / stop module to restart the ClickHouse cluster, for example, by inputting the parameter: restart.

[0145] S1302: Data backup.

[0146] For example, when performing step S1302, the server can perform data backup through the data backup module, that is, back up the files to be deleted by the physical deletion module (i.e., target copy data). Optionally, the input parameter is a backup flag (Y: backup, N: no backup).

[0147] S1303: Export DDL.

[0148] For example, when performing step S1303, the server can export the DDL module: first, export the DDL for creating the database to create_databases_${VERSION}.sql; second, export the DDL for creating the table to create_tables_${VERSION}.sql; and finally, export the DDL for clearing expired replicas to system_drop_replica_${VERSION}.sql.

[0149] Optionally, since the SQL files for both the data table and the clear copy contain special characters, a removal operation is also required. At the same time, it is also necessary to obtain the contents of the old shard copy in the macros.xml file and replace the placeholders in the SQL for the clear copy so that the SQL can be executed.

[0150] S1304: Delete the library.

[0151] S1305: Whether asynchronous deletion has ended. If yes, proceed to S1304; otherwise, proceed to S1307, and then proceed to S1305 again via S1307.

[0152] S1306: Clear invalid copies.

[0153] S1307: Sleep for n seconds.

[0154] For example, n is usually in the order of seconds, such as 1 second for sleep.

[0155] Obviously, based on the method steps S1304 to S1307 above, the server can export the database deletion DDL to drop_databases_${VERSION}.sql through the database deletion module, and then execute the database deletion operation. Since database deletion is executed asynchronously in ClickHouse, it is necessary to continuously check whether the file exists under ${CH_PATH} / metadata_dropped after a set time (e.g., 480s). If it exists, continue to wait for asynchronous deletion; otherwise, it means that the database deletion operation has been completed.

[0156] Furthermore, after the above steps are completed, the invalid replica cleanup module can be called. Optionally, the input parameter is ZKPATH, where ZKPATH provides application programming interfaces (APIs) for creating, querying, and deleting nodes. Therefore, after the module call is completed, the system_drop_replica_${VERSION}.sql generated in step S1303 can be executed to clear invalid replicas. Only then is the database deletion operation complete.

[0157] It should also be noted that this approach effectively solves the problems that may occur if processing is missed, such as cluster restart errors or cluster unavailability after switching. It also avoids the technical drawbacks of having to copy metadata, replace new cloud hosts and install services, and then copy the metadata of the old cloud host to the new cloud host through operating system-level data copying, which leads to low operational efficiency and longer processing time as the data volume increases.

[0158] S1308: Service process stopped.

[0159] For example, when performing step S1308, the server stops the ClickHouse service process by calling the service start / stop module, optionally with the input parameter: stop.

[0160] S1309: Modify sharding and replication.

[0161] For example, during step S1309, the server executes the modify shard replica module, optionally with the input parameters: new_shard, new_replica. This swaps the shard and replica to create a new shard and a new replica.

[0162] S1310: Physically delete useless data.

[0163] For example, when performing step S1310, the server physically deletes the files in the three subdirectories of metadata, data, and data warehouse under the specified directory in order to clean up useless data (i.e., clear the initial copy data).

[0164] S1311: Start the service process.

[0165] For example, during step S1311, the server starts the ClickHouse service process by calling the service start / stop module, optionally with the input parameter: start.

[0166] S1312: Wait m seconds, import DDL.

[0167] For example, when executing step S1312, after the server starts the ClickHouse service process, it needs to wait m seconds, where m is usually on the order of seconds, such as 5 seconds. This is mainly because the ClickHouse service will load data after restarting, and the service will only work normally after the data is loaded.

[0168] Therefore, after the waiting period ends, the DDL import module is invoked to execute the two files created by S1303, create_databases_${VERSION}.sql and create_tables_${VERSION}.sql, in sequence. After execution, the system waits for data recovery (i.e., recovery of the copy data); and after the data recovery is complete, the switched copy data can be seen.

[0169] It should be noted that, based on the method steps S1301 to S1312 above: step S1301 is an auxiliary step, which does not need to be repeated. It is sufficient to complete the logical relationship between the new shard replicas according to the pre-planned replica swap requirements; step S1302 is a preparation step, which does not need to be repeated. If the data has been backed up, it does not need to be backed up again; the invalid replica cleanup module is a cleanup module, which needs to clear invalid replicas from the deleted database module to ensure that the cluster can provide services normally after the replicas are rectified with the host machine; the export DDL module and import DDL module can be replaced by copying the metadata directory file (and replica data) of the new shard replicas. Although copying the metadata directory file has the problem of losing soft links and doubling the data growth during the copying process, DDL import and export are more flexible and the problem is easier to locate. Therefore, it is easy to improve or solve the located problem with an effective method.

[0170] In summary, the ClickHouse cluster data management method provided in this application embodiment obtains the initial shard replica information of each cloud host in the ClickHouse cluster, and selects sub-physical clusters that meet the preset shard replica co-location conditions from the physical clusters corresponding to the ClickHouse cluster; wherein, each cloud host has the same ClickHouse service capabilities, and each cloud host is virtualized from the corresponding physical machine; then, based on the initial shard replica information of multiple cloud hosts corresponding to the sub-physical clusters, and the preset shard replica non-co-location configuration rules, the target shard replica information of multiple cloud hosts is obtained; further, based on the obtained target shard replica information, the initial shard replica information of multiple cloud hosts is modified; finally, when it is determined that the initial shard replica information of multiple cloud hosts has been successfully modified to their respective target shard replica information, the target replica data associated with each of the multiple target shard replica information is stored on the multiple cloud hosts.

[0171] By employing this approach, sub-physical clusters that meet the preset co-location criteria for sharded replicas are selected from the physical clusters corresponding to the ClickHouse cluster. This allows for the identification of physical machines deploying multiple replicas of a single shard. Furthermore, based on preset non-co-location configuration rules for sharded replicas, the sharded replica information of multiple cloud hosts is reconfigured. This effectively avoids the problem of multiple replica data loss due to the failure of the physical machine deploying multiple replicas of a single shard. In addition, once the initial sharded replica information of multiple cloud hosts is modified to their respective target sharded replica information, the target replica data associated with each target sharded replica information can be stored on multiple cloud hosts. This also avoids the technical drawbacks of related technologies, where calling IaaS interfaces can only be performed serially and cannot be processed in parallel. Additionally, data synchronization is affected by the amount of data, resulting in a significant time consumption. Therefore, this greatly improves the data management efficiency of the ClickHouse cluster.

[0172] Furthermore, based on the same technical concept, embodiments of this application provide a data management device for a ClickHouse cluster, which is used to implement the above-described method flow of embodiments of this application. See also... Figure 14 As shown, the data management device for the ClickHouse cluster includes: an acquisition module 1401, a configuration module 1402, a modification module 1403, and a storage module 1404, wherein:

[0173] The acquisition module 1401 is used to acquire the initial sharded replica information of each cloud host in the ClickHouse cluster, and to select sub-physical clusters that meet the preset sharded replica co-location conditions from the physical clusters corresponding to the ClickHouse cluster; wherein, each cloud host has the same ClickHouse service capabilities, and each cloud host is obtained by virtualization of the corresponding physical machine;

[0174] Configuration module 1402 is used to obtain target shard replica information of multiple cloud hosts based on the initial shard replica information of multiple cloud hosts corresponding to the sub-physical cluster, and the preset shard replica non-same-machine configuration rules.

[0175] Modify module 1403 to modify the initial shard replica information of multiple cloud hosts based on the obtained multiple target shard replica information;

[0176] Storage module 1404 is used to store the target replica data associated with each of the multiple target shard replica information on multiple cloud hosts when it is determined that the initial shard replica information of multiple cloud hosts has been successfully modified to their respective target shard replica information.

[0177] In an optional embodiment, when selecting sub-physical clusters from the physical clusters corresponding to the ClickHouse cluster that meet the preset condition of sharded replicas on the same machine, the acquisition module 1401 is specifically used for:

[0178] For each physical machine in the physical cluster, perform the following operations:

[0179] Based on the initial shard replica information of at least two cloud hosts corresponding to the first physical machine, obtain the shard type to which each of the at least two initial shard replica information belongs; wherein, the first physical machine is any physical machine among the various physical machines;

[0180] If at least two initial shard replicas contain initial shard replicas of the same type, then the first physical machine is determined to meet the condition of having the same shard replica, and the first physical machine is designated as one of the physical machines in the sub-physical machine cluster.

[0181] In an optional embodiment, when obtaining target shard replica information of multiple cloud hosts based on the initial shard replica information of multiple cloud hosts corresponding to the sub-physical cluster and the preset shard replica non-co-location configuration rules, the configuration module 1402 is specifically used for:

[0182] Based on the initial shard replica information of multiple cloud hosts, obtain various shards and their corresponding replica counts;

[0183] Perform the following operations for each type of shard:

[0184] Based on the number of replicas in a shard, select multiple physical machines from the sub-physical cluster that meet the required number of replicas;

[0185] Each initial shard replica information corresponding to a shard is used as the target shard replica information for each cloud host corresponding to a physical machine.

[0186] In an optional embodiment, when modifying the initial shard replica information of multiple cloud hosts based on the obtained multiple target shard replica information, the modification module 1403 is specifically used for:

[0187] For multiple target shard replicas, perform the following operations respectively:

[0188] Parse the target shard replica information to obtain the target host identifier of the target cloud host and the target shard replica identifier corresponding to the target cloud host; wherein, the target shard replica identifier is the initial shard replica identifier of the corresponding cloud host;

[0189] Based on the target host identifier and the target shard replica identifier, the initial shard replica information of the corresponding cloud host is modified.

[0190] In an optional embodiment, when modifying the initial shard replica information of the corresponding cloud host based on the target host identifier and the target shard replica identifier, the modification module 1403 is specifically used for:

[0191] Obtain the logical relationship of the shard replicas to which the initial shard replica information belongs; wherein, the logical relationship of the shard replicas includes: the physical machine to which the corresponding cloud host belongs, and the host identifier of each cloud host;

[0192] In the sharded replica logical relationship, modify the initial host identifier associated with the target sharded replica identifier to the target host identifier.

[0193] In an optional embodiment, after modifying the initial shard replica information of the corresponding cloud host based on the target host identifier and the target shard replica identifier, the modification module 1403 is further configured to:

[0194] The initial shard replica information of the target cloud host is parsed to obtain the initial shard replica identifier of the target cloud host;

[0195] Change the initial shard replica identifier corresponding to the target cloud host to the target shard replica identifier.

[0196] In an optional embodiment, before modifying the initial shard replica information of multiple cloud hosts based on the obtained multiple target shard replica information, the modification module 1403 is further configured to:

[0197] The initial replica data stored on each of the multiple cloud hosts is copied to obtain the target replica data corresponding to each of the multiple initial replica data.

[0198] Once multiple target replica data are successfully obtained, multiple initial replica data are cleared, and multiple cloud hosts are checked according to a preset data check cycle until multiple initial replica data are completely cleared.

[0199] Based on the same technical concept, embodiments of this application also provide an electronic device that can implement the data management method flow of the ClickHouse cluster provided in the above embodiments of this application. In one embodiment, the electronic device can be a server, a terminal device, or other electronic devices. Figure 15 As shown, the electronic device may include:

[0200] At least one processor 1501 and a memory 1502 connected to at least one processor 1501. In this embodiment, the specific connection medium between the processor 1501 and the memory 1502 is not limited. Figure 15 The example shown is the connection between processor 1501 and memory 1502 via bus 1500. Bus 1500 is... Figure 15 The connections between other components are shown in thick lines only and are not intended to be limiting. The Bus 1500 can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 15 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 1501 can also be called a controller; there is no restriction on the name.

[0201] In this embodiment, memory 1502 stores instructions executable by at least one processor 1501. By executing the instructions stored in memory 1502, at least one processor 1501 can perform a ClickHouse cluster data management method described above. Processor 1501 can implement... Figure 14 The functions of each module in the device shown.

[0202] The processor 1501 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 1502 and calling data stored in memory 1502, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0203] In one possible design, processor 1501 may include one or more processing units. Processor 1501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 1501. In some embodiments, processor 1501 and memory 1502 may be implemented on the same chip; in some embodiments, they may also be implemented separately on separate chips.

[0204] Processor 1501 can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the ClickHouse cluster data management method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0205] Memory 1502, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 1502 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 1502 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 1502 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0206] By designing and programming the processor 1501, the code corresponding to the ClickHouse cluster data management method described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute it during runtime. Figure 4 The illustrated embodiment describes the steps of a ClickHouse cluster data management method. How to design and program the processor 1501 is a technique well-known to those skilled in the art and will not be described further here.

[0207] Based on the same inventive concept, embodiments of this application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a ClickHouse cluster data management method described above.

[0208] In some possible implementations, this application also provides a method for managing the data of a ClickHouse cluster that can also be implemented as a program product including program code that, when the program product is run on a device, causes the control device to perform the steps of a ClickHouse cluster data management method according to various exemplary embodiments of this application as described above.

[0209] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0210] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0211] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0212] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a server, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0213] Program code for performing the operations of this application can be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0214] In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0215] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0216] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0217] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.< / replica> < / shard>

Claims

1. A data management method for a ClickHouse cluster, characterized in that, include: Obtain the initial shard replica information of each cloud host in the ClickHouse cluster, and perform the following operations for each physical machine in the physical machine cluster: Based on the initial shard replica information of at least two cloud hosts corresponding to the first physical machine, obtain the shard type to which each of the at least two initial shard replica information belongs; wherein, the first physical machine is any one of the physical machines. If among the at least two initial shard replicas, there are initial shard replicas of the same type, then the first physical machine is determined to meet the condition of the same shard replica, and the first physical machine is included as a physical machine in the sub-physical machine cluster; wherein, each cloud host has the same ClickHouse service capability, and each cloud host is obtained by virtualization of the corresponding physical machine; Based on the initial shard replica information of the multiple cloud hosts, obtain various shards and their corresponding replica counts. For each of the aforementioned shards, perform the following operations respectively: Based on the number of replicas for a shard, multiple physical machines that meet the required number of replicas are selected from the sub-physical machine cluster. Each of the initial shard replicas corresponding to one type of sharding is used as the target shard replica information of each of the multiple physical machines corresponding to any cloud host. Based on the obtained target shard replica information, the initial shard replica information of the multiple cloud hosts is modified; When it is confirmed that the initial shard replica information of the multiple cloud hosts has been successfully modified to their respective target shard replica information, the target replica data associated with each of the multiple target shard replica information is stored on the multiple cloud hosts.

2. The method as described in claim 1, characterized in that, The modification of the initial shard replica information of the multiple cloud hosts based on the obtained multiple target shard replica information includes: For the multiple target shard replicas, perform the following operations respectively: Parse the target shard replica information to obtain the target host identifier of the target cloud host and the target shard replica identifier corresponding to the target cloud host; wherein, the target shard replica identifier is the initial shard replica identifier of the corresponding cloud host; Based on the target host identifier and the target shard replica identifier, the initial shard replica information of the corresponding cloud host is modified.

3. The method as described in claim 2, characterized in that, The modification of the initial shard replica information of the corresponding cloud host based on the target host identifier and the target shard replica identifier includes: Obtain the logical relationship of the shard replicas to which the initial shard replica information belongs; wherein, the logical relationship of the shard replicas includes: the physical machine to which the corresponding cloud host belongs, and the host identifier of each corresponding cloud host; In the sharded replica logical relationship, the initial host identifier associated with the target sharded replica identifier is modified to the target host identifier.

4. The method as described in claim 2, characterized in that, After modifying the initial shard replica information of the corresponding cloud host based on the target host identifier and the target shard replica identifier, the method further includes: The initial shard replica information of the target cloud host is parsed to obtain the initial shard replica identifier of the target cloud host; Modify the initial shard replica identifier corresponding to the target cloud host to the target shard replica identifier.

5. The method as described in claim 1, characterized in that, Before modifying the initial shard replica information of the multiple cloud hosts based on the obtained multiple target shard replica information, the process further includes: The initial replica data stored on each of the multiple cloud hosts is copied to obtain the target replica data corresponding to each of the multiple initial replica data. Upon successfully obtaining multiple target replica data, the multiple initial replica data are cleared, and the multiple cloud hosts are checked according to a preset data check cycle until the multiple initial replica data are completely cleared.

6. A data management device for a ClickHouse cluster, characterized in that, include: The acquisition module is used to acquire the initial shard replica information of each cloud host in the ClickHouse cluster, and perform the following operations for each physical machine in the physical machine cluster: based on the initial shard replica information of at least two cloud hosts corresponding to the first physical machine, obtain the shard type to which each of the at least two initial shard replica information belongs; wherein, the first physical machine is any one of the physical machines. If among the at least two initial shard replicas, there are initial shard replicas of the same type, then the first physical machine is determined to meet the condition of the same shard replica, and the first physical machine is included as a physical machine in the sub-physical machine cluster; wherein, each cloud host has the same ClickHouse service capability, and each cloud host is obtained by virtualization of the corresponding physical machine; The configuration module is used to obtain various shards and their corresponding replica counts based on the initial shard replica information of the multiple cloud hosts. For each of the aforementioned shards, perform the following operations respectively: Based on the number of replicas for a shard, multiple physical machines that meet the required number of replicas are selected from the sub-physical machine cluster. Each of the initial shard replicas corresponding to one type of sharding is used as the target shard replica information of each of the multiple physical machines corresponding to any cloud host. The modification module is used to modify the initial shard replica information of the multiple cloud hosts based on the obtained multiple target shard replica information; The storage module is used to store the target replica data associated with each of the multiple target shard replicas on the multiple cloud hosts when it is determined that the initial shard replica information of the multiple cloud hosts has been successfully modified to their respective target shard replica information.

7. The apparatus as claimed in claim 6, characterized in that, When modifying the initial shard replica information of the multiple cloud hosts based on the obtained multiple target shard replica information, the modification module is specifically used for: For the multiple target shard replicas, perform the following operations respectively: Parse the target shard replica information to obtain the target host identifier of the target cloud host and the target shard replica identifier corresponding to the target cloud host; wherein, the target shard replica identifier is the initial shard replica identifier of the corresponding cloud host; Based on the target host identifier and the target shard replica identifier, the initial shard replica information of the corresponding cloud host is modified.

8. The apparatus as claimed in claim 7, characterized in that, When modifying the initial shard replica information of the corresponding cloud host based on the target host identifier and the target shard replica identifier, the modification module is specifically used for: Obtain the logical relationship of the shard replicas to which the initial shard replica information belongs; wherein, the logical relationship of the shard replicas includes: the physical machine to which the corresponding cloud host belongs, and the host identifier of each corresponding cloud host; In the sharded replica logical relationship, the initial host identifier associated with the target sharded replica identifier is modified to the target host identifier.

9. The apparatus as claimed in claim 7, characterized in that, After modifying the initial shard replica information of the corresponding cloud host based on the target host identifier and the target shard replica identifier, the modification module is further configured to: The initial shard replica information of the target cloud host is parsed to obtain the initial shard replica identifier of the target cloud host; Modify the initial shard replica identifier corresponding to the target cloud host to the target shard replica identifier.

10. The apparatus as claimed in claim 6, characterized in that, Before modifying the initial shard replica information of the multiple cloud hosts based on the obtained multiple target shard replica information, the modification module is further configured to: The initial replica data stored on each of the multiple cloud hosts is copied to obtain the target replica data corresponding to each of the multiple initial replica data. Upon successfully obtaining multiple target replica data, the multiple initial replica data are cleared, and the multiple cloud hosts are checked according to a preset data check cycle until the multiple initial replica data are completely cleared.

11. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-5.

13. A computer program product, characterized in that, When the computer program product is invoked by a computer, it causes the computer to perform the method as described in any one of claims 1-5.

Images