A distributed database load balancing method, device, equipment and medium

Abstract

The embodiments of the present specification disclose a method, apparatus, device and medium for load balancing of a distributed database, including: recovering the missing shard copies of the distributed database and discarding the excess shard copies of the distributed database to obtain nodes that meet the requirements in the distributed database; traversing each node that meets the requirements to determine the number of shard copies in each node that meets the requirements; determining the load rate of each node that meets the requirements based on the number of shard copies in each node that meets the requirements; and adjusting the number of shard copies in each node that meets the requirements based on the load rate of each node that meets the requirements to complete load balancing of the distributed database.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and medium for load balancing of a distributed database. Background Art

[0002] With the development of the Internet, the scale of Internet data continues to expand, and the demand for data storage is increasing, which has led to the emergence of databases. A database is a computer software system that stores and manages data according to data structures. In practical applications, the requirements for data reliability are getting higher and higher. For this reason, it is necessary to further improve the load balancing of the database. Summary of the invention

[0003] One or more embodiments of the present specification provide a method, apparatus, device and medium for load balancing of a distributed database, which are used to solve the technical problems raised by the background technology.

[0004] One or more embodiments of this specification adopt the following technical solutions:

[0005] One or more embodiments of this specification provide a distributed database load balancing method, including:

[0006] Recovering the missing shard copies of the distributed database and discarding the redundant shard copies of the distributed database to obtain nodes in the distributed database that meet the requirements;

[0007] Traverse each node that meets the requirements and determine the number of shard copies in each node that meets the requirements;

[0008] Determine the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements;

[0009] According to the load rate of each node that meets the requirements, the number of shard copies in each node that meets the requirements is adjusted to achieve load balancing of the distributed database.

[0010] One or more embodiments of this specification provide a load balancing device for a distributed database, the device comprising:

[0011] A shard copy processing unit is used to recover the missing shard copies of the distributed database and discard the redundant shard copies of the distributed database to obtain nodes in the distributed database that meet the requirements;

[0012] A shard copy counting unit traverses each node that meets the requirements and determines the number of shard copies in each node that meets the requirements;

[0013] A load rate determination unit, which determines the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements;

[0014] The load balancing unit adjusts the number of shard copies in each of the nodes that meet the requirements according to the load rate of each of the nodes that meet the requirements to achieve load balancing of the distributed database.

[0015] One or more embodiments of this specification provide a distributed database load balancing device, including:

[0016] at least one processor; and,

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0019] Recovering the missing shard copies of the distributed database and discarding the redundant shard copies of the distributed database to obtain nodes in the distributed database that meet the requirements;

[0020] Traverse each node that meets the requirements and determine the number of shard copies in each node that meets the requirements;

[0021] Determine the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements;

[0022] According to the load rate of each node that meets the requirements, the number of shard copies in each node that meets the requirements is adjusted to achieve load balancing of the distributed database.

[0023] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to:

[0024] Recovering the missing shard copies of the distributed database and discarding the redundant shard copies of the distributed database to obtain nodes in the distributed database that meet the requirements;

[0025] Traverse each node that meets the requirements and determine the number of shard copies in each node that meets the requirements;

[0026] Determine the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements;

[0027] According to the load rate of each node that meets the requirements, the number of shard copies in each node that meets the requirements is adjusted to achieve load balancing of the distributed database.

[0028] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: The embodiments of this specification recover the missing shard copies of the distributed database and discard the redundant shard copies of the distributed database to obtain the nodes that meet the requirements in the distributed database, so as to facilitate the subsequent load balancing of the nodes of the distributed database. By determining the load rate of each node that meets the requirements, the number of shard copies in each node that meets the requirements is adjusted to complete the load balancing of the distributed database, thereby improving the data reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0030] Figure 1 A flow chart of a load balancing method for a distributed database provided in one or more embodiments of this specification;

[0031] Figure 2 A schematic diagram of recovery from missing shard replicas provided for one or more embodiments of this specification;

[0032] Figure 3 A schematic diagram of garbage fragmentation recovery provided for one or more embodiments of this specification;

[0033] Figure 4 A schematic diagram of data shard recovery provided for one or more embodiments of this specification;

[0034] Figure 5 A schematic diagram of a user accessing a distributed database provided in one or more embodiments of this specification;

[0035] Figure 6 A schematic diagram of the Master service provided for one or more embodiments of this specification;

[0036] Figure 7 A first Master selection flow chart provided for one or more embodiments of this specification;

[0037] Figure 8 Second Master Selection Flowchart for One or More Embodiments of This Specification

[0038] Fig. 9 A namespace creation flow chart provided for one or more embodiments of this specification;

[0039] Fig.10 A schematic diagram of the structure of a load balancing device for a distributed database provided in one or more embodiments of this specification;

[0040] Fig.11 A schematic diagram of the structure of a load balancing device for a distributed database provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0041] The embodiments of this specification provide a method, apparatus, device and medium for load balancing of a distributed database.

[0042] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0043] Figure 1 A schematic diagram of a flow chart of a load balancing method for a distributed database provided in one or more embodiments of this specification, the flow chart can be executed by a load balancing system for a distributed database, and can perform load balancing processing on a distributed database to improve data reliability. Certain input parameters or intermediate results in the flow chart allow manual intervention and adjustment to help improve accuracy.

[0044] The method steps of the embodiment of this specification are as follows:

[0045] S102, recovering the missing shard copies of the distributed database and discarding the redundant shard copies of the distributed database to obtain nodes in the distributed database that meet the requirements.

[0046] In an embodiment of the present specification, the missing shard copies of the distributed database are restored to prioritize the restoration of the missing shards as much as possible. During the process, the nodes in the distributed database are first scanned, and the number of shard copies of each node is determined; then, the missing shard copies are marked; then, the nodes of the distributed database are sorted according to the load rate; finally, the marked missing shard copies are added to the nodes whose load rate is lower than a preset value.

[0047] In the embodiment of the present specification, when the redundant shard copies of the distributed database are discarded, it is a process of garbage shard recycling to delete the redundant shard copies. In the process, the nodes in the distributed database are first scanned, and the number of shard copies of each node is determined; then, according to the shard copy data of each node, the non-compliant nodes whose number of shard copies exceeds the preset quota are determined; finally, the redundant shard copies are determined in the non-compliant nodes, and the redundant shard copies are discarded.

[0048] S104, traverse each node that meets the requirements, and determine the number of shard copies in each node that meets the requirements.

[0049] In the embodiment of the present specification, the number of shard copies in each node that meets the requirements can be determined by statistics.

[0050] S106: Determine the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements.

[0051] In an embodiment of the present specification, the load rate of each node that meets the requirements can be determined based on the number of shard copies in each node that meets the requirements and the preset capacity. The preset capacity can be the preset optimal capacity of each node that meets the requirements. When calculating the load rate of each node that meets the requirements, the capacity of the corresponding node can be divided by the number of shard copies in each node that meets the requirements.

[0052] S108, adjusting the number of shard copies in each of the nodes that meet the requirements according to the load rate of each of the nodes that meet the requirements, so as to achieve load balancing of the distributed database.

[0053] In an embodiment of the present specification, when adjusting the number of shard copies in each of the nodes that meet the requirements based on the load rate of each of the nodes that meet the requirements, the load rate of each of the nodes that meet the requirements can be traversed. If the load rate of the first specified node that meets the requirements is greater than the average load, the number of shard copies in the first specified node that meets the requirements is adjusted.

[0054] Furthermore, in an embodiment of the present specification, when adjusting the number of shard copies in each of the nodes that meet the requirements based on the load rate of each of the nodes that meet the requirements, the load rate of each of the nodes that meet the requirements can be traversed. If the load rate of the second designated node that meets the requirements is not greater than the average load, the number of shard copies in the first designated node that meets the requirements is adjusted to the second designated node that meets the requirements.

[0055] Furthermore, after the load balancing of the distributed database is completed, the node data in the nodes that meet the requirements can be stored in a preset tree-type storage middleware.

[0056] For the above-mentioned distributed database load balancing method, the node data balancing process can be called rebalance, the process of recovering missing shard copies can be called recover, and the process of discarding redundant copies can be called garbagecollect. Partition information is described using partition.

[0057] Recover is the process of data shard recovery. Each time you execute rebalance, you must first recover the data shards before rebalancing. The purpose of this is to give priority to recovering the missing shards as much as possible. For example, there are 4096 copies of partition 2. If a partition is missing and the copy of partition 1 is lost, executing the recover process can recover the missing copy of the partition. The schematic diagram of shard copy recovery is as follows Figure 2 shown.

[0058] The recover process first needs to scan all nodes in the distributed database and count the number of copies of all nodes; then mark the missing shards; then start to traverse all marked shards, sort the existing active data nodes according to the load (number of shards of the node / capacity of the node), and then select a data node that does not contain the missing shard from the nodes with lower load, and add the marked shard to the node.

[0059] Garbage collection is the process of recycling garbage fragments. Figure 3 As shown in the diagram of garbage shard recycling, for example, if the data has 2 copies, since rebalancing is to move a shard to other shards, this process will cause the number of shards to increase and may become 3 copies, so the redundant shards need to be deleted in the end.

[0060] The garbage collection process first scans all nodes in the distributed database and counts the number of copies of all shards on active nodes. It then extracts all shards whose number of copies exceeds the quota and removes non-primary shards that exceed the number of copies.

[0061] Rebalance is the core process of data load balancing, which requires moving shards with higher loads to shards with lower loads, so that data can be balanced among storage nodes as much as possible. Figure 4 As shown in the data shard recovery diagram, after rebalancing is executed, the number of replicas of some partitions will increase, and the redundant partitions will be deleted during garbage collection in the next rebalancing process.

[0062] Design of the specific process of rebalance:

[0063] (1) Execute the recover process. If a replica is missing, wait for the next rebalance cycle.

[0064] (2) Execute the garbage collect process. If there are redundant copies, wait for the next rebalance cycle.

[0065] (3) Traverse each shard and count the number of shards of each node.

[0066] (4) Traverse all active nodes. If the node is a low-load node and its load rate is less than the average load (number of shards of the node / capacity of the node < total number of shards * load balancing tolerance parameter / total capacity), then add it to the low-load set.

[0067] (5) Sort all active data nodes by load (number of shards of the node / capacity of the node).

[0068] (6) Traverse the data nodes from high load to low load, each node is n[i].

[0069] (7) If n[i] is a low-load node, then exit the rebalance. For each non-master shard of n[i], try to allocate it to a node in the low-load node set and execute (4).

[0070] When the rebalance process is completed, the metadata center will store the calculated partition data in Zookeeper.

[0071] Furthermore, the embodiments of this specification address some problems with existing storage and require a database that is easily accessible and elastically scalable. For the real-time storage part, it is necessary to achieve SSD-based storage, with hot data cached in memory and non-hot data generated by disks. For the offline storage part, it is necessary to solve problems such as data expansion, reduce the number of copies, and reduce storage costs.

[0072] The schematic diagram of user access to distributed database is as follows Figure 5 As shown in the figure, the client first needs to know the IP and port number of the metadata center Master in the cluster, and then access the Master to obtain the cluster metadata such as shard information and the IP and port number of the DataNode node. DataNode is a node that specifically stores data and is usually deployed on a single physical machine or container instance. The specific application types deployed by DataNode can be divided into offline storage and real-time storage.

[0073] The following embodiments of this specification will describe the design and implementation of the metadata center.

[0074] Special terms explanation:

[0075] Node: A physical machine or instance that constitutes a cluster. In the embodiments of this specification, it can be divided into DataNode or Master. DataNode can also be divided into BatchStorage and RealTimeStorage.

[0076] BatchStorage: Batch storage, offline storage, usually does not allow data changes, only provides read-only services, the data scale is relatively large, a single table usually has a few TB to hundreds of TB of data, the data file is produced by Hadoop MapReduce and the data is downloaded by the data importer, and the service is provided.

[0077] RealTimeStorage: Real-time storage, providing read and write services and supporting simple transactions, so its performance is not as high as offline storage.

[0078] Master: It can be compared to the NameNode node in Hadoop HDFS. The client usually obtains metadata through the Master and then communicates with each DataNode. In order to improve fault tolerance, multiple Masters are usually deployed. When the LeaderMaster fails, other Masters will take over the service.

[0079] NameSpace: The unit of resource configuration and data organization, a collection of a series of Tables. A DataNode can only be configured under one NameSpace to ensure that resources and IO are physically isolated. NameSpace can also be compared to the database concept of MySQL.

[0080] Table: A logical concept. Logically, data in a NameSpace can belong to different Tables and can be queried by Table. However, the underlying storage does not distinguish between Tables. Data from different Tables in the same DB are stored together, and the key has an additional attribute to identify the Table.

[0081] Partition: Sharding. The amount of data in a NameSpace is too large to be stored on a single machine. In SimpleDB, sharding is used to store the data in a DB into 4096 parts and distribute them to different machines. Each shard has two copies, primary and secondary, and provides a failover mechanism.

[0082] Eventual Consistency: Eventual consistency means that when there is no new update, the update will eventually propagate to all replicas through the network, and all replicas will eventually be consistent. The application layer cannot guarantee that it will see the newly written data in the intermediate process before a certain point in time.

[0083] Rebalance: Data balance. Due to some uncertain factors such as node online and offline, the shard data on each node is inconsistent. Therefore, the data needs to be recalculated according to the number of shards. The data on the nodes with heavier loads is transferred to the nodes with lighter loads, so that the number of requests received and the amount of data stored on each node are equal.

[0084] Since the resources of a single server are limited, a single-machine database cannot store massive amounts of data, so a distributed database needs to be introduced to store large-scale data. However, the introduction of distributed systems brings a series of distributed problems, such as consistency, availability, partition fault tolerance, etc. However, since the advertising library and feed stream data are not sensitive to data consistency requirements, but hope that the system can provide high availability, the storage requirements are selected according to the CAP law under the distributed system. Availability and partition fault tolerance, that is, the AP system. The advantage of the AP system is that the overhead of data synchronization in the distributed system is lower, it does not matter whether the version of the data is the latest, and the access results are always controllable. For example, a read operation may not read the latest value, but the data will not cause an access timeout because the data is waiting for synchronization to complete.

[0085] Massive amounts of data can only be stored in multiple servers, but how to store so much data and how the data is arranged and stored on multiple servers requires metadata to describe.

[0086] The following is a detailed description of the design and implementation of the metadata center:

[0087] 1. Overall structural design analysis

[0088] Metadata includes database name, table name, cluster node information, database shard information, etc. This information plays a critical role in the entire distributed database. For example, client access requires metadata to understand the number of nodes in the cluster and which data nodes to access when querying. Metadata sometimes needs to be changed, such as when a user creates a NameSpace or a table, so the database needs to provide an interface for metadata management.

[0089] Since metadata modification, table creation and modification are common operations for data access users, in order to improve the usability of the system and reduce data conflicts in distributed systems, metadata management should be an independent management service like GFS and other systems, and the metadata of the entire cluster should be controlled and issued through the metadata service.

[0090] For system administrators, it is necessary to understand the system status in a timely manner, so an RPC interface for collecting cluster information is needed. Because the system requires some expansion and reduction operations, it is also necessary to provide an operation interface for adding and deleting data nodes.

[0091] Since data storage consistency is not possible in a distributed system, the specific storage location of metadata also needs to be decided. Common metadata storage methods include centralized metadata, distributed metadata storage, and stateless metadata center service.

[0092] 1.1. Centralized metadata

[0093] A reliable metadata server is required in the cluster to record the coordination table and shards, and notify the data output to load the download files and switch versions within the entire cluster.

[0094] The advantage of centralized metadata is that the logic is simple and management is convenient, but the metadata server stored in it needs to maintain high availability. Once the metadata center unexpectedly goes down, the entire service will be unavailable.

[0095] 1.2 Metadata Distributed Storage

[0096] No Master node is provided. Each node in the cluster provides its own local storage table, shard information, and the file directory corresponding to the shard and records the download progress (offline data part). As long as the metadata of all nodes in the cluster are summarized, the overall status of the entire cluster can be obtained. Although there is no center, the metadata module has a master. Only the metadata module selected as the Master node can update the metadata information and then synchronize it to all other nodes in the cluster. When the Master goes down, the cluster will elect a new Master node.

[0097] Distributed metadata storage ensures high availability of the cluster. The failure of any metadata / query service node will not affect the retrieval service. However, in order to maintain the consistency of metadata of each node, the implementation complexity is relatively high. Some common solutions are to elect a metadata center node through paxos, and the center node is responsible for copying metadata to all nodes for persistence. After the center node fails, a new center node is elected. This ensures both metadata consistency and high availability of the cluster.

[0098] 1.3. Metadata Center Stateless Service

[0099] The cluster can delegate specific storage to other distributed components such as Zookeeper or ETCD cluster. The metadata center is only responsible for receiving requests and asynchronously caching, reducing the query pressure of the principle storage service. In addition, the service-oriented metadata center can facilitate capacity expansion, and issues such as metadata service leader election can also be solved by relying on distributed locks.

[0100] 2. Metadata service design

[0101] The existence of data nodes is usually transparent to users. Like Google File System, users usually need to know that the leader node in a cluster directly provides some basic services to users. This service is usually called metadata service. The metadata service records the node status of the entire cluster and maintains very important information such as NameSpace, Table, and Partition. Some user-created tables are also implemented through the modification interface provided by the metadata service.

[0102] This section includes the design of metadata service master selection, metadata storage, metadata class, and load balancing. Metadata service master selection is how the metadata service selects the master service to provide metadata writing services when multiple instance services are deployed. Metadata storage describes how to store metadata, and load balancing is how metadata is calculated by sharding and partitioning.

[0103] 2.1 Design of metadata service leader selection

[0104] In order to improve the reliability of metadata services, a stateless metadata service approach is adopted to manage metadata. Figure 6 As shown in the Master service diagram, MasterService starts multiple instances, but saves almost no data itself. All metadata is stored using Zookeeper.

[0105] In order to avoid conflicts in distributed systems, multiple MasterServices need to perform a master election operation. Only the successfully elected MasterService can receive write requests. Each MasterService periodically creates a temporary node under the Zookeeper root node. The content is that if it successfully obtains the lock, it is considered a Leader node and can accept metadata write requests and perform data rebalance. The remaining masters can only receive read requests. When the client queries data, it needs to communicate with the master to obtain shard information.

[0106] Once the master goes down, the temporary node applied by the master in Zookeeper will automatically disappear due to the interruption of communication between the master and Zookeeper. Other slaves will start to lock the node under Zookeeper. The successfully locked node will be upgraded to the master node to execute node detection, data rebalance and other logic. This process is as follows Figure 7 The first Master election flow chart is shown.

[0107] 2.2 Design of metadata storage

[0108] ZooKeeper is a tree-type storage middleware. Metadata is stored in or under a node. In order to improve the utilization of Zookeeper, a root node is usually built first.

[0109] The metadata is shown in Table 1 below. The metadata mainly includes NameSpace Table Node Partition, etc. These will be represented by C++ classes generated by protobuf.

[0110] Table 1 Metadata node design

[0111]

[0112] NameSpace is similar to the concept of database. Each NameSpace does not affect each other and is provided to different businesses. Multiple tables can be established under each NameSpace. Each DataNode can only be mounted to one NameSpace. Subsequent data load balancing will also be performed on a NameSpace basis. Table is similar to the concept of database table. Node refers to data node (BS / RS). Partition is partition information and represents sharding. Because metadata interfaces often need to be serialized into protobuf, protobuf message is used to describe metadata. Metadata is stored in Zookeeper.

[0113] Because metadata is mounted in Zookeeper, the mount path is shown in Table 2. Therefore, multiple Table nodes can be mounted under one NameSpace. Node nodes are different from Table nodes. Node nodes are directly stored in a single node after being serialized by protobuf.

[0114] Table 2 Metadata node path design

[0115]

[0116] 2.3 Metadata Class Design

[0117] Although metadata is stored in Zookeeper, in order to improve maintainability and portability, metadata needs to be represented by protobuf.

[0118] In NameSpace, name represents the name of the namespace, which is a byte array. In fact, it stores binary data in string utf-8. type represents the database type, which supports offline storage or real-time storage, and is represented by string bs or rs. partition_num is the number of data shards. The number of shards is more important. It is the standard for partitioning after the database hashes the key. Due to the current data rebalance strategy, this value cannot be changed at will after it is specified. It is generally set to 4096 shards. Replica is the number of replicas. For reliable services, the number of replicas should be greater than or equal to 2. Otherwise, the service may be unavailable at a certain moment due to data node failure. However, an increase in the number of replicas will lead to excessive data redundancy. The total number of shards is replica*partition_num. version is the metadata version. Each time the namespace is updated, a timestamp will be generated as the version number. A certain interface is reserved for metadata recovery and other operations in the future. Rebalance is a parameter for performing data load balancing.

[0119] The Table node mainly describes the metadata class diagram of the table. For real-time tables, only name, type, cache_ttl, and name_space are valid. name is the table name; type is the table type, which is also available in offline and real-time. The table type must be mounted under the same type of namespace; cache_ttl is the validity period of the database cache. For offline tables, url and priority are both required. url represents the import path of the offline table from HDFS; priority represents the priority of data download from HDFS. The higher the priority, the higher the priority of data download.

[0120] The Node node mainly describes the metadata of the node. All DataNode nodes need to be added to the Node node list. It is convenient for the Master to perform rebalance and management functions of the entire cluster. The IP represents the IP of the node, and the port is the node's interface to the system. Some RPC services exposed internally, such as node information collection and metadata distribution, communicate through this port. The query port is the system's external port, and the client and others perform service operations such as get / set through this port. The sync port is the data synchronization port, and real-time storage will use this port for data synchronization communication operations.

[0121] Partition is the sharding information of the cluster. pid is the partition number, and nodes is the node list of the partition. The sharding information mainly describes which nodes a data shard is located in under the distributed database. In order to prevent a single machine failure from causing service unavailability, each shard should be distributed under different nodes. The sharding information is calculated by the master node through the rebalance operation and stored in Zookeeper.

[0122] 2.4 Metadata Service Class Design

[0123] MasterService is the RPC interface provided by Master to the outside world, MasterServiceImpl is its specific implementation, ZKClient is the C++ API provided by Zookeeper, which provides some operation interfaces about ZooKeeper and basic functions of adding, deleting, modifying and checking nodes. Storage is mainly responsible for the storage of metadata in the system, which is the encapsulation of ZKClient, reducing the coupling between the MaserServiceImpl class and the API provided by Zookeeper, and facilitating the replacement of non-Zookeeper with metadata storage to reduce class modifications. Syncer is the class for MasterService to communicate with DataNode for metadata, and sends metadata to DataNode through the send_sync_meta() method. Rebalancer is mainly responsible for calculations related to data load balancing, and distributes metadata to DataNode through Syncer. Collector also communicates with DataNode, responsible for node detection, and obtaining information related to memory and SSD capacity.

[0124] In MasterService, alter_namespace is an RPC interface for modifying the NameSpace provided by the user, and performs namespace modification at the NameSpace level. Users can create and delete NameSpace through this interface.

[0125] In order to facilitate the maintenance and management of your own tables, MasterServier provides alter_table. Users can modify the table structure in the metadata service through this interface.

[0126] The rebalance of a cluster usually needs to be controlled by the system administrator, so it is necessary to provide the system administrator with a rebalance management interface, namely the rebalance and disrebalance interfaces, so that the system administrator can control the entire rebalance process. The system administrator can control the rebalance behavior of the cluster through this interface.

[0127] 3. Implementation of metadata services

[0128] 3.1. Implementation of metadata storage and reading

[0129] Master accesses Zookeeper through the API provided by Zookeeper, but some of the access functions are repeated frequently, which can be solved by using template functions in C++. To facilitate debugging, the data stored in Zookeeper is not binary data but text data, which is more readable. In addition, the template function is used to complete the replacement during compilation, so the efficiency will be higher than the virtual function.

[0130] Implementation of data writing to a node:

[0131] The input is the Zookeeper path. By designing the template function, any generated C++ class can write the DebugString to the Zookeeper node.

[0132] template<typename T>

[0133] Status Storage::write_node(conststd::string&path,const T&v){

[0134] int ret=_zk_client->WriteData(

[0135] path.c_str(),v.ShortDebugString().c_str(),v.ShortDebugString().size(),

[0136] v.has_version()? v.version():-1);

[0137] / / ellipsis some code

[0138] returnconvert_ret(ret);

[0139] }

[0140] Implementation of data reading:

[0141] The data reading process is the opposite of the data writing process. The input is the Zookeeper path and the entity Proto type of the data deserialized through DebugString. By designing the template function, any generated C++ class can deserialize DebugString into the target proto.

[0142]

[0143] When reading in data, you need to pay attention to the memory that the buffer needs to apply for in the heap, because as the number of tables increases, stack overflow may occur.

[0144] 3.2 Implementation of Master Election

[0145] Master election process:

[0146] The Master attempts to create a temporary node in a certain directory at regular intervals. If the node is successfully created, the Master is considered a Leader node. Otherwise, it is considered a Follower node. The process is as follows: Figure 8 The second Master election flow chart is shown.

[0147] The specific implementation of creating a temporary node through C++API is as follows:

[0148] int ret=_zk_client->CreateEphemeral(_node_path.data(),_ip.c_str(),_ip.length()).

[0149] 3.3 Metadata service external interface

[0150] Because brpc needs to provide service interfaces, mainly including common operations such as adding, deleting, modifying and querying metadata, and also provides rebalance interfaces for cluster management. It provides users with Alter_NameSpace, Alter_Table and other interfaces.

[0151] Alter_NameSpace is mainly used for database management changes and other operations. The specific operation is to change the nodes saved in Zookeeper. Create a namespace node under the namespace node of Zookeeper. Then create other nodes such as node. These nodes need to be created during the namespace creation process. Subsequent specific tables and other nodes need to be mounted under these nodes. This process is as follows Fig. 9 The namespace creation flowchart shown.

[0152]

[0153]

[0154] Storage calls the method to write the data into Zookeeper in 2.1 to complete the modification of Zookeeper. Other metadata requests are similar to namespace requests, which are also maintenance of Zookeeper nodes, so they will not be repeated here.

[0155] 3.4. Implementation of load balancing

[0156] Implementation of the Recover process:

[0157] Recover first needs to fill in the missing partitions. If the current partition definition of ns does not match the actual partition in the metadata, add the partition to be restored to the list of partitions to be restored. In addition to the initial process, the inactive nodes need to be treated as replicas that have been lost.

[0158] if(distribution.partitions_size()!=_ns->partition_num()){

[0159] for(inti=0;i<_ns->partition_num();++i){

[0160] _partition_to_missing_replica_num[i]=_ns->replica_num();

[0161] _partition_to_replicas[i].clear();

[0162] }

[0163] for(auto p:distribution.partitions()){

[0164] _partition_to_missing_replica_num.erase(p.pid());

[0165] }

[0166] }

[0167] Then, we restore each partition that needs to be restored, that is, sort the nodes according to their load, select a node with a smaller load, and try to add the missing partition to it. If successful, we record the partition and start restoring the next partition. When the whole process is completed, the missing partition will be restored.

[0168] for(auto&&iter:_partition_to_missing_replica_num){

[0169] auto partition=iter.first;

[0170] automissing_replica_num=iter.second;

[0171] / / priority to recover the partition to little usage node

[0172] sort_nodes_by_usage(_active_nodes);

[0173] for(auto node:_active_nodes){

[0174] / / skip node if it already has same partition

[0175] if(!match_distribute_partition_strategy(partition,node)){

[0176] continue;

[0177] }

[0178] if(missing_replica_num--==0){

[0179] break;

[0180] }

[0181] add_partition_to_node(partition,node);

[0182] rebalance_log(partition,node,RECOVER_ADD);

[0183] }

[0184] }

[0185] Implementation of Garbage Collact:

[0186] The garbage shard recovery process is similar to the Recover process, except that it counts redundant shards, so it will not be described in detail.

[0187] Implementation of Rebalance:

[0188] Rebalance first needs to calculate the load of all nodes and find out the nodes that are overloaded and underloaded.

[0189] / / 1.find disproportion nodes

[0190] for(auto node:_active_nodes){

[0191] if(is_usage_overbalance_node(node)){

[0192] _overbalance_nodes.insert(node);

[0193] }else if(is_usage_underbalance_node(node)){

[0194] _underbalance_nodes.insert(node);

[0195] }

[0196] }

[0197] Rebalance then needs to traverse all nodes with lower loads and sort all nodes by load from high to low. The partitions in the high-load nodes are transferred to the low-load nodes in sequence. After the whole process is completed, the high-load data can be transferred to the low-load nodes.

[0198]

[0199] It should be noted that the above metadata design principle and implementation are relatively simple, and the performance is also a good implementation compared with the existing databases. As a kind of SSD storage that saves memory costs, it is already a storage system metadata design that saves memory resources very much, and the correctness is guaranteed.

[0200] Fig.10 A structural diagram of a load balancing device for a distributed database provided for one or more embodiments of the present specification, the device comprising: a shard copy processing unit 1002, a shard copy statistics unit 1004, a load rate determination unit 1006 and a load balancing unit 1008.

[0201] The shard copy processing unit 1002 recovers the missing shard copies of the distributed database and discards the redundant shard copies of the distributed database to obtain nodes in the distributed database that meet the requirements;

[0202] The shard copy counting unit 1004 traverses each node that meets the requirements and determines the number of shard copies in each node that meets the requirements;

[0203] The load rate determining unit 1006 determines the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements;

[0204] The load balancing unit 1008 adjusts the number of shard copies in each of the nodes that meet the requirements according to the load rate of each of the nodes that meet the requirements to achieve load balancing of the distributed database.

[0205] Fig.11 A structural diagram of a load balancing device for a distributed database provided for one or more embodiments of this specification includes:

[0206] at least one processor; and,

[0207] a memory communicatively connected to the at least one processor; wherein,

[0208] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0209] Recovering the missing shard copies of the distributed database and discarding the redundant shard copies of the distributed database to obtain nodes in the distributed database that meet the requirements;

[0210] Traverse each node that meets the requirements and determine the number of shard copies in each node that meets the requirements;

[0211] Determine the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements;

[0212] According to the load rate of each node that meets the requirements, the number of shard copies in each node that meets the requirements is adjusted to achieve load balancing of the distributed database.

[0213] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to:

[0214] Recovering the missing shard copies of the distributed database and discarding the redundant shard copies of the distributed database to obtain nodes in the distributed database that meet the requirements;

[0215] Traverse each node that meets the requirements and determine the number of shard copies in each node that meets the requirements;

[0216] Determine the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements;

[0217] According to the load rate of each node that meets the requirements, the number of shard copies in each node that meets the requirements is adjusted to achieve load balancing of the distributed database.

[0218] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0219] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0220] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A load balancing method for a distributed database, characterized in that: The method comprises: Recovering the missing shard copies of the distributed database and discarding the redundant shard copies of the distributed database to obtain nodes in the distributed database that meet the requirements; Traverse each node that meets the requirements and determine the number of shard copies in each node that meets the requirements; Determine the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements; According to the load rate of each node that meets the requirements, the number of shard copies in each node that meets the requirements is adjusted to complete the load balancing of the distributed database; The step of adjusting the number of shard copies in each of the nodes that meet the requirements according to the load rate of each of the nodes that meet the requirements specifically includes: Traversing the load rates of the nodes that meet the requirements, if the load rate of the first specified node that meets the requirements is greater than the average load, adjusting the number of shard copies in the first specified node that meets the requirements; The step of adjusting the number of shard copies in each of the nodes that meet the requirements according to the load rate of each of the nodes that meet the requirements specifically includes: Traverse the load rates of the nodes that meet the requirements, and if the load rate of the second specified node that meets the requirements is not greater than the average load, adjust the number of shard copies in the first specified node that meets the requirements to the second specified node that meets the requirements.

2. The method according to claim 1, characterized in that The recovering of the missing shard copies of the distributed database specifically includes: Scan the nodes in the distributed database and determine the number of shard copies of each node; Marking the missing shard copy; sorting the nodes of the distributed database according to the load rate; Add the marked missing shard replicas to nodes whose load ratio is below a preset value.

3. The method according to claim 1, characterized in that The discarding of redundant shard copies of the distributed database specifically includes: Scan the nodes in the distributed database and determine the number of shard copies of each node; According to the shard copy data of each node, determine the non-compliant node whose number of shard copies exceeds the preset quota; Redundant shard copies are determined in the non-compliant node, and the redundant shard copies are discarded.

4. The method according to claim 1, characterized in that: Determining the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements specifically includes: The load rate of each node that meets the requirements is determined based on the number of shard copies in each node that meets the requirements and a preset capacity.

5. The method according to claim 1, characterized in that After completing the load balancing of the distributed database, the method further includes: The node data of each node that meets the requirements is stored in a preset tree storage middleware.

6. A load balancing device for a distributed database, characterized in that: The device comprises: A shard copy processing unit is used to recover the missing shard copies of the distributed database and discard the redundant shard copies of the distributed database to obtain nodes in the distributed database that meet the requirements; A shard copy counting unit traverses each node that meets the requirements and determines the number of shard copies in each node that meets the requirements; A load rate determination unit, which determines the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements; A load balancing unit, which adjusts the number of shard copies in each of the nodes that meet the requirements according to the load rate of each of the nodes that meet the requirements, so as to complete the load balancing of the distributed database; The step of adjusting the number of shard copies in each of the nodes that meet the requirements according to the load rate of each of the nodes that meet the requirements specifically includes: Traversing the load rates of the nodes that meet the requirements, if the load rate of the first specified node that meets the requirements is greater than the average load, adjusting the number of shard copies in the first specified node that meets the requirements; The step of adjusting the number of shard copies in each of the nodes that meet the requirements according to the load rate of each of the nodes that meet the requirements specifically includes: Traverse the load rates of the nodes that meet the requirements, and if the load rate of the second specified node that meets the requirements is not greater than the average load, adjust the number of shard copies in the first specified node that meets the requirements to the second specified node that meets the requirements.

7. A load balancing device for a distributed database, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Recovering the missing shard copies of the distributed database and discarding the redundant shard copies of the distributed database to obtain nodes in the distributed database that meet the requirements; Traverse each node that meets the requirements and determine the number of shard copies in each node that meets the requirements; Determine the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements; According to the load rate of each node that meets the requirements, the number of shard copies in each node that meets the requirements is adjusted to complete the load balancing of the distributed database; The step of adjusting the number of shard copies in each of the nodes that meet the requirements according to the load rate of each of the nodes that meet the requirements specifically includes: Traversing the load rates of the nodes that meet the requirements, if the load rate of the first specified node that meets the requirements is greater than the average load, adjusting the number of shard copies in the first specified node that meets the requirements; The step of adjusting the number of shard copies in each of the nodes that meet the requirements according to the load rate of each of the nodes that meet the requirements specifically includes: Traverse the load rates of the nodes that meet the requirements, and if the load rate of the second specified node that meets the requirements is not greater than the average load, adjust the number of shard copies in the first specified node that meets the requirements to the second specified node that meets the requirements.

8. A non-volatile computer storage medium, characterized in that: A computer executable instruction is stored, wherein the computer executable instruction is configured to: Recovering the missing shard copies of the distributed database and discarding the redundant shard copies of the distributed database to obtain nodes in the distributed database that meet the requirements; Traverse each node that meets the requirements and determine the number of shard copies in each node that meets the requirements; Determine the load rate of each node that meets the requirements according to the number of shard copies in each node that meets the requirements; According to the load rate of each node that meets the requirements, the number of shard copies in each node that meets the requirements is adjusted to complete the load balancing of the distributed database; The step of adjusting the number of shard copies in each of the nodes that meet the requirements according to the load rate of each of the nodes that meet the requirements specifically includes: Traversing the load rates of the nodes that meet the requirements, if the load rate of the first specified node that meets the requirements is greater than the average load, adjusting the number of shard copies in the first specified node that meets the requirements; The step of adjusting the number of shard copies in each of the nodes that meet the requirements according to the load rate of each of the nodes that meet the requirements specifically includes: Traverse the load rates of the nodes that meet the requirements, and if the load rate of the second specified node that meets the requirements is not greater than the average load, adjust the number of shard copies in the first specified node that meets the requirements to the second specified node that meets the requirements.

Images