Shrinking method, device and equipment of distributed database and medium

Abstract

A method, device, equipment and medium for scaling down a distributed database are disclosed. The method comprises: determining a data node to be scaled down from a plurality of data nodes, and determining a first target migration node to which a first replica data sub-node on the data node to be scaled down is to be migrated, and a second target migration node to which a second replica data sub-node on the first target migration node is to be migrated; migrating the first replica data sub-node to the first target migration node and the second replica data sub-node to the second target migration node respectively; determining a third target migration node to which a first master data sub-node on the data node to be scaled down is to be migrated; migrating the first master data sub-node to the third target migration node, and deleting the data node to be scaled down from the distributed database. The present application scales down the distributed database through an online scaling-down scheme, so that the distributed database can provide normal services without interruption during the scaling-down process.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a method, apparatus, device and medium for scaling down a distributed database. Background Technology

[0002] Distributed databases (such as Greenplum Database) are high-performance databases built on top of PostgreSQL and employ a massively parallel processing (MPP) architecture, offering advantages such as high availability, high scalability, and cost-effectiveness.

[0003] During use, to adapt to the scale of data and the demands of computing power, Greenplum databases typically need to be adaptively scaled down. Currently, most methods achieve this scaling down based on backup and recovery. The specific process involves backing up all data stored on each data node (segment) of the Greenplum database to other storage devices (such as hard drives or disks). Then, the Greenplum database is disassembled, and segments that do not meet the requirements are removed. The remaining segments are used to rebuild a new Greenplum database. After the new Greenplum database is running normally, the data temporarily stored on other storage devices is imported back into the new Greenplum database. During the data import, data from at least one removed segment is evenly distributed into each segment of the new Greenplum database, thus achieving the scaling down of the Greenplum database.

[0004] However, the above scaling-down method requires GreenPlum database service to be interrupted, causing GreenPlum database to become unavailable. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for scaling down a distributed database. Through an online scaling-down scheme, the distributed database is scaled down so that the distributed database can provide services normally without interruption during the scaling-down process.

[0006] In a first aspect, embodiments of the present invention provide a method for scaling down a distributed database, the distributed database including a management node and multiple data nodes, each data node including a primary data sub-node and replica data sub-nodes, wherein the primary data sub-node and the replica data sub-nodes store different data; the method is executed by the management node, and the method includes:

[0007] From multiple data nodes, determine the data node to be scaled down, and determine the first target migration node to which the first replica data sub-node on the data node to be scaled down is to be migrated, and the second target migration node to which the second replica data sub-node on the first target migration node is to be migrated;

[0008] The first replica data child node is migrated to the first target migration node, and the second replica data child node is migrated to the second target migration node, respectively.

[0009] Determine the third target migration node to which the first primary data sub-node on the data node to be scaled up should be migrated;

[0010] Migrate the first primary data child node to the third target migration node, and delete the data node to be scaled down from the distributed database.

[0011] Secondly, embodiments of the present invention provide a scaling-down device for a distributed database, the distributed database including a management node and multiple data nodes, each data node including a primary data sub-node and a replica data sub-node, wherein the primary data sub-node and the replica data sub-node store different data; the device is configured on the management node and includes:

[0012] The first determining module is used to determine the data node to be scaled down from multiple data nodes, and to determine the first target migration node to which the first replica data sub-node on the data node to be scaled down is to be migrated, and the second target migration node to which the second replica data sub-node on the first target migration node is to be migrated.

[0013] The first control module is used to migrate the first replica data sub-node to the first target migration node and the second replica data sub-node to the second target migration node, respectively.

[0014] The second determining module is used to determine the third target migration node to which the first primary data sub-node on the data node to be scaled down is to be migrated.

[0015] The second control module is used to migrate the first primary data sub-node to the third target migration node and delete the data node to be scaled down from the distributed database.

[0016] Thirdly, embodiments of the present invention also provide an electronic device, comprising:

[0017] One or more processors;

[0018] Storage device for storing one or more programs.

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the distributed database scaling method described in any of the embodiments of the present invention.

[0020] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the scaling-down method of any of the distributed databases described in the embodiments of the present invention.

[0021] The technical solutions disclosed in the embodiments of the present invention have the following beneficial effects:

[0022] The process involves identifying the data node to be scaled down from multiple data nodes, determining the first target migration node to which the first replica data child node on the data node to be scaled down will be migrated, and the second target migration node to which the second replica data child node on the first target migration node will be migrated. The first replica data child node is then migrated to the first target migration node, and the second replica data child node is migrated to the second target migration node. Finally, the third target migration node to which the first primary data child node on the data node to be scaled down will be migrated, and the primary primary data child node is migrated to the third target migration node. The data node to be scaled down is then deleted from the distributed database. This online scaling-down scheme allows for the scaling down of a distributed database without service interruption, ensuring continued normal service operation. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the structure of a distributed database provided in an embodiment of the present invention;

[0024] Figure 2 This is a flowchart illustrating a method for scaling down a distributed database according to an embodiment of the present invention.

[0025] Figure 3 This is a flowchart illustrating another method for scaling down a distributed database provided in an embodiment of the present invention.

[0026] Figure 4(a) is a schematic diagram of a specific determination of the first target migration node of the first replica data sub-node on the data node to be scaled down, provided by an embodiment of the present invention;

[0027] Figure 4(b) is a schematic diagram of a specific migration of a first replica data sub-node to a first target migration node and a second replica data sub-node to a second target migration node provided by an embodiment of the present invention;

[0028] Figure 5 This is a flowchart illustrating another method for scaling down a distributed database provided in an embodiment of the present invention.

[0029] Figure 6(a) is a schematic diagram of an exemplary disjoint-set data structure provided by an embodiment of the present invention;

[0030] Figure 6(b) is a schematic diagram of a disjoint-set data structure for constructing a distributed database according to an embodiment of the present invention;

[0031] Figure 6(c) is a schematic diagram of a distributed database after scaling down according to an embodiment of the present invention;

[0032] Figure 7 This is a schematic diagram of the structure of a distributed database scaling-down device provided in an embodiment of the present invention;

[0033] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0034] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the embodiments of the present invention, and not all structures.

[0035] To clearly illustrate the embodiments of the present invention, the following description is provided in conjunction with... Figure 1 Provide an explanation of distributed databases (such as Greenplum). Figure 1 As shown, a distributed database consists of a master node and multiple data nodes (segments), connected via an interconnect protocol. The master node manages the operation of the entire distributed database, while the data nodes handle data storage and computation. To prevent database outages due to data node failures, mirroring is typically used. This involves equipping each data node with a mirror segment. Each data node includes a primary segment and a mirror segment, and these segments are distributed across different data nodes to mitigate the risk of a single data node failure rendering the distributed database unusable.

[0036] The following detailed description of the distributed database scaling method, apparatus, device, and medium according to embodiments of the present invention is provided with reference to the accompanying drawings.

[0037] Figure 2This is a flowchart illustrating a method for scaling down a distributed database according to an embodiment of the present invention. This embodiment is applicable to scenarios involving scaling down distributed databases, and the method can be executed by a distributed database scaling-down device configured on the management node. Figure 2 As shown, the method specifically includes the following:

[0038] S101, determine the data node to be scaled down from multiple data nodes, and determine the first target migration node to which the first replica data sub-node on the data node to be scaled down is to be migrated, and the second target migration node to which the second replica data sub-node on the first target migration node is to be migrated.

[0039] Typically, the management node in a distributed database establishes a connection with the client, responsible for receiving requests or instructions sent by the user through the client, and executing corresponding operations based on the requests or instructions. For example, if a user sends a query request through the client, the management node parses the request, forms an execution plan, then distributes the execution plan to each data node in parallel, collects the execution results from each data node, and then returns the results to the client. As another example, if a user sends a scaling-down instruction through the client, the management node parses the scaling-down instruction and, based on the identification information of the data node to be scaled down carried in the instruction, determines the data node to be scaled down from multiple data nodes.

[0040] In other words, when determining the data node to be scaled down from multiple data nodes in this embodiment, it can receive a scaling-down command sent by the user, which carries the identification information of the data node to be scaled down. Based on the identification information of the data node to be scaled down carried in the scaling-down command, the data node to be scaled down is determined from the multiple data nodes. In this embodiment, the identification information refers to information that can uniquely identify the identity of the data node to be scaled down, such as a number, serial number, or digital certificate. For example, if the identification information of the data node to be scaled down carried in the scaling-down command is number 11, then the management node will determine the data node with number 11 as the data node to be scaled down.

[0041] Since data nodes in a distributed database are responsible for data computation and storage, the primary data child nodes and replica data child nodes (first replica data child nodes) on the data node to be scaled down, as determined by the management node according to the scaling-down command, also store data. To ensure the security and integrity of the distributed database, this invention migrates the primary data child nodes and replica data child nodes on the data node to be scaled down to other data nodes in the distributed database, so that there is no data on the data node to be scaled down. Then, the data to be scaled down that has no data is deleted, thereby achieving the scaling-down purpose of the distributed database.

[0042] In a distributed database, each data node has a primary data child node and replica data child nodes that store the same data. Typically, each data node is serviced by the primary data child node. Only when the primary data child node malfunctions and cannot provide service will the service switch to the replica data child node that stores the same data as the primary data child node, thus improving the disaster recovery capability of the distributed database. In other words, when the primary data child node of a data node is running normally, the replica data child node that stores the same data as the primary data child node does not provide service.

[0043] Based on this, in this embodiment of the invention, when migrating the primary data child node and replica data child node storing different data on the data node to be scaled down to other data nodes in the distributed database, the first replica data child node on the data node to be scaled down can be migrated first, and then the primary data child node on the data node to be scaled down can be migrated. This ensures that the primary data child node can continue to provide services while migrating the first replica data child node.

[0044] Before migrating the first replica data child node on the data node to be scaled down, it is necessary to first determine the target migration node (first target migration node) from the other data nodes besides the data node to be scaled down. After determining the first target migration node, the first replica data child node on the data node to be scaled down can be migrated to the first target migration node. Since the first target migration node also originally stores primary data child nodes and replica data nodes (second replica data nodes) with different data, to avoid a sudden increase in the number of replica data child nodes on the first target migration node due to directly migrating the first replica data child node on the data node to be scaled down, which would cause an imbalance in the amount of data stored on each data node in the distributed database, this embodiment further determines the second target migration node for the second replica data child node on the first target migration node after determining the first target migration node.

[0045] In this embodiment of the invention, determining the first target migration node of the first replica data sub-node on the data node to be scaled down, and determining the second target migration node of the second replica data sub-node on the first target migration node, can be based on the number of replica data sub-nodes corresponding to the primary data sub-node on the data node to be scaled down, or based on at least one of the data node's load, network distance, and hardware status data, or can be determined in other ways, which are not specifically limited here.

[0046] S102, migrate the first replica data sub-node to the first target migration node and the second replica data sub-node to the second target migration node, respectively.

[0047] Optionally, after determining the first target migration node and the second target migration node, in this embodiment of the invention, the management node can control the data node to be scaled down, migrate the first replica data sub-node to the first target migration node, and control the first target migration node to migrate the second replica data sub-node to the second target migration node.

[0048] When controlling the data node to be scaled down and the first target migration node to migrate their respective replica data child nodes to the corresponding target data node, the following scenarios may be included:

[0049] Scenario 1

[0050] The management node can send data child node migration instructions to the data node to be scaled down and the first target migration node in parallel, so that the data node to be scaled down and the first target migration node can migrate their respective replica data child nodes to the corresponding target migration node.

[0051] Scenario 2

[0052] The management node first sends a data sub-node migration instruction to the data node to be scaled down. After the data node to be scaled down migrates the first replica data sub-node to the first target migration node, it sends a data sub-node migration instruction to the first target migration node so that the first target migration node migrates the second replica data sub-node to the second target migration node.

[0053] Scenario 3

[0054] The management node first sends a data sub-node migration instruction to the first target migration node. After the first target migration node migrates the second replica data sub-node to the second target migration node, it sends a data sub-node migration instruction to the data node to be scaled down, so that the data node to be scaled down will migrate the first replica data sub-node to the first target migration node.

[0055] It should be noted that the above scenarios are merely illustrative examples of embodiments of the present invention and are not intended to limit the scope of the invention.

[0056] Preferably, in this embodiment of the invention, when migrating the first replica data child node on the data node to be scaled down and the second replica data child node on the first target migration node to their respective target migration nodes, a streaming replication method can be used.

[0057] S103, determine the third target migration node to which the first primary data sub-node on the data node to be scaled up is to be migrated.

[0058] Optionally, after migrating the first replica data child node on the data node to be scaled down and the second replica data child node on the first target migration node to their respective target migration nodes, the management node can obtain the mapping relationship between the primary data child nodes and replica data child nodes on each data node; construct a disjoint-set data structure (UnionFind) based on the mapping relationship; and then determine the third target migration node of the first primary data child node on the data node to be scaled down based on the constructed UnionFind. Here, the disjoint-set data structure is a tree-like data structure used to handle the merging and querying of disjoint sets. It has wide applications in graph theory algorithms for determining connectivity and cycles, and can describe the relationship between the set of vertices and the set of edges in a graph with a relatively small space when dealing with extremely large amounts of data. Based on this data structure, the time complexity of connectivity and cycle detection can be greatly reduced.

[0059] S104, migrate the first primary data child node to the third target migration node, and delete the data node to be scaled down from the distributed database.

[0060] Optionally, since the second replica data child node stored on the first target migration node stores the same data as the primary data child node on the data node to be scaled down, in order for the distributed database to operate normally and provide services when migrating the primary data child node on the data node to be scaled down, this embodiment of the invention can temporarily switch the second replica data child node storing the same data as the first primary data child node on the data node to be scaled down to a new primary primary data child node before migrating the first primary data child node to the third target migration node. This allows the distributed database to continue providing services through the new primary primary data child node when migrating the first primary data child node on the data node to be scaled down, ensuring continuous service without database interruption due to the migration of the primary primary data child node, thus improving database performance.

[0061] In this embodiment, when migrating the first primary data child node on the data node to be scaled down to the third target migration node, streaming replication can be used. After the migration is complete, there is no data left on the data node to be scaled down. At this point, the data node to be scaled down can be deleted from the distributed database to achieve the purpose of scaling down the distributed database.

[0062] The technical solution provided by this invention identifies a data node to be scaled down from multiple data nodes, and determines a first target migration node to which a first replica data child node on the data node to be scaled down is to be migrated, and a second target migration node to which a second replica data child node on the first target migration node is to be migrated. The first replica data child node is migrated to the first target migration node, and the second replica data child node is migrated to the second target migration node. A third target migration node to which a first primary data child node on the data node to be scaled down is to be migrated is determined, and the first primary data child node is migrated to the third target migration node. Finally, the data node to be scaled down is deleted from the distributed database. Thus, through an online scaling-down scheme, the distributed database is scaled down without service interruption during the scaling-down process, ensuring normal service provision.

[0063] Based on the above embodiments, after deleting the data node to be scaled down from the distributed database, the option may also be to restore the new first primary data child node as a replica data child node.

[0064] Specifically, the replica data child node refers to the first replica data child node on the data node to be scaled down.

[0065] By restoring the new primary data child node to a replica data child node, the primary data child node on the migrated data node to be scaled down can be restored to a normal working state to provide services to the outside world.

[0066] Based on the above embodiments, after deleting the data nodes to be scaled down from the distributed database, the following optional steps are also included: if the identification information of multiple data nodes is a number, then the identification information of multiple data nodes in the scaled-down distributed database is updated to make the multiple data nodes more regular and orderly.

[0067] Figure 3 This is a flowchart illustrating another distributed database scaling-down method provided in this embodiment of the invention. This embodiment further optimizes the above embodiments. Figure 3 As shown, the method specifically includes:

[0068] S201, determine the data node to be scaled down from multiple data nodes.

[0069] S202, based on the mapping relationship between primary data child nodes and replica data child nodes on at least two data nodes, determine the data node with a second replica data child node that stores the same data as the first primary data child node, and determine the data node as the first target migration node.

[0070] Since the management node can obtain the mapping relationship between the primary data child nodes and the replica data child nodes on each data node in the distributed database, this embodiment of the invention can determine the first target migration node among the data nodes other than the data node to be scaled down based on the obtained mapping relationship.

[0071] In practice, the process can be as follows: First, the first primary data child node and the first replica data child node to be scaled down can be determined. Second, based on the first primary data child node and the obtained mapping relationship, the replica data child node (second replica data child node) that stores the same data as the first primary data child node can be determined from the other data nodes besides the data node to be scaled down. Finally, the data node to which the second replica data child node belongs can be determined and the data node can be identified as the first target migration node.

[0072] For example, as shown in Figure 4(a), suppose the distributed database has one management node (Master) and 10 data nodes, namely Node1, Node2, ..., Node9 and Node10. The mapping relationships between primary data child nodes and replica data child nodes on the 10 data nodes are as follows: {(Node1,P1-P4<->M5-M8),(Node2,P5-P8<->M9-M12),(Node3,P9-P12<->M13-M16),(Node4,P13-P16<->M17-M20),(Node5,P17-P20<->M1-M4)},{(Node6,P21-P24<->M25-M28),(Node7,P25-P28<->M29-M32),(Node8,P29-P32<->M33-M36),(Node9,P33-P36<->M37-M40),(Node10,P37-P40<->M21-M24)}.

[0073] If the Master determines Node5 as the data node to be scaled down based on the scaling-down instruction, then according to the mapping relationship, the primary data child nodes on Node5 are determined to be: P17-P20; the first replica data child nodes are: M1-M4. Then, according to the mapping relationship, the second replica data child nodes storing the same data as P17-P20 are determined to be: M17-M20, and the data node to which M17-M20 belongs is: Node4. Therefore, Node4 can be determined as the first target migration node for M1-M4 on the data node to be scaled down, Node5.

[0074] S203, based on load, network distance and / or hardware status data, determine the second target migration node to which the second replica data sub-node on the first target migration node is to be migrated from among the other data nodes besides the data node to be scaled down.

[0075] The number of second target migration nodes is greater than or equal to 1 and less than or equal to the number of second replica data child nodes.

[0076] Since the first target migration node determined in S202 also originally stores a replica data node (second replica data node), this embodiment of the invention can also determine the second replica data child node on the first target migration node, so as to avoid directly migrating the first replica data child node on the data node to be scaled down to the first target migration node, which would cause a sudden increase in the number of replica data child nodes on the first target migration node, resulting in an imbalance in the amount of data stored on each data node in the distributed database.

[0077] Optionally, the Top K Minimum algorithm can be used to determine the second target migration node for the second replica data child node on the first target migration node, based on load, network distance, and / or hardware status data, among other data nodes besides the data node to be scaled down. In this embodiment of the invention, the Top K Maximum or Top K Minimum (TOPK) algorithm is an efficient algorithm for obtaining the top K elements of a certain dimension of data when processing massive amounts of data, where K can be any positive integer. Common application scenarios include finding the top K most frequent elements in massive amounts of data or finding the set of elements with the largest size (TOPK).

[0078] Among them, load is the total number of primary data sub-nodes and replica data sub-nodes running on each data node; network distance is the distance between racks where each data node is deployed. For example, if data node 1 and data node 5 are deployed on the same rack, it means that data node 1 and data node 5 are close in network distance; hardware status data refers to the hardware level of the data node, which includes at least one of the following: hard disk capacity, memory size, and CPU performance.

[0079] In specific implementation, the management node can determine the data node with the smallest load from among the data nodes other than the data node to be scaled down, based on the number W of second replica data child nodes on the first target migration node. If the number of data nodes with the smallest load is less than or equal to W, then the data node with the smallest load is determined as the second target migration node of the second replica data child node. If the number of data nodes with the smallest load is greater than W, then the data node with the smallest network distance is determined from among the data nodes with the smallest load. If the number of data nodes with the smallest network distance is less than or equal to W, then the data node with the smallest network distance is determined as the second target migration node of the second replica data child node. If the number of data nodes with the smallest network distance is greater than W, then the data node with the largest hardware status data is determined from among the data nodes with the smallest network distance. If the number of data nodes with the largest hardware status data is less than or equal to W, then the data node with the largest hardware status data is determined as the second target migration node of the second replica data child node. Otherwise, W+1 is controlled, and the process returns to the step of determining the data node with the smallest load.

[0080] It should be noted that in this embodiment, the load, network distance and / or hardware status data refer to the load; or, the load and network distance; or, the load, network distance and hardware status data, etc., and are not specifically limited here.

[0081] Continuing with Figure 4(a) as an example, when the first target migration node is Node4, the Master, based on the mapping relationship, determines that the second replica data child nodes on Node4 are M17-M20, with a total number of 4. Since there are 9 other data nodes in the distributed database besides the node to be scaled down (Node5), the Master can first select the 4 data nodes with the lowest load from these 9 nodes using the "top 4" algorithm. If the 4 data nodes with the lowest load are Node1, Node2, Node3, Node4, and Node7, where Node2 and Node3 have the same load, then there are 5 data nodes that satisfy the "top 4" load requirement, meaning the number of these 5 data nodes is greater than the number of second replica data child nodes. Therefore, the Master can select the 4 data nodes with the lowest network distance from these 5 data nodes using the "top 4" algorithm. If the four smallest data nodes in the network distance are Node1, Node2, Node3, and Node4, and the number of the four smallest data nodes in the network distance is equal to the number of second replica data child nodes, then Node1, Node2, Node3, and Node4 can be randomly determined as the second target migration node for any second replica data child node.

[0082] It is worth noting that, in this embodiment, after determining the second target migration node to which the second replica data child node on the first target migration node is to be migrated, in addition to determining the corresponding second target migration node for each second replica data child node in a random manner, the corresponding second target migration node can also be determined for each second replica data child node in the order of the second target migration nodes; or, the corresponding second target migration node can also be determined for each second replica data child node in other preset ways, etc., and no specific limitation is made here.

[0083] In one embodiment of the present invention, since the number of second target migration nodes determined based on load, network distance and / or hardware status data may be less than the number of second replica data child nodes, at least two second replica data child nodes need to be migrated in any second target migration node.

[0084] Specifically, the following method can be used to determine the corresponding target migration node from at least two second replica data child nodes among the determined second target migration nodes. For example, after determining the second target migration node for the second replica data child nodes on the first target migration node, the method may further include: if the number of second target migration nodes is less than the number of second replica data child nodes, then select a data node with low load and good hardware status data from the second target migration nodes, and use this data node as the target migration node for at least two second replica data child nodes on the first target migration node.

[0085] For example, the second target migration nodes include Node1, Node2, and Node3, and the number of second replica data child nodes is four. In this case, a data node with low load and good hardware status data can be selected from Node1, Node2, and Node3. If Node2 is selected, then Node2 is determined as the target migration node for any two replica data child nodes. That is, in this embodiment, one second replica data child node can be migrated to Node1, two second replica data child nodes can be migrated to Node2, and one second replica data child node can be migrated to Node3.

[0086] S204, migrate the first replica data child node to the first target migration node and the second replica data child node to the second target migration node, respectively.

[0087] In other words, after determining the first target migration node to which the first replica data child node on the data node to be scaled up should be migrated, and the second target migration node to which the second replica data child node on the first target migration node should be migrated, the management node can control the data node to be scaled up to migrate the first replica data child node to the first target migration node, and control the first target migration node to migrate the second replica data child node to the second target migration node. Specifically, controlling the first target migration node to migrate the second replica data child node to the second target migration node includes: if the number of second target migration nodes is equal to the number of second replica data child nodes, then the second replica data child nodes are migrated to their respective second target migration nodes.

[0088] Continuing with the example in Figure 4(a), after determining that the first target migration node for the first replica data child nodes M1-M4 on the data node to be scaled down is Node4, and the second target migration nodes for the second replica data child nodes on Node4 are Node1, Node2, Node3 and Node4 respectively, the Master can control Node5 to migrate the first replica data child nodes M1-M4 to Node4 in a streaming replication manner, and control Node4 to randomly migrate each of the second replica data child nodes M17-M20 to Node1, Node2, Node3 or Node4 in a streaming replication manner, so as to obtain the result shown in Figure 4(b).

[0089] S205, determine the third target migration node of the first primary data sub-node on the data node to be scaled down.

[0090] S206, migrate the first primary data child node to the third target migration node, and delete the data node to be scaled down from the distributed database.

[0091] The technical solution provided by this invention provides an online scaling-down scheme for distributed databases, ensuring uninterrupted service during the scaling-down process and guaranteeing normal service delivery. Furthermore, by determining a second target migration node for the second replica data child nodes on the first target migration node and migrating the second replica data child nodes to the second target migration node, it is ensured that the amount of data stored on each data node remains balanced, resulting in higher load balancing performance for the distributed database.

[0092] Figure 5 This is a flowchart illustrating another distributed database scaling-down method provided in this embodiment of the invention. This embodiment is further optimized based on the above embodiment. Figure 5 As shown, the method specifically includes the following:

[0093] S301, determine the data node to be scaled down from multiple data nodes, and determine the first target migration node to which the first replica data sub-node on the data node to be scaled down is to be migrated, and the second target migration node to which the second replica data sub-node on the first target migration node is to be migrated.

[0094] S302, migrate the first replica data sub-node to the first target migration node and the second replica data sub-node to the second target migration node, respectively.

[0095] S303, using the primary data child nodes on other data nodes and the replica data child nodes that store the same data as the primary data child nodes as tree nodes, and using the mapping relationship between the primary data child nodes on other data nodes and the replica data child nodes that store the same data as the primary data child nodes as edges, construct the disjoint-set data structure of the distributed database.

[0096] S304, based on the replica data sub-nodes that store the same data as the first primary data sub-node, determine the third target migration node of the first primary data sub-node in the disjoint lookup set.

[0097] In this embodiment of the invention, the disjoint-set data structure is a tree-like data structure used to handle the merging and querying of disjoint sets. It has wide applications in graph theory algorithms for determining connectivity and cycles, and can describe the relationship between a set of points and edges in a graph with a relatively small space, even with extremely large amounts of data. This data structure can significantly reduce the time complexity of connectivity and cycle detection. As shown in Figure 6(a), if there is a mapping relationship between primary data node A and replica data node B, and between primary data node C and replica data node D, then after adding primary data node C to primary data node A, replica data node D, which has a mapping relationship with C, also belongs to replica data node B.

[0098] Specifically, the third target migration node of the first primary data child node on the data node to be scaled up can be determined based on the constructed disjoint-set data structure. As an optional implementation, this embodiment determines the third target migration node of the first primary data node on the data node to be scaled up based on the constructed disjoint-set data structure, which may include: finding a third replica data child node from the disjoint-set data structure that merges with replica data child nodes storing the same data as the first primary data child node; determining the data node to which the second primary data child node storing the same data as the third replica data child node belongs based on the mapping relationship between primary data child nodes and replica data child nodes on other data nodes; and determining the data node to which the second primary data child node belongs as the third target migration node of the first primary data child node on the data node to be scaled up.

[0099] For example, as shown in Figure 4(b) of the above embodiment, the mapping relationships between the primary data child nodes and replica data child nodes on the 10 data nodes Node1, Node2, ..., Node9 and Node10 are as follows: {(Node1, P1-P4 <-> M5-M8, M17), (Node2, P5-P8 <-> M9-M12, M18), (Node3, P9-P12 <-> M13-M16, M19), (Node4, P1...} 3-P16<->M1-M4,M20),(Node5,P17-P20<->)},{(Node6,P21-P24<->M25-M28),(Node7,P25-P28<->M29 -M32), (Node8,P29-P32<->M33-M36), (Node9,P33-P36<->M37-M40), (Node10,P37-P40<->M21-M24)}. Therefore, based on the mapping relationship between the primary data child nodes and replica data child nodes on the other data nodes except for the data node to be scaled down Node5, and the primary data child nodes and replica data child nodes storing the same data, the Master constructs the disjoint-set data structure of the distributed database, with the primary data child nodes on other data nodes and the replica data child nodes storing the same data as the primary data child nodes as tree nodes, and the mapping relationship between the primary data child nodes on other data nodes and the replica data child nodes storing the same data as the primary data child nodes as edges. As shown in Figure 6(b).

[0100] After obtaining the disjoint-set data structure of the distributed database, the management node, based on the replica data sub-nodes (M17-M20) storing the same data as the first primary data sub-nodes (P17-P20) on the data node to be scaled down (Node5), determines in the disjoint-set data structure that M17 is merged with M5-M8, M18 with M9-M12, M19 with M13-M16, and M20 with M1-M4. Furthermore, based on the mapping relationships, it is determined that M17 and P17 store the same data, and M5-M8 and P5-P8 store the same data (i.e., a mapping relationship exists); M18 and P18 store the same data, and M9-M12 and P9-P12 store the same data; M19 and P19 store the same data, and M13-M16 and P13-P16 store the same data; M20 and P20 store the same data, and M1-M4 and P1-P4 store the same data.

[0101] Furthermore, according to Figures 4(b) and 6(b), the primary data child nodes P5-P8 mapped to the replica data child nodes M5-M8 on data node Node1 are located on data node Node2. Therefore, Node2 can be identified as the third target migration node for P17. Similarly, the primary data child nodes P9-P12 mapped to the replica data child nodes M9-M12 on data node Node2 are located on data node Node3. Therefore, Node3 can be identified as the third target migration node for P18. The three target migration nodes are as follows: the replica data child nodes M13-M16 on data node Node3 are mapped to the primary data child nodes P13-P16, which are located on data node Node4. Therefore, the data node to which P13-P16 belongs is Node4. Thus, Node4 can be identified as the third target migration node to which P19 will be migrated. The replica data child nodes M1-M4 on data node Node4 are mapped to the primary data child nodes P1-P4, which are located on data node Node1. Therefore, Node1 can be identified as the third target migration node to which P20 will be migrated.

[0102] S305, migrate the first primary data child node to the third target migration node, and delete the data node to be scaled down from the distributed database.

[0103] Continuing with Figure 4(b) above as an example, after migrating the first primary data child nodes: P17-P20 on Node5 to Node1, Node2, Node3 and Node4 in the form of streaming replication, there are no data child nodes on Node5. At this time, the management node can delete Node5 from the distributed database to achieve the purpose of shrinking the distributed database. The specific shrunk distributed database can be shown in Figure 6(c).

[0104] The technical solution provided by this invention offers an online scaling-up scheme for distributed databases, ensuring uninterrupted service during the scaling-up process. Furthermore, based on the constructed disjoint-set data structure, the speed and accuracy of determining the third target migration node for the primary data child nodes on the data nodes to be scaled up can be improved.

[0105] Figure 7This is a schematic diagram of a distributed database scaling-down device provided in an embodiment of the present invention. The distributed database includes a management node and multiple data nodes. Each data node includes a primary data sub-node and replica data sub-nodes, wherein the primary data sub-node and the replica data sub-node store different data. The security early warning device is configured on the management node and can be implemented in hardware and / or software, and configured in an electronic device. Figure 7 As shown, the distributed database scaling-down device 700 of this embodiment includes: a first determining module 710, a first control module 720, a second determining module 730, and a second control module 740.

[0106] The first determining module 710 is used to determine the data node to be scaled down from multiple data nodes, and to determine the first target migration node to which the first replica data sub-node on the data node to be scaled down is to be migrated, and the second target migration node to which the second replica data sub-node on the first target migration node is to be migrated.

[0107] The first control module 720 is used to migrate the first replica data sub-node to the first target migration node and the second replica data sub-node to the second target migration node, respectively.

[0108] The second determining module 730 is used to determine the third target migration node to which the first primary data sub-node on the data node to be scaled up is to be migrated.

[0109] The second control module 740 is used to migrate the first primary data sub-node to the third target migration node and delete the data node to be scaled down from the distributed database.

[0110] As an optional implementation of this invention, the distributed database scaling device 700 further includes: a third control module;

[0111] The third control module is used to switch the migrated second replica data sub-node to the new first primary data sub-node.

[0112] If the first primary data node storing the same data as the second replica data node is migrated to the third target migration node, then the new first primary data node will be restored as a replica data node.

[0113] As an optional implementation of this invention, the first determining module 710 is specifically used for:

[0114] Based on the mapping relationship between primary data child nodes and replica data child nodes on at least two data nodes, a data node with a second replica data child node storing the same data as the first primary data child node is determined, and the data node is determined as the first target migration node.

[0115] As an optional implementation of this invention, the first determining module 710 is specifically used for:

[0116] Based on load, network distance, and / or hardware status data, among the data nodes other than the data node to be scaled down, determine the second target migration node to which the second replica data child node on the first target migration node is to be migrated, wherein the number of the second target migration nodes is greater than or equal to 1 and less than or equal to the number of the second replica data child nodes.

[0117] As an optional implementation of this invention, the distributed database scaling-down device 700 further includes: a third determining module;

[0118] The third determining module is used to migrate the second replica data sub-nodes to the corresponding second target migration nodes if the number of the second target migration nodes is equal to the number of the second replica data sub-nodes.

[0119] If the number of the second target migration nodes is less than the number of the second replica data child nodes, then select a data node with a small load and good hardware status data from the second target migration nodes, and use this data node as the target migration node for at least two second replica data child nodes on the first target migration node.

[0120] As an optional implementation of this invention, the second determining module 730 is specifically used for:

[0121] Using the primary data child nodes on other data nodes and the replica data child nodes that store the same data as the primary data child nodes as tree nodes, and the mapping relationship between the primary data child nodes on other data nodes and the replica data child nodes that store the same data as the primary data child nodes as edges, the disjoint-set data structure of the distributed database is constructed.

[0122] Based on the replica data sub-nodes that store the same data as the first primary data sub-node, the third target migration node of the first primary data sub-node on the data node to be scaled up is determined in the disjoint lookup set.

[0123] As an optional implementation of this invention, the second determining module 730 is further configured to:

[0124] From the merge lookup set, find the third replica data sub-node that is merged with the replica data sub-node that stores the same data as the first primary data sub-node;

[0125] Based on the mapping relationship between primary data child nodes and replica data child nodes on other data nodes, determine the data node to which the second primary data child node that stores the same data as the third replica data child node belongs;

[0126] The data node to which the second primary data sub-node belongs is determined as the third target migration node of the first primary data sub-node on the data node to be scaled down.

[0127] It should be noted that the foregoing explanation of the distributed database scaling method embodiment also applies to the distributed database scaling device of this embodiment, and its implementation principle is similar, so it will not be repeated here.

[0128] The technical solution provided by this invention identifies a data node to be scaled down from multiple data nodes, and determines a first target migration node to which a first replica data child node on the data node to be scaled down will be migrated, and a second target migration node to which a second replica data child node on the first target migration node will be migrated. The first replica data child node is migrated to the first target migration node, and the second replica data child node is migrated to the second target migration node. A third target migration node is determined for the first primary data child node on the data node to be scaled down, and the first primary data child node is migrated to the third target migration node. Finally, the data node to be scaled down is deleted from the distributed database. Thus, through an online scaling-down scheme, the distributed database is scaled down without service interruption during the scaling-down process, ensuring normal service provision.

[0129] To achieve the above objectives, embodiments of the present invention also propose an electronic device.

[0130] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 8 A block diagram is shown of an exemplary electronic device 800 suitable for implementing embodiments of the present invention. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0131] like Figure 8 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: one or more processors or processing units 810, system memory 820, and bus 830 connecting different system components (including system memory 820 and processing unit 810).

[0132] Bus 830 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0133] Electronic device 800 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 800, including volatile and non-volatile media, removable and non-removable media.

[0134] System memory 820 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 821 and / or cache memory 822. Electronic device 800 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 823 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 8 Not shown; usually referred to as a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 830 via one or more data media interfaces. Memory 820 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0135] A program / utility 824 having a set (at least one) of program modules 825 may be stored, for example, in memory 820. Such program modules 825 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 825 typically perform the functions and / or methods described in the embodiments of the present invention.

[0136] Electronic device 800 can also communicate with one or more external devices 840 (e.g., keyboard, pointing device, display 841, etc.), and with one or more devices that enable a user to interact with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0137] Processing unit 810 executes various functional applications and data processing by running programs stored in system memory 820, such as implementing the distributed database scaling method provided in this embodiment of the invention. The distributed database includes a management node and multiple data nodes, each data node including a primary data sub-node and replica data sub-nodes, wherein the primary data sub-node and the replica data sub-nodes store different data. The method is executed by the management node and includes:

[0138] From multiple data nodes, determine the data node to be scaled down, and determine the first target migration node to which the first replica data sub-node on the data node to be scaled down is to be migrated, and the second target migration node to which the second replica data sub-node on the first target migration node is to be migrated;

[0139] The first replica data child node is migrated to the first target migration node, and the second replica data child node is migrated to the second target migration node, respectively.

[0140] Determine the third target migration node to which the first primary data sub-node on the data node to be scaled up should be migrated;

[0141] Migrate the first primary data child node to the third target migration node, and delete the data node to be scaled down from the distributed database.

[0142] It should be noted that the explanation of the aforementioned embodiment of the distributed database scaling method also applies to the electronic device of this embodiment, and its implementation principle is similar, so it will not be repeated here.

[0143] The technical solution provided by this invention identifies a data node to be scaled down from multiple data nodes, and determines a first target migration node to which a first replica data child node on the data node to be scaled down will be migrated, and a second target migration node to which a second replica data child node on the first target migration node will be migrated. The first replica data child node is migrated to the first target migration node, and the second replica data child node is migrated to the second target migration node. A third target migration node is determined for the first primary data child node on the data node to be scaled down, and the first primary data child node is migrated to the third target migration node. Finally, the data node to be scaled down is deleted from the distributed database. Thus, through an online scaling-down scheme, the distributed database is scaled down without service interruption during the scaling-down process, ensuring normal service provision.

[0144] To achieve the above objectives, the present invention also proposes a computer-readable storage medium.

[0145] The computer-readable storage medium provided in this embodiment of the invention stores a computer program thereon. When executed by a processor, the program implements the distributed database scaling method as described in this embodiment of the invention. The distributed database includes a management node and multiple data nodes. Each data node includes a primary data sub-node and a replica data sub-node, wherein the primary data sub-node and the replica data sub-node store different data. The method is executed by the management node and includes:

[0146] From multiple data nodes, determine the data node to be scaled down, and determine the first target migration node to which the first replica data sub-node on the data node to be scaled down is to be migrated, and the second target migration node to which the second replica data sub-node on the first target migration node is to be migrated;

[0147] The first replica data child node is migrated to the first target migration node, and the second replica data child node is migrated to the second target migration node, respectively.

[0148] Determine the third target migration node to which the first primary data sub-node on the data node to be scaled up should be migrated;

[0149] Migrate the first primary data child node to the third target migration node, and delete the data node to be scaled down from the distributed database.

[0150] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0151] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0152] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0153] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for scaling down a distributed database, characterized in that, The distributed database includes a management node and multiple data nodes. Each data node includes a primary data sub-node and a replica data sub-node, wherein the primary data sub-node and the replica data sub-node store different data. The method is executed by the management node, and the method includes: From multiple data nodes, determine the data node to be scaled down, and determine the first target migration node to which the first replica data sub-node on the data node to be scaled down is to be migrated, and the second target migration node to which the second replica data sub-node on the first target migration node is to be migrated; The first replica data child node is migrated to the first target migration node, and the second replica data child node is migrated to the second target migration node, respectively. Determine the third target migration node to which the first primary data sub-node on the data node to be scaled up should be migrated; Migrate the first primary data child node to the third target migration node, and delete the data node to be scaled down from the distributed database; Wherein, the first replica data sub-node is a replica data node on the data node to be scaled down, and the second replica data sub-node is a replica data node on the first target migration node; The step of determining the third target migration node to which the first primary data child node on the data node to be scaled up is to be migrated includes: Using the primary data child nodes on other data nodes and the replica data child nodes that store the same data as the primary data child nodes as tree nodes, and the mapping relationship between the primary data child nodes on other data nodes and the replica data child nodes that store the same data as the primary data child nodes as edges, the disjoint-set data structure of the distributed database is constructed. Based on the replica data sub-nodes that store the same data as the first primary data sub-node, the third target migration node of the first primary data sub-node on the data node to be scaled up is determined in the disjoint lookup set.

2. The method according to claim 1, characterized in that, Before migrating the first primary data child node to the third target migration node, the method further includes: Switch the migrated second replica data child node to the new first primary data child node; If the first primary data node storing the same data as the second replica data node is migrated to the third target migration node, then the new first primary data node will be restored as a replica data node.

3. The method according to claim 1, characterized in that, The step of determining the first target migration node to which the first replica data child node on the data node to be scaled up should be migrated includes: Based on the mapping relationship between primary data child nodes and replica data child nodes on at least two data nodes, a data node with a second replica data child node storing the same data as the first primary data child node is determined, and the data node is determined as the first target migration node.

4. The method according to claim 1, characterized in that, The step of determining the second target migration node to which the second replica data child node on the first target migration node is to be migrated includes: Based on load, network distance, and / or hardware status data, among the data nodes other than the data node to be scaled down, determine the second target migration node to which the second replica data child node on the first target migration node is to be migrated, wherein the number of the second target migration nodes is greater than or equal to 1 and less than or equal to the number of the second replica data child nodes.

5. The method according to claim 4, characterized in that, After determining the second target migration node to which the second replica data child node on the first target migration node is to be migrated, the method further includes: If the number of the second target migration nodes is equal to the number of the second replica data child nodes, then the second replica data child nodes will be migrated to the corresponding second target migration nodes. If the number of the second target migration nodes is less than the number of the second replica data child nodes, then select a data node with a small load and good hardware status data from the second target migration nodes, and use this data node as the target migration node for at least two second replica data child nodes on the first target migration node.

6. The method according to claim 1, characterized in that, The step of determining the third target migration node of the first primary data child node on the data node to be scaled down in the disjoint-setup includes: From the merge lookup set, find the third replica data sub-node that is merged with the replica data sub-node that stores the same data as the first primary data sub-node; Based on the mapping relationship between primary data child nodes and replica data child nodes on other data nodes, determine the data node to which the second primary data child node that stores the same data as the third replica data child node belongs; The data node to which the second primary data sub-node belongs is determined as the third target migration node of the first primary data sub-node on the data node to be scaled down.

7. A scaling-down device for a distributed database, characterized in that, The distributed database includes a management node and multiple data nodes. Each data node includes a primary data sub-node and a replica data sub-node, wherein the primary data sub-node and the replica data sub-node store different data. The device is configured on the management node and includes: The first determining module is used to determine the data node to be scaled down from multiple data nodes, and to determine the first target migration node to which the first replica data sub-node on the data node to be scaled down is to be migrated, and the second target migration node to which the second replica data sub-node on the first target migration node is to be migrated. The first control module is used to migrate the first replica data sub-node to the first target migration node and the second replica data sub-node to the second target migration node, respectively. The second determining module is used to determine the third target migration node to which the first primary data sub-node on the data node to be scaled down is to be migrated. The second control module is used to migrate the first primary data sub-node to the third target migration node and delete the data node to be scaled down from the distributed database. Wherein, the first replica data sub-node is a replica data node on the data node to be scaled down, and the second replica data sub-node is a replica data node on the first target migration node; Specifically, the second determining module is used for: Using the primary data child nodes on other data nodes and the replica data child nodes that store the same data as the primary data child nodes as tree nodes, and the mapping relationship between the primary data child nodes on other data nodes and the replica data child nodes that store the same data as the primary data child nodes as edges, the disjoint-set data structure of the distributed database is constructed. Based on the replica data sub-nodes that store the same data as the first primary data sub-node, the third target migration node of the first primary data sub-node on the data node to be scaled up is determined in the disjoint lookup set.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the scaling-down method of the distributed database as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, this program implements the scaling-down method for a distributed database as described in any one of claims 1-6.

Images