Data redistribution method and electronic device

Abstract

An embodiment of the present application discloses a data redistribution method and electronic device, the method being applied to a coordination node, the coordination node being used to manage multiple data nodes, the method comprising: determining, among multiple data nodes, whether there is a data table in a data table to be redistributed with a corresponding amount of data greater than or equal to a first threshold; determining the current disk IO resource usage of the multiple data nodes; and redistributing the data of the data table to be redistributed to some or all of the multiple data nodes based on the current disk IO resource usage; wherein the current disk IO resource usage is determined based on the current disk IO resource usage of some or all of the multiple data nodes, and the data table to be redistributed is a data table to be redistributed among the data tables included in the multiple data nodes. The embodiment of the present application can effectively utilize the currently remaining disk IO resources, thereby improving the efficiency of data redistribution.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data redistribution method and electronic equipment. Background Art

[0002] With the rapid development of network and communication technologies, massive amounts of data are impacting various industries at an unprecedented rate. To effectively manage, maintain, and process this data, users are increasingly demanding database storage capacity and performance. Distributed cluster databases, due to their high availability, scalability, and load balancing, are increasingly widely used.

[0003] Due to the scalability of distributed cluster databases, the number of nodes in a distributed cluster database can be adjusted based on business needs. For example, as data continues to grow, the database cluster can be expanded to support larger-scale data storage and queries by adding new nodes. Furthermore, to fully utilize the newly added nodes and achieve load balancing across each node, after adding the new node to the database cluster, the data stored on the original node can be redistributed across all the expanded nodes.

[0004] At the same time, how to design a suitable redistribution strategy, fully utilize disk input / output (IO) resources, and improve data redistribution efficiency is a problem that technical personnel are concerned about. Summary of the Invention

[0005] The embodiments of the present application disclose a data redistribution method and an electronic device for improving data redistribution efficiency.

[0006] The first aspect discloses a data redistribution method, which can be applied to a coordination node, or to a module (e.g., a chip) in a coordination node, or to a logic module or software that can implement all or part of the coordination node function. The coordination node can be used to manage multiple data nodes, and the following description will be given using the application to the coordination node as an example. The data redistribution method may include: determining that among the multiple data nodes, there is a data table in the data table to be redistributed whose corresponding data volume is greater than or equal to a first threshold; determining the current disk input and output IO resource utilization of the multiple data nodes; and based on the current disk IO resource utilization, redistributing the data in the data table to be redistributed to some or all of the data nodes in the multiple data nodes; wherein the current disk IO resource utilization is determined based on the current disk IO resource utilization of some or all of the data nodes in the multiple data nodes, and the data table to be redistributed is a data table to be redistributed among the data tables included in the multiple data nodes.

[0007] In the embodiment of the present application, due to the different data volumes in the data tables, the disk IO resources occupied during data redistribution are different. Furthermore, a data table with a data volume less than the first threshold value may be a small table, and substantially does not occupy disk IO resources during data redistribution. Therefore, when a corresponding data table with a data volume greater than or equal to the first threshold value exists in the data table to be redistributed, the coordination node may redistribute the data in the data table to be redistributed to some or all of the multiple data nodes based on the current disk input / output IO resource usage. In this way, the currently remaining disk IO resources can be effectively utilized, thereby improving the efficiency of data redistribution.

[0008] As a possible implementation, before determining that among the multiple data nodes, there is a data table in the data table to be redistributed with a corresponding data amount greater than or equal to a first threshold, the method may further include: creating M threads, where M is an integer greater than or equal to 1; and redistributing the data of the data table to be redistributed to some or all of the multiple data nodes includes: redistributing the data of the data table to be redistributed to some or all of the multiple data nodes through the M threads.

[0009] In an embodiment of the present application, the coordination node may perform data redistribution of the data table to be redistributed in parallel through M threads, thereby improving the efficiency of data redistribution.

[0010] As a possible implementation method, the data of the data table to be redistributed is distributed to some or all of the multiple data nodes based on the current disk IO resource utilization, specifically including: determining a first disk IO resource utilization interval corresponding to the first disk IO resource utilization, where the first disk IO resource utilization is the disk IO resource utilization at the first moment; determining a first data volume interval corresponding to the first disk IO resource utilization interval based on the correspondence between the disk IO resource utilization interval and the data volume interval of each data table to be redistributed; selecting at least one data table whose data volume is within the first data volume interval from the data table to be redistributed; and distributing the data in the at least one data table to some or all of the multiple data nodes.

[0011] In an embodiment of the present application, the coordination node can determine the current corresponding disk IO resource utilization interval based on the disk IO resource utilization at different times, and then determine a data volume interval based on the correspondence between the disk IO resource utilization interval and the data volume interval, and then select the data table whose data volume is within the data volume interval for redistribution processing. In this way, when the current disk IO resource utilization is high, the remaining disk IO resources can be used to redistribute large tables, and when the current disk IO resource utilization is low, the remaining disk IO resources can be used to redistribute small tables, thereby making full use of the disk IO resources and improving the efficiency of data redistribution.

[0012] As a possible implementation method, the data of the data table to be redistributed is distributed to some or all of the multiple data nodes according to the current disk IO resource utilization rate, including: when it is determined that the current disk IO resource utilization rate is less than or equal to a second threshold, the data of the data table to be redistributed is distributed to some or all of the multiple data nodes according to the current disk IO resource utilization rate.

[0013] In an embodiment of the present application, when it is determined that the current disk IO resource utilization is less than or equal to the second threshold, the coordination node can determine that there is still a certain amount of disk IO resources remaining and can support data redistribution. Therefore, the coordination node can select a suitable data table for data redistribution based on the current disk IO resource utilization. In this way, the impact on user business can be avoided.

[0014] As a possible implementation, the method may further include: when determining that the current disk IO resource usage is greater than the second threshold, stopping the redistribution processing of the first data table, the first data table being the table with the largest amount of data during the data redistribution process.

[0015] In an embodiment of the present application, when it is determined that the current disk IO resource utilization rate is greater than the second threshold, the coordination node can stop the redistribution processing of the first data table and release the disk IO resources. In this way, the normal operation of the user business can be guaranteed and slow user business response or congestion of the user business can be avoided.

[0016] As a possible implementation, the method may further include: when it is determined that the amount of data in the data table to be redistributed is less than the first threshold, selecting a fourth data table from the data table to be redistributed, the fourth data table being any data table in the data table to be redistributed; and redistributing the data of the fourth data table to some or all of the multiple data nodes.

[0017] In embodiments of the present application, data tables with data volumes less than a first threshold can be identified as small tables, which essentially consume no disk I / O resources during data redistribution. Therefore, when it is determined that the data volumes of all the data tables to be redistributed are less than the first threshold, the coordinating node does not need to determine the current disk I / O resources or perform data redistribution based on the current disk I / O resources. Instead, the coordinating node can directly select any data table from the data tables to be redistributed for data redistribution. This conserves the coordinating node's processing resources and improves data redistribution efficiency.

[0018] As a possible implementation, the method may further include: in the case of service congestion, determining a second data table, where the second data table is the data table causing the service congestion; and stopping redistribution processing of the second data table.

[0019] In an embodiment of the present application, when user service is congested, the coordination node may temporarily stop processing the data table causing the service congestion. In this way, the service congestion can be resolved in a timely manner and the impact on the user service can be avoided.

[0020] The second aspect discloses a data redistribution method, which can be applied to a coordination node, or to a module (e.g., a chip) in a coordination node, or to a logic module or software that can implement all or part of the coordination node's functions. The coordination node can be used to manage multiple data nodes, and the following description will take application to a coordination node as an example. The data redistribution method may include: redistributing data of a data table to be redistributed to some or all of the multiple data nodes based on the remaining available write rate of the disk; wherein the remaining available write rate of the disk is determined based on a write rate threshold and a write rate corresponding to the data table currently being redistributed, and the data table to be redistributed is a data table to be redistributed among the data tables included in the multiple data nodes.

[0021] In the embodiment of the present application, the coordinating node can redistribute the data of the data table to be redistributed to some or all of the multiple data nodes based on the current available disk write rate. In this way, the current available disk write rate can be effectively utilized, thereby improving the efficiency of data redistribution.

[0022] As a possible implementation, the method may further include: creating M threads, where M is an integer greater than or equal to 1; redistributing the data of the data table to be redistributed to some or all of the multiple data nodes includes: redistributing the data of the data table to be redistributed to some or all of the multiple data nodes through the M threads.

[0023] In an embodiment of the present application, the coordination node may perform data redistribution of the data table to be redistributed in parallel through M threads, thereby improving the efficiency of data redistribution.

[0024] As a possible implementation method, the data of the data table to be redistributed is redistributed to some or all of the multiple data nodes according to the remaining available write rate of the disk, including: selecting one or more data tables from the data table to be redistributed according to the remaining available write rate of the disk, and the write rate corresponding to the one or more data tables is less than or equal to the remaining available write rate; selecting a data table with the largest corresponding write rate from the one or more data tables, and redistributing the data of the data table to some or all of the multiple data nodes.

[0025] In this embodiment of the present application, the coordinating node can first screen out the data tables to be redistributed whose write rates are less than or equal to the current available write rate of the disk. Then, the coordinating node can select the table with the highest write rate among the screened data tables for redistribution. In this way, the current available write rate of the disk can be maximized, thereby improving redistribution efficiency.

[0026] As a possible implementation method, the data of the data table to be redistributed is redistributed to some or all of the multiple data nodes according to the remaining available write rate of the disk, including: when it is determined that there is a data table in the data table to be redistributed with a corresponding data volume greater than or equal to a first threshold, the data of the data table to be redistributed is redistributed to some or all of the multiple data nodes according to the remaining available write rate of the disk.

[0027] In the embodiments of the present application, due to the different data volumes of data tables, their corresponding write rates vary. Furthermore, a data table with a data volume less than a first threshold may be a small table, and its corresponding write rate may be 0. Therefore, when a corresponding data table with a data volume greater than or equal to the first threshold exists among the data tables to be redistributed, the coordinating node may redistribute the data of the data table to be redistributed to some or all of the multiple data nodes based on the current available disk write rate. This effectively utilizes the current available disk write rate, thereby improving data redistribution efficiency.

[0028] As a possible implementation, the method may further include: when it is determined that the amount of data in the data table to be redistributed is less than the first threshold, selecting a fourth data table from the data table to be redistributed, the fourth data table being any data table in the data table to be redistributed; and redistributing the data of the fourth data table to some or all of the multiple data nodes.

[0029] In the embodiment of the present application, a data table with a data volume less than a first threshold value can be determined as a small table, and its corresponding write rate can be 0. Therefore, when it is determined that the data volume of the data tables to be redistributed is less than the first threshold value, the coordinating node does not need to determine the current remaining available write rate of the disk, nor does it need to redistribute data based on the current remaining available write rate of the disk. Instead, it can directly select any data table from the data tables to be redistributed for data redistribution. In this way, the processing resources of the coordinating node can be saved and the efficiency of data redistribution can be improved.

[0030] The third aspect discloses an electronic device, which includes a processor, a memory and a communication interface, wherein the communication interface is used to receive information from other electronic devices outside the electronic device, and output information to other electronic devices outside the electronic device, and the processor calls the computer program stored in the memory to implement the data redistribution method provided in the first aspect and any possible implementation of the first aspect, or the data redistribution method provided in the second aspect and any possible implementation of the second aspect.

[0031] A fourth aspect discloses a computer-readable storage medium having a computer program or computer instructions stored thereon. When the computer program or computer instructions are executed, the data redistribution method disclosed in the above aspects is implemented.

[0032] The fifth aspect discloses a chip, including a processor for executing a program stored in a memory. When the program is executed, the chip executes the data redistribution method disclosed in the above aspects.

[0033] As a possible implementation, the memory is located outside the chip.

[0034] A sixth aspect discloses a computer program product, which includes a computer program code. When the computer program code is run, the data redistribution method disclosed in the above aspects is executed.

[0035] It is understandable that the electronic device provided in the third aspect, the computer-readable storage medium provided in the fourth aspect, the chip provided in the fifth aspect, and the computer program product provided in the sixth aspect are all used to execute the data redistribution method provided in the first aspect and any possible implementation of the first aspect, or the data redistribution method provided in the second aspect and any possible implementation of the second aspect. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0037] Figure 1 This is a schematic diagram of a scenario of database cluster expansion disclosed in an embodiment of the present application;

[0038] Figure 2 This is a schematic diagram of a system architecture disclosed in an embodiment of the present application;

[0039] Figure 3 This is a flow chart of a data redistribution method disclosed in an embodiment of the present application;

[0040] Figure 4 This is a schematic diagram of disk IO resource usage during a redistribution process disclosed in an embodiment of the present application;

[0041] Figure 5 This is a flow chart of another data redistribution method disclosed in an embodiment of the present application;

[0042] Figure 6 This is a flow chart of a method for handling user service congestion disclosed in an embodiment of the present application;

[0043] Figure 7 This is a schematic diagram of the software structure of a coordination node disclosed in an embodiment of the present application;

[0044] Figure 8 This is a structural diagram of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The embodiments of the present application disclose a data redistribution method and electronic device for improving data redistribution efficiency. The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings.

[0046] In order to better understand the embodiments of the present application, the relevant technologies of the embodiments of the present application are described below.

[0047] With the rapid development of network and communication technologies, massive amounts of data are impacting various industries at an unprecedented rate of growth. To effectively manage, maintain, and process this data, users are increasingly demanding database storage capacity and performance. Traditional centralized databases, consisting of only a single server node, are no longer able to meet user storage and computing needs. Distributed cluster databases, on the other hand, typically comprise multiple server nodes, forming a database cluster. These server nodes can work collaboratively with each other. Due to their high availability, scalability, and load balancing, distributed cluster databases are increasingly popular in practical applications.

[0048] In real-world business scenarios, due to the scalability of database clusters, the number of nodes in a database cluster can be adjusted based on business needs. For example, as data continues to grow, to support larger-scale data storage and queries, the database cluster can be expanded by adding new database nodes. Furthermore, to fully utilize the newly added nodes and achieve load balancing across each node, after adding the new database node to the database cluster, the data stored on the original node can be redistributed across all the expanded nodes.

[0049] See also Figure 1 , Figure 1 This is a schematic diagram of a scenario of database cluster expansion disclosed in an embodiment of this application. Figure 1 As shown, the original database cluster may include two database nodes (nodes 1 and 2). As data continues to grow, the disk usage of nodes 1 and 2 reaches 80%. To maintain normal business operations, the original database cluster can be expanded to add nodes 3 and 4. The metadata of the old nodes (nodes 1 and 2) can be synchronized to the new nodes (nodes 3 and 4). The data originally stored on the two nodes (nodes 1 and 2) can then be redistributed across four nodes (nodes 1-4 in the expanded database cluster). This reduces the disk usage of a single node to 40%, reducing the disk pressure on the single node.

[0050] For example, if a single node in a database cluster stores a small amount of data and has a low disk usage (e.g., 20%), scaling down the database cluster can be done to conserve storage resources and improve storage resource utilization. This involves removing some nodes from the cluster. Before removing these nodes from the cluster, the data stored on these nodes can be redistributed across all the nodes after the scaling down to achieve load balancing across each node.

[0051] It should be noted that data stored in a database cluster is typically stored in one or more data tables, each of which may contain one or more tuples. A tuple is a table entry in a database, such as a row in a row-oriented database, a column in a column-oriented database, or an object in an object-oriented database. A tuple is typically the smallest unit of data read or written. For example, the aforementioned distributed cluster database may be a relational database. In a relational database, data can be stored in rows and columns. A series of rows and columns can be considered a two-dimensional table, and a row of data typically represents a tuple in the relational database.

[0052] In a distributed cluster database, each data table can be evenly distributed on each node in the database cluster. Therefore, the above-mentioned data redistribution can be understood as the redistribution of each data table stored in the database cluster. For example, suppose that the original database cluster composed of the above-mentioned nodes 1 and 2 stores data table 1, which includes 20 tuples, tuples 1-10 are stored in node 1, and tuples 11-20 are stored in node 2. After the original database cluster is expanded, data table 1 can be redistributed to nodes 1-4 by using the strategy of creating a temporary table. Specifically, a temporary table T1 can be created on the database cluster after expansion. T1 has corresponding partitions on nodes 1-4. Assume that node 1 stores partition 1 of T1, node 2 stores partition 2 of T1, node 3 stores partition 3 of T1, and node 4 stores partition 4 of T1. Afterwards, the data in data table 1 can be relocated (i.e., written) to temporary table T1. For example, tuples 1-5 can be written (i.e., stored) to partition 1 of node 1, tuples 6-10 to partition 2 of node 2, tuples 11-15 to partition 3 of node 3, and tuples 16-20 to partition 4 of node 4. Afterward, the physical files and metadata of temporary table T1 and data table 1 can be exchanged, and indexes can be recreated. Through these operations, tuples 1-20 from nodes 1 and 2 can be redistributed to nodes 1-4. It should be understood that partitions 1-4 can be understood as the four temporary tables corresponding to T1 on nodes 1-4, respectively.

[0053] Understandably, during the data redistribution phase, data tables of different sizes consume different amounts of disk input / output (IO) resources per node. Tables with larger data volumes (hereinafter referred to as large tables) generally consume more disk IO resources during data migration, while tables with smaller data volumes (hereinafter referred to as small tables) generally consume fewer disk IO resources. Furthermore, large table redistribution operations require inserting more data into temporary tables, which typically takes a long time, potentially tens of minutes, one to two hours, or even longer. Small table redistribution operations require inserting less data into temporary tables, which typically takes a shorter time, perhaps tens of seconds or minutes.

[0054] When redistributing data tables in a database cluster, multiple threads are typically used to perform the redistribution of multiple tables in parallel to improve redistribution efficiency. These threads typically process tables in ascending order of data size, processing smaller tables first and then larger ones. This approach can result in multiple threads simultaneously redistributing data from multiple small tables during low-peak business periods, or from multiple threads simultaneously redistributing data from multiple large tables during peak business periods. In the former case, the disk I / O resources required for both user services and data redistribution are minimal, resulting in low disk I / O resource utilization. In the latter case, both the disk I / O resources required for user services and data redistribution are substantial, resulting in high disk I / O resource utilization. In this scenario, disk I / O resources may be insufficient, potentially impacting user services, leading to slow service responses and even congestion. Therefore, the above redistribution strategy, due to inefficient utilization of disk I / O resources, results in low overall data redistribution efficiency.

[0055] To solve the above problem, in an embodiment of the present application, the disk IO resource usage rate can be monitored, and data tables with appropriate data sizes can be selected for redistribution based on the disk IO resource usage rate. Specifically, when the disk IO resource usage rate is low, the remaining disk IO resources can be used to redistribute large tables. When the disk IO resource usage rate is high, the remaining disk IO resources can be used to redistribute small tables. In this way, the disk IO resources can be fully utilized, the average utilization rate of the disk IO resources during the data redistribution process can be improved, and thus the overall efficiency of data redistribution can be improved.

[0056] In order to better understand the embodiments of the present application, the system architecture used in the embodiments of the present application is described below.

[0057] See also Figure 2 , Figure 2 This is a schematic diagram of a system architecture disclosed in the embodiment of this application. Figure 2 As shown, the system architecture may include a database cluster 201, a network 202, a first electronic device 203, and a second electronic device 204. The database cluster 201 may include multiple server nodes. Figure 2 As shown are node 1 2011, node 2 2012, node 3 2013, ..., node N 2014, where N is an integer greater than or equal to 2, and each node may be one or more servers.

[0058] The database cluster 201 (i.e., nodes 1 to N), the first electronic device 203, and the second electronic device 204 can be connected to the network 202 and communicate with each other through the network 202. The N nodes in the database cluster 201 can also communicate with each other through the network.

[0059] It should be understood that the nodes in database cluster 201 can be divided into two categories: coordinator nodes (CN) and data nodes (DN). Data nodes are primarily used to store data, execute data operation instructions (such as data query and data modification) issued by the coordinator nodes, and return execution results to the coordinator nodes. For example, when storing a table in a distributed cluster database, the tuples in the table can be evenly distributed to each data node after hash calculation, ensuring that each data node stores the same number of tuples as much as possible to achieve load balancing.

[0060] The coordinating node is primarily used to manage and monitor each data node. It can issue data operation instructions to each data node, monitor the resource usage of each data node (such as processor resources, disk I / O resources, and remaining disk storage capacity), and maintain the entire database cluster to ensure it is in a healthy state. In some embodiments, the coordinating node can serve as the communication portal for the entire database cluster, and all database operations (such as adding, deleting, modifying, and querying data) can be completed through the coordinating node. The coordinating node can also be called a management node or a master node. In the embodiments of the present application, node 1 2011 can serve as the coordinating node, and the remaining nodes (i.e., nodes 2 through N) can serve as data nodes.

[0061] In the embodiment of the present application, the first electronic device 203 and the second electronic device 204 can send data operation instructions to the database cluster 201, such as data addition, data deletion, data modification, data query and other instructions. Accordingly, the database cluster 201 can coordinate the nodes (such as Figure 2 Node 1 (2011) shown receives data operation instructions from an electronic device. The coordination node can then send the received instructions synchronously or on demand to other data nodes. After receiving the instructions, each data node can perform corresponding processing and return the processing results to the coordination node. The coordination node can then summarize the results returned by the data nodes and return them to the electronic device that sent the data operation instructions.

[0062] In some embodiments, the first electronic device 203 and the second electronic device 204 may be mobile phones, tablet computers, laptop computers, smart cars, smart wearable devices, etc., which are not limited here.

[0063] In a possible implementation, the coordinating node may also serve as a data node.

[0064] In an embodiment of the present application, after the database cluster 201 is expanded, the coordination node can be responsible for the coordination and management functions in the data redistribution process, can execute the redistribution method provided in the embodiment of the present application, and can improve the efficiency of data redistribution. For details, please refer to the description in the following method embodiment.

[0065] It should be noted that Figure 2 The system architecture shown is only an example and does not constitute a limitation. In other embodiments of the present application, Figure 2 The system architecture shown may include more or fewer devices than shown in the figure, and is not limited to including only the database cluster 201, network 202, first electronic device 203, and second electronic device 204 shown in the figure.

[0066] Based on the above system architecture, please refer to Figure 3 , Figure 3 This is a flow chart of a data redistribution method disclosed in an embodiment of the present application. Figure 3 As shown, the data redistribution method may include but is not limited to the following steps:

[0067] 301. The coordination node creates M threads.

[0068] Specifically, after the original database cluster is expanded or reduced, that is, after one or more data nodes are added or removed, the coordinating node can initiate data redistribution. Data redistribution can redistribute one or more original tables to all nodes in the expanded or reduced database cluster, achieving load balancing. The original table is a data table stored in the original database cluster and may contain multiple tuples.

[0069] During the data redistribution process, in order to improve the efficiency of redistribution, the coordination node can generally create M threads and redistribute M data tables in parallel. Among them, these M data tables can be selected from the data tables to be redistributed based on the average disk input and output IO resource utilization rate. In addition, since the average disk IO resource utilization rate at different times may be different, the coordination node can select these M data tables in multiple times. For example, at the first moment, the current average disk IO resource utilization rate is the first disk IO resource utilization rate. The coordination node can first select N data tables based on the first disk IO resource utilization rate. In this way, the final M data tables can be obtained after multiple selections. Specifically, for any one of the M threads, it can perform the relevant operations of the following steps 302-312. M is an integer greater than or equal to 1, such as 2, 4, 6, etc., and N is an integer greater than or equal to 1 and less than or equal to M.

[0070] 302. The coordinating node determines whether the data volumes of the remaining tables to be redistributed are all less than a first threshold. If the data volumes of the remaining tables to be redistributed are all less than the first threshold, step 309 may be executed. If the data volumes of the remaining tables to be redistributed are not all less than the first threshold, step 303 may be executed.

[0071] Because the sizes of different original tables stored in the original database cluster may vary, small tables in the original database cluster consume essentially no disk IO resources during redistribution. Therefore, if the remaining tables to be redistributed (i.e., data tables to be redistributed) are all small tables, it is not necessary to select tables with appropriate data volumes for processing based on disk IO resource utilization, and step 309 can be performed. For non-small tables in the original database cluster, they may occupy different amounts of disk IO resources during redistribution depending on their data volumes. Generally speaking, the larger the data volume of a data table, the more disk IO resources it occupies. Therefore, if the remaining tables to be redistributed are not all small tables, in order to effectively utilize the remaining disk IO resources, tables with appropriate data volumes may be selected for processing based on disk IO resource utilization, and step 303 may be performed first. It should be understood that the remaining tables to be redistributed refer to all tables stored in the original database cluster that require redistribution and have not yet been redistributed.

[0072] In some embodiments, the coordination node may determine that a table with a data volume less than a first threshold is a small table. Therefore, the coordination node may determine whether the data volume of the remaining tables to be redistributed is less than the first threshold. If the data volume of the remaining tables to be redistributed is less than the first threshold, it may be determined that the remaining tables to be redistributed are all small tables, and step 309 may be executed. If the data volume of the remaining tables to be redistributed is not less than the first threshold, it may be determined that the remaining tables to be redistributed include non-small tables (i.e., tables with a data volume greater than or equal to the first threshold), which may also be referred to as large tables, and step 303 may be executed.

[0073] It should be noted that the size of the first threshold can be set according to actual conditions. In one possible implementation, the first threshold can be set according to the number of data nodes included in the database cluster after expansion, and the disk performance of each node. For example, in some embodiments, if the disk performance of each node (including data throughput, input / output per second (IOPS) etc.) is not much different, then the table with a single node data volume less than X1 can be determined as a small table based on the disk performance, that is, the table with a total data volume less than (X1*Q, Q is the number of data nodes included in the database cluster after expansion or reduction) can be determined as a small table. Among them, the value of X1 can be different depending on the disk performance, and the better the performance, the larger X1 can be.

[0074] It is understood that in other embodiments of the present application, the coordinating node may also determine whether a table is a small table based on the number of tuples included in the table. For example, a table whose number of tuples is less than a certain threshold (such as 10,000) may be determined to be a small table.

[0075] 303. The coordinating node determines whether the average disk IO resource usage is greater than a second threshold. If the average disk IO resource usage is greater than the second threshold, step 304 may be executed. If the average disk IO resource usage is less than or equal to the second threshold, step 310 may be executed.

[0076] Specifically, after determining that the amount of data in the remaining tables to be redistributed is not all less than the first threshold, the coordination node can obtain the disk IO resource usage of each data node in the database cluster after expansion or contraction, and then calculate the average disk IO resource usage of these data nodes. Afterwards, the coordination node can determine whether the average disk IO resource usage is greater than the second threshold. If it is determined that the average disk IO resource usage is greater than the second threshold, it can be determined that the current remaining disk IO resources are insufficient. Therefore, the coordination node can adjust the current redistribution business to reduce the disk IO resources occupied by the redistribution business, and can execute step 304. If it is determined that the average disk IO resource usage is less than or equal to the second threshold, the coordination node can determine that a certain amount of disk IO resources are still remaining and can support redistribution, and therefore, can execute step 310.

[0077] It is understandable that during data redistribution, only the data tables to be redistributed are redistributed to all data nodes in the expanded or shrunk database cluster. For example, Figure 1 The expansion shown (i.e., expanding from two data nodes to four data nodes) can redistribute the data tables to be redistributed on nodes 1 and 2 to nodes 1-4. At this time, nodes 1-4 will all write the redistributed data. For another example, for shrinking, if you go from Figure 1The four data nodes on the right are scaled down to the two data nodes on the left, and the data tables to be redistributed on nodes 1-4 can be redistributed to nodes 1 and 2. At this time, only nodes 1 and 2 will write the redistributed data. Therefore, when determining the disk IO resource utilization rate, only the current disk IO resource utilization rate of each data node in the database cluster after expansion or contraction is considered. For data nodes that do not need to write redistributed data during redistribution, their current disk IO resource utilization rate can be ignored. For example, when nodes 1-4 are scaled down, the current disk IO resource utilization rate of nodes 3 and 4 can be ignored. The average disk IO resource utilization rate can be calculated based on the current disk IO resource utilization rate of nodes 1 and 2, and then the appropriate data tables to be redistributed can be selected for processing based on the average disk IO resource utilization rate.

[0078] In addition, in the capacity expansion scenario, since there will be no user business on the newly added nodes during the data redistribution process, the disk IO resources of the newly added nodes can be fully used for data redistribution. Therefore, in some embodiments, when determining the average disk IO resource utilization, only the current disk IO resource utilization of the old nodes can be considered. For example, when the above two nodes (i.e., node 1 and node 2) are expanded to four nodes (i.e., node 1-node 4), when determining the average disk IO resource utilization, only the current disk IO resource utilization of node 1 and node 2 (i.e., old nodes) can be considered. The average disk IO resource utilization can be calculated based on the current disk IO resource utilization of node 1 and node 2, and then the appropriate data table to be redistributed can be selected for processing based on the average disk IO resource utilization.

[0079] It should be noted that the second threshold value can be set according to actual conditions. For example, if user traffic is relatively frequent under normal circumstances, the second threshold value can be set to 95%. For another example, if user traffic is relatively idle under normal circumstances, the second threshold value can be set to 99%.

[0080] 304. The coordinating node determines the first data table.

[0081] After determining that the average disk I / O resource usage is greater than the second threshold, the coordinating node can first determine the table with the largest data volume (i.e., the first data table) during data redistribution to prevent the data redistribution process from impacting user services. For example, suppose three threads are currently redistributing data tables 1, 2, and 3, respectively. Table 1 has a larger data volume than Table 2, and Table 2 has a larger data volume than Table 3. In this case, Table 1 can be determined as the first data table.

[0082] It should be noted that, in some embodiments, the coordination node may also determine other data tables as the first data table. For example, the data table with the second largest amount of data may be determined as the first data table. This is not limited here.

[0083] 305. The coordination node determines whether the baseline data of the first data table has been migrated. If the baseline data of the first data table has been migrated, step 306 may be executed. If the baseline data of the first data table has not been migrated, step 308 may be executed.

[0084] It is understood that when redistributing the first data table, after establishing a corresponding temporary table for the first data table, data from the first data table can be inserted into the temporary table. However, during the insertion process, updates to the first data table (such as additions, deletions, or modifications) may occur. In these cases, these updated data can be recorded. Subsequently, after the baseline data for the first data table is migrated, these updated data can be migrated. Specifically, data can be deleted from the temporary table, modified, or added to the temporary table based on the recorded updated data.

[0085] Normally, the first data table's baseline data will far outweigh the updated data. Therefore, after determining the first data table, the coordinating node can determine whether the first data table's baseline data has been migrated (i.e., whether all of the first data table's baseline data has been inserted into the temporary table). If so, to improve redistribution efficiency, step 306 can be executed. If not, step 308 can be executed directly. It should be understood that the first data table's baseline data refers to the data that existed before the first data table was redistributed.

[0086] 306. The coordinating node exits the thread processing the first data table without cleaning up related resources.

[0087] When the baseline data of the first data table has been migrated and only the updated data has not been migrated, the coordination node can exit the thread processing the first data table (that is, end the thread processing the first data table) and retain relevant resources such as temporary tables so that the migration of the updated data of the first data table can be completed quickly later.

[0088] 307. The coordinating node waits for resources to become free before resuming the transmission.

[0089] Since the coordination node retains the temporary table corresponding to the first data table in step 306, the coordination node can resume the transmission after determining that the disk IO resources are relatively idle (such as the average disk IO resources are less than or equal to the second threshold), that is, it can re-migrate the updated data of the first data table to the temporary table. In this way, the migration of all data in the first data table can be achieved.

[0090] 308. The coordinating node exits the thread processing the first data table and cleans up related resources.

[0091] If the baseline data for the first data table has not been migrated, the coordinating node can directly exit the thread processing the first data table, stop redistributing the first data table, and clean up related resources such as temporary tables. The coordinating node can then redistribute the first data table again when disk I / O resources are relatively free. It is understood that after the coordinating node terminates the thread processing the first data table, disk I / O resources occupied by data redistribution can be reduced.

[0092] For an example, see Figure 4 , Figure 4 This is a schematic diagram of disk IO resource occupancy during a redistribution process disclosed in an embodiment of the present application. Figure 4 As shown, the total disk IO resources of a single data node are 100%. However, at a certain point in the redistribution process, the user business needs to occupy 50% of the disk IO resources of each data node in the original database cluster, and the redistribution business needs to occupy 60% of the disk IO resources of each data node in the original database cluster. This 60% of disk IO resources includes the 20% disk IO resources required for redistributing the first data table. At this time, the coordination node can monitor that the average disk IO resource utilization rate is greater than the second threshold, close to 100%. In order not to affect the user business, the coordination node can adjust the redistribution business and exit the thread processing the first data table. Accordingly, since the redistribution of the first data table does not need to continue, the disk IO resources occupied by the redistribution business can be reduced to 40%, and the average disk IO resource utilization rate can be 90%. It can be seen that the above adjustments to the redistribution business can ensure the normal operation of user business.

[0093] 309. The coordinating node selects one of the remaining tables to be redistributed for redistribution.

[0094] Because redistributing small tables consumes little disk I / O resources, the coordinating node, upon determining that all remaining tables to be redistributed are small, can select a data table (i.e., the fourth data table) from the remaining tables to be redistributed for redistribution. After redistribution of the data table is complete, step 311 can be executed.

[0095] Specifically, the coordinating node may randomly select a table from the remaining tables to be redistributed for redistribution, or may select a table from the remaining tables to be redistributed for redistribution according to a preset rule. For example, the table with the largest amount of data among the remaining tables to be redistributed may be selected for redistribution. For another example, the table with the smallest amount of data among the remaining tables to be redistributed may be selected for redistribution.

[0096] 310. The coordinating node selects a table from the remaining tables to be redistributed based on the average disk IO resource usage for redistribution.

[0097] When the coordinating node determines that the average disk IO resource usage is less than or equal to the second threshold, it may determine that a certain amount of disk IO resources remain available for data redistribution. To effectively utilize the remaining disk resources, the coordinating node may select a data table with an appropriate amount of data from the remaining tables to be redistributed based on the average disk IO resource usage for redistribution. After redistribution of the data table is complete, step 311 may be executed.

[0098] In some embodiments, K disk IO resource usage intervals can be divided between 0 and a second threshold, and these K disk IO resource usage intervals can correspond to K data volume intervals respectively, where K is an integer greater than or equal to 2. Specifically, when the current average disk IO resource usage is in the first disk IO resource usage interval, a table whose data volume is in the first data volume interval can be preferentially selected from the remaining tables to be redistributed for redistribution. In this way, the average disk IO resource usage can be close to the second threshold, but not greater than the second threshold, so that the remaining disk IO resources can be fully utilized. The first disk IO resource usage interval can be any interval among the K disk IO resource usage intervals, and the first data volume interval can be a data volume interval corresponding to the K data volume intervals.

[0099] For example, assuming the second threshold is 95% and K is 4, [0, 95%] can be divided into four disk IO resource usage intervals. Interval 1 can be [0, 20%], interval 2 can be [20%, 45%], interval 3 can be [45%, 70%], and interval 4 can be [70%, 95%]. When the average disk IO resource usage is in interval 1, the coordination node can determine that the current user business and redistribution business use very few disk IO resources and can prioritize a table with a data volume greater than or equal to D1 from the remaining tables to be redistributed for redistribution. When the average disk IO resource usage is in interval 2, the coordination node can determine that the current user business and redistribution business use very few disk IO resources and can prioritize a table with a data volume greater than or equal to D2 and less than D1 from the remaining tables to be redistributed for redistribution. When the average disk IO resource usage is in interval 3, the coordination node can determine that the current user business and redistribution business use a lot of disk IO resources and can prioritize a table with a data volume greater than or equal to D3 and less than D2 from the remaining tables to be redistributed for redistribution. When the average disk IO resource usage is in interval 4, the coordinating node can determine that the current user business and redistribution business use a lot of disk IO resources. It can prioritize the redistribution of data from the remaining tables to be redistributed to a table with a data volume less than D3. Among them, D1>D2>D3.

[0100] It is understandable that when the average disk IO resource usage is in the first disk IO resource usage interval, if there is no table with a data volume in the first data volume interval among the remaining tables to be redistributed, the coordination node can select a table with a data volume in the second data volume interval from the remaining tables to be redistributed for redistribution processing. The data volume in the second data volume interval is smaller than the data volume in the first data volume interval. For example, when the average disk IO resource usage is in the above-mentioned interval 2, the coordination node can preferentially select a table with a data volume greater than or equal to D2 and less than D1 from the remaining tables to be redistributed for redistribution processing, but at this time if there is no table with a data volume greater than or equal to D2 and less than D1 among the remaining tables to be redistributed, the coordination node can select a table with a data volume less than D2 from the remaining tables to be redistributed for redistribution processing (such as selecting a table with a data volume greater than or equal to D3 and less than D2 for redistribution processing).

[0101] It should be noted that the larger the value of K, the smaller the interval between each interval. Correspondingly, the finer the division of each data volume interval will be. Therefore, when the coordination node selects a suitable table from the remaining tables to be redistributed based on the current average disk IO resource usage, it can select a table with a more appropriate data size, thereby effectively utilizing disk IO resources and improving redistribution efficiency.

[0102] 311. The coordinating node determines whether there are tables to be redistributed. If it is determined that there are tables to be redistributed, step 302 may be executed. If it is determined that there are no tables to be redistributed, step 312 may be executed.

[0103] After the coordination node has processed the previously selected table, it can determine whether there are still tables to be redistributed. If there are still tables to be redistributed, it can continue to select a table from the remaining tables to be redistributed for redistribution processing, and the above step 302 can be re-executed. If there are no tables to be redistributed, step 312 can be executed.

[0104] 312. The coordination node exits the current thread.

[0105] When the coordinating node determines that there is no table to be redistributed, the coordinating node may terminate the current thread.

[0106] It is understood that in steps 303-312 above, the operations performed by the coordinating node may be performed by a thread within the coordinating node. It is also understood that the redistribution process may also be performed using only a single thread. In this case, if the user business occupies fewer disk IO resources, to rationally utilize disk IO resources and improve redistribution efficiency, the redistribution of large tables may be prioritized. If the user business occupies more disk IO resources, to avoid impacting the user business, the redistribution of small tables may be prioritized.

[0107] It should be noted that in the above process, the coordinating node selects the appropriate table from the remaining tables to be redistributed for processing based solely on the average disk IO resource usage. However, in other embodiments of the present application, the coordinating node may select from the remaining tables to be redistributed based on more conditions, such as the average disk IO resource usage and the central processing unit (CPU) utilization.

[0108] It should also be noted that in the above process, the coordination node first creates M threads in step 301. However, in other embodiments of the present application, the coordination node may also first execute step 302 after the redistribution starts, and then, after selecting the table to be redistributed in step 309 or step 310, create a thread to redistribute the table. Afterwards, the coordination node may determine whether there are still tables to be redistributed. If so, step 302 may be executed again. Then, after the coordination node selects the table to be redistributed in step 309 or step 310, it may create another thread to process the table. At this time, the maximum number of threads M may be specified in advance. Alternatively, in another possible implementation method, a thread pool may be created in advance, and each time the coordination node selects a table to be redistributed, an idle thread may be selected from the thread pool to perform redistribution on the table.

[0109] In the above method process, the coordination node can perform adaptive data redistribution based on the amount of data in the remaining tables to be redistributed and the average disk IO resource utilization rate, which can effectively utilize disk IO resources, thereby improving data redistribution efficiency. Specifically, when the average disk IO resource utilization rate is less than or equal to the second threshold, each thread of the coordination node can select a most suitable table from the remaining tables to be redistributed for processing based on the current average disk IO resource utilization rate. In this way, the current remaining disk IO resources can be effectively utilized, thereby improving redistribution efficiency. Moreover, in the above process, when it is monitored that the average disk IO resource utilization rate is greater than the second threshold, the redistribution service can be adjusted in time to release disk IO resources, thereby ensuring the normal operation of user services and avoiding slow user service response or congestion in user services.

[0110] Based on the above system architecture, please refer to Figure 5 , Figure 5 This is a flow chart of another data redistribution method disclosed in the embodiment of this application. Figure 5 As shown, the data redistribution method may include but is not limited to the following steps:

[0111] 501. The coordination node creates M threads.

[0112] Step 501 is the same as step 301 , and reference may be made to the relevant description in step 301 .

[0113] 502. The coordinating node determines whether the data volumes of all remaining tables to be redistributed are less than a first threshold. If the data volumes of all remaining tables to be redistributed are less than the first threshold, step 505 may be executed, similar to step 309 above. If the data volumes of all remaining tables to be redistributed are not less than the first threshold, step 503 may be executed.

[0114] Since the original tables stored in the original database cluster may vary in size, for small tables in the original database cluster, their write rate on a single node is low and their write time is short during redistribution. Therefore, the write rate on a single node for small tables during redistribution can be defaulted to 0. Therefore, if the remaining tables to be redistributed are all small tables, it is not necessary to select one of the remaining tables to be redistributed for redistribution based on the current remaining available write rate, and step 505 can be performed. For non-small tables in the original database cluster, their write rate on a single node can vary depending on the data volume during redistribution. Generally speaking, the larger the data volume of a data table, the greater its write rate on a single node. Therefore, if the remaining tables to be redistributed are not all small tables, to improve data redistribution efficiency, a single table to be redistributed can be selected for redistribution based on the current remaining available write rate, and step 503 can be performed first. It should be understood that the write rate is the rate at which the data table to be redistributed is written to the disk of a single data node.

[0115] The specific method of determining the small table in step 502 is the same as that in step 302 , and reference may be made to the relevant description in step 302 .

[0116] 503. The coordinating node calculates the current remaining available write rate.

[0117] After the coordinating node determines that the remaining tables to be redistributed are not all small tables, to improve redistribution efficiency, the coordinating node can calculate the current remaining available write rate. This is the remaining write rate of a single node available for data redistribution, or the average remaining write rate of all data nodes in the database cluster after expansion or contraction. The coordinating node can calculate the current remaining available write rate by subtracting the sum of the write rates of tables currently being redistributed by other threads from the write rate threshold. The write rate threshold can be the write rate that a single data node (i.e., its disk) can use for data redistribution and can be less than or equal to the write rate cap for a single data node (i.e., the maximum write rate a single node can achieve).

[0118] For example, assume that the write rate cap for a single node is 500 megabytes per second (MB / s) and the write rate threshold for a single node is 250 MB / s. In this case, the coordinating node can create four threads (thread 1, thread 2, thread 3, and thread 4) to perform data redistribution in parallel. Threads 1, 2, and 3 are redistributing data tables 1, 2, and 3, respectively. The write rate for table 1 on a single data node is 100 MB / s, the write rate for table 2 on a single data node is 70 MB / s, and the write rate for table 3 on a single data node is 50 MB / s. The coordinating node can then subtract the write rates already consumed by tables 1, 2, and 3 from the write rate threshold to obtain a remaining available write rate of 30 MB / s (i.e., 250 - 100 - 70 - 50).

[0119] It can be understood that in an embodiment of the present application, the coordinating node can calculate or test in advance the write rate of each table to be redistributed on a single data node in a single thread in the new cluster (i.e., the database cluster after expansion or reduction) without interference from other user businesses, and then save it.

[0120] 504. The coordinating node selects a table from the remaining tables to be redistributed for redistribution based on the current remaining available write rate.

[0121] To improve the overall efficiency of redistribution, after the coordinating node calculates the current remaining available write rate, it can select the most appropriate table from the remaining tables to be redistributed based on the current remaining available write rate for redistribution. Specifically, the coordinating node can first filter out the tables to be redistributed whose write rates are less than or equal to the current remaining available write rate. Then, it can select the table with the highest write rate among these filtered tables for redistribution. This effectively utilizes the write rate threshold and improves redistribution efficiency.

[0122] For example, referring to the example in step 503 above, the sum of the write rates of data table 1, data table 2, and data table 3 is 220 MB / s, and the current remaining available write rate is 30 MB / s. The remaining data tables to be redistributed include data table 4, data table 5, data table 6, and data table 7. Among them, the write rate of data table 4 is 10 MB / s, the write rate of data table 5 is 25 MB / s, the write rate of data table 6 is 15 MB / s, and the write rate of data table 7 is 0 MB / s. At this time, the coordination node can determine that the write rates of data tables 4-7 are all less than the current remaining available write rate, but the write rate of data table 5 is greater than the write rates of data tables 4, data table 6, and data table 7. Therefore, thread 4 of the coordination node can redistribute data table 5, so as to maximize the use of the current remaining available write rate.

[0123] 505. The coordinating node selects one of the remaining tables to be redistributed for redistribution.

[0124] Step 505 is the same as step 309 , and reference may be made to the relevant description in the above step 309 , which will not be repeated here.

[0125] 506. The coordinating node determines whether there are tables to be redistributed. If it is determined that there are tables to be redistributed, step 502 may be executed. If it is determined that there are no tables to be redistributed, step 507 may be executed.

[0126] The coordination node can determine whether there are still tables to be redistributed. If there are still tables to be redistributed, it can continue to select a table from the remaining tables to be redistributed for redistribution processing and re-execute the above step 502. If there are no tables to be redistributed, step 507 can be executed.

[0127] 507. The coordination node exits the current thread.

[0128] When the coordinating node determines that there is no table to be redistributed, the coordinating node may terminate the current thread.

[0129] It can be understood that, in the above steps 502 - 507 , the operations performed by the coordination node may be operations performed by a thread in the coordination node.

[0130] It is understandable that the coordination node performs the above Figure 3 or Figure 5 When using the corresponding redistribution method, user service congestion may occur during the data redistribution process, especially during peak hours. For example, a data table may be locked for a short period of time during the redistribution process, which will block some user operations on the table (such as update operations on the data table), which may cause user services to not proceed normally and cause congestion. Therefore, the coordination node can monitor user service congestion. If congestion is detected, the coordination node can perform the following operations: Figure 6 The relevant steps shown are:

[0131] 601. The coordinating node diagnoses the second data table causing user service congestion through kernel instructions.

[0132] When the coordination node detects that user traffic is congested, the coordination node can diagnose the second data table causing the user traffic congestion through kernel instructions. The second data table can be a table currently being redistributed.

[0133] 602. The coordination node determines whether the amount of data in the second data table is less than a first threshold. If the amount is less than the first threshold, step 606 may be executed. If the amount is not less than the first threshold, step 603 may be executed.

[0134] After the coordination node determines the second data table, it can determine whether the data volume of the second data table is less than a first threshold. If the data volume of the second data table is less than the first threshold, the coordination node can determine that the second data table is a small table, directly suspend the current redistribution of the second data table, and execute step 606. If the data volume of the second data table is greater than or equal to the first threshold, the coordination node can determine that the second data table is not a small table and execute step 603 to improve overall redistribution efficiency.

[0135] In some embodiments, the coordination node may not execute step 602 but directly execute step 603 .

[0136] 603. The coordination node determines whether the baseline data of the second data table has been migrated. If the migration has been completed, step 604 may be executed; if the migration has not been completed, step 606 may be executed.

[0137] 604. The coordinating node exits the thread processing the second data table without cleaning up related resources.

[0138] 605. The coordinating node waits for resources to become free before resuming the transmission.

[0139] 606. The coordinating node exits the thread processing the second data table and cleans up related resources.

[0140] Steps 603-606 are similar to the above steps 305-308, and reference may be made to the relevant descriptions of the above steps 305-308.

[0141] It should be noted that the relevant information (ie, the same information or similar information) and related descriptions in the above different embodiments can refer to each other.

[0142] It should be understood that the above Figure 3 、 Figure 5 and Figure 6 The above processing flow is illustrated by taking the coordination node as the execution subject of the interactive indication as an example, but this application does not limit the execution subject of the interactive indication. Figure 3 、 Figure 5 and Figure 6 The coordination node in the method may also be a chip, a chip system, or a processor that supports the coordination node to implement the method, or may be a logic module or software that can implement all or part of the coordination node functions.

[0143] Based on the above system architecture, please refer to Figure 7 , Figure 7 Schematic diagram of the software structure of a coordination node disclosed in an embodiment of the present application. The coordination node 700 may include a redistribution tool 701 and a resource monitoring platform 702. The redistribution tool 701 and the resource monitoring platform 702 may communicate with each other.

[0144] The redistribution tool 701 and the resource monitoring platform 702 may be two software modules deployed in the coordination node. The resource monitoring platform 702 may monitor each data node in the database cluster, including monitoring resource usage of each data node. For example, CPU utilization, memory utilization, disk I / O resource utilization, and remaining disk storage capacity may be monitored.

[0145] The redistribution tool 701 can be responsible for the coordination and management function of the data redistribution process. In the embodiment of the present application, the redistribution tool 701 can obtain the disk IO resource usage through the resource monitoring platform 702, and then select the appropriate table for processing according to the disk IO resource usage. In some embodiments, the redistribution tool 701 can perform the above Figure 3 、 Figure 5 and Figure 6 The operations performed by the coordination node in the illustrated method embodiment may refer to the above-mentioned related descriptions and will not be described in detail here.

[0146] It should be noted that Figure 7 The software structure shown is only an example and does not constitute a limitation. In other embodiments of the present application, Figure 7 The software architecture shown may include more or fewer software modules than shown.

[0147] Based on the above system architecture, please refer to Figure 8 , Figure 8 800 is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. The electronic device 800 may include: a processor 801, a communication interface 802, and a memory 803. The processor 801, the communication interface 802, and the memory 803 may be interconnected or connected to each other via a bus 804.

[0148] Exemplarily, the memory 803 is used to store computer programs and data of the electronic device 800. The memory 803 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). The communication interface 802 is used to support the electronic device 800 to communicate, such as receiving or sending data.

[0149] Exemplarily, the processor 801 may be a CPU, a complex programmable logic device, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and the like.

[0150] In one embodiment, the electronic device 800 may be a data node, which may be used to store data, execute data operation instructions (such as data query, data modification, etc.) issued by the coordination node, and return execution results to the coordination node.

[0151] In one embodiment, the electronic device 800 may be a coordination node, and the processor 801 may be used to read the program stored in the memory 803 and execute the above Figure 3 、 Figure 5 and Figure 6 The operations performed by the coordination node in the illustrated method embodiment may refer to the above-mentioned related descriptions and will not be described in detail here.

[0152] It should be noted that Figure 8 The electronic device 800 shown is only one implementation of the embodiment of the present application. In actual applications, the electronic device 800 may also include more or fewer components, which is not limited here.

[0153] An embodiment of the present application further discloses a computer-readable storage medium having instructions stored thereon, which, when executed, execute the method in the above method embodiment.

[0154] The embodiments of the present application further disclose a computer program product comprising instructions, which, when executed, perform the method in the above method embodiments.

[0155] Obviously, the embodiments described above are only some of the embodiments of this application, and not all of them. Reference to "embodiments" herein means that the specific features, structures, or characteristics described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive with other embodiments. It is understood, both explicitly and implicitly, that the embodiments described herein can be combined with other embodiments. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. In the specification, claims, and accompanying drawings of this application, the terms "first," "second," "third," and so on are used to distinguish between different objects, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a list of steps or elements may be included, or alternatively, steps or elements not listed may be included, or alternatively, other steps or elements inherent to the process, method, product, or device may be included. It is understandable that the equal sign in the above conditional judgment can be greater than or less than. For example, the above conditional judgment for a threshold greater than, less than, or equal to can also be changed to a conditional judgment for the threshold greater than, equal to, or less than. This is not limited here. Specifically, for the above first threshold, the above table with a data volume less than the first threshold is determined to be a small table, and the table with a data volume greater than or equal to the first threshold is determined to be a non-small table. However, in other embodiments of the present application, a table with a data volume less than or equal to the first threshold can be determined to be a small table, and a table with a data volume greater than the first threshold can be determined to be a non-small table.

[0156] It will be appreciated that the accompanying drawings illustrate only portions relevant to the present application, and not all of the content. It will be appreciated that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts depict the various operations (or steps) as sequential processes, many of the operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated upon completion of its operations, but may also have additional steps not shown in the accompanying drawings. The process may correspond to a method, function, procedure, subroutine, subprogram, or the like.

[0157] As used in this specification, the terms "component," "module," "system," "unit," and the like are used to refer to computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a unit can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or distributed between two or more computers. In addition, these units can be executed from various computer-readable media having various data structures stored thereon. Units can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from a second unit interacting with another unit in a local system, a distributed system, and / or a network, such as the Internet, which interacts with other systems via signals).

[0158] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of this application should be included in the scope of protection of this application.

Claims

1. A data redistribution method, characterized in that: Applied to a coordinating node, the coordinating node is used to manage multiple data nodes, the method includes: Determining, among the plurality of data nodes, whether there is a data table in the data table to be redistributed, the corresponding data amount being greater than or equal to a first threshold; Determine current disk input and output IO resource usage of the multiple data nodes; Redistributing the data in the to-be-redistributed data table to some or all of the multiple data nodes according to the current disk IO resource usage; The current disk IO resource usage is determined based on the current disk IO resource usage of some or all of the multiple data nodes, and the data table to be redistributed is a data table to be redistributed among the data tables included in the multiple data nodes; The redistributing the data of the to-be-redistributed data table to some or all of the multiple data nodes according to the current disk IO resource usage includes: When it is determined that there is a data table in the data tables to be redistributed whose corresponding data volume is greater than or equal to the first threshold and the current disk IO resource usage is greater than the second threshold, terminating the thread processing the first data table, the first data table being the data table with the largest data volume or the data table with the second largest data volume in the data redistribution process; When it is determined that there is a data table in the data table to be redistributed with a corresponding data volume greater than or equal to the first threshold and the current disk IO resource usage is less than or equal to the second threshold, the remaining disk IO resources are used to redistribute the data of at least one data table with a data volume within the first data volume range to some or all of the multiple data nodes.

2. The method according to claim 1, characterized in that Before determining that, among the plurality of data nodes, there is a data table in the to-be-redistributed data table with a corresponding amount of data greater than or equal to the first threshold, the method further includes: Create M threads, where M is an integer greater than or equal to 1; Redistributing the data of the to-be-redistributed data table to some or all of the multiple data nodes includes: The data in the to-be-redistributed data table is redistributed to some or all of the multiple data nodes through the M threads.

3. The method according to claim 1 or 2, characterized in that The utilizing of the remaining disk IO resources to redistribute the data of at least one data table whose data volume is within the first data volume range to some or all of the multiple data nodes specifically includes: Determine a first disk IO resource usage interval corresponding to a first disk IO resource usage, where the first disk IO resource usage is a disk IO resource usage at a first moment; determining, based on a correspondence between disk IO resource usage intervals and data volume intervals of each data table to be redistributed, a first data volume interval corresponding to the first disk IO resource usage interval; Selecting at least one data table whose data volume is within the first data volume range from the data tables to be redistributed; The data in the at least one data table is distributed to some or all of the multiple data nodes.

4. The method according to claim 1 or 2, characterized in that The method further comprises: When it is determined that the current disk IO resource usage is greater than the second threshold, the redistribution processing of the first data table is stopped, and the first data table is the table with the largest amount of data during the data redistribution process.

5. The method according to claim 1 or 2, characterized in that The method further comprises: When it is determined that the data amounts of the data tables to be redistributed are all less than the first threshold, selecting a fourth data table from the data tables to be redistributed, where the fourth data table is any data table in the data tables to be redistributed; Redistribute the data of the fourth data table to some or all of the multiple data nodes.

6. The method according to any one of claims 1 or 2, characterized in that The method further comprises: In the case of traffic congestion, determining a second data table, where the second data table is the data table causing the traffic congestion; Stop the redistribution process of the second data table.

7. A data redistribution method, characterized in that: Applied to a coordinating node, the coordinating node is used to manage multiple data nodes, the method includes: Redistributing the data of the data table to be redistributed to some or all of the multiple data nodes according to the remaining available write rate of the disk; The remaining available write rate of the disk is determined according to a write rate threshold and a write rate corresponding to a data table currently being redistributed, and the data table to be redistributed is a data table to be redistributed among the data tables included in the multiple data nodes; The redistributing the data of the data table to be redistributed to some or all of the multiple data nodes according to the remaining available write rate of the disk includes: Select one or more data tables from the data table to be redistributed according to the remaining available write rate of the disk, where the write rate corresponding to the one or more data tables is less than or equal to the remaining available write rate; A data table with the largest corresponding write rate is selected from the one or more data tables, and data in the data table is redistributed to some or all of the multiple data nodes.

8. The method according to claim 7, characterized in that The method further comprises: Create M threads, where M is an integer greater than or equal to 1; Redistributing the data of the data table to be redistributed to some or all of the multiple data nodes includes: The data of the data table to be redistributed is redistributed to some or all of the multiple data nodes through the M threads.

9. The method according to claim 7 or 8, characterized in that The redistributing the data of the data table to be redistributed to some or all of the multiple data nodes according to the remaining available write rate of the disk includes: When it is determined that there is a data table in the data table to be redistributed with a corresponding data amount greater than or equal to the first threshold, the data of the data table to be redistributed is redistributed to some or all of the multiple data nodes according to the remaining available write rate of the disk.

10. The method according to claim 9, characterized in that The method further comprises: When it is determined that the data amounts of the data tables to be redistributed are all less than the first threshold, selecting a fourth data table from the data tables to be redistributed, where the fourth data table is any data table in the data tables to be redistributed; Redistribute the data of the fourth data table to some or all of the multiple data nodes.

11. An electronic device, characterized in that: The electronic device includes a processor, a memory, and a communication interface, wherein the communication interface is used to receive information from other electronic devices outside the electronic device and output information to other electronic devices outside the electronic device, and the processor calls the computer program stored in the memory to implement the method according to any one of claims 1 to 6 or claims 7 to 10.

Images