Distributed database cluster data redistribution method and device and medium

Abstract

The embodiment of the invention discloses a distributed database cluster data redistribution method and device and a medium, belongs to the technical field of data processing, and solves the problem of low manual redistribution efficiency along with continuous expansion of data volume. Obtaining a data volume corresponding to the PostgreSQL distributed cluster, and determining a corresponding number of nodes based on the data volume; determining a data type corresponding to the to-be-redistributed data; based on different data types, matching different node sequence number calculation strategies so as to perform node sequence number determination on the to-be-redistributed data; acquiring to-be-redistributed data in batches through a corresponding number of nodes, and performing data pulling based on a corresponding relationship between a node sequence number corresponding to the to-be-redistributed data and a current node sequence number; and determining that the data redistribution is completed until the corresponding number of nodes cannot obtain the data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a distributed database cluster data redistribution method, device and medium. Background Art

[0002] With the rapid development of big data technology, the scale and complexity of data are increasing. Traditional stand-alone database systems can no longer meet the requirements of high concurrency, high availability and high scalability.

[0003] As a popular relational database, PostgreSQL's distributed cluster architecture can effectively solve the needs of large-scale data processing. The introduction of the distributed cluster architecture, by sharding data and distributing it on multiple nodes, not only significantly improves the system's data processing capabilities, but also enhances the system's fault tolerance and scalability.

[0004] However, the deployment of distributed clusters is not easy, especially the data redistribution process. Traditional redistribution methods often rely on manual operations, which are time-consuming and laborious. Moreover, when faced with massive data, as the amount of data continues to expand, manual redistribution is not only difficult to cope with the changing data types, but may also cause data consistency problems due to improper operations, resulting in low redistribution efficiency. Summary of the invention

[0005] The embodiments of the present application provide a distributed database cluster data redistribution method, device and medium for solving the following technical problems: as the amount of data continues to expand, manual redistribution is not only difficult to cope with the changing data types, but may also cause data consistency problems due to improper operation, resulting in low redistribution efficiency.

[0006] The present application embodiment adopts the following technical solutions:

[0007] The embodiment of the present application provides a distributed database cluster data redistribution method, including obtaining the data volume corresponding to a PostgreSQL distributed cluster, determining a corresponding number of nodes based on the data volume; determining the data type corresponding to the data to be redistributed; matching different node sequence number calculation strategies based on different data types to determine the node sequence number of the data to be redistributed; batch acquiring the data to be redistributed through a corresponding number of nodes, and pulling data based on the corresponding relationship between the node sequence number corresponding to the data to be redistributed and the current node sequence number; until the corresponding number of nodes cannot obtain the data, determining that the data redistribution is completed.

[0008] The embodiment of the present application improves resource utilization by accurately assessing the amount of data, reasonably allocating node resources, avoiding the situation of excess or insufficient resources, and determining the number of nodes according to the amount of data to ensure that the cluster has sufficient computing power and storage space when processing large amounts of data, thereby improving overall performance. The embodiment of the present application selects different computing strategies through different data types, and evenly distributes data to each node through a reasonable node sequence number calculation strategy, avoiding the situation where some nodes are overloaded while other nodes are idle, and achieving load balancing. The embodiment of the present application obtains data in batches and pulls data through a clear correspondence between node sequences. On the one hand, it reduces the number of data transmissions and improves data transmission efficiency. On the other hand, it automatically determines the node ownership through the node sequence number, solving the problem of requiring a large amount of manual intervention. The redistribution process of the embodiment of the present application does not require the export of data files, so it also solves the problem of a large amount of extra space being occupied.

[0009] In one implementation of the present application, a corresponding number of nodes is determined based on the data volume, specifically including: determining the data input volume, the total data volume and the data access volume within a preset time period based on the data volume; comparing the data input volume within the preset time period with the data input volume within the previous preset time period to determine the input-output difference, performing a ratio processing on the input-output difference with the data input volume within the previous preset time period, and determining the reference first node number based on the ratio result and the number of historical nodes corresponding to the input volume within the previous preset time period; comparing the total data volume with the historical total volume, and determining the reference second node number based on the comparison result and the number of historical nodes corresponding to the historical total volume; comparing the data access volume with the historical access volume, and determining the reference third node number based on the comparison result and the number of historical nodes corresponding to the historical access volume; determining the corresponding number of nodes based on the reference first node number, the reference second node number and the reference third node number.

[0010] In one implementation of the present application, different node sequence number calculation strategies are matched based on different data types, specifically including: when the data type is text type, determining the data features corresponding to the text type; wherein the data features include at least one of the data size, the data access mode, and whether the data includes geographic information; based on the data features, dividing the data corresponding to the text type into different data groups; comparing the different data groups with the preset node sequence number calculation strategy table, and based on the comparison results, determining the node sequence number calculation strategies corresponding to the different data groups, so as to obtain the node number; wherein the preset node sequence number calculation strategy table includes data groups with different features, and also includes the node sequence number calculation strategies corresponding to the data groups with different features; wherein the node sequence number calculation strategy includes at least one of hash remainder, interval sharding, random distribution, and geographic distribution.

[0011] In one implementation of the present application, different node serial number calculation strategies are matched based on different data types, specifically including: when the data type is int type or bigint type, it is determined to be a bit-by-bit processing strategy; based on the business type and business data size corresponding to the data to be redistributed, the bit-by-bit processing strategy is matched; wherein the bit-by-bit processing strategy at least includes a mask-based processing strategy and an offset processing strategy; the result data obtained after the bit-by-bit processing is subjected to modulo processing, and the node serial number is determined based on the modulo processing result.

[0012] In one implementation of the present application, different node number calculation strategies are matched based on different data types, specifically including: when the data type is NULL value, determining the total amount of data of NULL value; comparing the total amount of data of NULL value with the preset data amount threshold; when the total amount of data of NULL value is not greater than the preset data amount threshold, adopting the node number direct allocation strategy; when the total amount of data of NULL value is greater than the preset data amount threshold, adopting the node number indirect allocation strategy.

[0013] In one implementation of the present application, when the total amount of data of NULL values ​​is not greater than the preset data amount threshold, a node number direct allocation strategy is adopted, specifically including: when the total amount of data of NULL values ​​is greater than the preset data amount threshold, a random number is generated for each NULL value, and the node number is determined based on the random number; or based on a preset allocation order, the obtained NULL value is allocated to each node in turn.

[0014] In one implementation of the present application, when the total amount of data of NULL values ​​is greater than a preset data amount threshold, an indirect node number allocation strategy is adopted, specifically including: when the total amount of data of NULL values ​​is not greater than the preset data amount threshold, generating a random value for each NULL value; determining the random value type corresponding to the random value; wherein the random value types include text type, int type, and bigint type; matching different random value types with the data types corresponding to the data to be redistributed, and matching different node number calculation strategies based on the corresponding relationship to determine the node number for the NULL value.

[0015] In one implementation of the present application, data to be redistributed is obtained in batches through a corresponding number of nodes, and data is pulled based on the correspondence between the node serial number corresponding to the data to be redistributed and the current node serial number, specifically including: establishing a connection between the node that needs to be redistributed and the node that needs to receive data; obtaining the data to be redistributed corresponding to the nodes that need to be redistributed in batches through a corresponding number of nodes in accordance with a preset data acquisition order; matching the node serial number corresponding to the obtained data to be redistributed with the current node, and if the match is successful, pulling the data to be redistributed to the current node.

[0016] An embodiment of the present application provides a distributed database cluster data redistribution device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: obtain the data volume corresponding to the PostgreSQL distributed cluster, and determine the corresponding number of nodes based on the data volume; determine the data type corresponding to the data to be redistributed; match different node sequence number calculation strategies based on different data types to determine the node sequence number of the data to be redistributed; batch obtain the data to be redistributed through the corresponding number of nodes, and pull the data based on the corresponding relationship between the node sequence number corresponding to the data to be redistributed and the current node sequence number; until the corresponding number of nodes cannot obtain the data, it is determined that the data redistribution is completed.

[0017] A non-volatile computer storage medium provided by an embodiment of the present application stores computer executable instructions, wherein the computer executable instructions are configured to: obtain a data volume corresponding to a PostgreSQL distributed cluster, and determine a corresponding number of nodes based on the data volume; determine a data type corresponding to the data to be redistributed; match different node sequence number calculation strategies based on different data types to determine the node sequence number of the data to be redistributed; obtain the data to be redistributed in batches through a corresponding number of nodes, and pull data based on a corresponding relationship between a node sequence number corresponding to the data to be redistributed and a current node sequence number; and determine that data redistribution is completed until the corresponding number of nodes cannot obtain data.

[0018] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the embodiments of the present application improve resource utilization by accurately assessing the amount of data, reasonably allocating node resources, avoiding excess or insufficient resources, and determining the number of nodes according to the amount of data to ensure that the cluster has sufficient computing power and storage space when processing large amounts of data, thereby improving overall performance. The embodiments of the present application select different computing strategies through different data types, and evenly distribute data to each node through a reasonable node sequence number calculation strategy, avoiding the situation where some nodes are overloaded while other nodes are idle, and achieving load balancing. The embodiments of the present application obtain data in batches and pull data through a clear correspondence between node sequences. On the one hand, it reduces the number of data transmissions and improves data transmission efficiency. On the other hand, it automatically determines the node ownership through the node sequence number, solving the problem of requiring a lot of manual intervention. The redistribution process of the embodiments of the present application does not require the export of data files, so it also solves the problem of a large amount of extra space being occupied. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0020] Figure 1 A flow chart of a distributed database cluster data redistribution method provided in an embodiment of the present application;

[0021] Figure 2 A flowchart of a node sequence number determination process provided in an embodiment of the present application;

[0022] Figure 3 A data extraction process flow chart provided in an embodiment of the present application;

[0023] Figure 4 A flow chart of overall data redistribution provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the structure of a distributed database cluster data redistribution device provided in an embodiment of the present application.

[0025] Reference numerals:

[0026] 200: distributed database cluster data redistribution device, 201: processor, 202: memory. DETAILED DESCRIPTION

[0027] Embodiments of the present application provide a distributed database cluster data redistribution method, device, and medium.

[0028] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0029] The technical solution proposed in the embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0030] Figure 1 A flow chart of a distributed database cluster data redistribution method provided in an embodiment of the present application, such as Figure 1 As shown, the distributed cluster data redistribution method includes the following steps:

[0031] S101. Obtain the data volume corresponding to the PostgreSQL distributed cluster, and determine a corresponding number of nodes based on the data volume.

[0032] In one embodiment of the present application, based on the data volume, the amount of data entering the warehouse, the total amount of data, and the amount of data access within a preset time length are determined. The amount of data entering the warehouse within the preset time length is compared with the amount entering the warehouse within the previous preset time length to determine the difference in the amount of data entering the warehouse, and the difference in the amount of data entering the warehouse is compared with the amount entering the warehouse within the previous preset time length. Based on the ratio result and the number of historical nodes corresponding to the amount entering the warehouse within the previous preset time length, the number of reference first nodes is determined. The total amount of data is compared with the historical total amount, and the number of reference second nodes is determined based on the comparison result and the number of historical nodes corresponding to the historical total amount. The amount of data access is compared with the historical access amount, and the number of reference third nodes is determined based on the comparison result and the number of historical nodes corresponding to the historical access amount. Based on the reference first node number, the reference second node number, and the reference third node number, a corresponding number of nodes is determined.

[0033] Specifically, the amount of data corresponding to the PostgreSQL distributed cluster is obtained, and based on the data amount, the amount of new data stored in the database within a set time period, such as daily, weekly or monthly, is determined, as well as the total amount of all data stored in the database. In addition, the number of times or the amount of data that the database is queried or accessed within the set time period is also determined.

[0034] Furthermore, the amount of data entering the warehouse within the current preset time period is compared with the amount of data entering the warehouse within the previous preset time period, and the difference in the amount of data entering the warehouse is calculated. The difference is divided by the amount of data entering the warehouse within the previous preset time period to obtain a ratio, which reflects the growth rate or decrease rate of the amount of data entering the warehouse. The number of historical nodes corresponding to the amount of data entering the warehouse within the previous preset time period is obtained, and the product between the ratio and the number of historical nodes is determined, and the number of reference first nodes is determined by the product result.

[0035] Furthermore, the current total amount of data is compared with the total amount of historical data to determine a total amount of reference historical data that is similar to the current total amount of data, and the reference second node number is determined based on the number of historical nodes corresponding to the reference total amount of historical data, that is, the number of nodes required to process the total amount of historical data.

[0036] Furthermore, the current data access volume is compared with the historical data access volume to determine the total amount of reference historical data similar to the current data access volume, and the reference third node number is determined based on the number of historical nodes corresponding to the reference historical data access volume, that is, the number of nodes required to process the historical data access volume.

[0037] The reference first node quantity, the reference second node quantity and the reference third node quantity are weightedly calculated to determine the required corresponding number of nodes.

[0038] S102: Determine the data type corresponding to the data to be redistributed.

[0039] In one embodiment of the present application, the data type of the data to be redistributed is determined to include text type, int type, bigint type, and NULL value.

[0040] S103: Based on different data types, different node sequence number calculation strategies are matched to determine the node sequence numbers of the data to be redistributed.

[0041] In one embodiment of the present application, when the data type is text type, the data features corresponding to the text type are determined; wherein the data features include at least one of the data size, the data access mode, and whether the data includes geographic information. Based on the data features, the data corresponding to the text type is divided into different data groups. The different data groups are compared with the preset node sequence number calculation strategy table, and based on the comparison results, the node sequence number calculation strategies corresponding to the different data groups are determined to obtain the node sequence numbers. Among them, the preset node sequence number calculation strategy table includes data groups with different features, and also includes node sequence number calculation strategies corresponding to data groups with different features. Among them, the node sequence number calculation strategy includes at least one of hash remainder, interval sharding, random distribution, and geographic distribution.

[0042] Specifically, the text type is implemented in C language according to the original distribution logic. The original distribution logic in the embodiment of the present application is the distribution scheme of the business itself. For example, in some scenarios, it may be based on hash modulus, interval sharding, random distribution, geographic distribution, etc. The redistribution must also be consistent with the original to ensure the consistency of the data access logic.

[0043] Specifically, when the data type is text type, determine the data characteristics corresponding to the text type, for example, determine the length of the text data or the storage space requirement, the frequency of data access, the time distribution or user access pattern, and whether the text data contains geographic location information, such as address, longitude and latitude, etc.

[0044] Based on data features, text type data is divided into different data groups. For example:

[0045] Group A: contains a large number of short text messages, high access frequency, and no geographic information;

[0046] Group B: contains articles of medium length, with medium access frequency and geographic information;

[0047] Group C: contains a small number of long documents with low access frequency and no geographic information.

[0048] The embodiment of the present application creates a preset node number calculation strategy table, which lists data groups with different characteristics and their corresponding node number calculation strategies. For each data group, its characteristics are compared with the preset node number calculation strategy table to find the most suitable node number calculation strategy. For example:

[0049] Group A (short text, high-frequency access) is suitable for the hash remainder strategy, which can quickly and evenly distribute data and reduce conflicts;

[0050] Group B (medium length, with geographic information) is suitable for using a geographic distribution strategy, which can optimize data access based on the geographic location of the data;

[0051] Group C (long documents, infrequent access) is suitable for the interval sharding strategy, which can simplify data management and reduce the complexity of data migration.

[0052] Furthermore, after calculation using the node sequence number calculation strategy, the node sequence number corresponding to each data can be obtained.

[0053] In one embodiment of the present application, when the data type is int type or bigint type, a bitwise processing strategy is determined. The bitwise processing strategy is matched based on the business type and business data size corresponding to the data to be redistributed; wherein the bitwise processing strategy at least includes a mask-based processing strategy and an offset processing strategy. The result data obtained after the bitwise processing is subjected to a modulus processing, and the node sequence number is determined based on the modulus processing result.

[0054] Specifically, when the data type is int type or bigint type, mask processing can be used or offset processing can be used, and the result after processing is modulo operation according to the final number of nodes. Specifically, when the data type is int or bigint, the most appropriate bitwise processing strategy is matched according to the business type and business data size corresponding to the data to be redistributed. The bitwise processing strategy in the embodiment of the present application includes at least two types: mask-based processing strategy and offset processing strategy. The mask-based processing strategy includes: using a specific binary mask to perform bitwise AND (AND), bitwise OR (OR), bitwise negation (NOT) and other operations with the data to extract or modify certain bits of the data. This strategy is suitable for situations where distribution needs to be performed according to certain bit patterns of the data. Offset processing strategy: Left or right shift the data to change the bit position of the data. This strategy is usually used to adjust the size or range of the data to make it more suitable for specific distribution requirements.

[0055] Further, after selecting a suitable bitwise processing strategy, the processed result is subjected to a remainder operation. The data is mapped to a limited range through a remainder operation, and the range in the embodiment of the present application corresponds to the number of nodes in the distributed cluster. The result of the remainder operation is the node sequence number, which determines which node the data should be stored on. For example, if the cluster has 10 nodes, the result of the remainder operation will be an integer between 0 and 9, representing the target node of the data.

[0056] In one embodiment of the present application, when the data type is a NULL value, the total amount of data of the NULL value is determined. The total amount of data of the NULL value is compared with a preset data amount threshold. When the total amount of data of the NULL value is not greater than the preset data amount threshold, a node sequence number direct allocation strategy is adopted. When the total amount of data of the NULL value is greater than the preset data amount threshold, a node sequence number indirect allocation strategy is adopted.

[0057] Specifically, the embodiment of the present application performs special processing on NULL so that it can be evenly distributed on all nodes. Specifically, the number of NULL values ​​in the data to be redistributed is determined, and the number of NULL values ​​determined is compared with the preset data volume threshold. When the total amount of NULL value data is not greater than the preset data volume threshold, a direct allocation strategy is adopted, that is, each data item will be directly allocated to a node. When the total amount of NULL value data is greater than the preset data volume threshold, an indirect allocation strategy is adopted, such as grouping based on certain characteristics of the data and then allocating it to the final processing node.

[0058] In one embodiment of the present application, when the total amount of data of NULL values ​​is greater than a preset data amount threshold, a random number is generated for each NULL value, and the node sequence number is determined based on the random number; or based on a preset allocation order, the obtained NULL value is allocated to each node in turn.

[0059] Specifically, for each NULL value, a random number is generated. This random number can be an integer in a certain range, such as 0 to N-1, where N is the number of processing nodes, and the generated random number is used as the node sequence number. Alternatively, a distribution order is preset, such as node 1, node 2, node 3, node 4, and then this order is cycled, and node sequences are assigned to NULL values ​​in sequence according to the preset distribution order.

[0060] In one embodiment of the present application, when the total amount of NULL value data is not greater than a preset data amount threshold, a random value is generated for each NULL value. The random value type corresponding to the random value is determined; wherein the random value type includes a text type, an int type, and a bigint type. Different random value types are matched with the data types corresponding to the data to be redistributed, and different node sequence number calculation strategies are matched based on the corresponding relationship to determine the node sequence number for the NULL value.

[0061] Specifically, for each NULL value, a random value is generated. The type of this random value can be: text type, int type and bigint type. The type of the random value is matched with the data type corresponding to the data to be redistributed. When the corresponding data type is text type, the data feature corresponding to the random value is determined. Based on the data feature, the data of the NULL value is divided into different data groups. For each data group, its feature is compared with the preset node sequence number calculation strategy table to find the most suitable node sequence number calculation strategy. When the corresponding data type is int type or bigint type, the NULL value can be processed with a mask or an offset. The result after processing is modulo operation according to the final number of nodes, and the corresponding node number is determined based on the result obtained by the modulo operation.

[0062] Figure 2 A flowchart of a node sequence number determination process provided in an embodiment of the present application is as follows: Figure 2 As shown, first determine the data type of the data to be redistributed: including text type, int type, and bigint type, and perform corresponding logical processing according to the data type: the text type is implemented in C language according to the original distribution logic; the int and bigint types are processed by bit calculation. Processing NULL values: If it is a NULL value, it may cause calculation anomalies, causing all null data to enter a node, thereby causing data skew and affecting efficiency during use, so NULL is specially processed, where the special processing may include: generating random numbers, generating random values, and sending fixed value null data to each node in turn; so that it can be evenly distributed on all nodes. Based on this process, the corresponding node serial number (code) is calculated according to the distribution rule.

[0063] S104. Acquire the data to be redistributed in batches through a corresponding number of nodes, and pull the data based on the correspondence between the node serial numbers corresponding to the data to be redistributed and the current node serial numbers.

[0064] In one embodiment of the present application, a connection is established between a node that needs to be redistributed and a node that needs to receive data. The data to be redistributed corresponding to the nodes that need to be redistributed are acquired in batches through a corresponding number of nodes in accordance with a preset data acquisition order. The node sequence number corresponding to the acquired data to be redistributed is matched with the current node, and if the match is successful, the data to be redistributed is pulled to the current node.

[0065] Specifically, Figure 3 A data extraction process flow chart provided in an embodiment of the present application is as follows: Figure 3 As shown, first establish a connection between the node that needs to be redistributed and the node that needs to receive data, and batch obtain the data of the node that needs to be redistributed. If the data cannot be obtained, it means that the redistribution is completed and the process ends. If the data can be obtained, determine whether the data belongs to the current node according to the node sequence number. If it does, pull it to the current node in batches. If it does not belong to the current node.

[0066] S105: until a corresponding number of nodes cannot obtain data, it is determined that data redistribution is completed.

[0067] In one embodiment of the present application, Figure 3 As shown, data is acquired in batches through node loops until data redistribution is completed.

[0068] Figure 4 A flow chart of overall data redistribution provided in an embodiment of the present application, such as Figure 4 As shown in the figure, it is assumed that there are three original data nodes, and the data is calculated by some algorithm logic implemented in Java, and then inserted into the corresponding partition. As the amount of data increases, three nodes can no longer meet business needs, and the number of nodes needs to be expanded from the original three to six. However, the data of the original three nodes should also be redistributed to the six nodes according to the rules of six nodes to maintain logical consistency. After that, the data is calculated by the data processing module for the node number, and the data is pulled to the corresponding node through the data transmission module.

[0069] Figure 5 A schematic diagram of the structure of a distributed database cluster data redistribution device provided in an embodiment of the present application. Figure 5As shown, a distributed database cluster data redistribution device 200 includes: at least one processor 201; and a memory 202 in communication with the at least one processor 201; wherein the memory 202 stores instructions that can be executed by the at least one processor 201, and the instructions are executed by the at least one processor 201, so that the at least one processor 201 can: obtain the data volume corresponding to the PostgreSQL distributed cluster, and determine the corresponding number of nodes based on the data volume; determine the data type corresponding to the data to be redistributed; match different node sequence number calculation strategies based on different data types to determine the node sequence number of the data to be redistributed; batch obtain the data to be redistributed through a corresponding number of nodes, and pull data based on the corresponding relationship between the node sequence number corresponding to the data to be redistributed and the current node sequence number; until the corresponding number of nodes cannot obtain the data, it is determined that the data redistribution is completed.

[0070] A non-volatile computer storage medium provided by an embodiment of the present application stores computer executable instructions, wherein the computer executable instructions are configured to: obtain a data volume corresponding to a PostgreSQL distributed cluster, and determine a corresponding number of nodes based on the data volume; determine a data type corresponding to the data to be redistributed; match different node sequence number calculation strategies based on different data types to determine the node sequence number of the data to be redistributed; obtain the data to be redistributed in batches through a corresponding number of nodes, and pull data based on a corresponding relationship between a node sequence number corresponding to the data to be redistributed and a current node sequence number; and determine that data redistribution is completed until the corresponding number of nodes cannot obtain data.

[0071] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0072] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various changes and variations. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A distributed database cluster data redistribution method, characterized in that: The method comprises: Obtaining a data volume corresponding to a PostgreSQL distributed cluster, and determining a corresponding number of nodes based on the data volume; Determine the data type corresponding to the data to be redistributed in the PostgreSQL distributed cluster; Based on the different data types, different node sequence number calculation strategies are matched to determine the node sequence number of the data to be redistributed; Acquire the data to be redistributed in batches through the corresponding number of nodes, and pull the data based on the corresponding relationship between the node serial number corresponding to the data to be redistributed and the current node serial number; Until the corresponding number of nodes cannot obtain data, it is determined that the data redistribution is completed.

2. A distributed database cluster data redistribution method according to claim 1, characterized in that: Determining a corresponding number of nodes based on the data volume specifically includes: Based on the data volume, determine the amount of data stored, the total amount of data, and the amount of data accessed within a preset time period; Compare the data input amount within the preset time period with the input amount within the previous preset time period to determine the input-output difference, perform a ratio processing on the input difference and the input amount within the previous preset time period, and determine the reference first node number based on the ratio result and the number of historical nodes corresponding to the input amount within the previous preset time period; Comparing the total amount of data with the historical total amount, and determining the number of reference second nodes based on the comparison result and the number of historical nodes corresponding to the historical total amount; Comparing the data access volume with the historical access volume, and determining the reference third node number based on the comparison result and the number of historical nodes corresponding to the historical access volume; Based on the reference first node quantity, the reference second node quantity and the reference third node quantity, the corresponding number of nodes is determined.

3. A distributed database cluster data redistribution method according to claim 1, characterized in that: The matching of different node sequence number calculation strategies based on different data types specifically includes: In the case where the data type is a text type, determining a data feature corresponding to the text type; wherein the data feature includes at least one of a data size, a data access mode, and whether the data includes geographic information; Based on the data features, the data corresponding to the text type is divided into different data groups; Comparing the different data groups with the preset node sequence number calculation strategy table, and determining the node sequence number calculation strategies corresponding to the different data groups respectively based on the comparison result to obtain the node sequence number; The preset node number calculation strategy table includes data groups with different characteristics, and also includes node number calculation strategies corresponding to the data groups with different characteristics; The node number calculation strategy includes at least one of hash modulus, interval sharding, random distribution and geographical distribution.

4. A distributed database cluster data redistribution method according to claim 1, characterized in that: The matching of different node sequence number calculation strategies based on different data types specifically includes: When the data type is an int type or a bigint type, a bit-wise processing strategy is determined; Based on the service type and service data size corresponding to the data to be redistributed, the bit-by-bit processing strategy is matched; wherein the bit-by-bit processing strategy at least includes a mask-based processing strategy and an offset processing strategy; The result data obtained after the bitwise processing is subjected to modulo processing, and the node sequence number is determined based on the modulo processing result.

5. A distributed database cluster data redistribution method according to claim 1, characterized in that: The matching of different node sequence number calculation strategies based on different data types specifically includes: When the data type is a NULL value, determining the total amount of data of the NULL value; Compare the total amount of data of the NULL value with a preset data amount threshold; When the total amount of data of the NULL value is not greater than the preset data amount threshold, a node sequence number direct allocation strategy is adopted; When the total amount of data of the NULL value is greater than the preset data amount threshold, the node sequence number indirect allocation strategy is adopted.

6. A distributed database cluster data redistribution method according to claim 5, characterized in that: When the total amount of data of the NULL value is not greater than the preset data amount threshold, the node sequence number direct allocation strategy is adopted, specifically including: When the total amount of data of the NULL value is not greater than the preset data amount threshold, a random number is generated for each of the NULL values, and a node sequence number is determined based on the random number; or Based on the preset allocation order, the obtained NULL value is allocated to each node in turn.

7. A distributed database cluster data redistribution method according to claim 5, characterized in that: When the total amount of data of the NULL value is greater than the preset data amount threshold, the node sequence number indirect allocation strategy is adopted, specifically including: When the total amount of data of the NULL value is greater than the preset data amount threshold, generating a random value for each of the NULL values; Determine a random value type corresponding to the random value; wherein the random value type includes text type, int type and bigint type; The different random value types are matched with the data types corresponding to the data to be redistributed, and different node sequence number calculation strategies are matched based on the corresponding relationship to determine the node sequence number of the NULL value.

8. A distributed database cluster data redistribution method according to claim 1, characterized in that: The step of batch-acquiring the data to be redistributed through the corresponding number of nodes and pulling data based on the corresponding relationship between the node serial number corresponding to the data to be redistributed and the current node serial number specifically includes: Establish connections between nodes that need to be redistributed and nodes that need to receive data; Acquire the data to be redistributed corresponding to the nodes to be redistributed in batches through the corresponding number of nodes according to a preset data acquisition order; The obtained node serial number corresponding to the data to be redistributed is matched with the current node, and if the match is successful, the data to be redistributed is pulled to the current node.

9. A distributed database cluster data redistribution device, characterized in that: The device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the method according to any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions can execute the method according to any one of claims 1 to 8.

Images