Distributed database data splitting method and device, equipment and medium

Abstract

The invention relates to the technical field of data processing, and discloses a distributed database data splitting method, which comprises the following steps of: obtaining a distributed database table to be split, and analyzing and processing the distributed database table to obtain metadata; evaluating a data distribution condition of a distributed database table according to the metadata to obtain an evaluation result; and selecting a splitting strategy according to the evaluation result, and carrying out balanced splitting on the data according to the splitting strategy to generate a data segmentation interval. According to the method, metadata of a distributed database table is dynamically obtained through a coordination node, the data distribution condition is evaluated based on the metadata, and if data distribution is balanced, continuous equal splitting is carried out according to a fixed step length; and if the data are not balanced, uniform segmentation is realized through hash value partitioning, and balanced splitting of the data is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a distributed database data splitting method, device, equipment and storage medium. BACKGROUND

[0002] With the acceleration of the localization process in key fields, the financial, securities and banking industries generally face the demand for data migration and synchronization between heterogeneous databases. However, these industries are highly sensitive to data, and ensuring eventual consistency after data migration relies on accurate data comparison and verification. Therefore, the industry usually adopts a data splitting strategy for comparison, that is, grouping data before querying and comparing. There are three main core splitting schemes: the first is based on key value sorting, that is, sorting by primary key or unique index and then dividing by record number; the second is full materialized cache division, which first queries and materializes the source data to the cache, and then divides; the third is mapping physical sharding division, which directly uses the logic of database and table splitting to divide.

[0003] However, in the distributed database scenario, these schemes have significant limitations. First, the key value sorting based scheme lacks universality, and not all distributed databases support efficient global sorting operations on primary keys or indexes. Second, the full materialized cache division scheme cannot be used in real-time synchronization scenarios because comparison cannot be performed until migration is complete, and full data caching will result in significant memory overhead and failure risk. Finally, the core problem of the mapping physical sharding division scheme is its strong binding to the physical sharding structure. Even if the table design is reasonable, due to the uncertainty of actual business data distribution (such as the existence of hot keys), it is easy to cause serious imbalance in data and load between comparison shards. This imbalance will further cause significant fluctuations in the execution time of grouped query operations, and even time out, resulting in a significant increase in the overall comparison delay. SUMMARY

[0004] The main purpose of the present application is to provide a distributed database data splitting method, device, equipment and storage medium, which solves the performance bottleneck problem caused by data skew in a distributed environment.

[0005] To achieve the above purpose, the present application provides a distributed database data splitting method, comprising: Obtaining a distributed database table to be split, and analyzing and processing the distributed database table to obtain metadata; According to the metadata, the data distribution of the distributed database table is evaluated to obtain an evaluation result; According to the evaluation result, a splitting strategy is selected, and the data is evenly split according to the splitting strategy to generate a data segment interval.

[0006] Further, to achieve the above object, the present application provides a distributed database data splitting device, comprising: a metadata initialization module, configured to acquire a distributed database table to be split, and analyze and process the distributed database table to acquire metadata; a data dispersion evaluation module, configured to evaluate data distribution of the distributed database table according to the metadata, and obtain an evaluation result; a data adaptive splitting module, configured to select a splitting strategy according to the evaluation result, and perform balanced splitting on data according to the splitting strategy to generate a data segmentation interval.

[0007] Further, to achieve the above object, the present application also provides a computer device, which comprises a memory, a processor, and a distributed database data splitting program stored in the memory and capable of running on the processor, and the distributed database data splitting program realizes the steps of the distributed database data splitting method when executed by the processor.

[0008] Further, to achieve the above object, the present application also provides a computer readable storage medium, which stores a distributed database data splitting program, and the distributed database data splitting program realizes the steps of the distributed database data splitting method when executed by a processor.

[0009] Beneficial effects: The present application relates to the technical field of data processing, and discloses a distributed database data splitting method, which comprises the following steps: acquiring a distributed database table to be split, and analyzing and processing the distributed database table to acquire metadata; evaluating data distribution of the distributed database table according to the metadata, and obtaining an evaluation result; selecting a splitting strategy according to the evaluation result, and performing balanced splitting on data according to the splitting strategy to generate a data segmentation interval. The present application dynamically acquires metadata of a distributed database table through a coordination node, evaluates data distribution based on the metadata, performs continuous equal division splitting according to a fixed step length if the data distribution is balanced, and performs uniform partitioning through a hash value if the data distribution is unbalanced, so that balanced splitting of data is realized. BRIEF DESCRIPTION OF DRAWINGS

[0010] The present application will be further described below in combination with the drawings and embodiments, and the drawings show: Figure 1 An application environment schematic diagram of a distributed database data splitting method in an embodiment of the present application; Figure 2 A flowchart of the distributed database data splitting method in an embodiment of the present application; Figure 3 A functional module schematic diagram of a preferred embodiment of the distributed database data splitting device of the present application; Figure 4 Figure 1 is a structural schematic diagram of a computer device in an embodiment of the present application; Figure 5 Figure 2 is another structural schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0011] It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

[0012] The distributed database data splitting method provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 , wherein a user end communicates with a service end through a network. The service end can acquire a distributed database table to be split through the user end, analyze and process the distributed database table, acquire metadata, evaluate data distribution of the distributed database table according to the metadata, obtain an evaluation result, select a splitting strategy according to the evaluation result, perform balanced splitting on data according to the splitting strategy, and generate a data segmentation interval. The present application dynamically acquires metadata of the distributed database table through a coordination node, evaluates data distribution based on the metadata, performs continuous equal division splitting according to a fixed step length if the data distribution is balanced, and performs uniform partitioning through a hash value if the data distribution is unbalanced, thereby realizing balanced splitting of data. The user end can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The service end can be realized by an independent server or a server cluster composed of multiple servers. The present application will be described in detail through specific embodiments.

[0013] Please refer to Figure 2 , Figure 2 Figure 3 is a flowchart of an embodiment of the distributed database data splitting method provided by the present application. It should be noted that although a logical order is shown in the flowchart, the steps shown or described herein can be performed in an order different from that shown in some cases.

[0014] As shown in Figure 2 , the distributed database data splitting method provided by the present application includes the following steps: S100, acquiring a distributed database table to be split, analyzing and processing the distributed database table, and acquiring metadata; S200, evaluating data distribution of the distributed database table according to the metadata, and obtaining an evaluation result; S300, selecting a splitting strategy according to the evaluation result, performing balanced splitting on data according to the splitting strategy, and generating a data segmentation interval.

[0015] In this embodiment, in a distributed database environment, in order to split the table and optimize data distribution, a series of steps are needed to obtain metadata, evaluate data distribution, and select a suitable splitting strategy.

[0016] First, the client sends a request to the coordination node (CN) through the JDBC API, specifying the distributed database table that needs to be split. After receiving the request, the CN node parses the request content and determines the table that needs to be processed. Subsequently, the CN node queries the metadata information in the database according to the request, which includes the table distribution strategy, data node topology structure, etc. Specifically, the CN node obtains the following information from the metadata table: total data amount (COUNT), which is obtained by aggregating the local count results of each data node; global extreme values (MAX and MIN), which are obtained by aggregating the local extreme value results of each data node; table distribution strategy (such as Hash, Range, List, Duplicate); and data node topology structure, including node ID, host address, port number, and storage shard mapping relationship.

[0017] Next, the CN node evaluates the data distribution based on the obtained metadata. First, set the dispersion threshold (default is 0.9). Then, calculate the dispersion based on the total data amount (COUNT), the maximum value (MAX) and the minimum value (MIN) of the global extreme values. If the dispersion is greater than or equal to the dispersion threshold, the data distribution is determined to be balanced; if the dispersion is less than the dispersion threshold, the data distribution is determined to be unbalanced. The evaluation result includes the specific value of the dispersion and the determination of the data distribution. According to the evaluation result, the CN node selects a suitable splitting strategy and performs balanced splitting on the data. If the data distribution is balanced (dispersion ≥ 0.9), the dynamic range sharding strategy is adopted. The specific steps are as follows: obtain the maximum value (MAX) and the minimum value (MIN) of the global extreme values from the metadata, determine the expected number of shards according to the actual demand or configuration, Then, starting from the minimum value (MIN), the data interval is divided by step (STEP) in turn until the maximum value (MAX), to generate data segmentation interval. The corresponding query SQL statement is generated for each segmentation interval, which is used for subsequent data query and operation, and the generated segmentation interval is stored. If the data distribution is uneven (dispersion <0.9), a composite key hash redistribution strategy is adopted. The specific steps are as follows: splice the global extreme value into a string, and perform hash calculation on the spliced string to generate a hash value. The number of hash buckets is determined according to the data volume and the expected number of shards, and the hash buckets are encoded. The hash value is used to take the modulus of the number of hash buckets, and the data is distributed to the corresponding hash bucket. The data is uniformly segmented according to the hash bucket code to generate the data segmentation interval. The corresponding query SQL statement is generated for each hash bucket, which is used for subsequent data query and operation, and the generated segmentation interval is stored.

[0018] Through the above steps, the balanced distribution of data in the distributed database is ensured, and the query efficiency and resource utilization are optimized.

[0019] In one embodiment, the step S100 comprises: S101, decomposing the query task according to the query request to generate a subquery task by the coordination node; S102, assigning each subquery task to each data node; S103, each data node counts the number of rows of the table segment according to the subquery task; S104, aggregating the number of rows of each table segment by the coordination node to generate the total number of rows of the table, i.e. the total amount of data.

[0020] In this embodiment, in the distributed database environment, the query request of the client can be implemented by decomposing the task into multiple subquery tasks to achieve parallel execution, thereby improving the query efficiency. The client sends a query request to the coordination node (CN) through the JDBC API, requesting to obtain the total number of rows (COUNT) of the table. After receiving the request, the CN node parses the SQL statement to determine the operation to be performed. According to the topology structure of the distributed database and the distribution strategy of the table, the CN node decomposes the query task into multiple subquery tasks, each of which corresponds to a table segment on a data node (DN). The CN node generates a subquery task for each data node to ensure that each subquery task only queries the table segment on the data node, and distributes the decomposed subquery tasks to each data node for execution.

[0021] After receiving the subquery task, the data node starts to prepare for executing the query operation. Each data node parses the content of the subquery task, executes the task to count the number of rows of the table segment stored by the data node, and returns the counting result to the CN node. The returned result is usually a number representing the number of rows of the table segment on the data node.

[0022] The CN node receives the partial counting results returned by all data nodes, aggregates the results, and generates the total number of rows of the table (total data amount). Specifically, the CN node accumulates all the partial counting results to obtain the final total number of rows (COUNT), and returns the result to the client.

[0023] Through the above steps, the coordination node (CN) decomposes the query task into multiple subquery tasks, and distributes the subquery tasks to various data nodes (DN) for execution. Each data node counts the number of rows of the table segment stored by the data node, and returns the result to the CN node. The CN node aggregates all the partial counting results to generate the total number of rows of the table (total data amount). This process avoids full table scanning, significantly reduces I / O overhead, and thus improves query efficiency.

[0024] In an embodiment, the step S100 further comprises: S105, each data node calculates the primary key or unique index column extreme value of the table segment according to the subquery task; S106, the coordination node aggregates the primary key or unique index column extreme value of each table segment to generate a global extreme value.

[0025] In this embodiment, in a distributed database environment, in order to efficiently calculate the global extreme value (minimum value and maximum value) of the primary key or unique index column of a table, first, the coordination node (CN) generates a subquery task according to the topology structure of the distributed database and the distribution strategy of the table, to calculate the extreme value (maximum value and minimum value) of the primary key or unique index column of the table segment stored on each data node. The CN node sends the subquery tasks to the corresponding data nodes (DN) respectively. After receiving the subquery task, the data node starts to prepare for executing the query operation.

[0026] Each data node calculates the extreme value of the primary key or unique index column of the table segment stored by the data node according to the subquery task, and returns the calculated partial extreme value result to the CN node. The returned result is usually a tuple representing the minimum value and maximum value of the primary key or unique index column of the table segment on the data node.

[0027] The CN node receives the local extremum results returned by all data nodes, aggregates these results, and generates global extrema (global minimum and global maximum). The specific aggregation logic is that the global minimum (MIN) takes the minimum value among all local minimum values, and the global maximum (MAX) takes the maximum value among all local maximum values. Finally, the CN node returns the global extrema (MIN and MAX) to the client.

[0028] Through the above steps, each data node calculates the extremum of the primary key or unique index column of the table segment stored by it according to the subquery task, and returns the result to the coordination node (CN). The CN node aggregates all local extremum results to generate global extrema (global minimum and global maximum). This process avoids full table scanning, significantly reduces I / O overhead, and improves query efficiency. At the same time, it also provides an important basis for subsequent data dispersion evaluation and split strategy selection.

[0029] In one embodiment, the step S100 further comprises: S107, querying metadata information in the database according to the query request by the coordination node; S108, obtaining the distribution strategy of the current table according to the metadata information; S109, obtaining the topology structure of all data nodes according to the metadata information.

[0030] In this embodiment, in a distributed database environment, the coordination node (CN) is responsible for processing the query request of the client and providing metadata information to support subsequent data splitting and query optimization. The client sends a query request to the coordination node (CN) through the JDBC API, requesting to obtain the metadata information of the table. After receiving the query request, the CN node parses the SQL statement to determine the metadata information that needs to be queried. Subsequently, the CN node queries the metadata information in the database, which is usually stored in the metadata table of the system and includes the distribution strategy of the table and the topology structure of the data nodes. The CN node obtains the distribution strategy of the current table and the topology structure of all data nodes from the metadata table.

[0031] The CN node parses the queried metadata information and extracts the distribution strategy of the current table. The distribution strategy can include the following types: Hash distribution, that is, data is distributed to different data nodes according to the hash value of the primary key or specified column; Range distribution, that is, data is distributed to different data nodes according to the range of the primary key or specified column; List distribution, that is, data is distributed to different data nodes according to the enumeration value of the primary key or specified column; Duplicate distribution, that is, data is replicated on multiple data nodes to improve query performance. The CN node returns the distribution strategy of the current table to the client.

[0032] Meanwhile, the CN node parses the queried metadata information and extracts the topology structure of all data nodes. The topology structure information usually includes node ID (unique identifier of each data node), host address (IP, i.e. IP address of the data node), port number (Port, i.e. service port number of the data node), and storage shard mapping relationship (table fragment information stored on each data node). The CN node returns the topology structure of all data nodes to the client.

[0033] Through the above steps, the coordination node (CN) queries the metadata information in the database according to the query request of the client, and obtains the distribution strategy of the current table and the topology structure of all data nodes. These metadata information provides an important basis for subsequent data splitting and query optimization.

[0034] In one embodiment, the step S200 comprises: S201, obtaining the total amount of data, the maximum value of the global extreme value, and the minimum value of the global extreme value of the metadata; S202, calculating the dispersion degree according to the total amount of data, the maximum value, and the minimum value; S203, comparing the dispersion degree with a dispersion threshold; S204, if the dispersion degree is greater than or equal to the dispersion threshold, determining that the data distribution is balanced according to the data distribution; S205, if the dispersion degree is less than the dispersion threshold, determining that the data distribution is unbalanced according to the data distribution; S206, constructing an evaluation result from the dispersion degree and the data distribution.

[0035] In this embodiment, in a distributed database environment, the coordination node (CN) evaluates the data distribution through a series of steps to determine whether the data is evenly distributed and provides a basis for subsequent data splitting strategy.

[0036] Firstly, the coordination node (CN) obtains the total amount of data (COUNT), the maximum value (MAX) and the minimum value (MIN) of the global extreme value from the metadata. These information is obtained by aggregating the local count results and local extreme value results returned by each data node through the previous steps. Specifically, the CN node has calculated the total number of rows (COUNT) of the table and the global extreme value (MAX and MIN).

[0037] Next, the CN node calculates the dispersion degree according to the total amount of data (COUNT), the maximum value (MAX) and the minimum value (MIN) of the global extreme value. The calculation formula of the dispersion degree is:

[0038] When the dispersion approaches 1, it indicates that the primary key values are fully utilized, and the data distribution is continuous without obvious holes or hotspots. When the dispersion approaches 0, it indicates that the primary key values are wasted, and the data has holes or hotspots, and the data distribution is unbalanced.

[0039] The CN node compares the calculated dispersion with a preset dispersion threshold to determine whether the data distribution is balanced. The dispersion threshold is a preset value configured according to actual needs, and the default value can be set to 90% (0.9). If the dispersion is greater than or equal to the dispersion threshold, it is determined that the data distribution is balanced; if the dispersion is less than the dispersion threshold, it is determined that the data distribution is unbalanced.

[0040] According to the dispersion and the data distribution, the CN node generates the final evaluation result. The evaluation result includes the specific value of the dispersion and the determination of the data distribution. The CN node returns these evaluation results to the client for subsequent data splitting strategy selection.

[0041] Through the above steps, the coordination node (CN) calculates the dispersion according to the total amount of data (COUNT), the maximum value (MAX) and the minimum value (MIN) of the global extreme value in the metadata, and compares the dispersion with the preset dispersion threshold to finally determine whether the data distribution is balanced. This process provides an important basis for subsequent selection of appropriate data splitting strategies.

[0042] In one embodiment, the step S300 includes: S3011, if the evaluation result is that the data distribution is balanced, calculating the step size according to the metadata and the expected number of shards; S3012, continuously and equally splitting the data according to the step size to generate data segmentation intervals.

[0043] In this embodiment, in a distributed database environment, when the data distribution is evaluated as balanced (dispersion ≥ 90%), the coordination node (CN) will split the data according to the metadata and the expected number of shards to optimize subsequent data query and comparison operations.

[0044] First, the CN node obtains the evaluation result from the previous step, including the dispersion and the data distribution. If the evaluation result shows that the data distribution is balanced (dispersion ≥ 90%), the next step is entered.

[0045] The CN node obtains the maximum value (MAX) and the minimum value (MIN) of the global extreme value from the metadata. At the same time, according to actual needs or configuration, the expected number of shards is determined. Based on these information, the step size is calculated:

[0046] According to the calculated step size, the CN node will continuously divide the data into equal segments to generate data segment intervals. The specific operation is to start from the minimum value (MIN) and divide the data interval by step size (STEP) in turn until the maximum value (MAX). A corresponding query SQL statement is generated for each segment interval for subsequent data query and operation. These query statements will explicitly specify the range of each segment interval to ensure the accuracy and efficiency of the query.

[0047] Finally, the CN node stores the generated segment intervals for subsequent data query and comparison operations. These segment intervals provide explicit range division for subsequent query tasks, which helps to optimize query efficiency and resource utilization.

[0048] Through the above steps, when the evaluation result shows that the data distribution is balanced, the coordination node (CN) calculates the step size according to the metadata and the expected number of shards, and continuously divides the data into equal segments according to the step size to generate data segment intervals. These segment intervals will be used for subsequent data query and comparison operations to ensure the optimization of query efficiency and resource utilization.

[0049] In one embodiment, the step S300 further comprises: S3021, if the evaluation result is data distribution imbalance, concatenate the global extreme value in the metadata to generate a string; S3022, hash calculation is performed on the string to generate a hash value; S3023, determine the number of hash buckets according to the data volume and the expected number of shards, and encode the hash buckets; S3024, distribute the data into the hash buckets according to the hash value; S3025, uniformly divide the data according to the hash bucket encoding to generate data segment intervals.

[0050] In this embodiment, in a distributed database environment, when the data distribution is evaluated as unbalanced (dispersion < 90%), the coordination node (CN) will adopt a hash-based strategy to re-split and organize the data to optimize subsequent data query and comparison operations. First, the CN node obtains the evaluation result from the previous step, including the dispersion and data distribution. If the evaluation result shows that the data distribution is unbalanced (dispersion < 90%), proceed to the next step.

[0051] The CN node obtains the maximum value (MAX) and the minimum value (MIN) of the global extreme value from the metadata. Concatenate the two global extreme values into a string for hash calculation. Then, hash calculation is performed on the concatenated string to generate a hash value. This hash value will be used for subsequent data distribution.

[0052] Next, the CN node determines the number of hash buckets based on the data volume and the expected number of shards, and encodes the hash buckets. The number of hash buckets is usually determined based on the data volume and the expected number of shards to ensure that the data can be reasonably divided. The encoding of the hash bucket usually uses numbers or strings to facilitate subsequent operations.

[0053] The CN node allocates data to the hash buckets based on the hash value. The specific method is to use the hash value to take the modulus of the number of hash buckets, and allocate the data to the corresponding hash bucket. In this way, it can ensure that the data is evenly distributed among the hash buckets.

[0054] The CN node stores the data of each hash bucket for subsequent operations. Then, the data is uniformly divided according to the hash bucket encoding to generate data segment intervals. A corresponding query SQL statement is generated for each hash bucket for subsequent data query and operation. These query statements will explicitly specify the range of each hash bucket to ensure the accuracy and efficiency of the query.

[0055] Finally, the CN node stores the generated segment intervals for subsequent data query and comparison operations. These segment intervals provide explicit range division for subsequent query tasks, which helps to optimize query efficiency and resource utilization.

[0056] Through the above steps, when the evaluation result shows that the data distribution is uneven, the coordination node (CN) generates a hash value based on the global extreme value in the metadata, determines the number of hash buckets and encodes them, allocates data to the hash buckets, and uniformly divides the data according to the hash bucket encoding to generate data segment intervals. These segment intervals will be used for subsequent data query and comparison operations to ensure the optimization of query efficiency and resource utilization.

[0057] In an embodiment, a distributed database data splitting apparatus is provided, which corresponds to the distributed database data splitting method in the above embodiment. Referring to Figure 3 , Figure 3 The functional module diagram of a preferred embodiment of the distributed database data splitting apparatus of the present application is shown. The metadata initialization module 10, the data dispersion evaluation module 20, and the data adaptive splitting module 30. The detailed description of each functional module is as follows: The metadata initialization module 10 is used to obtain the distributed database table to be split and analyze and process the distributed database table to obtain the metadata; The data dispersion evaluation module 20 is used to evaluate the data distribution of the distributed database table based on the metadata to obtain an evaluation result; The data adaptive splitting module 30 is configured to select a splitting strategy according to the evaluation result, and split the data according to the splitting strategy to generate data segment intervals.

[0058] In one embodiment, the metadata initialization module 10 comprises: The task decomposition unit is configured to decompose the query task into sub-query tasks according to the query request by the coordination node. The task allocation unit is configured to allocate each sub-query task to a respective data node. The table segment row number unit is configured to count the number of rows of a table segment according to the sub-query task by the respective data node. The data total amount unit is configured to aggregate the number of rows of each table segment to generate a total number of rows, i.e., a data total amount, by the coordination node.

[0059] In one embodiment, the metadata initialization module 10 further comprises: The extreme value calculation unit is configured to calculate the primary key or unique index column extreme value of a table segment according to the sub-query task by the respective data node. The global extreme value unit is configured to aggregate the primary key or unique index column extreme value of each table segment to generate a global extreme value by the coordination node.

[0060] In one embodiment, the metadata initialization module 10 further comprises: The metadata information unit is configured to query metadata information in a database according to the query request by the coordination node. The distribution strategy unit is configured to obtain a distribution strategy of the current table according to the metadata information. The topology structure unit is configured to obtain a topology structure of all data nodes according to the metadata information.

[0061] In one embodiment, the data dispersion degree evaluation module 20 comprises: The data acquisition unit is configured to acquire a data total amount of the metadata, a maximum value of the global extreme value, and a minimum value of the global extreme value. The dispersion degree unit is configured to calculate a dispersion degree according to the data total amount, the maximum value, and the minimum value. The dispersion evaluation unit is configured to compare the dispersion degree with a dispersion threshold value. The distribution balance unit is configured to determine that the data distribution is balanced if the dispersion degree is greater than or equal to the dispersion threshold value. The distribution imbalance unit is configured to determine that the data distribution is unbalanced if the dispersion degree is less than the dispersion threshold value. The evaluation result unit is configured to constitute an evaluation result from the dispersion degree and the data distribution.

[0062] In one embodiment, the data adaptive splitting module 30 comprises: a step unit, configured to calculate a step according to the metadata and the expected number of shards if the evaluation result is that the data distribution is balanced; a balanced distribution table unit, configured to continuously and equally split the data according to the step to generate data segment intervals.

[0063] In one embodiment, the data adaptive splitting module 30 further comprises: a string unit, configured to splice a global extreme value in the metadata to generate a string if the evaluation result is that the data distribution is unbalanced; a hash value unit, configured to perform hash calculation on the string to generate a hash value; a hash bucket number unit, configured to determine the number of hash buckets according to the data volume and the expected number of shards, and encode the hash buckets; a hash value distribution unit, configured to distribute the data into the hash buckets according to the hash value; an unbalanced distribution table unit, configured to uniformly segment the data according to the hash bucket encoding to generate data segment intervals.

[0064] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 4 The computer device comprises a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile and / or volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with an external user terminal through a network connection. The computer program is executed by the processor to implement the functions or steps of a distributed database data splitting method server side.

[0065] In one embodiment, a computer device is provided, which can be a user terminal, and an internal structure diagram of the computer device can be as shown in Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external server through the network connection. The computer program is executed by the processor to realize the functions or steps of the user side of the distributed database data splitting method In one embodiment, a computer device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the computer program to implement the following steps: Obtain a distributed database table to be split, and analyze and process the distributed database table to obtain metadata; According to the metadata, the data distribution of the distributed database table is evaluated to obtain an evaluation result; According to the evaluation result, a splitting strategy is selected, and the data is balanced according to the splitting strategy to generate a data segmentation interval.

[0066] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the following steps: Obtain a distributed database table to be split, and analyze and process the distributed database table to obtain metadata; According to the metadata, the data distribution of the distributed database table is evaluated to obtain an evaluation result; According to the evaluation result, a splitting strategy is selected, and the data is balanced according to the splitting strategy to generate a data segmentation interval.

[0067] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement correspond to the related descriptions of the server side and the user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0068] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0069] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0070] It should be noted that if non-company software tools or components appear in the embodiments of the present application, they are only used for example introduction and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A distributed database data partitioning method, characterized in that, Includes the following steps: Obtain the distributed database tables to be split, analyze and process the distributed database tables, and obtain metadata; The data distribution of the distributed database tables is evaluated based on the aforementioned metadata to obtain the evaluation results. Based on the evaluation results, a splitting strategy is selected, and the data is split evenly according to the splitting strategy to generate data segment intervals.

2. The distributed database data splitting method as described in claim 1, characterized in that, The process of obtaining the distributed database table to be split and analyzing and processing the distributed database table to obtain metadata includes: The coordinating node decomposes the query task into sub-query tasks based on the query request; Distribute each subquery task to the respective data nodes; Each data node counts the number of rows in the table segment based on the subquery task; The coordinating node aggregates the row counts of each table fragment to generate the total number of rows in the table, i.e., the total amount of data.

3. The distributed database data splitting method as described in claim 2, characterized in that, The process of obtaining the distributed database table to be split, analyzing and processing the distributed database table, and obtaining metadata also includes: Each data node calculates the extreme value of the primary key or unique index column of the table fragment based on the subquery task; The coordinating node aggregates the extreme values ​​of the primary key or unique index column of each table fragment to generate a global extreme value.

4. The distributed database data splitting method as described in claim 2, characterized in that, The process of obtaining the distributed database table to be split, analyzing and processing the distributed database table, and obtaining metadata also includes: The coordinating node queries the database for metadata information based on the query request; The distribution strategy for the current table is obtained based on the metadata information; The topology of all data nodes is obtained based on the metadata information.

5. The distributed database data splitting method as described in claim 3, characterized in that, The step of evaluating the data distribution of the distributed database tables based on the metadata and obtaining the evaluation result includes: Obtain the total amount of the metadata, the maximum value of the global extremum, and the minimum value of the global extremum; The dispersion is calculated based on the total amount of data, the maximum value, and the minimum value. The dispersion is compared with the dispersion threshold; If the dispersion is greater than or equal to the dispersion threshold, the data distribution is determined to be balanced. If the dispersion is less than the dispersion threshold, the data distribution is determined to be unbalanced. The evaluation result is composed of the dispersion and the data distribution.

6. The distributed database data splitting method as described in claim 1, characterized in that, The step of selecting a splitting strategy based on the evaluation results, and then splitting the data evenly according to the splitting strategy to generate data segment intervals includes: If the evaluation result indicates that the data distribution is balanced, then the step size is calculated based on the metadata and the expected number of shards; The data is continuously divided into equal segments based on the step size to generate data segment intervals.

7. The distributed database data splitting method as described in claim 1, characterized in that, The step of selecting a splitting strategy based on the evaluation results, and performing balanced splitting of the data according to the splitting strategy to generate data segment intervals, further includes: If the evaluation result indicates that the data distribution is unbalanced, then the global extreme values ​​in the metadata are concatenated to generate a string; Perform a hash calculation on the string to generate a hash value; The number of hash buckets is determined based on the data volume and the expected number of shards, and the hash buckets are then encoded. The data is allocated to hash buckets based on the hash value; The data is evenly divided according to the hash bucket encoding to generate data segment intervals.

8. A distributed database data splitting device, characterized in that, The distributed database data splitting device includes: The metadata initialization module is used to obtain the distributed database tables to be split, analyze and process the distributed database tables, and obtain metadata. The data dispersion assessment module is used to assess the data distribution of the distributed database tables based on the metadata and obtain the assessment results. The data adaptive splitting module is used to select a splitting strategy based on the evaluation results, split the data evenly according to the splitting strategy, and generate data segment intervals.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a distributed database data splitting program stored in the memory and executable on the processor. When executed by the processor, the distributed database data splitting program implements the steps of the distributed database data splitting method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a distributed database data splitting program, which, when executed by a processor, implements the steps of the distributed database data splitting method as described in any one of claims 1-7.

Images