Cross-database migration data uniform distribution rule recommendation method and device
Through multi-dimensional information collection and classification processing, the appropriate distribution rule priority and distribution key are selected, which solves the problems of storage uniformity and load balancing in data migration, and realizes flexible distribution rule adaptation and efficient migration process.
Patent Information
- Application Number
- CN202411225782.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-12-20
AI Technical Summary
The existing technology lacks effective distribution rules during data migration, which makes it difficult to achieve data storage uniformity and business processing load balancing, and the distribution rules are relatively mechanized and cannot be adjusted dynamically.
Multi-dimensional information collection and classification processing are adopted to select appropriate distribution rules priorities, group and distribution key selection is carried out based on table association relationship and storage calculation cost, comprehensively determine the table distribution scope, and provide recommended methods for uniform distribution rules for data migration across databases.
It realizes data storage uniformity and business processing load balancing after data migration, provides flexible distribution rules adaptability, can dynamically adjust according to the data status and business model, and improves migration efficiency and processing efficiency.
Smart Images
Figure CN119166614A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of distributed databases, and in particular to a method for recommending uniform distribution rules for data migration across databases. Background Art
[0002] A distributed database is a database system in which data is physically distributed but logically centrally managed. Distributed databases have transparency, data redundancy, and scalability. Scalability is an important feature of distributed databases. Distributed databases rely on scalability to improve the processing power of the database. Data is dynamically distributed to newly added DNs (Data Nodes) to eliminate data skew, and the improvement of scalability depends on data partitioning.
[0003] Data is split into multiple DNs and stored and processed on multiple DNs. This technology is called data partitioning, also known as data sharding. There are two types of data partitioning: horizontal partitioning and vertical partitioning. Horizontal partitioning is to store table data in multiple DNs according to rules, and each DN contains part of the table data. Vertical partitioning is to store different tables in different DNs, or to distribute different columns of table information to different DNs. Distributed database distribution rules (partitioning rules) are the problem of mapping the entire data set to multiple DNs according to the partitioning rules, that is, dividing the data set into multiple DNs, and each DN is responsible for a subset of the overall data. The partitioning rules and distribution range of table data will affect the storage distribution and business processing efficiency of the data in the destination distributed database before and after migration. Therefore, reasonable data distribution rules are of great significance to distributed databases.
[0004] With the development of online Internet services, the number of concurrent business accesses has increased exponentially, and the demand for massive data computing and analysis has become increasingly common. Traditional stand-alone systems face huge challenges in terms of business, cost, and openness. The emergence of distributed databases can better meet these challenges. Therefore, distributed databases are becoming more and more popular, and data interactions with other databases are increasing. Data migration to distributed databases is also becoming more and more common. In order to ensure the integrity, consistency, and standardization of data operations, the migration evaluation system plays an important role. For distributed databases, how data is reasonably distributed is of great research significance, and providing reasonable distribution rules is of great significance for data migration to distributed databases.
[0005] Through patent search, it is found that there are few inventions related to database distribution rules. The existing distribution methods have the following problems: the distribution rule table is generated based on table information, with little reference information and lack of reliability; the distribution method and distribution key selection are not considered, and the capacity of the destination DN is not considered. There are also methods that take into account data capacity and business model to optimize distribution rules; but their distribution rules are relatively mechanical and rigid, requiring manual specification of rules, and cannot be adjusted dynamically according to the current data status; and the distribution range is determined by the single dependent table data volume. Therefore, it is of great value to provide a reliable data distribution rule based on the current data status, business model and target distributed database, which can ensure the storage uniformity of the destination database and the load balancing of business processing after data migration. Summary of the invention
[0006] The purpose of the present invention is to provide a reasonable distribution rule recommendation method for data migration to a distributed database based on multi-dimensional information, so as to ensure the uniformity of data storage and the load balance of business processing after data migration, and solve the difficulties of data storage uniformity and business processing load balance after distribution.
[0007] To achieve the above-mentioned purpose of the invention, the embodiment of the present application first provides a method for recommending uniform distribution rules for data migration across databases, which is characterized by comprising: Collect basic information of the source database; Categorize the collected basic information and filter out the services that can and cannot be supported by the destination database; Select the distribution rule priority for the database tables supported by the target database, including the priority of the number of target DNs and the priority of the capacity of the target DNs; Group tables according to their associations; Select table distribution key based on table grouping information; Select table distribution method based on table grouping information; Considering the storage cost and computing cost, determine the table distribution range and give an evaluation of the cross-database migration results.
[0008] In order to achieve the above-mentioned object of the invention, the embodiment of the present application further provides a device for recommending uniform distribution rules of data for cross-database migration, which is characterized by comprising: The collection module is used to collect basic information of the source database; The classification module is used to classify the collected basic information and filter out the services that can and cannot be supported by the destination database; A distribution rule selection module is used to select a distribution rule priority for the database table supported by the destination database, including selecting a destination DN quantity priority and a destination DN capacity priority; The grouping module is used to group tables according to their association relationships; A distribution key selection module, used for selecting a table distribution key according to table grouping information; A distribution mode selection module is used to select a table distribution mode according to table grouping information; The evaluation module is used to comprehensively consider the storage cost and the computational cost, determine the table distribution range, and provide a result evaluation of the cross-database migration. To achieve the above-mentioned invention purpose, the embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores a program that can be run on the processor, characterized in that when the program is executed by the processor, the steps of the method for recommending uniform data distribution rules for cross-database migration are implemented.
[0009] Also, a computer-readable storage medium is provided, storing at least one program, characterized in that the at least one program can be executed by at least one processor to implement the steps of the above-mentioned method for recommending uniform distribution rules for data migration across databases.
[0010] The present invention has the following beneficial effects: a) The method of the present invention firstly performs distribution rule priority planning from the overall perspective, performs characteristic analysis and processing according to the characteristics of the table group, and adapts from the segmented scenarios to provide two distribution priority options, one of which is the priority of the number of destination DNs, and the other is the priority of the capacity of the destination DNs, so that the method of the present invention can be flexibly applied to multiple scenarios and has wide adaptability.
[0011] b) The method of the present invention selects distribution keys according to table structure, business frequency and table association information, thereby ensuring the efficiency of data migration and subsequent business processing.
[0012] c) The method of the present invention is based on the storage cost and computing cost of the target database, comprehensively considers the storage cost and computing cost of the table, performs a uniformity check after processing each table group, and splits or reallocates the table groups that may exceed the limit, etc., to ensure the storage uniformity and computing uniformity of the target database before and after migration, and to ensure business processing efficiency.
[0013] d) The method of the present invention realizes automatic processing of the distribution rules of the table according to the characteristics of the table group and the overall plan, including but not limited to the automatic action of expansion or splitting, and can generate different distribution rules according to the configuration scheme, which has strong flexibility. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 A flowchart of a method for recommending uniform distribution rules for data migration across databases according to an embodiment of the present application; Figure 2 A logical diagram for implementing the recommended uniform data distribution rule for the embodiment of the present application; Figure 3 A logical diagram for implementing the priority selection of distribution rules in an embodiment of the present application; Figure 4 A logical schematic diagram for determining the distribution range of a single-node replication table for an embodiment of the present application; Figure 5 A logical schematic diagram for determining a hash table distribution range for an embodiment of the present application; Figure 6 This is a functional schematic diagram of a device for recommending uniform distribution rules for data migration across databases according to an embodiment of the present application; Figure 7 A schematic diagram of the structure of a device for recommending uniform distribution rules for data migration across databases according to an embodiment of the present application; Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0015] In order to make the technical solution of the present invention more clearly understood, the technical solution of the present invention is further described in detail below according to specific embodiments and in combination with the accompanying drawings. Obviously, the following embodiments are only part of the embodiments of the present invention and do not constitute a limitation on the technical solution of the present invention. For those of ordinary skill in the art, other embodiments can also be obtained according to the embodiments of the present application without paying creative labor.
[0016] It should be noted that the order of steps described in the method embodiments in the following embodiments does not constitute a limitation on the technical solution of the present invention. For ordinary technicians in this field, without paying any creative labor, they can also reasonably adjust the order of certain steps according to the embodiments of the present application. Example 1
[0017] This embodiment provides a method for recommending rules for uniform distribution of data for cross-database migration, which is applied to the cross-database data migration assessment scenario from a source database to a distributed target database. In order to achieve uniform data storage and load balancing of business processing after data migration, reasonable distribution rules are provided based on the information collected by the migration assessment system and the target database information, thereby solving the problems of uniform data storage and load balancing of business processing after distribution.
[0018] The method of the present invention relies on the information collection and analysis results of the target database, collects and analyzes the basic information of the database, data definition language (DDL), data query language (DQL) and data management language (DML) according to the type of the source database, and uses the table dimension to count the structural information of the table, the information of the associated table and the associated fields, the where condition field and the table data volume, and generates distribution rules in combination with the target database information and configuration information.
[0019] like Figure 1 , Figure 2 As shown, the method for recommending uniform distribution rules for data migration across databases in this embodiment includes the following steps: Step 101: Collect basic information of the source database.
[0020] Specifically, the basic information of the source database that needs to be collected mainly includes the basic structure information of the source table, the storage status of the table, the business model of the table, and the correlation between the tables.
[0021] Step 102: Classify and process the collected basic information of the source database.
[0022] Since it involves cross-database migration, we first perform lexical and grammatical analysis on the basic information of the source database, thereby screening out the services that the target database can and cannot support. For example, for services such as views and object classes, we need to consider whether the target database syntax or storage engine supports it.
[0023] Step 103, based on the screening result of step 102, select the distribution rule priority for the service library table that the destination database can support, including selecting the destination DN quantity priority and selecting the destination DN capacity priority.
[0024] Specifically, the logic for implementing the priority selection of distribution rules is as follows: Figure 3 As shown: (1) Priority on the number of DNs at the destination: When the number of DNs in the distributed database at the destination is determined, that is, the planned number of DNs (optimized to an even number) has been determined and the number of DNs cannot be increased or reduced, the priority selection method of DN number priority is selected.
[0025] When the number of destination DNs is prioritized, perform the following operations: Configure the required number of DNs; The total data volume of the existing table and the number of DNs required for configuration are evenly divided to obtain the estimated storage capacity of each DN; Finally, the number of DNs under the priority principle and the estimated storage capacity of a single DN are given, where the number of DNs remains unchanged in the subsequent allocation process.
[0026] (2) Destination DN capacity priority: When the number of DNs in the destination distributed database is uncertain and can be increased or decreased, and the planned storage capacity of a single DN has an upper limit, the DN storage capacity priority selection method is used.
[0027] When the destination DN capacity is prioritized, the following operations are performed: Configure the storage capacity of a single DN; The total data volume of the existing table and the configured single DN storage capacity are evenly divided, and the required number of DNs is estimated (optimized to an even number). The single DN storage capacity is determined based on the estimated number of DNs. Finally, the single DN storage capacity and the estimated number of DNs under the priority principle are given, where the single DN storage capacity remains unchanged in the subsequent allocation process.
[0028] (3) In the scenario where the number of DNs deployed in the destination distributed database is uncertain and there is no upper limit on the storage capacity of a single DN, two different priority selection methods are run. The final selection of the destination DN number priority or the destination DN capacity priority is determined based on the final uniformity result.
[0029] Step 104: group the tables according to their association relationships.
[0030] Specifically, according to the association relationship of the table, the table is divided into a table without foreign keys and no association relationship, a table with foreign keys but no association relationship, and a table with association relationship, and the table index, foreign key information, association column information and where condition fields are classified and stored.
[0031] Step 105: Select a table distribution key according to the table grouping information.
[0032] Specifically include: (1) For tables without foreign keys and relationships, if there is only one unique index, the first field of the index is used as the distribution key. If there is more than one unique index, determine whether the index field and the where condition field overlap, and select the overlapping field as the distribution key. If there is no overlap, randomly select the first field of a unique index as the distribution key. If there is no unique index, take the key with the most searches. If there is no where condition, take the first non-large type field as the distribution key.
[0033] (2) For tables with foreign keys but no associated relationships, if there is only one foreign key, the first field of the foreign key is used as the distribution key, and all associated tables use this as the distribution key. If there are multiple foreign keys associated with multiple tables, determine whether a common field can be found. If so, the common field is the distribution key. If not, a single-node replicated table is used.
[0034] (3) For tables with associated relationships, take all associated fields in the associated group as candidate distribution keys; determine whether the table has a foreign key. If not, take the candidate distribution key with the largest number of searches as the distribution key; if a foreign key exists and exists among the candidate distribution keys, take the foreign key as the distribution key. Otherwise, take the candidate distribution key with the largest number of searches as the distribution key.
[0035] Step 106: Select a table distribution method according to the table grouping information.
[0036] Specifically include: (1) For tables without foreign keys and association relationships, if they do not reach the hash table limit, the single-node replication table distribution method is used, otherwise the hash table method is used; (2) For tables with foreign keys but no association relationships and tables with association relationships, analyze them in table groups. Try to ensure that tables in the same table group are distributed on the same DN within the allowed DN storage capacity. If the allowed DN storage capacity is exceeded, distribute the tables in the same table group on multiple DNs, and give priority to single-node table replication.
[0037] Step 107, comprehensively consider the storage space required for the table (i.e., storage cost) and the business processing frequency of the table (i.e., computing cost), determine the table distribution range, determine on which DN the table is distributed according to the DN configuration and capacity estimation, and provide a result evaluation of the cross-database migration.
[0038] When determining the table distribution range, single-node replication tables are allocated first, followed by hash tables.
[0039] Specifically, for a single-node replication table group, the logic for determining the distribution range is as follows: Figure 4 As shown: First, the total cost is obtained based on the storage cost and the computing cost according to the set weights, and the table groups are sorted by the total cost; DN is sorted according to the consumed storage cost, and the unprocessed table group with the largest total cost is assigned to the DN with the smallest storage cost in order. Each time a table group is processed, it is calculated whether the current DN storage and computing cost is within the set upper limit range. If it is not exceeded, the DN is re-sorted according to the consumed storage cost and the next table group is processed; otherwise, the next DN is reselected, or the table group is split and continued to be processed in order.
[0040] Specifically, for a hash table group, the logic for determining the distribution range is as follows: Figure 5As shown: First, the total cost is obtained based on the storage cost and the computing cost according to the set weights, and the table groups are sorted by the total cost; the remaining DNs that do not exceed the limit are sorted according to the consumed storage cost, and the unprocessed maximum total cost table groups are evenly distributed to the DNs that do not exceed the limit. Each time a table group is processed, it is calculated whether the current DN storage and computing cost is within the set upper limit range. If it is not exceeded, continue to process the next table group; otherwise, the DN that has exceeded the limit is eliminated, and the current processing table group is evenly distributed to the remaining DNs.
[0041] The distribution range is selected in the above manner, taking into account the storage cost and computing cost. An upper limit range is set for the storage cost and computing cost of a single DN. The storage cost and computing cost of the DN are checked each time a table group is processed, ensuring storage uniformity and business processing efficiency.
[0042] After the distribution scope is determined, the evaluation information of the distribution rules can be given, including but not limited to the following information: executable SQL statements, the number of DNs required for the destination database and the storage capacity of a single DN, table distribution summary information (including table group correlation files and single table distribution rule information), the storage cost and computing cost difference rate of each DN after allocation, and the overall DN storage cost and computing cost volatility, providing strong support for the distribution rule recommendation for migration evaluation. Example 2
[0043] This embodiment provides a device for recommending uniform distribution rules for data migration across databases. Figure 6 As shown, the device collects basic information of the source database, and through a series of analysis and processing, gives a result evaluation of the cross-database migration, providing migration guidance for the cross-database migration from the source database to the target database.
[0044] like Figure 7 As shown, the device for recommending uniform distribution rules for data migration across databases in this embodiment includes: The acquisition module 21 is used to acquire basic information of the source database.
[0045] The classification module 22 is used to classify the basic information collected by the collection module 21 and filter out supported and unsupported services.
[0046] The distribution rule selection module 23 is used to select the distribution rule priority for the supported library table screened by the classification module 22, including selecting the destination DN quantity priority and selecting the destination DN capacity priority. Specifically: (1) Priority on the number of DNs at the destination: When the number of DNs in the distributed database at the destination is determined, that is, the planned number of DNs has been determined and cannot be increased or reduced, the priority selection method of DN number priority is selected.
[0047] When the distribution rule selection module 23 selects the number of destination DNs to be prioritized, the following operations are performed: Configure the required number of DNs; The total data volume of the existing table and the number of configured DNs are evenly divided to obtain the estimated storage capacity of each DN.
[0048] Therefore, the distribution rule selection module 23 provides the number of DNs and the estimated storage capacity of a single DN under the priority principle, wherein the number of DNs remains unchanged in the subsequent allocation process.
[0049] (2) Destination DN capacity priority: When the number of DNs in the destination distributed database is uncertain and can be increased or decreased, and the planned storage capacity of a single DN has an upper limit, the DN storage capacity priority selection method is used.
[0050] When the distribution rule selection module 23 selects the destination DN capacity priority, the following operations are performed: Configure the storage capacity of a single DN; The total data volume of the existing table and the configured single DN storage capacity are evenly divided, the required number of DNs is estimated, and optimized to an even number. The single DN storage capacity is determined based on the estimated number of DNs.
[0051] Therefore, the distribution rule selection module 23 provides the single DN storage capacity and the estimated number of DNs under the priority principle. The single DN storage capacity remains unchanged in the subsequent allocation process.
[0052] (3) In the scenario where the number of DNs deployed in the destination distributed database is uncertain and there is no upper limit on the storage capacity of a single DN, the distribution rule selection module 23 runs two different priority selection methods and determines the final distribution rule based on the final uniformity result.
[0053] The grouping module 24 is used to group the tables according to the association relationship between the tables.
[0054] Specifically, according to the association relationship of the table, the grouping module 24 divides the table into a table without foreign keys and association relationships, a table with foreign keys but no association relationships, and a table with association relationships, and classifies and stores the table index, foreign key information, association column information, and where condition fields.
[0055] The distribution key selection module 25 is used to select a table distribution key according to the table grouping information.
[0056] Specifically: (1) For tables without foreign keys and relationships, if there is only one unique index, the first field of the index is used as the distribution key. If there is more than one unique index, determine whether the index field and the where condition field overlap, and select the overlapping field as the distribution key. If there is no overlap, randomly select the first field of a unique index as the distribution key. If there is no unique index, select the key with the most searches. If there is no where condition, select the first non-large type field as the distribution key. (2) For tables with foreign keys but no associated relationships, if there is only one foreign key, the first field of the foreign key is used as the distribution key, and all associated tables use this as the distribution key; if there are multiple foreign keys associated with multiple tables, determine whether a common field can be found. If it can be found, the common field is the distribution key; if it cannot be found, it is used as a single-node replicated table; (3) For tables with associated relationships, take all associated fields in the associated group as candidate distribution keys; determine whether the table has a foreign key. If not, take the candidate distribution key with the largest number of searches as the distribution key; if a foreign key exists and exists among the candidate distribution keys, take the foreign key as the distribution key. Otherwise, take the candidate distribution key with the largest number of searches as the distribution key.
[0057] The distribution mode selection module 26 is used to select a table distribution mode according to the table grouping information.
[0058] Specifically: (1) For tables without foreign keys and association relationships, if they do not reach the hash table limit, the single-node replication table distribution method is used, otherwise the hash table method is used; (2) For tables with foreign keys but no association relationships and tables with association relationships, analysis is performed on a table group basis. Within the allowed DN storage capacity, tables in the same table group are distributed on the same DN as much as possible. If the capacity exceeds the limit, the tables are distributed on multiple DNs. Single-node replication is preferred.
[0059] As an example, Figure 7 As shown in the dotted box, the functions implemented by the above-mentioned classification module 22, distribution rule selection module 23, grouping module 24, distribution key selection module 25, and distribution method selection module 26 can be integrated into one analysis and processing module.
[0060] Evaluation module 27 comprehensively considers the storage space required for the table (i.e., storage cost) and the business processing frequency of the table (i.e., computing cost), sets a volatility limit range for the storage cost and computing cost of a single DN, determines the table distribution range, determines on which DNs the table is distributed based on DN configuration and capacity estimation, and provides a result evaluation of cross-database migration.
[0061] When determining the table distribution range, single-node replication tables are allocated first, followed by hash tables.
[0062] Specifically, for a single-node replication table group, the logic for determining the distribution range is as follows: Figure 4 As shown: First, the total cost is obtained based on the storage cost and the computing cost according to the set weights, and the table groups are sorted by the total cost; DN is sorted according to the consumed storage cost, and the unprocessed table group with the largest total cost is assigned to the DN with the smallest storage cost in order. Each time a table group is processed, it is calculated whether the current DN storage and computing cost is within the set upper limit range. If it is not exceeded, the DN is re-sorted according to the consumed storage cost and the next table group is processed; otherwise, the next DN is reselected, or the table group is split and continued to be processed in order.
[0063] Specifically, for a hash table group, the logic for determining the distribution range is as follows: Figure 5 As shown: First, the total cost is obtained based on the storage cost and the computing cost according to the set weights, and the table groups are sorted by the total cost; the remaining DNs that do not exceed the limit are sorted according to the consumed storage cost, and the unprocessed maximum total cost table groups are evenly distributed to the DNs that do not exceed the limit. Each time a table group is processed, it is calculated whether the current DN storage and computing cost is within the set upper limit range. If it is not exceeded, continue to process the next table group; otherwise, the DN that has exceeded the limit is eliminated, and the current processing table group is evenly distributed to the remaining DNs.
[0064] The distribution range is selected in the above manner, taking storage cost and computing cost into comprehensive consideration. Storage cost and computing cost verification of DN are performed each time a table group is processed, thus ensuring storage uniformity and business processing efficiency.
[0065] The distribution range is selected in the above manner, taking into account the storage cost and computing cost. An upper limit range is set for the storage cost and computing cost of a single DN. The storage cost and computing cost of the DN are checked each time a table group is processed, ensuring storage uniformity and business processing efficiency.
[0066] After the distribution scope is determined, the evaluation module 27 can provide evaluation information of the distribution rules, including but not limited to the following information: executable SQL statements, the number of DNs required for the destination database and the storage capacity of a single DN, table distribution summary information (including table group correlation files and single table distribution rule information), the storage cost and computing cost difference rate of each DN after allocation, and the overall DN storage cost and computing cost volatility, providing strong support for distribution rule recommendation for migration evaluation. Example 3
[0067] This embodiment provides an electronic device, such as Figure 8As shown, it includes a processor 31 and a memory 32 in communication with the processor 31. The memory 32 stores a program that can be run on the processor 31, and when the program is executed by the processor 31, the steps of the method for recommending uniform distribution rules for data migration across databases provided in the above embodiment are implemented.
[0068] The above program can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. Example 4
[0069] This embodiment provides a computer-readable storage medium storing at least one program. The at least one program can be executed by at least one processor to implement the steps of the method for recommending uniform distribution rules for data migration across databases provided in the above embodiment.
[0070] The above storage media include: U disk, mobile hard disk, read-only memory, random access memory, disk or optical disk and other media that can store program code and their combination. With the development of science and technology, the meaning of storage media may become more and more extensive and not limited to tangible media.
[0071] Those skilled in the art can understand that the above-mentioned embodiments are specific embodiments for implementing the present invention. In addition to the above-mentioned embodiments, the present invention may also have other implementation modes; any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the protection scope required by the present invention.
Claims
1. A method for recommending uniform distribution rules for data migration across databases, characterized in that: include: Collect basic information of the source database; Categorize the collected basic information and filter out the services that can and cannot be supported by the destination database; Select the distribution rule priority for the database tables supported by the target database, including the priority of the number of target DNs and the priority of the capacity of the target DNs; Group tables according to their associations; Select table distribution key based on table grouping information; Select table distribution method based on table grouping information; Considering the storage cost and computing cost, determine the table distribution range and give an evaluation of the cross-database migration results.
2. The method for recommending uniform distribution rules for data migration across databases according to claim 1, characterized in that: When the number of destination DNs is selected first, the method further includes: Configure the required number of DNs; The total data volume of the table and the required number of DNs are evenly divided to obtain the estimated storage capacity of each DN.
3. The method for recommending uniform distribution rules for data migration across databases according to claim 1, characterized in that: When the destination DN capacity is selected first, the method further includes: Configure the storage capacity of a single DN; The total data volume of the table and the configured single DN storage capacity are evenly divided, the required number of DNs is estimated and optimized to an even number, and the single DN storage capacity is determined based on the estimated number of DNs.
4. The method for recommending uniform distribution rules for data migration across databases according to claim 1, characterized in that: Group tables according to their associations, including: According to the association relationship of the table, the table is divided into a table without foreign keys and no association relationship, a table with foreign keys but no association relationship, and a table with association relationship, and the table index, foreign key information, associated column information and where condition fields are classified and stored.
5. The method for recommending uniform distribution rules for data migration across databases according to claim 1, characterized in that: Select table distribution keys based on table grouping information, including: For tables without foreign keys and association relationships, if there is only one unique index, the first field of the index is used as the distribution key; if there is more than one unique index, determine whether the index field and the where condition field overlap, and select the overlapping field as the distribution key; if there is no overlap, randomly select the first field of a unique index as the distribution key; if there is no unique index, take the key with the most searches; if there is no where condition, take the first non-large type field as the distribution key; For tables with foreign keys but no association relationships, if there is only one foreign key, the first field of the foreign key is used as the distribution key, and all associated tables use this as the distribution key; if there are multiple foreign keys associated with multiple tables, determine whether a common field can be found. If it can be found, the common field is the distribution key; if it cannot be found, it is used as a single-node replicated table; For a table with an associated relationship, take all associated fields in the associated group as candidate distribution keys; determine whether the table has a foreign key. If not, take the candidate distribution key with the largest number of retrievals as the distribution key; if a foreign key exists and the foreign key exists among the candidate distribution keys, take the foreign key as the distribution key, otherwise take the candidate distribution key with the largest number of retrievals as the distribution key.
6. The method for recommending uniform distribution rules for data migration across databases according to claim 1, characterized in that: Select the table distribution method based on the table grouping information, including: For tables without foreign keys and association relationships, if they do not reach the hash table limit, the single-node replication table distribution method is used, otherwise the hash table method is used; For tables with foreign keys but no association relationships and tables with association relationships, analysis is performed on a table group basis. Within the allowed DN storage capacity, try to ensure that tables in the same table group are distributed on the same DN. If the allowed DN storage capacity is exceeded, distribute the tables in the same table group on multiple DNs, and give priority to single-node table replication.
7. The method for recommending uniform distribution rules for data migration across databases according to claim 1, characterized in that: Consider the storage cost and computing cost to determine the table distribution range, including: Single-node replication tables are allocated first, followed by hash tables; For single-node replication table groups, the total cost is calculated based on the storage cost and computing cost according to the set weights, and the table groups are sorted by the total cost; DNs are sorted according to the consumed storage cost, and the unprocessed table groups with the largest total cost are assigned to the DNs with the smallest consumed storage cost in order. Each time a table group is processed, it is calculated whether the current DN storage and computing cost is within the set upper limit. If it is within the upper limit, the next table group is processed; otherwise, the next DN is reselected, or the table group is split and processed in order; For hash table groups, the total cost is calculated based on the storage cost and computing cost according to the set weights, and the table groups are sorted by the total cost; the unprocessed table groups with the largest total cost are evenly distributed to the DNs that have not exceeded the limit in order. Each time a table group is processed, it is calculated whether the storage and computing cost of the current DN is within the set upper limit range. If it is not exceeded, the next table group is processed; otherwise, the DN that has exceeded the limit range is eliminated, and the currently processed table group is evenly distributed to the remaining DNs.
8. A device for recommending uniform distribution rules for data migration across databases, characterized in that: include: The collection module is used to collect basic information of the source database; The classification module is used to classify the collected basic information and filter out the services that can and cannot be supported by the destination database; A distribution rule selection module is used to select a distribution rule priority for the database table supported by the destination database, including selecting a destination DN quantity priority and a destination DN capacity priority; The grouping module is used to group tables according to their association relationships; A distribution key selection module, used for selecting a table distribution key according to table grouping information; A distribution mode selection module is used to select a table distribution mode according to table grouping information; The evaluation module is used to comprehensively consider storage costs and computing costs, determine the table distribution range, and provide an evaluation of the results of cross-database migration.
9. An electronic device comprising a processor and a memory, wherein the memory stores a program that can be run on the processor, characterized in that: When the program is executed by the processor, the steps of the method for recommending uniform distribution rules for data migration across databases as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing at least one program, characterized in that: The at least one program can be executed by at least one processor to implement the steps of the method for recommending uniform distribution rules for data migration across databases as described in any one of claims 1 to 7.