Data migration method and device

Abstract

The invention provides a data migration method and device, and belongs to the technical field of databases. The method comprises the steps that under the condition that to-be-migrated data is migrated from a first database to a second database, static metadata information of the to-be-migrated data is obtained from the first database, N initial fragmentation keys are determined based on the static metadata information, N is a positive integer, dynamic load data of the to-be-migrated data are obtained according to the N initial fragmentation keys, and the to-be-migrated data are migrated according to the dynamic load data. The method comprises the steps of determining N initial fragmentation keys according to dynamic load data, determining M candidate fragmentation keys from the N initial fragmentation keys based on the dynamic load data, determining a balance evaluation index corresponding to the candidate fragmentation key for any candidate fragmentation key, determining a target fragmentation key from the M candidate fragmentation keys based on the balance evaluation indexes corresponding to the M candidate fragmentation keys, and carrying out data migration on the to-be-migrated data according to the target fragmentation key. In this way, the technical problems that in the data migration process, due to the fact that hot spot data are centrally accessed, data skew is caused, and part of nodes are overloaded can be solved.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data migration method and apparatus. Background Technology

[0002] With the development of big data technology, the demand for data storage and processing in various industries is growing exponentially. Due to the powerful data storage and processing capabilities of distributed databases, migrating from centralized databases to distributed databases has become an important trend in the development of the database field.

[0003] In related technologies, existing data migration methods mainly determine sharding keys by analyzing the static characteristics of the data, and then migrate data from a centralized database to various nodes of a distributed database based on these sharding keys. While this method can achieve data sharding and migration to a certain extent, when a large number of users access a particular piece of data simultaneously, a large amount of data will flood the node storing that data, causing data skew. This can lead to excessive node load, pushing data processing capacity to its limit, increasing response time, and even potentially causing node crashes. Summary of the Invention

[0004] The purpose of this application is to provide a data migration method and apparatus to solve the technical problems of data skew and excessive load on some nodes caused by concentrated access to hot data. The specific technical solution is as follows:

[0005] In a first aspect of this application, a data migration method is provided, the method comprising:

[0006] When data to be migrated is migrated from the first database to the second database, static metadata information of the data to be migrated is obtained from the first database, and N initial sharding keys are determined based on the static metadata information, where N is a positive integer;

[0007] The dynamic load data of the data to be migrated is obtained based on the N initial sharding keys, and M candidate sharding keys are determined from the N initial sharding keys based on the dynamic load data, where M is a positive integer;

[0008] For any of the candidate partitioning keys, determine the balance evaluation index corresponding to the candidate partitioning key;

[0009] Based on the balance evaluation index corresponding to the M candidate sharding keys, a target sharding key is determined from the M candidate sharding keys, and the data to be migrated is migrated according to the target sharding key.

[0010] In an optional implementation, determining N initial sharding keys based on the static metadata information includes:

[0011] Obtain the field cardinality of all fields in the data to be migrated from the static metadata information;

[0012] The field whose cardinality in the data to be migrated is greater than a preset cardinality is used as the initial sharding key.

[0013] In an optional implementation, obtaining the dynamic load data of the data to be migrated based on the N initial sharding keys includes:

[0014] Obtain the historical operation records of the data to be migrated;

[0015] The operation records containing the N initial sharding keys in the historical operation records are used as the dynamic load data.

[0016] In an optional implementation, determining M candidate sharding keys from the N initial sharding keys based on the dynamic load data includes:

[0017] For any of the initial sharding keys, the operand rate of the initial sharding key is determined based on the dynamic load data;

[0018] The initial sharding key with an operation rate greater than a preset operation threshold is determined as the candidate sharding key.

[0019] In an optional implementation, determining the balance evaluation metric corresponding to the candidate sharding key includes:

[0020] Determine the discrete entropy value corresponding to the candidate fragmentation key;

[0021] And / or,

[0022] Determine the write skew rate corresponding to the candidate sharding key.

[0023] In an optional implementation, determining the discrete entropy value corresponding to the candidate fragmentation key includes:

[0024] Obtain the operation record corresponding to the candidate sharding key from the dynamic load data;

[0025] Obtain multiple values ​​corresponding to the candidate sharding key from the operation record, and determine the probability of occurrence of each value corresponding to the candidate sharding key;

[0026] For any value corresponding to the candidate sharding key, calculate the distribution rate of the value based on the probability of the value's occurrence;

[0027] The distribution rates of each value corresponding to the candidate fragmentation key are summed to obtain the initial entropy value corresponding to the candidate fragmentation key;

[0028] The negative value of the initial entropy value corresponding to the candidate partition key is used as the discrete entropy value corresponding to the candidate partition key.

[0029] In an optional implementation, calculating the distribution rate of the value based on the probability of its occurrence includes:

[0030] The probability of the value occurring is input into the distribution formula to calculate the distribution of the value. The distribution formula is as follows:

[0031] G = p i *log2p i ;

[0032] Where G is the distribution rate of the value, p i The probability of the value occurring is given.

[0033] In an optional implementation, determining the write skew rate corresponding to the candidate shard key includes:

[0034] The data to be migrated is simulated by splitting it according to the preset number of shards and the candidate sharding key, and a simulation operation record is obtained.

[0035] Obtain the write operation data volume of each fragment in the preset fragment from the simulation operation record;

[0036] Based on the amount of write operation data in each of the preset shards, determine the average amount of operation data and the maximum amount of write operation data in the preset shards;

[0037] The write skew rate corresponding to the candidate shard key is determined by the average operation data volume and the maximum write operation data volume.

[0038] In an optional implementation, after migrating the data to be migrated according to the target shard key, the method further includes:

[0039] Retrieve business operation data of the migrated data from the second database;

[0040] The number of cross-node operations is determined based on the aforementioned business operation data;

[0041] The cross-node operation rate is determined based on the number of cross-node operations and the business operation data.

[0042] If the cross-shard operation rate exceeds a preset cross-shard threshold, obtain the static metadata information of the migrated data in the second database;

[0043] The step of determining N initial sharding keys is performed based on the static metadata information of the data that has been migrated from the second database.

[0044] In a second aspect of this application, a data migration apparatus is also provided, the apparatus comprising:

[0045] The information acquisition module is used to acquire static metadata information of the data to be migrated from the first database when the data to be migrated is migrated from the first database to the second database.

[0046] An initial sharding key determination module is used to determine N initial sharding keys based on the static metadata information, where N is a positive integer.

[0047] The candidate sharding key determination module is used to obtain dynamic load data of the data to be migrated based on the N initial sharding keys, and determine M candidate sharding keys from the N initial sharding keys based on the dynamic load data, where M is a positive integer;

[0048] The balance evaluation module is used to determine the balance evaluation index corresponding to any of the candidate sharding keys.

[0049] The target sharding key determination module is used to determine the target sharding key from the M candidate sharding keys based on the balance evaluation index corresponding to the M candidate sharding keys;

[0050] The data migration module is used to migrate the data to be migrated based on the target shard key.

[0051] In a third aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0052] Memory, used to store computer programs;

[0053] When a processor executes a program stored in memory, it implements the data migration method described in any one of the first aspects above.

[0054] In a fourth aspect of the embodiments of this application, a storage medium is also provided, wherein the storage medium stores instructions that, when run on a computer, cause the computer to perform any of the data migration methods described in the first aspect above.

[0055] In a fifth aspect of the embodiments of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the data migration methods described in the first aspect above.

[0056] The technical solution provided in this application, when migrating data from a first database to a second database, involves obtaining static metadata information of the data to be migrated from the first database, determining N initial sharding keys (N being a positive integer) based on the static metadata information, obtaining dynamic load data of the data to be migrated based on the N initial sharding keys, and determining M candidate sharding keys (M being a positive integer) from the N initial sharding keys based on the dynamic load data. For each candidate sharding key, determining the corresponding balance evaluation index, and determining the target sharding key from the M candidate sharding keys based on the balance evaluation index, and then migrating the data to be migrated according to the target sharding key. This combination of static metadata information and dynamic load data to determine candidate sharding keys, and determining the target sharding key based on the balance evaluation index of the candidate sharding keys, effectively migrates the data to be migrated. This solves the technical problems of data skew and excessive load on some nodes caused by concentrated access to hot data during data migration. Attached Figure Description

[0057] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 A schematic diagram illustrating the implementation process of a data migration method provided in this application embodiment;

[0060] Figure 2 A schematic diagram illustrating the implementation process of another data migration method provided in this application embodiment;

[0061] Figure 3 A schematic diagram illustrating the implementation process of a method for determining discrete entropy values ​​provided in this application embodiment;

[0062] Figure 4 A schematic diagram illustrating the implementation process of a method for determining write skew rate provided in this application embodiment;

[0063] Figure 5 A schematic diagram illustrating the implementation process of a method for adjusting a fragmentation strategy provided in an embodiment of this application;

[0064] Figure 6 This is a schematic diagram of the structure of a data migration device provided in an embodiment of this application;

[0065] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.

[0068] like Figure 1 The diagram shown is a schematic representation of the implementation process of a data migration method provided in this application. The method may specifically include the following steps:

[0069] S101, when the data to be migrated is migrated from the first database to the second database, obtain the static metadata information of the data to be migrated from the first database, and determine N initial sharding keys based on the static metadata information, where N is a positive integer.

[0070] The data to be migrated refers to the set of data that needs to be migrated from the source database (first database) to the target database (second database).

[0071] The aforementioned first database refers to the source database for data migration, i.e., the database currently storing the data to be migrated, such as a centralized database. This application embodiment does not limit this.

[0072] The aforementioned second database refers to the target database for data migration, i.e., the database to which the data to be migrated will be migrated. This application embodiment does not limit this.

[0073] The aforementioned static metadata information is used to describe the data structure and attributes of the data to be migrated, and may include field type, field length, field cardinality, etc., which are not limited in this embodiment of the application.

[0074] The aforementioned initial sharding key is a field selected based on static metadata information and used for sharding the data to be migrated. For example, a field with a cardinality greater than 10 million in the static metadata information can be used as the initial sharding key; however, this embodiment of the application does not limit this.

[0075] In this embodiment of the application, when the data to be migrated is migrated from the first database to the second database, static metadata information of the data to be migrated is obtained from the first database, including field type, field length, field cardinality, etc. Based on the static metadata information, N initial sharding keys are determined from the fields of the data to be migrated, where N is a positive integer (e.g., N is 1, 2, 3, etc.). This embodiment of the application does not limit this.

[0076] For example, based on static metadata information, user ID, order ID, and product ID are determined as initial sharding keys, resulting in three initial sharding keys (N is 3).

[0077] S102, obtain the dynamic load data of the data to be migrated based on the N initial sharding keys, and determine M candidate sharding keys from the N initial sharding keys based on the dynamic load data, where M is a positive integer.

[0078] The aforementioned dynamic load data refers to the frequency and distribution of operations on various fields in the data to be migrated during the actual operation of the first database, such as historical operation records and SQL statement execution status. This application embodiment does not limit this.

[0079] The aforementioned candidate sharding keys were selected from N initial sharding keys based on dynamic load data.

[0080] In this embodiment of the application, dynamic load data of the data to be migrated is obtained based on N initial sharding keys, and M candidate sharding keys are determined from the N initial sharding keys based on the dynamic load data, where M is a positive integer (e.g., M is 1, 2, 3, etc.).

[0081] For example, the initial sharding keys are User ID, Order ID, and Product ID. The historical operation records related to User ID, Order ID, and Product ID are retrieved. Based on the historical operation records, the fields that appear most frequently in User ID, Order ID, and Product ID are selected as candidate sharding keys.

[0082] At the same time, some unnecessary fields (such as enumeration fields) and fields containing abnormal records need to be removed from the initial sharding key to ensure the validity and rationality of the sharding key.

[0083] S103, for any candidate partitioning key, determine the balance evaluation index corresponding to the candidate partitioning key.

[0084] The aforementioned balance evaluation metrics are used to evaluate the degree of data distribution balance among the shards after data sharding of the candidate sharding key, such as entropy value, write skew rate, and conflict probability. This application embodiment does not limit these metrics.

[0085] In this embodiment of the application, for any candidate sharding key, a balance evaluation index corresponding to the candidate sharding key is determined. The degree of data dispersion corresponding to each candidate sharding key can be determined by simulating sharding.

[0086] S104. Based on the balance evaluation index corresponding to the M candidate sharding keys, determine the target sharding key from the M candidate sharding keys, and perform data migration on the data to be migrated according to the target sharding key.

[0087] The aforementioned target sharding key is a sharding key selected from M candidate sharding keys based on the balance evaluation index corresponding to the M candidate sharding keys. It is used to shard the data to be migrated and realize data migration.

[0088] In this embodiment, a target sharding key is determined from the M candidate sharding keys based on the balance evaluation index corresponding to the M candidate sharding keys, and the data to be migrated is migrated according to the target sharding key. This embodiment does not limit the scope of the application.

[0089] For example, if the candidate sharding key corresponds to the fields User ID and Order ID, the target sharding key is determined to be User ID based on the entropy value of the candidate sharding key. The data to be migrated is then sharded according to the User ID, thereby performing data migration.

[0090] Based on the above description of the technical solution provided in the embodiments of this application, when data to be migrated is moved from a first database to a second database, static metadata information of the data to be migrated is obtained from the first database. Based on the static metadata information, N initial sharding keys (N being a positive integer) are determined. Dynamic load data of the data to be migrated is obtained based on the N initial sharding keys. Based on the dynamic load data, M candidate sharding keys (M being a positive integer) are determined from the N initial sharding keys. For any candidate sharding key, a balance evaluation index corresponding to the candidate sharding key is determined. Based on the balance evaluation index corresponding to the M candidate sharding keys, a target sharding key is determined from the M candidate sharding keys. Data migration is then performed based on the target sharding key. This method of combining static metadata information and dynamic load data to determine candidate sharding keys, and determining the target sharding key based on the balance evaluation index of the candidate sharding keys, thereby performing data migration, can solve the technical problems of data skew and excessive load on some nodes caused by concentrated access to hot data during data migration.

[0091] like Figure 2 The diagram shown illustrates the implementation flow of another data migration method provided in this application, which may specifically include the following:

[0092] S201, when the data to be migrated is being migrated from the first database to the second database, obtain the static metadata information of the data to be migrated from the first database, and obtain the field cardinality of all fields in the data to be migrated from the static metadata information.

[0093] The data to be migrated refers to the set of data that needs to be migrated from the source database (first database) to the target database (second database).

[0094] The aforementioned first database refers to the source database for data migration, i.e., the database currently storing the data to be migrated, such as a centralized database. This application embodiment does not limit this.

[0095] The aforementioned second database refers to the target database for data migration, i.e., the database to which the data to be migrated will be migrated. This application embodiment does not limit this.

[0096] The aforementioned static metadata information is used to describe the data structure and attributes of the data to be migrated, and may include field type, field length, field cardinality, etc., which are not limited in this embodiment of the application.

[0097] The aforementioned field cardinality can be the number of unique values ​​for each field in the data to be migrated, but this application embodiment does not limit this.

[0098] In this embodiment of the application, when the data to be migrated is migrated from the first database to the second database, the static metadata information of the data to be migrated is obtained from the first database, and the field cardinality (number of unique values) of all fields in the data to be migrated is obtained from the static metadata information. During the acquisition process, some unnecessary fields (such as enumeration fields) need to be removed.

[0099] For example, obtain the number of unique values ​​for User ID, Order ID, Product ID, and Region ID in the data to be migrated from static metadata information, and remove the gender field.

[0100] S202, use the field whose cardinality in the data to be migrated is greater than the preset cardinality as the initial sharding key.

[0101] The aforementioned preset base number is the minimum value of the pre-set field base number, such as 10 million, 12 million, etc.

[0102] In this embodiment of the application, the field whose cardinality in the data to be migrated is greater than a preset cardinality is used as the initial sharding key.

[0103] For example, to obtain the cardinality of each field in the data to be migrated: the cardinality of user ID is 15 million, the cardinality of order ID is 20 million, the cardinality of product ID is 14 million, the cardinality of region ID is 12 million, and the preset cardinality is 13 million, then user ID, order ID, and product ID will be used as the initial sharding keys.

[0104] S203, retrieve historical operation records of the data to be migrated.

[0105] In this embodiment of the application, historical operation records of the data to be migrated are obtained. These historical operation records refer to the records of various operations (such as querying, updating, and deleting) performed on the data to be migrated during the actual operation of the first database. These records include information such as the time, type, and fields involved in the operation.

[0106] For example, retrieve the operation records corresponding to fields such as User ID, Order ID, Product ID, Gender, and Region ID in the data to be migrated, and use them as historical operation records.

[0107] S204, take the operation record containing N initial sharding keys in the historical operation record as dynamic load data, where N is a positive integer.

[0108] In this embodiment, the operation records containing N initial sharding keys in the historical operation records are used as dynamic load data, where N is a positive integer. The dynamic load data reflects the frequency and distribution of operations on various fields in the data to be migrated during the actual operation of the first database, including but not limited to operation records involving initial sharding keys in the historical operation records.

[0109] For example, if the historical operation records contain operation records corresponding to user ID, order ID, product ID, gender, and region ID, and the initial sharding key is user ID, order ID, and product ID, then the operation records containing user ID, order ID, and product ID in the historical operation records will be used as dynamic load data.

[0110] S205, for any initial sharding key, determine the operand rate of the initial sharding key based on dynamic load data.

[0111] In this embodiment, for any initial sharding key, the operation rate of the initial sharding key is determined based on dynamic load data. The operation rate is the frequency or proportion of operations performed on the initial sharding key in the dynamic load data, reflecting the activity level of the field corresponding to the initial sharding key in actual business operations.

[0112] Specifically, for each initial sharding key, the number of times the field corresponding to that initial sharding key is operated on (or involved in) in the dynamic load data is counted. The number of operations for each initial sharding key is divided by the total number of operations in the dynamic load data (or the number of operations within a specific time period) to obtain the operation ratio of that field, and the operation ratio is used as the operation rate.

[0113] In addition, before calculating the operation rate, the dynamic load data needs to be cleaned to remove abnormal or invalid operation records to ensure the accuracy of the operation rate.

[0114] For example, dynamic load data contains operation records corresponding to user ID, order ID, and product ID. There are 15 million operation records corresponding to user ID, 5 million corresponding to order ID, and 10 million corresponding to product ID, for a total of 30 million operation records in the dynamic load data. Therefore, the operation rate for user ID is 15 million / 30 million = 0.5, the operation rate for order ID is 5 million / 30 million ≈ 0.17, and the operation rate for product ID is 10 million / 30 million ≈ 0.33.

[0115] Furthermore, when calculating the operation rate, if a field corresponding to an initial sharding key contains a large number of abnormal or invalid operation records, these records may distort the calculated operation rate. Therefore, it is necessary to clean the dynamic load data and remove these abnormal records before calculation.

[0116] S206, the initial sharding key with an operation rate greater than a preset operation threshold is determined as a candidate sharding key.

[0117] In this embodiment, initial sharding keys with an operation rate greater than a preset operation threshold are identified as candidate sharding keys. The preset operation threshold is used to filter out initial sharding keys with high operation frequency in actual business operations as candidate sharding keys. The selection of the preset operation threshold should be based on business requirements and performance considerations.

[0118] For example, the preset operation threshold is 0.3. The operation rate of user ID is 0.5, which is greater than the preset operation threshold of 0.3. The operation rate of product ID is 0.33, which is greater than the preset operation threshold of 0.3. However, the operation rate of order ID is 0.25, which is less than the preset operation threshold of 0.3. Therefore, user ID and product ID are selected as candidate sharding keys, while order ID is not selected as a candidate sharding key.

[0119] S207, for any candidate partitioning key, determine the balance evaluation index corresponding to the candidate partitioning key.

[0120] In this embodiment of the application, this step is similar to step S103 above, and will not be described in detail here.

[0121] The balance evaluation metric includes at least one of the discrete entropy value and the write skew rate. The aforementioned balance evaluation metric for determining the candidate shard key may include determining the discrete entropy value corresponding to the candidate shard key, and / or determining the write skew rate corresponding to the candidate shard key.

[0122] For details on how to determine the discrete entropy value corresponding to the candidate fragmentation key, please refer to... Figure 3 The method shown. (As illustrated) Figure 3 The diagram shown is a schematic representation of an implementation flow of a method for determining discrete entropy values ​​according to an embodiment of this application. Specifically, it may include the following steps:

[0123] S301, retrieve the operation record corresponding to the candidate sharding key from the dynamic load data.

[0124] In this embodiment, operation records corresponding to candidate sharding keys are obtained from dynamic load data. These operation records contain SQL statements (such as query, insert, update, and delete statements) involving fields corresponding to the candidate sharding keys.

[0125] For example, if the candidate sharding key is the user ID, the operation record corresponding to the user ID can be obtained from the dynamic load data.

[0126] S302, retrieve multiple values ​​corresponding to the candidate sharding key from the operation record, and determine the probability of occurrence of each value corresponding to the candidate sharding key.

[0127] In this embodiment, multiple values ​​corresponding to candidate sharding keys are obtained from the operation records, and the probability of occurrence of each value corresponding to the candidate sharding key is determined. Specifically, multiple values ​​corresponding to candidate sharding keys are obtained from the operation records, the occurrence frequency of each value corresponding to the candidate sharding key is counted, and the probability of occurrence of each value corresponding to the candidate sharding key is determined based on the occurrence frequency of each value corresponding to the candidate sharding key and the total number of operation records. Here, the occurrence frequency of each value refers to the number of times this value (e.g., user 1) appears in the operation records (the number of SQL statements involved), the total number of operation records is the sum of the occurrence frequencies of all different values ​​(e.g., all different user IDs), and the occurrence probability can be the frequency of each value appearing in all operation records.

[0128] Specifically, the occurrence count of each value corresponding to the candidate sharding key and the total number of operation records can be input into the occurrence probability formula to calculate the occurrence probability of each value. The occurrence probability formula is:

[0129]

[0130] Where, p i Let q be the probability of the i-th value of the candidate sharding key. i Let represent the number of occurrences of the i-th value, and Q represent the total number of operation records.

[0131] For example, if the field corresponding to the candidate sharding key is user ID, and the total number of operation records is 6000, different user ID values ​​are obtained from the operation records to get user 1, user 2, and user 3. Based on the number of times each value appears in the operation records and the total number of operation records, the probability of each value is obtained: the probability of user 1 is 1000 times / 6000 times = 0.1667 (or 16.67%), the probability of user 2 is 2000 times / 6000 times = 0.3333 (or 33.33%), and the probability of user 3 is 3000 times / 6000 times = 0.5 (or 50%).

[0132] S303, for any value corresponding to a candidate sharding key, calculate the distribution rate of the value based on the probability of the value's occurrence.

[0133] In this embodiment of the application, for any value corresponding to a candidate sharding key, the distribution rate of the value is calculated based on the probability of the value's occurrence. The distribution rate is used to quantify the degree of dispersion of the values.

[0134] Specifically, based on the probability of a value occurring, the distribution of that value is calculated. This can be done by inputting the probability of a value occurring into the distribution formula, which is:

[0135] G = p i *log2p i ;

[0136] Where G is the distribution of the value, p i The probability of the value being .

[0137] For example, if the probability of user 1 occurring is 0.1667, input the probability of user 1 occurring into the distribution formula to calculate the distribution of the value.

[0138] S304, sum the distribution rates of each value corresponding to the candidate partitioning key to obtain the initial entropy value corresponding to the candidate partitioning key.

[0139] In this embodiment of the application, the distribution rate of each value corresponding to the candidate partitioning key is summed to obtain the initial entropy value corresponding to the candidate partitioning key.

[0140] For example, if the user IDs correspond to user 1, user 2, and user 3, the distribution rates of user 1, user 2, and user 3 are summed to obtain the initial entropy value corresponding to the candidate sharding key.

[0141] S305, take the negative value of the initial entropy value corresponding to the candidate partition key as the discrete entropy value corresponding to the candidate partition key.

[0142] In this embodiment of the application, the negative value of the initial entropy value corresponding to the candidate partitioning key is used as the discrete entropy value corresponding to the candidate partitioning key.

[0143] For example, the initial entropy value corresponding to the candidate partition key is 0.05, and its negative value is -0.05. -0.05 is used as the discrete entropy value corresponding to the candidate partition key.

[0144] In another embodiment of this application, the probability of different values ​​of the candidate partitioning key can be input into the discrete entropy formula to calculate the discrete entropy value corresponding to the candidate partitioning key. The discrete entropy formula is as follows:

[0145]

[0146] Where H is the discrete entropy value corresponding to the candidate partition key, k is the number of different values ​​of the candidate partition key, and p i Let be the probability of the i-th value of the candidate sharding key.

[0147] For details on how to determine the write skew rate corresponding to the candidate sharding key, please refer to... Figure 4 The method shown. (As illustrated) Figure 4 The diagram shown illustrates the implementation flow of a method for determining write skew rate according to an embodiment of this application, which may specifically include the following steps:

[0148] S401, simulate sharding the data to be migrated according to the preset number of shards and the candidate sharding key, and obtain the simulation operation record.

[0149] The aforementioned preset partitions are independent data storage units (such as database tables, file partitions, etc.).

[0150] The number of preset shards mentioned above is set in advance based on the data to be migrated.

[0151] The aforementioned simulated sharding refers to simulating the process of data to be migrated being allocated to different shards based on candidate sharding keys and sharding algorithms (such as hash sharding, range sharding, and list sharding) without performing actual data migration.

[0152] The aforementioned simulation operation record refers to all write operation information recorded during the simulation sharding process, including which shard each piece of data was written to and the amount of data written to the corresponding shard.

[0153] In this embodiment of the application, the data to be migrated is simulated by splitting it according to the preset number of shards and the candidate shard keys to obtain a simulated operation record.

[0154] For example, if the candidate sharding key is the order ID, the simulated sharding results in the following simulated operation records: Order 1 is distributed to shard 1 (80 records); Order 2 is distributed to shard 2 (40 records); Order 3 is distributed to shard 3 (50 records); and Order 4 is distributed to shard 2 (30 records).

[0155] S402, obtain the write operation data volume of each fragment in the preset fragment from the simulation operation record.

[0156] In this embodiment of the application, the write operation data volume of each fragment in the preset fragment is obtained from the simulation operation record. Here, the write operation data volume refers to the number of data entries written to each fragment.

[0157] For example, based on the simulated operation records, order 1 is distributed to shard 1 (80 records); order 2 is distributed to shard 2 (40 records); order 3 is distributed to shard 3 (50 records); and order 4 is distributed to shard 2 (30 records). Thus, the write operation data volume of shard 1 is 80 records; the write operation data volume of shard 2 is 70 records; and the write operation data volume of shard 3 is 50 records.

[0158] S403, based on the write operation data volume of each segment in the preset segment, determine the average operation data volume and the maximum write operation data volume of the preset segment.

[0159] The above average operation data volume is the average of the preset shard write operation data volume, used to measure the balance of the overall write load.

[0160] The maximum write operation data volume mentioned above refers to the data write volume of the shard with the largest write operation data volume among the preset shards. This shard can be used as a hotspot shard.

[0161] In this embodiment of the application, the average and maximum write operation data volumes of the preset partitions are determined based on the write operation data volumes of each partition in the preset partitions.

[0162] For example, if the write operation data volume of each shard in the preset shard is as follows: 80 write operation data volume for shard 1; 70 write operation data volume for shard 2; and 50 write operation data volume for shard 3, then the average operation data volume can be obtained as (80+70+50) / 3≈66.67, and the maximum write operation data volume is the write operation data volume of shard 1 (80 data volume).

[0163] S404 determines the write skew rate corresponding to the candidate shard key by using the average operation data volume and the maximum write operation data volume.

[0164] In this embodiment, the write skew rate corresponding to the candidate shard key is determined by the average operation data volume and the maximum write operation data volume. The write skew rate is an indicator used to measure the degree of balance in the distribution of write operations of the data to be migrated across shards.

[0165] Specifically, the average amount of data operated and the maximum amount of data written can be input into the first write skewness formula to calculate the write skewness corresponding to the candidate shard key. The first write skewness formula is as follows:

[0166]

[0167] Where R is the write skew rate corresponding to the candidate shard key, max is the maximum write operation data volume, and μ is the average operation data volume.

[0168] In another embodiment of this application, the write operation data volume of each fragment in the preset fragment, the number of preset fragments, and the average operation data volume can be input into the second write skew rate formula to calculate the write skew rate corresponding to the candidate fragment key. The second write skew rate formula is as follows:

[0169]

[0170] Where R is the write skew rate corresponding to the candidate shard key, and S j The number of write operations is the j-th shard in the preset shards (j = 1, 2, ..., n), where n is the number of preset shards and μ is the average amount of data operated.

[0171] S208: Based on the balance evaluation index corresponding to the M candidate sharding keys, determine the target sharding key from the M candidate sharding keys, and perform data migration on the data to be migrated according to the target sharding key.

[0172] In this embodiment of the application, this step is similar to step S104 above, and will not be described in detail here.

[0173] Specifically, based on the balance evaluation index corresponding to the M candidate sharding keys, the target sharding key is determined from the M candidate sharding keys. The M candidate sharding keys can be sorted according to the balance evaluation index of the M candidate sharding keys to obtain a candidate sharding key sequence, and the target sharding key is determined from the candidate sharding key sequence.

[0174] In another embodiment of this application, the M candidate partition keys can be sorted based on the discrete entropy values ​​corresponding to the M candidate partition keys to obtain a candidate partition key sequence, and the target partition key can be determined from the candidate partition key sequence.

[0175] In another embodiment of this application, the M candidate shard keys can be sorted based on the write skew rates corresponding to the M candidate shard keys to obtain a candidate shard key sequence, and the target shard key can be determined from the candidate shard key sequence.

[0176] In another embodiment of this application, the M candidate shard keys can be sorted based on the discrete entropy values ​​and write skew rates corresponding to the M candidate shard keys to obtain a candidate shard key sequence, and the target shard key can be determined from the candidate shard key sequence.

[0177] In addition, after migrating the data to be migrated based on the target sharding key, it is necessary to monitor the migrated data in real time and adjust the sharding strategy when data skew occurs.

[0178] For details on how to adjust the sharding strategy, please refer to the following: Figure 5 The method shown. (As illustrated) Figure 5 The diagram shown illustrates the implementation flow of a method for adjusting a fragmentation strategy according to an embodiment of this application, which may specifically include the following steps:

[0179] S501: Obtain business operation data of the migrated data in the second database.

[0180] The aforementioned migrated data refers to all data sets successfully migrated from the first database (centralized database) to the second database (distributed database), including tables, fields, indexes, and relationships.

[0181] The aforementioned business operation data refers to the actual business operation records performed by users or applications on the migrated data in the second database after the migration of the data to be migrated is completed. It is dynamic data that reflects data access patterns, operation frequency, and load distribution.

[0182] In this embodiment, business operation data of the migrated data in the second database is obtained. After the data to be migrated is migrated from the centralized database (first database) to the distributed database (second database), operation records of the migrated data in the second database during actual business operations are continuously collected. This includes all business behaviors on the migrated data, including but not limited to user-initiated queries (SELECT), data updates (UPDATE), inserts (INSERT), and deletes (DELETE), as well as details such as the timestamp of the operation, the tables / fields involved, and the execution time. This embodiment does not limit these details.

[0183] S502 determines the number of cross-node operations based on business operation data.

[0184] In this embodiment of the application, the number of times all cross-node operations are screened and counted from the business operation data to obtain the cross-node count. Cross-node operation refers to an operation that requires accessing two or more physical nodes (shards) in the distributed database to complete a single business operation (e.g., performing a JOIN query on a table stored in different shards, updating the same record distributed across multiple nodes, etc.). The cross-node count refers to the total number of operations that require crossing multiple physical nodes (shards) in the distributed database to complete a business operation. It is used to directly reflect the "dispersion" impact of the sharding strategy on business operations. The higher the count, the more unreasonable the sharding strategy may be (e.g., too many cross-shard associations).

[0185] S503 determines the cross-shard operation rate based on the number of cross-node operations and business operation data.

[0186] In this embodiment, the cross-shard operation rate is determined based on the number of cross-node operations and business operation data. The cross-shard operation rate reflects the proportion of cross-node operations in all business operations and is one of the core indicators for evaluating the rationality of the sharding strategy. A higher proportion indicates a greater cost for distributed transactions and cross-node communication caused by the sharding strategy, potentially leading to increased latency and performance degradation.

[0187] Specifically, the total amount of business operation data can be counted, and the number of cross-node operations and the total amount of business operation data can be input into the cross-shard operation rate formula to calculate the cross-shard operation rate:

[0188]

[0189] Where CrossShardRatio is the cross-shard operation rate, C is the number of cross-node operations, and T is the total amount of business operation data.

[0190] S504: If the cross-shard operation rate exceeds the preset cross-shard threshold, obtain the static metadata information of the migrated data in the second database.

[0191] In this embodiment, when the cross-shard operation rate exceeds a preset cross-shard threshold, static metadata information of the migrated data in the second database is obtained. The preset cross-shard threshold is a pre-defined critical value for the cross-shard operation rate (e.g., 15%, 20%, etc.), determined by business performance requirements. When the cross-shard operation rate exceeds the preset cross-shard threshold, it indicates that the current sharding strategy needs optimization (selecting a new target shard key).

[0192] S505, perform the step of determining N initial sharding keys based on the static metadata information of the data migrated in the second database.

[0193] In this embodiment, the step of determining N initial sharding keys is performed based on the static metadata information of the migrated data in the second database. Fields with a cardinality greater than a preset cardinality (e.g., 10 million) can be selected from all fields of the migrated data as new initial sharding keys (N, where N is a positive integer). Subsequently, candidate sharding keys are further filtered in conjunction with dynamic load data to finally determine the new target sharding key, i.e., the new sharding strategy.

[0194] Corresponding to the above method embodiments, this application also provides a data migration apparatus, such as... Figure 6 As shown, the device may include an information acquisition module 601, an initial sharding key determination module 602, a candidate sharding key determination module 603, a balance evaluation module 604, a target sharding key determination module 605, and a data migration module 606.

[0195] The information acquisition module 601 is used to acquire static metadata information of the data to be migrated from the first database when the data to be migrated is migrated from the first database to the second database.

[0196] The initial sharding key determination module 602 is used to determine N initial sharding keys based on static metadata information, where N is a positive integer;

[0197] The candidate sharding key determination module 603 is used to obtain dynamic load data of the data to be migrated based on N initial sharding keys, and determine M candidate sharding keys from the N initial sharding keys based on the dynamic load data, where M is a positive integer;

[0198] The balance evaluation module 604 is used to determine the balance evaluation index corresponding to any candidate sharding key.

[0199] The target partitioning key determination module 605 is used to determine the target partitioning key from the M candidate partitioning keys based on the balance evaluation index corresponding to the M candidate partitioning keys;

[0200] The data migration module 606 is used to migrate the data to be migrated based on the target shard key.

[0201] This application also provides an electronic device, such as... Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0202] Memory 703 is used to store computer programs;

[0203] In one embodiment of this application, when the processor 701 executes a program stored in the memory 703, it performs the following steps:

[0204] When migrating data from the first database to the second database, the static metadata information of the data to be migrated is obtained from the first database. Based on the static metadata information, N initial sharding keys (N being a positive integer) are determined. Dynamic load data of the data to be migrated is obtained based on the N initial sharding keys. Based on the dynamic load data, M candidate sharding keys (M being a positive integer) are determined from the N initial sharding keys. For any candidate sharding key, a balance evaluation index corresponding to the candidate sharding key is determined. Based on the balance evaluation index corresponding to the M candidate sharding keys, a target sharding key is determined from the M candidate sharding keys. Data migration is then performed on the data to be migrated according to the target sharding key.

[0205] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0206] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0207] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0208] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0209] In another embodiment provided in this application, a storage medium is also provided, which stores instructions that, when run on a computer, cause the computer to execute any of the data migration methods described in the above embodiments.

[0210] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the data migration methods described in the above embodiments.

[0211] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a storage medium or transmitted from one storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0212] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0213] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0214] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A data migration method, characterized in that, The method includes: When data to be migrated is migrated from the first database to the second database, static metadata information of the data to be migrated is obtained from the first database, and N initial sharding keys are determined based on the static metadata information, where N is a positive integer; The dynamic load data of the data to be migrated is obtained based on the N initial sharding keys, and M candidate sharding keys are determined from the N initial sharding keys based on the dynamic load data, where M is a positive integer; For any of the candidate partitioning keys, determine the balance evaluation index corresponding to the candidate partitioning key; Based on the balance evaluation index corresponding to the M candidate sharding keys, a target sharding key is determined from the M candidate sharding keys, and the data to be migrated is migrated according to the target sharding key.

2. The method according to claim 1, characterized in that, The process of determining N initial sharding keys based on the static metadata information includes: Obtain the field cardinality of all fields in the data to be migrated from the static metadata information; The field whose cardinality in the data to be migrated is greater than a preset cardinality is used as the initial sharding key.

3. The method according to claim 1, characterized in that, The step of obtaining dynamic load data of the data to be migrated based on the N initial sharding keys includes: Obtain the historical operation records of the data to be migrated; The operation records containing the N initial sharding keys in the historical operation records are used as the dynamic load data.

4. The method according to claim 1, characterized in that, The step of determining M candidate sharding keys from the N initial sharding keys based on the dynamic load data includes: For any of the initial sharding keys, the operand rate of the initial sharding key is determined based on the dynamic load data; The initial sharding key with an operation rate greater than a preset operation threshold is determined as the candidate sharding key.

5. The method according to claim 1, characterized in that, The step of determining the balance evaluation index corresponding to the candidate partitioning key includes: Determine the discrete entropy value corresponding to the candidate fragmentation key; And / or, Determine the write skew rate corresponding to the candidate sharding key.

6. The method according to claim 5, characterized in that, Determining the discrete entropy value corresponding to the candidate fragmentation key includes: Obtain the operation record corresponding to the candidate sharding key from the dynamic load data; Obtain multiple values ​​corresponding to the candidate sharding key from the operation record, and determine the probability of occurrence of each value corresponding to the candidate sharding key; For any value corresponding to the candidate sharding key, calculate the distribution rate of the value based on the probability of the value occurring; The distribution rates of each value corresponding to the candidate fragmentation key are summed to obtain the initial entropy value corresponding to the candidate fragmentation key; The negative value of the initial entropy value corresponding to the candidate partition key is used as the discrete entropy value corresponding to the candidate partition key.

7. The method according to claim 6, characterized in that, The calculation of the distribution rate of the value based on the probability of its occurrence includes: The probability of the value occurring is input into the distribution formula to calculate the distribution of the value. The distribution formula is as follows: G=p i *log2p i ; Where G is the distribution rate of the value, p i The probability of the value occurring is given.

8. The method according to claim 5, characterized in that, Determining the write skew rate corresponding to the candidate shard key includes: The data to be migrated is simulated by splitting it according to the preset number of shards and the candidate sharding key, and a simulation operation record is obtained. Obtain the write operation data volume of each fragment in the preset fragment from the simulation operation record; Based on the amount of write operation data in each of the preset shards, determine the average amount of operation data and the maximum amount of write operation data in the preset shards; The write skew rate corresponding to the candidate shard key is determined by the average operation data volume and the maximum write operation data volume.

9. The method according to claim 1, characterized in that, After migrating the data to be migrated based on the target shard key, the process further includes: Retrieve business operation data of the migrated data from the second database; The number of cross-node operations is determined based on the aforementioned business operation data; The cross-node operation rate is determined based on the number of cross-node operations and the business operation data. If the cross-shard operation rate exceeds a preset cross-shard threshold, obtain the static metadata information of the migrated data in the second database; The step of determining N initial sharding keys is performed based on the static metadata information of the data that has been migrated from the second database.

10. A data migration device, characterized in that, The device includes: The information acquisition module is used to acquire static metadata information of the data to be migrated from the first database when the data to be migrated is migrated from the first database to the second database. An initial sharding key determination module is used to determine N initial sharding keys based on the static metadata information, where N is a positive integer. The candidate sharding key determination module is used to obtain dynamic load data of the data to be migrated based on the N initial sharding keys, and determine M candidate sharding keys from the N initial sharding keys based on the dynamic load data, where M is a positive integer; The balance evaluation module is used to determine the balance evaluation index corresponding to any of the candidate sharding keys. The target sharding key determination module is used to determine the target sharding key from the M candidate sharding keys based on the balance evaluation index corresponding to the M candidate sharding keys; The data migration module is used to migrate the data to be migrated based on the target shard key.

Images