Data table creation method and apparatus

Abstract

Embodiments of the present application provide a data table creation method and device, the method comprising: obtaining a creation instruction of a first partition table, the creation instruction comprising annotation information, the annotation information being used to indicate creation of a partition table of a skip consistent hashing type; creating the first partition table according to the creation instruction, and generating disk-falling information, the disk-falling information comprising the annotation information; and generating a table definition cache corresponding to the first partition table according to the disk-falling information, the table definition cache comprising: the type of the first partition table being the skip consistent hashing type. Based on the partition table of the skip consistent hashing type, the efficiency of migrating data is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for creating data tables. Background Technology

[0002] A database server can include partitioned tables, which can contain multiple partitions, and each partition can store multiple data entries.

[0003] In related technologies, partitioned tables are typically modulo hash type partitioned tables. When the number of partitions in a partitioned table changes, all data in the partitioned table needs to be migrated according to the number of partitions. However, in this process, the large amount of data to be migrated leads to low data migration efficiency. Summary of the Invention

[0004] This application provides a data table creation method and apparatus in several aspects to improve the efficiency of data migration.

[0005] In a first aspect, embodiments of this application provide a method for creating a data table, including:

[0006] Obtain the creation instruction for the first partition table, the creation instruction including comment information, the comment information being used to indicate the creation of a partition table of the skip consistent hash type;

[0007] The first partition table is created according to the creation instruction, and disk write information is generated, including the comment information;

[0008] Based on the disk write information, a table definition cache corresponding to the first partition table is generated. The table definition cache includes: the first partition table is of type jump consistent hash.

[0009] In one possible implementation, generating a table definition cache corresponding to the first partition table based on the disk write information includes:

[0010] Based on the disk write information, the type of the first partition table is determined to be the skip consistent hash type;

[0011] Generate the table definition cache based on the type of the first partition table.

[0012] In one possible implementation, the method further includes:

[0013] Obtain the operation instructions for the first partition table;

[0014] According to the operation instruction, the type of the first partition table is obtained from the table definition cache corresponding to the first partition table;

[0015] The first partition table is processed according to its type and the operation instructions.

[0016] In one possible implementation, the operation instruction is a partition update instruction, which is either a partition addition instruction or a partition deletion instruction.

[0017] Based on the type of the first partition table and the operation instructions, the first partition table is processed through operations, including:

[0018] The target number of partitions for the first partition table is determined based on the type of the first partition table and the partition update instruction.

[0019] Based on the partition update instruction and the target number of partitions, the data to be migrated and the target partition of the data to be migrated are determined in the first partition table, wherein the number of data to be migrated is less than the number of data in the first partition table;

[0020] The data to be migrated is migrated to the corresponding target partition.

[0021] In one possible implementation, determining the data to be migrated and the target partitions of the data to be migrated in the first partition table according to the partition update instruction and the target number of partitions includes:

[0022] If the partition update instruction is a partition addition instruction, then based on the data in the first partition table and the target number of partitions, the data to be migrated and the target partition of the data to be migrated are determined in the first partition table;

[0023] If the partition update instruction is a partition deletion instruction, then the partition to be deleted is determined in the first partition table according to the partition deletion instruction, the data in the partition to be deleted is determined as the data to be migrated, and the target partition of the data to be migrated is determined according to the data to be migrated and the target number of partitions.

[0024] In one possible implementation, determining the data to be migrated and the target partitions of the data to be migrated in the first partition table based on the data in the first partition table and the number of target partitions includes:

[0025] Based on the data in the first partition table and the target number of partitions, determine the migration partition for each piece of data in the first partition table, wherein the migration partition is either the partition where the data is currently located or a newly added partition;

[0026] Based on the migration partition of each data in the first partition table, determine the data to be migrated and the target partition of the data to be migrated in the first partition table.

[0027] In one possible implementation, determining the data to be migrated and the target partition of the data to be migrated in the first partition table based on the migration partition of each piece of data in the first partition table includes:

[0028] For any data in the first partition table, if the migration partition of the data is the newly added partition, then the data is determined as the data to be migrated, and the migration partition of the data is determined as the target partition of the data to be migrated.

[0029] In one possible implementation, for any data in the first partition table; determining the migration partition for the data based on the data and the target number of partitions includes:

[0030] Based on the data and the original seed, generate pseudo-random numbers corresponding to the data;

[0031] The ratio of the number of newly added partitions to the target number of partitions is determined as the comparison value;

[0032] If the pseudo-random number is greater than or equal to the comparison value, then the migration partition of the data is determined to be the partition where the data is currently located;

[0033] If the pseudo-random number is less than the comparison value, then the migration partition of the data is determined to be the newly added partition.

[0034] In one possible implementation, determining the target partitions for the data to be migrated, based on the data to be migrated and the number of target partitions, includes:

[0035] The data to be migrated and the target number of partitions are processed using a skip consistent hashing algorithm to obtain partition identifiers;

[0036] The partition corresponding to the partition identifier is determined as the target partition for the data to be migrated.

[0037] In one possible implementation, migrating the data to be migrated to the corresponding target partition includes:

[0038] During the first time period, the data to be migrated is processed in parallel on the first partition table by at least two threads to obtain the updated first partition table.

[0039] The data manipulation language (DML) operations performed on the first partition table during the first time period are determined, and the DML operations include insert operations, update operations, and delete operations.

[0040] The DML operation is synchronized in the updated first partition table.

[0041] In one possible implementation, the DML operation is an insert operation; synchronizing the DML operation in the updated first partition table includes:

[0042] Determine the first data corresponding to the insertion operation, and the original insertion partition of the first data in the first partition table;

[0043] Based on the target number of partitions, determine the target partition in the first partition table where the first data will be inserted;

[0044] If the original insertion partition is different from the target insertion partition, insert the first data into the target insertion partition, delete the first data in the original insertion partition, and update the operation log corresponding to the insertion operation.

[0045] In one possible implementation, the DML operation is a delete operation; synchronizing the DML operation in the updated first partition table includes:

[0046] Determine the second data corresponding to the deletion operation, and the original deletion partition of the second data in the first partition table;

[0047] Based on the target number of partitions, determine the target partition for deletion of the second data in the first partition table;

[0048] If the original partition to be deleted is different from the target partition to be deleted, the operation log corresponding to the deletion operation is updated based on the original partition to be deleted and the target partition to be deleted.

[0049] In one possible implementation, the DML operation is an update operation; synchronizing the DML operation in the updated first partition table includes:

[0050] Determine the third data corresponding to the update operation, and the original update partition of the third data in the first partition table;

[0051] Based on the target number of partitions, determine the target update partition of the third data in the first partition table;

[0052] If the original update partition is different from the target update partition, the updated third data is inserted into the target update partition, the updated first data is deleted from the original update partition, and the operation log corresponding to the update operation is updated.

[0053] In one possible implementation, the ratio of the amount of data to be migrated to the amount of current data in the first partition table is N / (M+N);

[0054] Wherein, M is the current number of partitions in the first partition table, and N is the number of partitions added or deleted from the first partition table.

[0055] Secondly, embodiments of this application provide a data table creation apparatus, comprising: a first acquisition module, a creation module, and a generation module, wherein,

[0056] The first acquisition module is used to acquire the creation instruction of the first partition table, the creation instruction including comment information, the comment information being used to indicate the creation of a partition table of the skip consistent hash type;

[0057] The creation module is used to create the first partition table according to the creation instruction and to generate disk write information, wherein the disk write information includes the annotation information;

[0058] The generation module is used to generate a table definition cache corresponding to the first partition table based on the disk information. The table definition cache includes: the first partition table is of type jump consistent hash.

[0059] In one possible implementation, the generation module is specifically used for:

[0060] Based on the disk write information, the type of the first partition table is determined to be the skip consistent hash type;

[0061] Generate the table definition cache based on the type of the first partition table.

[0062] In one possible implementation, the device further includes a second acquisition module and an operation module, wherein,

[0063] The first acquisition module is used to acquire the operation instructions of the first partition table;

[0064] The second acquisition module is used to acquire the type of the first partition table from the table definition cache corresponding to the first partition table according to the operation instruction;

[0065] The operation module is used to perform operation processing on the first partition table according to the type of the first partition table and the operation instruction.

[0066] In one possible implementation, the operation instruction is a partition update instruction, which is either a partition addition instruction or a partition deletion instruction; the operation module is specifically used for:

[0067] The target number of partitions for the first partition table is determined based on the type of the first partition table and the partition update instruction.

[0068] Based on the partition update instruction and the target number of partitions, the data to be migrated and the target partition of the data to be migrated are determined in the first partition table, wherein the number of data to be migrated is less than the number of data in the first partition table;

[0069] The data to be migrated is migrated to the corresponding target partition.

[0070] In one possible implementation, the operation module is specifically used for:

[0071] If the partition update instruction is a partition addition instruction, then based on the data in the first partition table and the target number of partitions, the data to be migrated and the target partition of the data to be migrated are determined in the first partition table;

[0072] If the partition update instruction is a partition deletion instruction, then the partition to be deleted is determined in the first partition table according to the partition deletion instruction, the data in the partition to be deleted is determined as the data to be migrated, and the target partition of the data to be migrated is determined according to the data to be migrated and the target number of partitions.

[0073] In one possible implementation, the operation module is specifically used for:

[0074] Based on the data in the first partition table and the target number of partitions, determine the migration partition for each piece of data in the first partition table, wherein the migration partition is either the partition where the data is currently located or a newly added partition;

[0075] Based on the migration partition of each data in the first partition table, determine the data to be migrated and the target partition of the data to be migrated in the first partition table.

[0076] In one possible implementation, the operation module is specifically used for:

[0077] For any data in the first partition table, if the migration partition of the data is the newly added partition, then the data is determined as the data to be migrated, and the migration partition of the data is determined as the target partition of the data to be migrated.

[0078] In one possible implementation, for any data in the first partition table; the operation module is specifically used for:

[0079] Based on the data and the original seed, generate pseudo-random numbers corresponding to the data;

[0080] The ratio of the number of newly added partitions to the target number of partitions is determined as the comparison value;

[0081] If the pseudo-random number is greater than or equal to the comparison value, then the migration partition of the data is determined to be the partition where the data is currently located;

[0082] If the pseudo-random number is less than the comparison value, then the migration partition of the data is determined to be the newly added partition.

[0083] In one possible implementation, the operation module is specifically used for:

[0084] The data to be migrated and the target number of partitions are processed using a skip consistent hashing algorithm to obtain partition identifiers;

[0085] The partition corresponding to the partition identifier is determined as the target partition for the data to be migrated.

[0086] In one possible implementation, the operation module is specifically used for:

[0087] During the first time period, the data to be migrated is processed in parallel on the first partition table by at least two threads to obtain the updated first partition table.

[0088] The data manipulation language (DML) operations performed on the first partition table during the first time period are determined, and the DML operations include insert operations, update operations, and delete operations.

[0089] The DML operation is synchronized in the updated first partition table.

[0090] In one possible implementation, the DML operation is an insert operation; the operation module is specifically used for:

[0091] Determine the first data corresponding to the insertion operation, and the original insertion partition of the first data in the first partition table;

[0092] Based on the target number of partitions, determine the target partition in the first partition table where the first data will be inserted;

[0093] If the original insertion partition is different from the target insertion partition, insert the first data into the target insertion partition, delete the first data in the original insertion partition, and update the operation log corresponding to the insertion operation.

[0094] In one possible implementation, the DML operation is a delete operation; the operation module is specifically used for:

[0095] Determine the second data corresponding to the deletion operation, and the original deletion partition of the second data in the first partition table;

[0096] Based on the target number of partitions, determine the target partition for deletion of the second data in the first partition table;

[0097] If the original partition to be deleted is different from the target partition to be deleted, the operation log corresponding to the deletion operation is updated based on the original partition to be deleted and the target partition to be deleted.

[0098] In one possible implementation, the DML operation is an update operation; the operation module is specifically used for:

[0099] Determine the third data corresponding to the update operation, and the original update partition of the third data in the first partition table;

[0100] Based on the target number of partitions, determine the target update partition of the third data in the first partition table;

[0101] If the original update partition is different from the target update partition, the updated third data is inserted into the target update partition, the updated first data is deleted from the original update partition, and the operation log corresponding to the update operation is updated.

[0102] In one possible implementation, the ratio of the amount of data to be migrated to the amount of current data in the first partition table is N / (M+N);

[0103] Wherein, M is the current number of partitions in the first partition table, and N is the number of partitions added or deleted from the first partition table.

[0104] Thirdly, embodiments of this application provide a database server, including: a memory and a processor;

[0105] The memory stores computer-executed instructions;

[0106] The processor executes computer execution instructions stored in the memory, causing the processor to perform the data table creation method according to any one of the first aspects.

[0107] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data table creation method described in any of the first aspects.

[0108] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the data table creation method shown in any of the first aspects.

[0109] This application provides a data table creation method and device. A database server can obtain a creation instruction for a first partition table, and create the first partition table and generate disk write information according to the creation instruction. The database server can generate a table definition cache corresponding to the first partition table based on the disk write information. The table definition cache includes: the first partition table is of type jump-consistent hash. Because the first partition table is of type jump-consistent hash, operations on the first partition table can significantly reduce the amount of data to be migrated; and the data to be migrated can be processed in parallel by at least two threads, improving the migration speed; furthermore, the number of operations when adding or deleting partitions in the first partition table is reduced, reducing the generated log files, thus comprehensively improving the efficiency of data migration. Attached Figure Description

[0110] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0111] Figure 1 A schematic diagram illustrating an application scenario provided for an exemplary embodiment of this application;

[0112] Figure 2A A schematic diagram illustrating the addition of partitions to the first partition table in related technologies;

[0113] Figure 2B This is a schematic diagram illustrating the deletion of a partition in the first partition table in related technologies;

[0114] Figure 3 A flowchart illustrating a data table creation method provided for an exemplary embodiment of this application;

[0115] Figure 4 A schematic diagram illustrating the process of creating a data table as an exemplary embodiment of this application;

[0116] Figure 5 A flowchart illustrating a data processing method provided for an exemplary embodiment of this application;

[0117] Figure 6 A schematic diagram of parallel migration processing and DML operations during a first time period provided as an exemplary embodiment of this application;

[0118] Figure 7 A flowchart illustrating another data processing method provided for an exemplary embodiment of this application;

[0119] Figure 8 A schematic diagram illustrating the determination of migration partitions provided for an exemplary embodiment of this application;

[0120] Figure 9AAn illustration of the amount of data to be migrated provided for an exemplary embodiment of this application. Figure 1 ;

[0121] Figure 9B This is a diagram illustrating the amount of data to be migrated in related technologies. Figure 1 ;

[0122] Figure 10 An illustration of a parallel migration provided as an exemplary embodiment of this application Figure 1 ;

[0123] Figure 11 An illustration of updating a first partition table provided for an exemplary embodiment of this application Figure 1 ;

[0124] Figure 12 A flowchart illustrating yet another data processing method provided for an exemplary embodiment of this application;

[0125] Figure 13A Schematic diagram 2 showing the amount of data to be migrated as provided in an exemplary embodiment of this application;

[0126] Figure 13B Schematic diagram 2 showing the amount of data to be migrated in related technologies;

[0127] Figure 14 A schematic diagram of a parallel migration provided for an exemplary embodiment of this application;

[0128] Figure 15 A schematic diagram 2 illustrating an update of a first partition table, provided as an exemplary embodiment of this application;

[0129] Figure 16 A schematic diagram of a data table creation apparatus provided for an exemplary embodiment of this application;

[0130] Figure 17 A schematic diagram of another data table creation apparatus provided for an exemplary embodiment of this application;

[0131] Figure 18 This is a schematic diagram of the structure of a database server provided as an exemplary embodiment of this application. Detailed Implementation

[0132] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0133] Figure 1 This is a schematic diagram illustrating an application scenario provided for an exemplary embodiment of this application. Please refer to [link / reference]. Figure 1 The database server may include a first partition table, which can be a skip consistent hash type partition table. The first partition table may include multiple partitions, namely partition 1, partition 2, partition 3, ..., partition M.

[0134] The database server can obtain operation instructions for the first partition table and process the first partition table according to the operation instructions. For example, if the operation instruction is a partition update instruction, the database server can determine the target number of partitions in the first partition table based on the partition update instruction. For example, if the current number of partitions in the first partition table is M, and the partition update instruction is to add N partitions to the first partition table, then the target number of partitions in the first partition table is M+N. The updated first partition table can include multiple partitions, namely partition 1, partition 2, partition 3, ..., partition M, partition M+1, ..., partition M+N.

[0135] After updating the number of partitions in the first partition table to the target number of partitions, in order to balance the data load in each partition, some data in the original M partitions can be migrated to the newly added N partitions. For example, some data in partitions 1 and 2 can be migrated to partition M+1; some data in partitions 3, ..., M can be migrated to partition M+N.

[0136] In related technologies, partitioned tables are typically of the modulo hash type. When the number of partitions in a partitioned table changes, all data in the table needs to be migrated according to the new number of partitions. However, this process is inefficient due to the large amount of data to be migrated.

[0137] In this embodiment, the database server can obtain the creation instruction for the first partition table and create the first partition table according to the creation instruction. The type of the first partition table can be a skip-consistent hash type. Based on the fact that the first partition table is a skip-consistent hash type partition table, the database server can significantly reduce the amount of data to be migrated when operating on the first partition table, without needing to migrate all the data in the first partition table, thus improving the efficiency of data migration.

[0138] To facilitate understanding of the technical solution of this application, firstly, in conjunction with... Figure 2A This section explains the situation where partitions are added to the first partition table in related technologies; combined with... Figure 2B This section explains the situation where a partition is deleted from the first partition table in the relevant technologies.

[0139] Figure 2A This is a schematic diagram illustrating the addition of a partition to the first partition table in related technologies. Please refer to [link / reference]. Figure 2A This includes process 1 and process 2.

[0140] If the first partition table is Table 1, and Table 1 includes three partitions: Partition 1, Partition 2, and Partition 3, each partition stores multiple data entries. If one partition needs to be added to Table 1, then the target number of partitions in Table 1 will be 4.

[0141] In process 1, the database server can create a temporary table based on the target number of partitions, 4. This temporary table can include four partitions: partition 1, partition 2, partition 3, and partition 4. The database server can read all data from table 1 row by row and write all the data to the temporary table. For example... Figure 2A As shown, data from partition 1 of Table 1 can be read and written to partitions 1, 2, 3, and 4 of the temporary table; data from partition 2 of Table 1 can be read and written to partitions 1, 2, 3, and 4 of the temporary table; data from partition 3 of Table 1 can be read and written to partitions 1, 2, 3, and 4 of the temporary table, thus enabling all data in Table 1 to be written to the temporary table.

[0142] In process 2, Table 1 can be renamed to "Backup Table" and the temporary table can be renamed to "Table 1". After renaming, the "Backup Table" (the original Table 1) can be deleted, while the temporary table (the updated "Table 1") can be kept. This allows the addition of one partition to Table 1, increasing the number of partitions from 3 to 4.

[0143] Figure 2B This is a schematic diagram illustrating the deletion of a partition in the first partition table in related technologies. Please refer to [link / reference]. Figure 2B This includes process 1 and process 2.

[0144] If the first partition table is Table 2, and Table 2 contains 4 partitions: partition 1, partition 2, partition 3, and partition 4, then if one partition needs to be deleted from Table 2, the target number of partitions in Table 2 will be 3.

[0145] In process 1, the database server can create a temporary table with three partitions, designated as partition 1, partition 2, and partition 3. The database server can then read all data from table 2 one by one and write all the data to the temporary table.

[0146] In process 2, Table 2 can be renamed to "Backup Table" and the temporary table can be renamed to "Table 2". After renaming, the "Backup Table" (the original Table 2) can be deleted, while the temporary table (the updated "Table 2") can be kept. This allows you to delete one partition in Table 2 and update Table 2 from four partitions to three partitions.

[0147] In the above Figure 2A and Figure 2B In the embodiments shown, when updating the first partition table in related technologies, that is, when adding or deleting partitions in the first partition table, all data in the first partition table needs to be migrated. The amount of data to be migrated is large, resulting in low efficiency of data migration.

[0148] The technical solutions shown in this application will now be described in detail through specific embodiments. It should be noted that the following embodiments may exist independently or in combination with each other; for identical or similar content, the description will not be repeated in different embodiments.

[0149] The embodiments of this application may include the process of creating a first partition table and the process of operating on the first partition table.

[0150] Below, firstly, in combination with Figures 3-4 The process of creating the first partition table will be explained.

[0151] Figure 3 This is a flowchart illustrating a data table creation method provided for an exemplary embodiment of this application. Please refer to [link / reference]. Figure 3 The method may include:

[0152] S301. Obtain the creation command for the first partition table.

[0153] The execution entity in this application embodiment can be a database server or a data table creation device installed in the database server. The data table creation device can be implemented in software or a combination of software and hardware. The data table creation device can be a processor in the database server. For ease of understanding, the following description will use a database server as the execution entity.

[0154] The creation command can include comments. Comments can be used to instruct the creation of a partitioned table of the skip consistent hash type.

[0155] The database server can execute the create statement for the first partition table to obtain the creation instructions for the first partition table. The create statement can be a Structured Query Language (SQL) statement.

[0156] For example, if the creation statement is: CREATE TABLE test (id INT), COMMENT='JUMP_CONSISTENT_HASH=ON', PARTITION BY HASH (id) PARTITIONS 3, then the database server can execute this creation statement to obtain the creation instructions for the first partition table.

[0157] S302. Create the first partition table and generate disk write information according to the creation command.

[0158] Optionally, after receiving the creation instruction, the database server can create the first partition table according to the creation instruction. For example, if the creation instruction indicates that table 1 is to be created as the first partition table, and the type of the first partition table is skip consistent hash, then the database server can create table 1 as a skip consistent hash type partition table.

[0159] In an optional embodiment, the database server can parse the creation statement using Lex Yacc syntax to obtain relevant information about the partitions in the first partition table to be created. This relevant information may include comment information. It can then generate disk write information based on the partition information, which may also include comment information. Finally, it can save the disk write information as a binary disk file. The disk file can be a tab_obj file.

[0160] For example, if the creation statement for the first partition table includes: "Create the first partition table as table 1, where table 1 is a skip-consistent hash partition table. Table 1 includes partition 1, partition 2, partition 3, and partition 4," then the database server can execute the creation statement for the first partition table to obtain the creation instructions. The database server can parse this creation statement using LexYacc syntax to obtain information about the partitions in the first partition table to be created. Assuming that the partition information may include: the first partition table is a skip-consistent hash partition table, partition identifier 1, partition identifier 2, partition identifier 3, and partition identifier 4, this information can be used to generate disk persistence information 1, which is then saved as disk persistence file 1.

[0161] S303. Generate the table definition cache corresponding to the first partition table based on the disk information.

[0162] The table definition cache may include: the partition table type of the first partition table is a jump consistent hash type.

[0163] In one optional embodiment, the table definition cache corresponding to the first partition table can be generated based on the disk write information as follows: based on the disk write information, determine that the type of the first partition table is a jump consistent hash type; generate the table definition cache based on the type of the first partition table.

[0164] For example, if the disk write information includes information such as Table 1 being a partition table of the skip consistent hash type, partition identifier 1, partition identifier 2, partition identifier 3, and partition identifier 4, the database server can identify the annotation information in the disk write information to determine that the type of Table 1 is skip consistent hash type, and generate the table definition cache corresponding to Table 1 based on the type of the first partition table.

[0165] Optionally, the table definition cache can be a static table cache, corresponding to a unique first partition table. The first partition table can include multiple partitions. Threads can operate on any partition in the first partition table, that is, they can perform operations such as reading data, inserting data, deleting data, and updating data within the partition.

[0166] In this embodiment, the database server can obtain the creation instruction for the first partition table, and create the first partition table and generate disk write information according to the creation instruction. The database server can generate a table definition cache corresponding to the first partition table based on the disk write information. Since the creation instruction includes the partition table type of the first partition table, and the partition table type can be a skip consistent hash partition table, the first partition table with the skip consistent hash partition table type can be created according to the creation instruction. Based on the skip consistent hash partition table, when updating the first partition table, the amount of data to be migrated can be significantly reduced, improving migration efficiency.

[0167] Below, in conjunction with Figure 4 The method for creating the first partition table described above will be explained in further detail.

[0168] Figure 4 This is a schematic diagram illustrating a method for creating a data table, provided as an exemplary embodiment of this application. Please refer to [link / reference]. Figure 4 It includes process 1 and process 2. Process 1 is the process before the plate is placed, and process 2 is the process after the plate is placed.

[0169] In process 1, the database server can execute a create statement for the first partition table to obtain the creation instructions. The database server can parse the create statement using Lex Yacc syntax to obtain information about the partitions in the first partition table to be created, and generate disk write information, which may include comment information. The database server can save the disk write information as a binary file.

[0170] Optionally, the database server can employ a two-level caching mechanism: the first level is the table definition cache, and the second level is the first partition table. In process 2, the database server can create the first partition table according to the creation command and generate the table definition cache corresponding to the first partition table based on the disk write information.

[0171] Optionally, the table definition cache may include a first partition table and its partition table type. The partition table type of the first partition table may be a skip-consistent hash partition table.

[0172] Threads can operate on any partition in the first partition table, meaning they can perform operations such as reading, inserting, deleting, and updating data within the partition.

[0173] In this embodiment, the database server can obtain the creation instruction for the first partition table, and create the first partition table and generate disk write information according to the creation instruction. The database server can generate a table definition cache corresponding to the first partition table based on the disk write information. Since the creation instruction includes the partition table type of the first partition table, and the partition table type can be a skip consistent hash partition table, the first partition table with the skip consistent hash partition table type can be created according to the creation instruction. Based on the skip consistent hash partition table, when updating the first partition table, the amount of data to be migrated can be significantly reduced, improving migration efficiency.

[0174] Below, in conjunction with Figures 5-15 The process of operating on the first partition table will be explained.

[0175] Figure 5 This is a schematic flowchart illustrating a data processing method provided for an exemplary embodiment of this application. Please refer to [link / reference]. Figure 5 The method may include:

[0176] S501. Obtain the operation instruction for the first partition table. The operation instruction is a partition update instruction.

[0177] Partition update commands can be either partition addition commands or partition deletion commands. Partition addition commands are used to instruct the creation of a partition in the first partition table; partition deletion commands are used to instruct the deletion of a portion of a partition in the first partition table.

[0178] Optionally, the database server can execute computer program statements to obtain operation instructions for the first partition table. For example, if computer program statement 1 is `ALTER TABLE TEST ADD PARTITION PARTITIONS 2`, the database server can execute this statement to obtain operation instructions for the first partition table, which is a partition addition instruction, indicating that two partitions are added to the first partition table; if computer program statement 2 is `ALTER TABLE TESTCOALESCE PARTITION 1`, the database server can execute this statement to obtain operation instructions for the first partition table, which is a partition deletion instruction, indicating that one partition is deleted from the first partition table.

[0179] S502. According to the partition update instruction, obtain the type of the first partition table from the table definition cache corresponding to the first partition table.

[0180] The database server may include a table definition cache for the first partition table.

[0181] Since the table definition cache can include the first partition table as a skip consistent hash partition table, the database server can obtain the type of the first partition table from the table definition cache corresponding to the first partition table according to the operation instructions.

[0182] For example, if the first partition table is table 1 and the corresponding table definition cache is 1, then the database server can obtain the type of table 1 as skip consistent hash from table definition cache 1.

[0183] S503. Determine the target number of partitions for the first partition table based on the type of the first partition table and the partition update instruction.

[0184] Optionally, after the database server obtains the type of the first partition table, if the operation instruction is a partition update instruction, the database server can determine the current number of partitions in the first partition table, and determine the target number of partitions in the first partition table based on the type of the first partition table, the partition update instruction, and the current number of partitions.

[0185] For example, if the current number of partitions in the first partition table is 3, and the partition update instruction is a partition add instruction, indicating that 2 new partitions are added to the first partition table, then the target number of partitions in the first partition table can be determined to be 5 based on the current number of partitions 3 and the partition update instruction; if the partition update instruction is a partition delete instruction, indicating that 1 partition is deleted from the first partition table, then the target number of partitions in the first partition table can be determined to be 2 based on the current number of partitions 3 and the partition update instruction.

[0186] S504. Based on the partition update instruction and the target number of partitions, determine the data to be migrated and the target partition of the data to be migrated in the first partition table.

[0187] The amount of data to be migrated can be less than the amount of data in the first partition table.

[0188] Optionally, the ratio of the number of data to be migrated to the number of current data in the first partition table can be N / (M+N). Where M can be the current number of partitions in the first partition table, and N can be the number of partitions added or deleted in the first partition table.

[0189] For example, if the current number of partitions M in the first partition table is 3, and the number of new partitions N added to the first partition table can be 2, then the ratio of the number of data to be migrated to the current number of data in the first partition table can be 2 / 5. That is, if the current number of data in the first partition table is 100, then the number of data to be migrated can be 40. If the current number of partitions M in the first partition table is 3, and the number of partitions N deleted from the first partition table can be 1, then the ratio of the number of data to be migrated to the current number of data in the first partition table can be 1 / 2. That is, if the current number of data in the first partition table is 100, then the number of data to be migrated can be 50.

[0190] Optionally, determining the data to be migrated and the target partition for the data to be migrated in the first partition table can include the following two cases:

[0191] Scenario 1: The partition update command is a partition addition command.

[0192] In this case, the data to be migrated and the target partitions of the data to be migrated can be determined in the first partition table based on the data in the first partition table and the number of target partitions.

[0193] For example, if the first partition table includes partition 1, partition 2, and partition 3, meaning the current number of partitions in the first partition table is 3, and the partition update instruction is to add two partitions, namely partition 4 and partition 5, then the target number of partitions is 5. The first partition table can include partition 1, partition 2, partition 3, partition 4, and partition 5. Therefore, based on the data in the first partition table and the target number of partitions (5), the data to be migrated and its target partitions can be determined within the first partition table. Assuming there are 20 pieces of data to be migrated, the target partitions for these 20 pieces of data could include partition 4 and partition 5.

[0194] Scenario 2: The partition update command is a partition deletion command.

[0195] In this case, the partition to be deleted can be determined in the first partition table according to the partition deletion command, the data in the partition to be deleted can be determined as the data to be migrated, and the target partition of the data to be migrated can be determined according to the data to be migrated and the number of target partitions.

[0196] If the first partition table includes partition 1, partition 2, and partition 3, meaning the current number of partitions in the first partition table is 3, and if the partition update instruction is a partition deletion instruction indicating the deletion of partition 3, then partition 3 can be determined to be deleted in the first partition table based on the partition deletion instruction. If the target number of partitions is 2, then the data in partition 3 can be determined as the data to be migrated. Based on the data to be migrated and the target number of partitions 2, the target partitions for the data to be migrated can include partition 1 and partition 2.

[0197] Optionally, in the above two cases, for any data to be migrated, the target partition of the data to be migrated can be further determined in the following way: the data to be migrated and the number of target partitions are processed by the jump consistent hash algorithm to obtain the partition identifier; the partition corresponding to the partition identifier is determined as the target partition of the data to be migrated.

[0198] Jump Consistent Hash is a uniformly mapped and consistent hashing algorithm. It maps data to different partitions, avoiding the need to remap all data and ensuring that as little data as possible is remapped.

[0199] Each partition has a partition identifier. For example, partition 1 can be identified by the identifier 1, and partition 2 can be identified by the identifier 2.

[0200] For example, if the data to be migrated is data 1 and the target number of partitions is 5, that is, the first partition table includes partition 1, partition 2, partition 3, partition 4 and partition 5, then the skip consistent hash algorithm can be used to process data 1 and the target number of partitions 5 to obtain partition identifier 3. Then the partition corresponding to partition identifier 3, i.e. partition 3, can be determined as the target partition of data 1.

[0201] S505. Migrate the data to be migrated to the corresponding target partition.

[0202] In an optional embodiment, the data to be migrated is migrated to the corresponding target partition: during a first time period, at least two threads perform parallel migration processing on the data to be migrated in the first partition table to obtain an updated first partition table; the Data Manipulation Language (DML) operation on the first partition table is determined during the first time period; and the DML operation is synchronized in the updated first partition table.

[0203] Since there are multiple data sets to be migrated, at least two threads can be used to perform parallel migration of the data in the first partition table within the first time period, which can reduce migration time and improve migration efficiency.

[0204] For example, if the data to be migrated includes data 1, data 2, data 3, data 4, and data 5, and the target partition corresponding to data 1, data 2, and data 3 is partition 4, and the target partition corresponding to data 4 and data 5 is partition 5, then in the first time period, data 1, data 2, and data 3 can be migrated to partition 4 through thread 1; in parallel, data 4 and data 5 can be migrated to partition 5 through thread 2 to obtain the updated first partition table. In the updated first partition table, partition 4 stores data 1, data 2, and data 3, and partition 5 stores data 4 and data 5.

[0205] Optionally, during the first time period, there may be Data Manipulation Language (DML) operations performed on the first partition table.

[0206] Data Manipulation Language (DML) can be used to manipulate data in the first partition table, including inserting, modifying, deleting, and querying data. For example, a DML statement could be "update table nameset field1=value 1".

[0207] Below, in conjunction with Figure 6The parallel migration processing and DML operations described above in the first time period will be explained.

[0208] Figure 6 This is a schematic diagram illustrating parallel migration processing and DML operations during a first time period, provided as an exemplary embodiment of this application. Please refer to... Figure 6 The first time period is from t1 to t2. If the first partition table includes partition 1, partition 2, and partition 3, event 1 within the first time period can be the parallel migration of the data to be migrated in the first partition table, resulting in an updated first partition table. The updated first partition table can include partition 1, partition 2, partition 3, and partition 4. Event 2 is the performance of DML operations on the data in the first partition table and the generation of corresponding operation logs. DML operations can include insert, update, and delete operations.

[0209] After completing Event 1 within the first time period, since Event 2 also involves DML operations on the unupdated first partition table within the same time period, to avoid execution errors, after obtaining the updated first partition table, it is necessary to synchronize the DML operations performed on the first partition table within the first time period in the updated first partition table. For example, if Event 2 involves inserting data 1 into partition 1 of the first partition table, and if the updated first partition table determines that data 1 should be inserted into partition 2, then data 1 can be inserted into partition 2 and deleted from partition 1 in the updated first partition table. This achieves synchronization of the DML operations performed on the first partition table within the first time period in the updated first partition table.

[0210] In this embodiment, the database server can obtain operation instructions for the first partition table and, based on these instructions, retrieve the type of the first partition table from the table definition cache corresponding to it. The operation instructions can be partition update instructions. The database server can determine the target number of partitions for the first partition table based on its type and the partition update instructions. Based on the partition update instructions and the target number of partitions, the database server can determine the data to be migrated and its target partitions within the first partition table, and then migrate the data to the corresponding target partitions. Since the first partition table is a skip-consistent hash partition table, the amount of data to be migrated can be less than the amount of data in the first partition table, reducing the amount of data to be migrated. Parallel migration of the data to be migrated can be performed using at least two threads, improving migration speed. The number of operations when adding or deleting partitions is reduced, the number of generated log files is reduced, and the master-slave synchronization speed is accelerated, comprehensively improving the efficiency of data migration.

[0211] Since updating the first partition table includes both adding and deleting partitions, the following... Figure 5 Based on the illustrated embodiment, by Figure 7Further details are provided regarding the addition of partitions in the first partition table; through Figure 12 The scenario of deleting a partition in the first partition table is explained in further detail.

[0212] Figure 7 A flowchart illustrating another data processing method provided for an exemplary embodiment of this application. Please refer to... Figure 7 The method may include:

[0213] S701, Obtain the partition update command for the first partition table.

[0214] Optionally, the database server can execute computer program statements to obtain partition update instructions for the first partition table. These partition update instructions can add instructions to the partition.

[0215] For example, if the computer program statement 1 is ALTER TABLE TEST ADD PARTITION PARTITIONS 2, the database server can execute the computer program statement 1 to obtain the partition update instruction for the first partition table as a partition addition instruction, indicating the addition of 2 partitions.

[0216] S702. According to the partition update instruction, obtain the type of the first partition table from the table definition cache corresponding to the first partition table.

[0217] The database server may include a table definition cache for the first partition table.

[0218] Since the table definition cache can include the first partition table's partition table type as a skip consistent hash partition table, the database server can obtain the type of the first partition table from the table definition cache corresponding to the first partition table based on the partition update instruction.

[0219] For example, if the first partition table is table 1 and the corresponding table definition cache is 1, then the database server can obtain the type of table 1 as skip consistent hash from table definition cache 1.

[0220] S703. Determine the target number of partitions for the first partition table based on the type of the first partition table and the partition update instruction.

[0221] Optionally, after the database server obtains the type of the first partition table, it can determine the target number of partitions for the first partition table based on the type of the first partition table and the partition update instruction.

[0222] For example, if the current number of partitions in the first partition table is 3, and the partition update instruction is a partition addition instruction, indicating the addition of 2 partitions, then the target number of partitions in the first partition table can be determined to be 5 based on the current number of partitions of 3 and the partition update instruction.

[0223] For example, if the partition update instruction is a partition addition instruction, indicating that the target number of partitions in the first partition table is 5, then the target number of partitions in the first partition table can be determined to be 5 based on the partition update instruction.

[0224] S704. Based on the data in the first partition table and the number of target partitions, determine the migration partition for each piece of data in the first partition table.

[0225] A migration partition refers to the partition where data will be migrated. For any data in the first partition table, the migration partition can be the partition where the data currently resides or a newly added partition.

[0226] In an optional embodiment, the migration partition of the data can be determined as follows: generate pseudo-random numbers corresponding to the data based on the data and the original seed; determine the ratio of the number of newly added partitions to the number of target partitions as the comparison value; if the pseudo-random number is greater than or equal to the comparison value, then the migration partition of the data is determined to be the partition where the data is currently located; if the pseudo-random number is less than the comparison value, then the migration partition of the data is determined to be the newly added partition.

[0227] Optionally, a pseudo-random number generator can be configured in the database server. The pseudo-random number generator can be used to generate pseudo-random numbers.

[0228] For any given data, a pseudo-random number generator can be used to generate pseudo-random numbers corresponding to the data, based on the data and the original seed. The original seed can be a sequence of numbers. For example, the original seed could be 2862933555777941757ULL.

[0229] For example, if the data 1 is 0.5 and the original seed is 2862933555777941757ULL, then a pseudo-random number generator can be used to generate the pseudo-random number 1 corresponding to 0.5 based on 0.5 and 2862933555777941757ULL. Assume the generated pseudo-random number 1 can be 0.365.

[0230] Optionally, the ratio of the number of new partitions to the target number of partitions can be determined as the comparison value. For example, if the number of new partitions is 2 and the target number of partitions is 5, then the comparison value can be determined as 2 / 5.

[0231] For any data in the first partition table, determining the migration partition based on a pseudo-random number and a comparison value can include the following two cases:

[0232] Case 1: The pseudo-random number is greater than or equal to the comparison value.

[0233] In this case, the data migration partition can be determined to be the partition where the data is currently located.

[0234] For example, if the partition where data 1 is currently located is partition 1, the pseudo-random number corresponding to data 1 is 0.516, and the comparison value is 2 / 5, then since the pseudo-random number 0.516 is greater than the comparison value 2 / 5, it can be determined that the migration partition of data 1 is the partition where the data is currently located, i.e., partition 1.

[0235] Case 2: The pseudo-random number is less than the comparison value.

[0236] In this case, the data migration partition can be determined to be the newly added partition.

[0237] For example, if the current partition of data 2 is partition 1, the newly added partition is partition 4, the pseudo-random number corresponding to data 2 is 0.315, and the comparison value is 2 / 5, then since the pseudo-random number 0.315 is less than the comparison value 2 / 5, it can be determined that the migration partition of data 2 is the newly added partition, i.e., partition 4.

[0238] Below, in conjunction with Figure 8 This section explains the process of determining the migration partitions for the data.

[0239] Figure 8 This is a schematic diagram illustrating the determination of a migration partition as provided in an exemplary embodiment of this application. Please refer to... Figure 8 If the current number of partitions in the first partition table is 1, and data K is in partition 1. If a new partition 2 is added to the first partition table, and the original seed is 2862933555777941757ULL, then a pseudo-random number 1 corresponding to data K can be generated based on data K and the original seed. Assume that the pseudo-random number 1 is 0.516. Based on the newly added partition number 1 and the target partition number 2, the comparison value 1 is determined to be 1 / 2. Since the pseudo-random number 1 is greater than the comparison value 1, the migration partition for data K can be determined to be the newly added partition 2. When the target partition number of the first partition table changes from 2 to 3, a pseudo-random number 2 corresponding to data K can be generated based on data K and the original seed. Assuming the pseudo-random number 2 is 0.358, the comparison value 2 is determined to be 1 / 3 based on the newly added partition number 1 and the target partition number 3. Since the pseudo-random number 2 is greater than the comparison value 2, the migration partition for data K can be determined to be the current partition, i.e., partition 2. When the target partition number of the first partition table changes from 3 to 4, a pseudo-random number 3 corresponding to data K can be generated based on data K and the original seed. Assuming the pseudo-random number 3 is 0.135, the comparison value 3 is determined to be 1 / 4 based on the newly added partition number 1 and the target partition number 4. Since the pseudo-random number 3 is less than the comparison value 3, the migration partition for data K can be determined to be the newly added partition, i.e., partition 4.

[0240] S705. Based on the migration partition of each data in the first partition table, determine the data to be migrated and the target partition of the data to be migrated in the first partition table.

[0241] Optionally, for any data in the first partition table, if the data's migration partition is a newly added partition, then the data is identified as the data to be migrated, and the data's migration partition is identified as the target partition of the data to be migrated.

[0242] For example, if the migration partition of data 2 is the newly added partition 4, then data 2 can be identified as the data to be migrated, and the migration partition of data 2, i.e. partition 4, can be identified as the target partition of data 2.

[0243] Optionally, the amount of data to be migrated can be less than the amount of data in the first partition table.

[0244] The ratio of the number of data to be migrated to the current number of data in the first partition table can be N / (M+N). Where M can be the current number of partitions in the first partition table, and N can be the number of new partitions added to the first partition table.

[0245] For example, if the current number of partitions M in the first partition table is 3, and the number of new partitions N added to the first partition table can be 2, then the ratio of the number of data to be migrated to the current number of data in the first partition table can be 2 / 5. That is, if the current number of data in the first partition table is 100, then the number of data to be migrated can be 40.

[0246] Below, in conjunction with Figure 9A This application specifies the amount of data to be migrated; combined with... Figure 9B This section explains the amount of data to be migrated in the relevant technologies.

[0247] Figure 9A An illustration of the amount of data to be migrated provided for an exemplary embodiment of this application. Figure 1 Please see. Figure 9A If the first partition table includes partition 1, partition 2, partition 3, and partition 4, and if partition 4 is a newly added partition, and if partition 1, partition 2, and partition 3 each store 100 data entries, then the number of data entries to be migrated can be 300. 1 / 4 = 75. Assuming partitions 1, 2, and 3 each contain 25 data entries to be migrated, then we can read 25 data entries from each partition, for a total of 75 data entries to be migrated. These 75 data entries can then be written to partition 5. Specifically, the partitions for reading data include partitions 1, 2, 3, and 4, with a total of 75 data entries read; the partition for writing data is partition 5, with a total of 75 data entries written.

[0248] Figure 9B This is a diagram illustrating the amount of data to be migrated in related technologies. Figure 1 Please see. Figure 9BIf the first partition table includes partition 1, partition 2, partition 3, and partition 4, and partition 4 is a newly added partition, and if partition 1, partition 2, and partition 3 each store 100 data entries, then all 300 data entries (partitions 1, 2, and 3) need to be read. 75 data entries can be written to each partition to write these 300 data entries to partition 1, partition 2, partition 3, partition 4, and partition 5 respectively. Specifically, the partitions for which data needs to be read are partition 1, partition 2, partition 3, and partition 4, and the number of data entries to be read is 300; the partitions for which data needs to be written are partition 1, partition 2, partition 3, partition 4, and partition 5, and the number of data entries to be written is also 300.

[0249] Depend on Figure 9A and Figure 9B It is known that this application only needs to read 75 data points, compared to 300 data points in related technologies, which significantly reduces the amount of data to be migrated and reduces the number of partitions that need to be written to. That is, related technologies require 5 partitions to be written to, while this application only requires 5 partitions to be written to, which reduces the operations when adding partitions, reduces the number of log files generated, and speeds up the master-slave synchronization.

[0250] S706. During the first time period, at least two threads perform parallel migration processing on the data to be migrated in the first partition table to obtain the updated first partition table.

[0251] Since there are multiple data sets to be migrated, once the data sets to be migrated and their target partitions are determined, the data sets to be migrated can be processed in parallel using at least two threads to obtain the updated first partition table.

[0252] Specifically, the data to be migrated can be migrated in the following way: Create a hidden new partition in the first partition table; determine the original metadata and new metadata corresponding to the first partition table; open the first partition table according to the original metadata and new metadata respectively, and migrate the data to be migrated in the first partition table to the target partition through at least two threads; rename the original metadata corresponding to the first partition table to "backup metadata" and rename the new metadata to "metadata"; delete "backup metadata" (original metadata) and keep "metadata" (new metadata).

[0253] Here, the original metadata refers to the metadata corresponding to the first partition table before the update; the new metadata refers to the metadata that will be updated on the first partition table.

[0254] Below, in conjunction with Figure 10 This section explains the parallel migration process.

[0255] Figure 10 An illustration of a parallel migration provided as an exemplary embodiment of this application Figure 1 Please see. Figure 10 If the first partition table includes partition 1, partition 2, partition 3, partition 4, and partition 5, and partition 5 is a newly added partition, then thread 1 can read the data to be migrated from partition 1 and migrate the data to be migrated from partition 1 to partition 5; in parallel, thread 2 can read the data to be migrated from partition 2 and migrate the data to be migrated from partition 2 to partition 5; thread 3 can read the data to be migrated from partition 3 and migrate the data to be migrated from partition 3 to partition 5; and thread 4 can read the data to be migrated from partition 4 and migrate the data to be migrated from partition 4 to partition 5.

[0256] Below, in conjunction with Figure 11 This section explains how to update the first partition table.

[0257] Figure 11 An illustration of updating a first partition table provided for an exemplary embodiment of this application Figure 1 Please see. Figure 11 This includes the first partition table and the updated first partition table.

[0258] Before the update, the first partition table includes partition 1, partition 2, and partition 3, each storing multiple data entries. Once the data to be migrated in the first partition table and its target partition are determined, the data can be migrated. If the target partition number in the first partition table is 4, a hidden partition 4 can be created. This indicates that the original metadata of the first partition table includes 3 partitions, and the new metadata includes 4 partitions. The first partition table can be opened based on the original metadata, the data to be migrated can be read, and the data can be migrated to the target partition in the first partition table opened according to the new metadata. If the data to be migrated is 4 in partition 1, 13 in partition 2, and 15 in partition 3, and the corresponding target partition is partition 4, then at least two threads can be used to migrate 4 in partition 1, 13 in partition 2, and 15 in partition 3 to partition 4. Figure 8 As shown, the updated first partition table can include four partitions: partition 1, partition 2, partition 3, and partition 4.

[0259] After the migration of the data is completed, rename the original metadata corresponding to the first partition table to "backup metadata" and rename the new metadata to "metadata". Delete "backup metadata", which is the original metadata, and keep "metadata", which is the new metadata. This will complete the update of the first partition table from 3 partitions to 4 partitions.

[0260] S707. Determine the DML operations on the first partition table within the first time period.

[0261] Optionally, DML operations include insert, update, and delete operations.

[0262] For example, if DML operation 1 on the first partition table in the first time period is to insert data 1 in partition 1, DML operation 2 is to update data 2 in partition 1, and DML operation 3 is to delete data 3 in partition 1, then the database server can determine the three DML operations on the first partition table in the first time period, namely DML operation 1, DML operation 2, and DML operation 3.

[0263] S708. Synchronize DML operations in the updated first partition table.

[0264] Since DML operations include insert, update, and delete operations, synchronizing DML operations in the updated first partition table can include the following three scenarios:

[0265] Case 1: The DML operation is an insert operation.

[0266] In this case, the first data corresponding to the insertion operation and the original insertion partition of the first data in the first partition table can be determined; based on the target partition number, the target insertion partition of the first data in the first partition table can be determined; if the original insertion partition is different from the target insertion partition, the first data is inserted into the target insertion partition, the first data is deleted from the original insertion partition, and the operation log corresponding to the insertion operation is updated.

[0267] For example, if in the first time period, before the first partition table is updated, DML operation 1 is performed on the first partition table to insert data 1 into partition 1, then the first data corresponding to DML operation 1 is determined to be data 1, and the original insertion partition of the first data in the first partition table is partition 1. The corresponding operation log 1 is: Insert data 1 into partition 1. If the target number of partitions in the first partition table is 5, and assuming that based on the target number of partitions 5, the target insertion partition for data 1 in the first partition table is determined to be partition 2, then since the original insertion partition and the target insertion partition are different, after updating the first partition table, data 1 can be inserted into the target insertion partition, i.e., partition 2, and data 1 can be deleted from the original insertion partition, i.e., partition 1. The operation log corresponding to DML operation 1 can be updated, i.e., operation log 1 can be updated to: Insert data 1 into partition 2.

[0268] Scenario 2: The DML operation is a delete operation.

[0269] In this case, the second data corresponding to the deletion operation and the original deletion partition of the second data in the first partition table can be determined; the target deletion partition of the second data in the first partition table can be determined according to the target partition number; if the original deletion partition is different from the target deletion partition, the operation log corresponding to the deletion operation can be updated according to the original deletion partition and the target deletion partition.

[0270] For example, if in the first time period, before the first partition table is updated, DML operation 2 on the first partition table is to delete data 2 in partition 1, then it can be determined that the second data corresponding to DML operation 2 is data 2, and the original deletion partition of the second data in the first partition table is partition 1. The corresponding operation log 2 is: Delete data 2 in partition 1. If the target number of partitions in the first partition table is 5, assuming that based on the target number of partitions 5, the target deletion partition for data 2 in the first partition table should be partition 2, then since data 2 has already been deleted in the original deletion partition, i.e., partition 1, it is only necessary to update operation log 2 based on the original deletion partition and the target deletion partition, so that operation log 2 can be updated to: Delete data 2 in partition 2.

[0271] Scenario 3: The DML operation is an update operation.

[0272] In this case, the third data corresponding to the update operation and the original update partition of the third data in the first partition table can be determined; the target update partition of the third data in the first partition table can be determined according to the target partition number; if the original update partition is different from the target update partition, the updated third data is inserted into the target update partition, the updated first data is deleted from the original update partition, and the operation log corresponding to the update operation is updated.

[0273] For example, if in the first time period, before the first partition table is updated, DML operation 3 updates data 1 to 0.5 in partition 1, then the third data corresponding to DML operation 3 is determined to be data 1, and the original update partition for the third data in the first partition table is partition 1. The corresponding operation log 3 is: "Updated data 1 to 0.5 in partition 1". If the target number of partitions in the first partition table is 5, and assuming that based on the target number of partitions 5, the target update partition for the third data in the first partition table should be partition 2, then since the original update partition and the target update partition are different, and data 1 has already been updated to 0.5 in the original update partition (partition 1), the updated data 1 (0.5) can be inserted into the target update partition (partition 2), and 0.5 can be deleted from the original update partition (partition 1). Accordingly, the operation log 3 corresponding to this DML operation 3 can be updated, that is, the operation log 3 can be updated to: "Updated data 1 to 0.5 in partition 2".

[0274] In this embodiment, the database server can obtain the partition update instruction of the first partition table and, based on the partition update instruction, retrieve the type of the first partition table from the table definition cache corresponding to the first partition table. The database server can determine the target number of partitions for the first partition table based on its type and the partition update instruction. The database server can determine the migration partition for each piece of data in the first partition table based on the data and the target number of partitions, and, based on the migration partition for each piece of data in the first partition table, determine the data to be migrated and its target partition in the first partition table. The database server can perform parallel migration processing on the data to be migrated in the first partition table using at least two threads within a first time period to obtain the updated first partition table. It can also determine the DML operations performed on the first partition table within the first time period and synchronize these DML operations in the updated first partition table. Since the first partition table is a skip-consistent hash partition table, the number of data to be migrated can be less than the number of data in the first partition table, reducing the amount of data to be migrated; and parallel migration processing can be performed using at least two threads, improving the migration speed; furthermore, the number of operations when adding partitions is reduced, the number of generated log files is reduced, and the master-slave synchronization speed is accelerated, comprehensively improving the efficiency of data migration.

[0275] Figure 12 A flowchart illustrating yet another data processing method provided for an exemplary embodiment of this application. Please refer to [link / reference]. Figure 12 The method may include:

[0276] S1201. Obtain the partition update command for the first partition table.

[0277] Optionally, the database server can execute computer program statements to obtain partition update instructions for the first partition table. These partition update instructions can be partition deletion instructions.

[0278] For example, if the computer program statement 1 is ALTER TABLE TEST COALESCE PARTITION 1, the database server can execute the computer program statement 1 to obtain the partition update instruction for the first partition table as a partition deletion instruction, indicating the deletion of one partition.

[0279] S1202. According to the partition update instruction, obtain the type of the first partition table from the table definition cache corresponding to the first partition table.

[0280] It should be noted that the execution process of step S1202 can be referred to the execution process of step S702, and will not be repeated here.

[0281] S1203. Determine the target number of partitions for the first partition table based on the type of the first partition table and the partition update instruction.

[0282] Optionally, after the database server obtains the type of the first partition table, it can determine the target number of partitions for the first partition table based on the type of the first partition table and the partition update instruction.

[0283] For example, if the current number of partitions in the first partition table is 4, and the partition update instruction is a partition deletion instruction indicating the deletion of 1 partition, then the target number of partitions in the first partition table can be determined to be 3 based on the current number of partitions of 4 and the partition update instruction.

[0284] For example, if the partition update instruction is a partition deletion instruction, indicating that the target number of partitions in the first partition table is 3, then the target number of partitions in the first partition table can be determined to be 3 based on the partition update instruction.

[0285] S1204. Based on the partition update instruction, determine the partition to be deleted in the first partition table, and determine the data in the partition to be deleted as the data to be migrated.

[0286] For example, if the current number of partitions in the first partition table is 4, that is, the first partition table includes 4 partitions, namely partition 1, partition 2, partition 3 and partition 4, and if the partition update instruction is a partition deletion instruction, indicating the deletion of 1 partition, then the partition to be deleted in the first partition table can be determined as partition 4, and the data in partition 4 can be determined as the data to be migrated.

[0287] Optionally, the amount of data to be migrated can be less than the amount of data in the first partition table. The ratio of the amount of data to be migrated to the amount of current data in the first partition table can be N / (M+N). Where M can be the current number of partitions in the first partition table, and N can be the number of new partitions added to the first partition table.

[0288] For example, if the current number of partitions M in the first partition table is 3, and the number of partitions N to be deleted from the first partition table can be 1, then the ratio of the number of data to be migrated to the current number of data in the first partition table can be 1 / 3. That is, if the current number of data in the first partition table is 150, then the number of data to be migrated can be 50.

[0289] Below, in conjunction with Figure 13A This application specifies the amount of data to be migrated; combined with... Figure 13B This section explains the amount of data to be migrated in the relevant technologies.

[0290] Figure 13A Schematic diagram two illustrating the amount of data to be migrated as provided in an exemplary embodiment of this application. Please refer to... Figure 13A ,

[0291] If the first partition table includes partition 1, partition 2, partition 3, and partition 4, and partition 4 is the partition to be deleted, and partition 4 stores 120 pieces of data, then these 120 pieces of data in partition 4 can be identified as data to be migrated, resulting in a total of 120 pieces of data to be migrated. Assuming that 40 pieces of data can be written to partition 1, partition 2, and partition 3 respectively, then the 120 pieces of data to be migrated in partition 4 can be read and written to partition 1, partition 2, and partition 3 respectively. Specifically, the partitions for reading data include partition 4, and the number of pieces of data read is 120; the partitions for writing data are partition 1, partition 2, and partition 3, and the number of pieces of data written is 120.

[0292] Figure 13B This is illustration 2, illustrating the amount of data to be migrated in the relevant technologies. Please refer to [link / reference]. Figure 13B If the first partition table includes partition 1, partition 2, partition 3, and partition 4, storing a total of 420 data entries, and partition 4 is the partition to be deleted, then all 420 data entries from partitions 1, 2, 3, and 4 need to be read. This can be achieved by writing 140 data entries to each of partitions 1, 2, and 3, thus writing the 420 data entries to partitions 1, 2, and 3 respectively. Specifically, the partitions for which data needs to be read are partitions 1, 2, 3, and 4, and the number of data entries to be read is 420; the partitions for which data needs to be written are partitions 1, 2, and 3, and the number of data entries to be written is also 420.

[0293] Depend on Figure 13A and Figure 13B As can be seen, this application only needs to read 120 data points, compared to 420 data points in related technologies, which significantly reduces the amount of data to be migrated and reduces the number of partitions that need to be read. That is, related technologies require reading 4 partitions, while this application only requires reading 4 partitions, which reduces the operation of deleting partitions, reduces the number of log files generated, and speeds up the master-slave synchronization.

[0294] S1205. Determine the target partitions for the data to be migrated based on the data to be migrated and the number of target partitions.

[0295] Optionally, after determining the data to be migrated, for any data, the data to be migrated and the target partition number can be processed by the jump consistent hash algorithm to obtain the partition identifier, and the partition corresponding to the partition identifier is determined as the target partition of the data to be migrated.

[0296] For example, if the target number of partitions in the first partition table is 3, and data 1 is the data to be migrated, then the jump consistent hash algorithm can be used to process data 1 and the target number of partitions 3 to obtain partition identifier 2. Then the partition corresponding to partition identifier 2, i.e. partition 2, can be determined as the target partition corresponding to data 1.

[0297] S1206. During the first time period, at least two threads perform parallel migration processing on the data to be migrated in the first partition table to obtain the updated first partition table.

[0298] Below, in conjunction with Figure 14 This section explains the parallel migration process.

[0299] Figure 14 A second schematic diagram illustrating a parallel migration provided as an exemplary embodiment of this application. Please refer to... Figure 14 If the first partition table includes partition 1, partition 2, partition 3, partition 4 and partition 5, and partition 4 and partition 5 are partitions to be deleted, then thread 1 can read the data in partition 4 and migrate the data in partition 4 to partition 1, partition 2 and partition 3; in parallel, thread 2 can read the data in partition 5 and migrate the data in partition 5 to partition 1, partition 2 and partition 3.

[0300] Below, in conjunction with Figure 15 This section explains how to update the partition table.

[0301] Figure 15 This is a second schematic diagram illustrating an update of a first partition table, provided as an exemplary embodiment of this application. Please refer to [link to schematic diagram]. Figure 15 This includes the first partition table and the updated first partition table.

[0302] Before the update, the first partition table includes partitions 1, 2, 3, and 4, each storing multiple data entries. After determining the data to be migrated in the first partition table and its target partitions, at least two threads can be used to migrate the data. If the target partition number in the first partition table is 3, data in partition 4 can be read and migrated to partitions 1, 2, and 3. For example, if partition 4 corresponds to partition 1, partition 13 to partition 2, and partition 15 to partition 3, then at least two threads can be used to migrate partitions 4, 13, and 15 to their respective target partitions, as shown in Figure 13. The updated first partition table includes partitions 1, 2, and 3, effectively deleting partition 4 from the first partition table.

[0303] S1207. Determine the DML operations on the first partition table within the first time period.

[0304] It should be noted that the execution process of step S1207 can be referred to the execution process of step S707, and will not be repeated here.

[0305] S1208. Synchronize DML operations in the updated first partition table.

[0306] It should be noted that the execution process of step S1208 can be referred to the execution process of step S708, and will not be repeated here.

[0307] In this embodiment, the database server can obtain the partition update instruction of the first partition table and, based on the partition update instruction, retrieve the type of the first partition table from the table definition cache corresponding to the first partition table. The database server can determine the target number of partitions for the first partition table based on its type and the partition update instruction. The database server can also determine the partitions to be deleted in the first partition table based on the partition update instruction, identify the data in the partitions to be deleted as the data to be migrated, and determine the target partitions for the data to be migrated based on the data to be migrated and the target number of partitions. Within a first time period, the database server can perform parallel migration processing on the data to be migrated in the first partition table using at least two threads to obtain the updated first partition table, and determine the DML operations performed on the first partition table within the first time period, synchronizing the DML operations in the updated first partition table. Since the first partition table is a skip-consistent hash partition table, the amount of data to be migrated can be less than the amount of data in the first partition table, reducing the amount of data to be migrated; and parallel migration processing can be performed using at least two threads, improving the migration speed; furthermore, fewer operations are performed when deleting partitions, fewer log files are generated, and the master-slave synchronization speed is accelerated, comprehensively improving the efficiency of data migration.

[0308] Figure 16 This is a schematic diagram of a data table creation apparatus provided for an exemplary embodiment of this application. Please refer to [link / reference]. Figure 16 The data table creation device 10 may include: a first acquisition module 11, a creation module 12, and a generation module 13, wherein,

[0309] The first acquisition module 11 is used to acquire the creation instruction of the first partition table, the creation instruction including comment information, the comment information being used to indicate the creation of a partition table of the skip consistent hash type;

[0310] The creation module 12 is used to create the first partition table according to the creation instruction and to generate disk write information, wherein the disk write information includes the annotation information;

[0311] The generation module 13 is used to generate a table definition cache corresponding to the first partition table based on the disk information. The table definition cache includes: the first partition table is of type jump consistent hash.

[0312] The data table creation apparatus provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0313] In one possible implementation, the generation module 13 is specifically used for:

[0314] Based on the disk write information, the type of the first partition table is determined to be the skip consistent hash type;

[0315] Generate the table definition cache based on the type of the first partition table.

[0316] The data table creation apparatus provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0317] Figure 17 A schematic diagram of another data table creation apparatus provided for an exemplary embodiment of this application. Please refer to... Figure 17 ,exist Figure 16 Based on the illustrated embodiment, the data table creation device 10 may further include a second acquisition module 14 and an operation module 15, wherein,

[0318] The first acquisition module 11 is used to acquire the operation instructions of the first partition table;

[0319] The second acquisition module 14 is used to acquire the type of the first partition table from the table definition cache corresponding to the first partition table according to the operation instruction;

[0320] The operation module 15 is used to perform operation processing on the first partition table according to the type of the first partition table and the operation instruction.

[0321] The data table creation apparatus provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0322] In one possible implementation, the operation instruction is a partition update instruction, which is either a partition addition instruction or a partition deletion instruction; the operation module 15 is specifically used for:

[0323] The target number of partitions for the first partition table is determined based on the type of the first partition table and the partition update instruction.

[0324] Based on the partition update instruction and the target number of partitions, the data to be migrated and the target partition of the data to be migrated are determined in the first partition table, wherein the number of data to be migrated is less than the number of data in the first partition table;

[0325] The data to be migrated is migrated to the corresponding target partition.

[0326] In one possible implementation, the operation module 15 is specifically used for:

[0327] If the partition update instruction is a partition addition instruction, then based on the data in the first partition table and the target number of partitions, the data to be migrated and the target partition of the data to be migrated are determined in the first partition table;

[0328] If the partition update instruction is a partition deletion instruction, then the partition to be deleted is determined in the first partition table according to the partition deletion instruction, the data in the partition to be deleted is determined as the data to be migrated, and the target partition of the data to be migrated is determined according to the data to be migrated and the target number of partitions.

[0329] In one possible implementation, the operation module 15 is specifically used for:

[0330] Based on the data in the first partition table and the target number of partitions, determine the migration partition for each piece of data in the first partition table, wherein the migration partition is either the partition where the data is currently located or a newly added partition;

[0331] Based on the migration partition of each data in the first partition table, determine the data to be migrated and the target partition of the data to be migrated in the first partition table.

[0332] In one possible implementation, the operation module 15 is specifically used for:

[0333] For any data in the first partition table, if the migration partition of the data is the newly added partition, then the data is determined as the data to be migrated, and the migration partition of the data is determined as the target partition of the data to be migrated.

[0334] In one possible implementation, for any data in the first partition table; the operation module 15 is specifically used for:

[0335] Based on the data and the original seed, generate pseudo-random numbers corresponding to the data;

[0336] The ratio of the number of newly added partitions to the target number of partitions is determined as the comparison value;

[0337] If the pseudo-random number is greater than or equal to the comparison value, then the migration partition of the data is determined to be the partition where the data is currently located;

[0338] If the pseudo-random number is less than the comparison value, then the migration partition of the data is determined to be the newly added partition.

[0339] In one possible implementation, the operation module 15 is specifically used for:

[0340] The data to be migrated and the target number of partitions are processed using a skip consistent hashing algorithm to obtain partition identifiers;

[0341] The partition corresponding to the partition identifier is determined as the target partition for the data to be migrated.

[0342] In one possible implementation, the operation module 15 is specifically used for:

[0343] During the first time period, the data to be migrated is processed in parallel on the first partition table by at least two threads to obtain the updated first partition table.

[0344] The data manipulation language (DML) operations performed on the first partition table during the first time period are determined, and the DML operations include insert operations, update operations, and delete operations.

[0345] The DML operation is synchronized in the updated first partition table.

[0346] In one possible implementation, the DML operation is an insert operation; the operation module 15 is specifically used for:

[0347] Determine the first data corresponding to the insertion operation, and the original insertion partition of the first data in the first partition table;

[0348] Based on the target number of partitions, determine the target partition in the first partition table where the first data will be inserted;

[0349] If the original insertion partition is different from the target insertion partition, insert the first data into the target insertion partition, delete the first data in the original insertion partition, and update the operation log corresponding to the insertion operation.

[0350] In one possible implementation, the DML operation is a delete operation; the operation module 15 is specifically used for:

[0351] Determine the second data corresponding to the deletion operation, and the original deletion partition of the second data in the first partition table;

[0352] Based on the target number of partitions, determine the target partition for deletion of the second data in the first partition table;

[0353] If the original partition to be deleted is different from the target partition to be deleted, the operation log corresponding to the deletion operation is updated based on the original partition to be deleted and the target partition to be deleted.

[0354] In one possible implementation, the DML operation is an update operation; the operation module 15 is specifically used for:

[0355] Determine the third data corresponding to the update operation, and the original update partition of the third data in the first partition table;

[0356] Based on the target number of partitions, determine the target update partition of the third data in the first partition table;

[0357] If the original update partition is different from the target update partition, the updated third data is inserted into the target update partition, the updated first data is deleted from the original update partition, and the operation log corresponding to the update operation is updated.

[0358] In one possible implementation, the ratio of the amount of data to be migrated to the amount of current data in the first partition table is N / (M+N);

[0359] Wherein, M is the current number of partitions in the first partition table, and N is the number of partitions added or deleted from the first partition table.

[0360] The data table creation apparatus provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0361] This application provides an exemplary embodiment of a database server structure diagram. Please refer to [link / reference]. Figure 18 The database server 20 may include a processor 21 and a memory 22. Exemplarily, the processor 21 and the memory 22 are interconnected via a bus 23.

[0362] The memory 22 stores computer-executed instructions;

[0363] The processor 21 executes the computer execution instructions stored in the memory 22, causing the processor 21 to execute the data table creation method as shown in the above method embodiment.

[0364] Accordingly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data table creation method described in the above method embodiments.

[0365] Accordingly, embodiments of this application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the data table creation method shown in the above method embodiments.

[0366] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0367] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0368] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0369] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0370] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0371] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0372] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0373] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0374] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for creating a data table, characterized in that, include: Obtain the creation instruction for the first partition table, the creation instruction including comment information, the comment information being used to indicate the creation of a partition table of the skip consistent hash type; The first partition table is created according to the creation instruction, and disk write information is generated, including the comment information; Based on the disk write information, a table definition cache corresponding to the first partition table is generated, wherein the table definition cache includes: the type of the first partition table is a jump consistent hash type; Obtain the operation instructions for the first partition table; According to the operation instruction, the type of the first partition table is obtained from the table definition cache corresponding to the first partition table; The first partition table is processed according to its type and the operation instructions.

2. The method according to claim 1, characterized in that, Generate a table definition cache corresponding to the first partition table based on the disk write information, including: Based on the disk write information, the type of the first partition table is determined to be the skip consistent hash type; Generate the table definition cache based on the type of the first partition table.

3. The method according to claim 1, characterized in that, The operation command is a partition update command, which is either a partition addition command or a partition deletion command; Based on the type of the first partition table and the operation instructions, the first partition table is processed through operations, including: The target number of partitions for the first partition table is determined based on the type of the first partition table and the partition update instruction. Based on the partition update instruction and the target number of partitions, the data to be migrated and the target partition of the data to be migrated are determined in the first partition table, wherein the number of data to be migrated is less than the number of data in the first partition table; The data to be migrated is migrated to the corresponding target partition.

4. The method according to claim 3, characterized in that, Based on the partition update instruction and the target number of partitions, the data to be migrated and the target partitions of the data to be migrated are determined in the first partition table, including: If the partition update instruction is a partition addition instruction, then based on the data in the first partition table and the target number of partitions, the data to be migrated and the target partition of the data to be migrated are determined in the first partition table; If the partition update instruction is a partition deletion instruction, then the partition to be deleted is determined in the first partition table according to the partition deletion instruction, the data in the partition to be deleted is determined as the data to be migrated, and the target partition of the data to be migrated is determined according to the data to be migrated and the target number of partitions.

5. The method according to claim 4, characterized in that, Based on the data in the first partition table and the target number of partitions, determine the data to be migrated and the target partitions for the data to be migrated in the first partition table, including: Based on the data in the first partition table and the target number of partitions, determine the migration partition for each piece of data in the first partition table, wherein the migration partition is either the partition where the data is currently located or a newly added partition; Based on the migration partition of each data in the first partition table, determine the data to be migrated and the target partition of the data to be migrated in the first partition table.

6. The method according to claim 5, characterized in that, Based on the migration partition of each piece of data in the first partition table, determine the data to be migrated and the target partition of the data to be migrated in the first partition table, including: For any data in the first partition table, if the migration partition of the data is the newly added partition, then the data is determined as the data to be migrated, and the migration partition of the data is determined as the target partition of the data to be migrated.

7. The method according to claim 6, characterized in that, For any data in the first partition table; determine the migration partition for the data based on the data and the target number of partitions, including: Based on the data and the original seed, generate pseudo-random numbers corresponding to the data; The ratio of the number of newly added partitions to the target number of partitions is determined as the comparison value; If the pseudo-random number is greater than or equal to the comparison value, then the migration partition of the data is determined to be the partition where the data is currently located; If the pseudo-random number is less than the comparison value, then the migration partition of the data is determined to be the newly added partition.

8. The method according to claim 4, characterized in that, Determining the target partitions for the data to be migrated based on the data to be migrated and the number of target partitions includes: The data to be migrated and the target number of partitions are processed using a skip consistent hashing algorithm to obtain partition identifiers; The partition corresponding to the partition identifier is determined as the target partition for the data to be migrated.

9. The method according to any one of claims 3-8, characterized in that, Migrating the data to be migrated to the corresponding target partition includes: During the first time period, the data to be migrated is processed in parallel on the first partition table by at least two threads to obtain the updated first partition table. The data manipulation language (DML) operations performed on the first partition table during the first time period are determined, and the DML operations include insert operations, update operations, and delete operations. The DML operation is synchronized in the updated first partition table.

10. The method according to claim 9, characterized in that, The DML operation is an insert operation; Synchronizing the DML operations in the updated first partition table includes: Determine the first data corresponding to the insertion operation, and the original insertion partition of the first data in the first partition table; Based on the target number of partitions, determine the target partition in the first partition table where the first data will be inserted; If the original insertion partition is different from the target insertion partition, insert the first data into the target insertion partition, delete the first data in the original insertion partition, and update the operation log corresponding to the insertion operation.

11. The method according to claim 9, characterized in that, The DML operation is a delete operation; Synchronizing the DML operations in the updated first partition table includes: Determine the second data corresponding to the deletion operation, and the original deletion partition of the second data in the first partition table; Based on the target number of partitions, determine the target partition for deletion of the second data in the first partition table; If the original partition to be deleted is different from the target partition to be deleted, the operation log corresponding to the deletion operation is updated based on the original partition to be deleted and the target partition to be deleted.

12. The method according to claim 9, characterized in that, The DML operation is an update operation; Synchronizing the DML operations in the updated first partition table includes: Determine the third data corresponding to the update operation, and the original update partition of the third data in the first partition table; Based on the target number of partitions, determine the target update partition of the third data in the first partition table; If the original update partition is different from the target update partition, the updated third data is inserted into the target update partition, the updated first data is deleted from the original update partition, and the operation log corresponding to the update operation is updated.

13. The method according to any one of claims 3-8 and 10-12, characterized in that, The ratio of the number of data to be migrated to the number of current data in the first partition table is N / (M+N); Wherein, M is the current number of partitions in the first partition table, and N is the number of partitions added or deleted from the first partition table.

14. A database server, characterized in that, include: Memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the data table creation method as described in any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the data table creation method according to any one of claims 1 to 13.

16. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data table creation method according to any one of claims 1 to 13.

Images