Data redistribution method, device, equipment and storage medium

Abstract

The present invention discloses a data redistribution method, apparatus, device, and storage medium. The method comprises: determining a tablespace to be migrated, a source node where the tablespace is located, and a target node to be migrated; copying data files in the tablespace to be migrated to the target node; and sending log files corresponding to the tablespace to be migrated to the target node and replaying them. The present invention provides a simple and efficient data redistribution method that, through online migration, copies tablespace data files online, and replays log files generated by users during the migration process, allows users to continue accessing and modifying data in tables within the tablespace during the migration process, with little impact from the tablespace migration.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data redistribution method, apparatus, device and storage medium. Background Art

[0002] Usually, when the number of nodes in a distributed database system changes, in order to fully utilize the node resources within the system, achieve balanced data storage, and improve resource utilization and overall performance, it is necessary to redistribute the original data based on the distribution properties of the data table.

[0003] In existing technologies, data redistribution is typically performed using a specific algorithm, such as a hash consistency algorithm. This process involves scanning the data in the table to be migrated, recalculating the target node for each row, and then distributing the data. This process is complex and time-consuming, impacting overall efficiency. Summary of the Invention

[0004] The present invention provides a data redistribution method, apparatus, device and storage medium to achieve the purpose of simply and efficiently completing data redistribution.

[0005] According to one aspect of the present invention, a data redistribution method is provided, comprising:

[0006] Determine the tablespace to be migrated, the source node where the tablespace to be migrated is located, and the target node to which the tablespace will be migrated;

[0007] Copying the data files in the tablespace to be migrated to the target node;

[0008] The log file corresponding to the tablespace to be migrated is sent to the target node and replayed.

[0009] Furthermore, copying the data files in the tablespace to be migrated to the target node includes:

[0010] Blocking the tablespace to be migrated and each table in the tablespace to be migrated;

[0011] Obtaining a definition of the tablespace to be migrated on the source node, and creating a new tablespace on the target node according to the definition;

[0012] The data files in the tablespace to be migrated are copied to the new tablespace.

[0013] Furthermore, blocking the tablespace to be migrated and each table in the tablespace to be migrated includes:

[0014] Using an exclusive lock to block the tablespace to be migrated;

[0015] Obtain the partition tables in each table, use intention shared locks to lock the main table in the partition table, and use intention exclusive locks to lock the sub-tables in the partition table and the remaining tables in the table space to be migrated.

[0016] Furthermore, sending the log file corresponding to the tablespace to be migrated to the target node and replaying the log file includes:

[0017] Obtaining a first target log file during the copying of the data files in the tablespace to be migrated;

[0018] Sending the first target log file to the target node, and performing corresponding operations on the data file in the target node according to the first target log file;

[0019] Obtaining a second target log file during the replay of the first target log file;

[0020] The second target log file is sent to the target node, and corresponding operations are performed on the data files in the target node according to the second target log file.

[0021] Furthermore, obtaining the first target log file during the copying of the data file in the tablespace to be migrated includes:

[0022] Obtaining a checkpoint log sequence number at a time when the data files in the tablespace to be migrated start copying, and determining the checkpoint log sequence number as a starting log sequence number;

[0023] Writing the log in the memory into a log file, and determining the file log sequence number at the time when the log file is written as the end log sequence number;

[0024] A log file whose log sequence number is greater than or equal to the start log sequence number and less than or equal to the end log sequence number is determined as the first target log file.

[0025] Furthermore, obtaining a second target log file during the replay of the first target log file includes:

[0026] Blocking each table in the tablespace to be migrated using a shared lock;

[0027] Determine the end log sequence number as a new start log sequence number;

[0028] Writing the log in the memory into the log file, and determining the file log sequence number at the time when the log file is written as the new end log sequence number;

[0029] A log file whose log sequence number is greater than or equal to the new starting log sequence number and less than or equal to the new ending log sequence number is determined as the second target log file.

[0030] Furthermore, after sending the log file corresponding to the tablespace to be migrated to the target node and replaying the log file, the method further includes:

[0031] Modify the node dictionary information in the metadata node.

[0032] Furthermore, the node dictionary information in the metadata node is modified, including:

[0033] Blocking each table in the tablespace to be migrated using an exclusive lock;

[0034] Modify the node where the tablespace to be migrated is located in the node dictionary information to the target node;

[0035] Remove the dictionary cache information on the tablespace to be migrated, and delete the tablespace to be migrated on the source node.

[0036] According to another aspect of the present invention, there is provided a data redistribution device, comprising:

[0037] A module for determining a tablespace to be migrated, a source node, and a target node, configured to determine a tablespace to be migrated, a source node where the tablespace to be migrated is located, and a target node to which the tablespace is to be migrated;

[0038] A data file copy module, used for copying the data files in the tablespace to be migrated to the target node;

[0039] The log file sending and replay module is used to send the log file corresponding to the table space to be migrated to the target node and replay it.

[0040] Optionally, the data file copy module is also used to:

[0041] Blocking the tablespace to be migrated and each table in the tablespace to be migrated;

[0042] Obtaining a definition of the tablespace to be migrated on the source node, and creating a new tablespace on the target node according to the definition;

[0043] The data files in the tablespace to be migrated are copied to the new tablespace.

[0044] Optionally, the data file copy module is also used to:

[0045] Using an exclusive lock to block the tablespace to be migrated;

[0046] Obtain the partition tables in each table, use intention shared locks to lock the main table in the partition table, and use intention exclusive locks to lock the sub-tables in the partition table and the remaining tables in the table space to be migrated.

[0047] Optionally, the log file sending and replay module is also used to:

[0048] Obtaining a first target log file during the copying of the data files in the tablespace to be migrated;

[0049] Sending the first target log file to the target node, and performing corresponding operations on the data file in the target node according to the first target log file;

[0050] Obtaining a second target log file during the replay of the first target log file;

[0051] The second target log file is sent to the target node, and corresponding operations are performed on the data files in the target node according to the second target log file.

[0052] Optionally, the log file sending and replay module is also used to:

[0053] Obtaining a checkpoint log sequence number at a time when the data files in the tablespace to be migrated start copying, and determining the checkpoint log sequence number as a starting log sequence number;

[0054] Writing the log in the memory into a log file, and determining the file log sequence number at the time when the log file is written as the end log sequence number;

[0055] A log file whose log sequence number is greater than or equal to the start log sequence number and less than or equal to the end log sequence number is determined as the first target log file.

[0056] Optionally, the log file sending and replay module is also used to:

[0057] Blocking each table in the tablespace to be migrated using a shared lock;

[0058] Determine the end log sequence number as a new start log sequence number;

[0059] Writing the log in the memory into the log file, and determining the file log sequence number at the time when the log file is written as the new end log sequence number;

[0060] A log file whose log sequence number is greater than or equal to the new starting log sequence number and less than or equal to the new ending log sequence number is determined as the second target log file.

[0061] Optionally, the data redistribution device further includes a node dictionary information modification module, which is used to modify the node dictionary information in the metadata node.

[0062] Optionally, the node dictionary information modification module is also used to:

[0063] Blocking each table in the tablespace to be migrated using an exclusive lock;

[0064] Modify the node where the tablespace to be migrated is located in the node dictionary information to the target node;

[0065] Remove the dictionary cache information on the tablespace to be migrated, and delete the tablespace to be migrated on the source node.

[0066] According to another aspect of the present invention, an electronic device is provided, comprising:

[0067] at least one processor; and

[0068] a memory communicatively connected to the at least one processor; wherein,

[0069] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data redistribution method described in any embodiment of the present invention.

[0070] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data redistribution method described in any embodiment of the present invention when executed.

[0071] The embodiment of the present invention first determines the tablespace to be migrated, the source node where the tablespace is located, and the target node to be migrated. Then, the data files in the tablespace to be migrated are copied to the target node. Finally, the log files corresponding to the tablespace to be migrated are sent to the target node and replayed. The data redistribution method provided by the embodiment of the present invention uses online migration, online copying of tablespace data files, and replaying of log files generated by users during the migration process. This allows users to continue accessing and modifying data in the tables within the tablespace during the migration process, with little impact from the tablespace migration, thereby achieving a simple and efficient technical effect.

[0072] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0074] Figure 1 This is a flow chart of a data redistribution method provided according to the first embodiment of the present invention;

[0075] Figure 2 This is a flow chart of a data redistribution method provided according to the second embodiment of the present invention;

[0076] Figure 3 This is a structural diagram of a data redistribution device provided according to a third embodiment of the present invention;

[0077] Figure 4 It is a structural diagram of an electronic device for implementing the data redistribution method of embodiment 4 of the present invention. DETAILED DESCRIPTION

[0078] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0079] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0080] Example 1

[0081] Figure 1A flowchart of a data redistribution method is provided for the first embodiment of the present invention. This embodiment is applicable to the case where two or more sensors are used for target tracking. The method can be executed by a data redistribution device. The data redistribution device can be implemented in the form of hardware and / or software. The data redistribution device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0082] S110: Determine the tablespace to be migrated, the source node where the tablespace to be migrated is located, and the target node to which the tablespace will be migrated.

[0083] In a database, a tablespace (TS) consists of one or more data files. All database objects are logically stored in a tablespace and physically stored in the data files within the tablespace. A tablespace serves as a portable unit of data, allowing for rapid file migration of subtables between data storage nodes (BPs). Data storage nodes are the actual storage nodes for user data. A cluster has multiple BPs, supporting dynamic addition and deletion.

[0084] In this embodiment, when the number of BPs changes, node data in the distributed system needs to be migrated, i.e., redistributed. The tablespace to be migrated can be a tablespace on the BP that requires data migration. Furthermore, the source node can be the BP where the tablespace to be migrated was located before the migration, and the target node can be the BP where the tablespace to be migrated is located after the migration.

[0085] Optionally, redistribution can use two types of tablespace migration modes: read-only and read-write. In read-only mode, users can only read tables within the tablespace, but not write to them. In read-write mode, users can still write to tables within the tablespace most of the time, such as during file copying. Write operations to tables within the tablespace are blocked only during the short final node switch. Read-write operations have less impact on the system and are more user-friendly.

[0086] For example, if you want to migrate tablespace TS1 on node BP1 to node BP2, the SQL statement executed by the user is: alter tablespace TS1 move to BP2, where the tablespace to be migrated is TS1, the source node is BP1, and the target node is BP2.

[0087] S120: Copy the data files in the tablespace to be migrated to the target node.

[0088] In this embodiment, based on the known tablespace to be migrated and the corresponding target node, the tablespace to be migrated can be migrated to the target node, which is physically manifested as copying the data files in the tablespace to be migrated to the target node. Preferably, the tablespace to be migrated can be the entire tablespace on the source node.

[0089] Optionally, the data files in the tablespace to be migrated can be copied to the target node by: blocking the tablespace to be migrated and each table in the tablespace to be migrated; obtaining the definition of the tablespace to be migrated on the source node, and creating a new tablespace on the target node according to the definition; and copying the data files in the tablespace to be migrated to the new tablespace.

[0090] Specifically, the tablespace to be migrated may contain multiple tables. When copying data files, in order to prevent destructive interactions between multiple transactions that access shared data, including incorrect data updates or incorrect changes to basic data structures, the database's locking mechanism can be used to block the tablespace to be migrated and each table in the tablespace to be migrated. With the help of the locking mechanism, the definition of the tablespace to be migrated can be prevented from being modified. Then, the definition of the tablespace to be migrated can be obtained from the source node, and then a new tablespace identical to the tablespace to be migrated can be created on the target node based on the obtained definition. Finally, all data files of the tablespace to be migrated can be copied, that is, from the source node to the target node. Preferably, a multi-threaded parallel copy method can be used when copying data to improve migration efficiency.

[0091] Furthermore, the method of locking the tablespace to be migrated and each table in the tablespace to be migrated can be: using an exclusive lock to lock the tablespace to be migrated; obtaining the partition table in each table, using an intention shared lock to lock the main table in the partition table, and using an intention exclusive lock to lock the sub-tables in the partition table and the remaining tables in the tablespace to be migrated.

[0092] Typically, a database management system can provide four types of locks: shared locks (S), exclusive locks (X), intention shared locks (IS), and intention exclusive locks (IX). IS / IS, IS / IX, IS / S, and IX / IX locks are compatible. IX locks are incompatible with S locks, and X locks are incompatible with IS / IX / S / X locks. The compatibility characteristics of the four lock types can be represented by a matrix:

[0093]

[0094] Among them, 1 means that the two lock types are compatible, and 0 means they are incompatible and a lock wait will occur.

[0095] Specifically, when locking the tablespace to be migrated and the tables within it, you can perform an X lock on the tablespace to be migrated. Then, perform an IS lock on the primary table in the partitioned table within the tablespace to prevent deletion. Furthermore, perform an IX lock on the child tables within the partitioned table and the remaining tables in the tablespace to be migrated to prevent table definition modifications. With an IX or IS lock, different transactions can simultaneously add, delete, modify, and query data within the table, but modifications to the table definition are prohibited.

[0096] S130: Send the log file corresponding to the tablespace to be migrated to the target node and replay it.

[0097] Among them, log files are record files or file collections used to record system operation events. They can be divided into event logs and message logs. They play important roles in processing historical data, tracing diagnostic problems, and understanding system activities.

[0098] In this embodiment, after the data files in the tablespace to be migrated are copied to the target node, all logs in memory can be archived into log files. The archived logs are then sent to the target node and the logs are replayed. Replaying the logs involves modifying the data pages based on the log contents. Specifically, the log files record all operations performed on the data files in the corresponding tablespace. The same operations can be performed on the data copied to the target node based on the contents of the log files.

[0099] Optionally, after sending the log file corresponding to the tablespace to be migrated to the target node and replaying it, you can also modify the node dictionary information in the metadata node.

[0100] After a tablespace is created, the table creation information is saved in the data dictionary. The dictionary can store database metadata, such as the table creator, creation time, and the index of the node where the tablespace is located. The Metadata Processor (MP) is the node that stores and manages the dictionary information.

[0101] In this embodiment, after the data files of the tablespace to be migrated are copied, the data storage node where the data files are located is changed. Accordingly, the node dictionary information in the metadata node should be modified.

[0102] Optionally, the node dictionary information in the metadata node can be modified by: using an exclusive lock to block each table in the tablespace to be migrated; changing the node where the tablespace to be migrated is located in the node dictionary information to the target node; removing the dictionary cache information on the tablespace to be migrated, and deleting the tablespace to be migrated on the source node.

[0103] Specifically, all tables X in the migration tablespace can be locked, preventing read and write access for a short period before the dictionary switch. No other transactions, except the migration transaction, can modify or access the object. Then, the dictionary information for the node where the tablespace to be migrated resides, recorded on the MP, is modified, replacing the source node with the target node. The dictionary cache information for all table objects involved in the migration tablespace is removed, and the plans associated with these table objects in the plan cache are cleared. Finally, the migration tablespace on the source node is deleted. After completing these steps, the transaction is committed, which automatically unlocks the locks on the tablespace and its individual tables.

[0104] The embodiment of the present invention first determines the tablespace to be migrated and the source node where the tablespace to be migrated is located and the target node to be migrated, then copies the data files in the tablespace to be migrated to the target node, and finally sends the log files corresponding to the tablespace to be migrated to the target node and replays them. The data redistribution method provided by the embodiment of the present invention uses online migration, online copying of tablespace data files, and replaying of log files generated by users during the migration process, so that during the migration process, users can continue to access and modify the data in the tables in the tablespace, and are almost unaffected by the tablespace migration. In addition, a multi-threaded parallel copy method is used when copying data to improve migration efficiency, making the data redistribution process simpler and more efficient.

[0105] Example 2

[0106] Figure 2 This is a flow chart of a data redistribution method provided in the second embodiment of the present invention. This embodiment is a refinement of the above embodiment S130. Figure 2 As shown, the method includes:

[0107] S210: Determine the tablespace to be migrated, the source node where the tablespace to be migrated is located, and the target node to which the tablespace will be migrated.

[0108] In this embodiment, when the number of data storage nodes changes, it is necessary to migrate the node data in the distributed system, that is, redistribute it.

[0109] Optionally, based on the executed SQL statement, the tablespace to be migrated, the source node where the tablespace to be migrated is located, and the target node to which the tablespace is to be migrated may be determined.

[0110] S220: Copy the data files in the tablespace to be migrated to the target node.

[0111] In this embodiment, based on the known tablespace to be migrated and the corresponding target node, the tablespace to be migrated can be migrated to the target node, which is physically manifested as copying the data files in the tablespace to be migrated to the target node.

[0112] Alternatively, the tablespace to be migrated and each table within it can be locked to prevent table definitions from being modified. The definition of the tablespace to be migrated is then obtained on the source node, a new tablespace is created on the target node based on the definition, and the data files in the tablespace to be migrated are copied to the new tablespace. Preferably, the tablespace to be migrated can be locked with an exclusive lock, the main table of the partitioned table within the tablespace can be locked with an intent shared lock, and the child tables within the partitioned table and the remaining tables within the tablespace can be locked with an intent exclusive lock.

[0113] S230: Obtain the first target log file during the copying of the data files in the tablespace to be migrated.

[0114] In this embodiment, the log files corresponding to the tablespace to be migrated can be sent and replayed twice. The first time is the log file during the data file copy period in the tablespace to be migrated, and the second time is the incremental log file generated during the first log replay period.

[0115] Optionally, the method for obtaining the first target log file during the copying of the data files in the tablespace to be migrated may be: obtaining the checkpoint log sequence number of the time when the copying of the data files in the tablespace to be migrated starts, and determining the checkpoint log sequence number as the starting log sequence number; writing the log in the memory to the log file, and determining the file log sequence number of the time when the log file is written as the ending log sequence number; determining the log file whose log sequence number is greater than or equal to the starting log sequence number and less than or equal to the ending log sequence number as the first target log file.

[0116] The Log Sequence Number (LSN) is a Bigint value automatically maintained by the system. It is auto-incrementing and globally unique. Each LSN represents a physical transaction generated within the database system. A physical transaction (PTX) is a collection of operations that modify physical data pages within the database. It corresponds to the concept of a transaction in the database management system and is atomic, ordered, and irreversible. The Checkpoint Log Sequence Number (CKPT_LSN) is the checkpoint LSN. All data pages modified by physical transactions with an LSN <= CKPT_LSN have been written from the buffer to the tablespace data file. The CKPT_LSN is adjusted by the checkpoint thread. When replaying logs, the REDO logs before CKPT_LSN do not need to be replayed. Replaying only from CKPT_LSN onwards will restore the system to the latest state. The File Log Sequence Number (FILE_LSN) is the highest LSN value that has been flushed to the disk in the log file. The content from CKPT_LSN to FILE_LSN has been written to the log file but has not yet been written to the tablespace data file. That is, this part of the log needs to be replayed.

[0117] Specifically, before the first transmission and replay of the log file, a full checkpoint can be executed. A checkpoint is a database event whose function is to write dirty pages in the buffer buffer to disk in the order in which the data pages are modified. During this process, the CKPT_LSN value is dynamically adjusted, and the CKPT_LSN at the time when the data file in the tablespace to be migrated begins to be copied is used as the starting log sequence number (BEGIN_LSN). This indicates that all modified data smaller than this LSN has been flushed to the new tablespace file. Subsequently, the log file is copied to the target node starting from CKPT_LSN for replay. Furthermore, the first target log file is the log file during the data file copy period in the tablespace to be migrated. During the data file copy, the log in the memory is written to the log file, and the FILE_LSN at the time the log file is written is determined as the ending log sequence number (END_LSN). The first target log file is the portion of the LSN range that is within [BEGIN_LSN, END_LSN].

[0118] S240: Send the first target log file to the target node, and perform corresponding operations on the data files in the target node according to the first target log file.

[0119] In this embodiment, after the first target log file is determined, it can be sent to the target node and the log replayed. This is the first replay, and its purpose is to replay the logs generated during the copy of the tablespace data file to be migrated. File copying is the most time-consuming operation in the entire process, and a period of logs may accumulate during this period. Replaying this portion of logs first does not affect user write operations, minimizing the impact on the system.

[0120] S250: Acquire a second target log file during the replay of the first target log file.

[0121] In this embodiment, the object of the second replay is the incremental log file generated during the first replay of the log.

[0122] Optionally, the method for obtaining the second target log file during the replay of the first target log file can be: using a shared lock to block each table in the table space to be migrated; determining the end log sequence number as the new start log sequence number; writing the log in the memory to the log file, and determining the file log sequence number at the time of writing the log file as the new end log sequence number; determining the log file with a log sequence number greater than or equal to the new start log sequence number and less than or equal to the new end log sequence number as the second target log file.

[0123] Specifically, after completing the first replay, you can add an S lock to all tables in the migration tablespace. Since the IS / IX lock has been applied before, the lock of each table is now IX+S or IS+S, which can prevent users from modifying the table and writing data. After adding the S lock, other transactions are not allowed to modify the table definition and the data in the table, but other transactions are allowed to share access to the data in the table. Furthermore, you can use the previous END_LSN as the new BEGIN_LSN, and write all the logs in the memory to the log file again, and obtain the FILE_LSN at this time as the new END_LSN. The second target log file is the part of the LSN range that is in [BEGIN_LSN, END_LSN], where BEGIN_LSN and END_LSN are both updated values.

[0124] S260: Send the second target log file to the target node, and perform corresponding operations on the data file in the target node according to the second target log file.

[0125] In this embodiment, after the second target log file is determined, it can be sent to the target node and the log can be replayed.

[0126] The embodiment of the present invention first determines the tablespace to be migrated and the source node where the tablespace to be migrated is located and the target node to be migrated, then copies the data files in the tablespace to be migrated to the target node, then obtains the first target log file during the copying of the data files in the tablespace to be migrated, then sends the first target log file to the target node, and performs corresponding operations on the data files in the target node according to the first target log file, then obtains the second target log file during the replay of the first target log file, and finally sends the second target log file to the target node, and performs corresponding operations on the data files in the target node according to the second target log file. The data redistribution method provided by the embodiment of the present invention can migrate the tablespace online, copy the tablespace data files online, and complete the replay of the log files in two steps, so that during the migration process, the user can continue to access and modify the data of the table in the tablespace, making the data redistribution process simpler and more efficient.

[0127] Example 3

[0128] Figure 3 This is a structural diagram of a data redistribution device provided in the third embodiment of the present invention. Figure 3 As shown, the device includes: a module 310 for determining a tablespace to be migrated and source and target nodes, a data file copying module 320 and a log file sending and replaying module 330 .

[0129] The tablespace to be migrated, source node, and target node determination module 310 is used to determine the tablespace to be migrated, the source node where the tablespace to be migrated is located, and the target node to which the tablespace will be migrated.

[0130] The data file copy module 320 is used to copy the data files in the tablespace to be migrated to the target node.

[0131] Optionally, the data file copy module 320 is further configured to:

[0132] Block the tablespace to be migrated and all tables in it; obtain the definition of the tablespace to be migrated on the source node, and create a new tablespace on the target node based on the definition; copy the data files in the tablespace to be migrated to the new tablespace.

[0133] Optionally, the data file copy module 320 is further configured to:

[0134] Use exclusive locks to block the tablespace to be migrated; obtain the partition tables in each table, use intention shared locks to block the main table in the partition table, and use intention exclusive locks to block the sub-tables in the partition table and the rest of the tables in the tablespace to be migrated.

[0135] The log file sending and replay module 330 is used to send the log file corresponding to the tablespace to be migrated to the target node and replay it.

[0136] Optionally, the log file sending and replay module 330 is further configured to:

[0137] Obtain a first target log file during the copying of data files in the tablespace to be migrated; send the first target log file to the target node, and perform corresponding operations on the data files in the target node according to the first target log file; obtain a second target log file during the replay of the first target log file; send the second target log file to the target node, and perform corresponding operations on the data files in the target node according to the second target log file.

[0138] Optionally, the log file sending and replay module 330 is further configured to:

[0139] Obtain the checkpoint log sequence number of the time when the data file in the tablespace to be migrated starts to be copied, and determine the checkpoint log sequence number as the starting log sequence number; write the log in the memory to the log file, and determine the file log sequence number of the time when the log file is written as the ending log sequence number; determine the log file with a log sequence number greater than or equal to the starting log sequence number and less than or equal to the ending log sequence number as the first target log file.

[0140] Optionally, the log file sending and replay module 330 is further configured to:

[0141] Use a shared lock to block each table in the tablespace to be migrated; determine the end log sequence number as the new start log sequence number; write the log in the memory to the log file, and determine the file log sequence number at the time of writing the log file as the new end log sequence number; determine the log file with a log sequence number greater than or equal to the new start log sequence number and less than or equal to the new end log sequence number as the second target log file.

[0142] Optionally, the data redistribution device further includes a node dictionary information modification module 340 for modifying the node dictionary information in the metadata node.

[0143] Optionally, the node dictionary information modification module 340 is further configured to:

[0144] Use exclusive locks to block all tables in the tablespace to be migrated; change the node where the tablespace to be migrated is located in the node dictionary information to the target node; remove the dictionary cache information on the tablespace to be migrated, and delete the tablespace to be migrated on the source node.

[0145] The data redistribution device provided in the embodiment of the present invention can execute the data redistribution method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0146] Example 4

[0147] Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0148] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0149] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0150] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the data redistribution method.

[0151] In some embodiments, the data redistribution method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data redistribution described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data redistribution method in any other suitable manner (e.g., by means of firmware).

[0152] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0154] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0157] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0158] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0159] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data redistribution method, characterized in that: include: Determine the tablespace to be migrated, the source node where the tablespace to be migrated is located, and the target node to which the tablespace will be migrated; Using an exclusive lock to block the tablespace to be migrated; Obtain the partition table of each table in the tablespace to be migrated, lock the main table in the partition table with an intention shared lock, and lock the sub-tables in the partition table and the remaining tables in the tablespace to be migrated with an intention exclusive lock; Obtaining a definition of the tablespace to be migrated on the source node, creating a new tablespace on the target node according to the definition; and copying data files in the tablespace to be migrated to the new tablespace. The log file corresponding to the tablespace to be migrated is sent to the target node and replayed.

2. The method according to claim 1, characterized in that Sending the log file corresponding to the tablespace to be migrated to the target node and replaying the log file includes: Obtaining a first target log file during the copying of the data files in the tablespace to be migrated; Sending the first target log file to the target node, and performing corresponding operations on the data file in the target node according to the first target log file; Obtaining a second target log file during the replay of the first target log file; The second target log file is sent to the target node, and corresponding operations are performed on the data files in the target node according to the second target log file.

3. The method according to claim 2, characterized in that Obtaining the first target log file during the copying of the data file in the tablespace to be migrated includes: Obtaining a checkpoint log sequence number at a time when the data files in the tablespace to be migrated start copying, and determining the checkpoint log sequence number as a starting log sequence number; Writing the log in the memory into a log file, and determining the file log sequence number at the time when the log file is written as the end log sequence number; A log file whose log sequence number is greater than or equal to the start log sequence number and less than or equal to the end log sequence number is determined as the first target log file.

4. The method according to claim 3, characterized in that Acquiring a second target log file during the replay of the first target log file includes: Blocking each table in the tablespace to be migrated using a shared lock; Determine the end log sequence number as a new start log sequence number; Writing the log in the memory into the log file, and determining the file log sequence number at the time when the log file is written as the new end log sequence number; A log file whose log sequence number is greater than or equal to the new starting log sequence number and less than or equal to the new ending log sequence number is determined as the second target log file.

5. The method according to claim 1, characterized in that After sending the log file corresponding to the tablespace to be migrated to the target node and replaying the log file, the method further includes: Modify the node dictionary information in the metadata node.

6. The method according to claim 5, characterized in that Modify the node dictionary information in the metadata node, including: Blocking each table in the tablespace to be migrated using an exclusive lock; Modify the node where the tablespace to be migrated is located in the node dictionary information to the target node; Remove the dictionary cache information on the tablespace to be migrated, and delete the tablespace to be migrated on the source node.

7. A data redistribution device, characterized in that: include: A module for determining a tablespace to be migrated, a source node, and a target node, configured to determine a tablespace to be migrated, a source node where the tablespace to be migrated is located, and a target node to which the tablespace is to be migrated; A data file copy module, configured to lock the tablespace to be migrated using an exclusive lock; Obtain the partition table of each table in the tablespace to be migrated, lock the main table in the partition table with an intention shared lock, and lock the sub-tables in the partition table and the remaining tables in the tablespace to be migrated with an intention exclusive lock; Obtaining a definition of the tablespace to be migrated on the source node, creating a new tablespace on the target node according to the definition; and copying data files in the tablespace to be migrated to the new tablespace. The log file sending and replay module is used to send the log file corresponding to the table space to be migrated to the target node and replay it.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the data redistribution method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data redistribution method according to any one of claims 1 to 6 when executed.

Images