Database table data restructuring method and device, medium and computer equipment

Abstract

Embodiments of the present disclosure provide a database table data restructuring method and device, medium and computer device, the method comprising: in response to a DDL operation on a source table, reading data in the source table; the DDL operation can change the storage structure of the source table; sorting the read data, writing the sorted data to the disk of the database node where the target table is located according to data blocks, and recording the storage addresses of each data block in the disk of the database node where the target table is located; after each data block is written, generating an SSTable of the target table based on the storage addresses of each data block, wherein the SSTable of the target table comprises the data blocks.

Description

Technical Field

[0001] This disclosure relates to the field of database technology, and in particular to methods, apparatus, media and computer equipment for data reorganization of database tables. Background Technology

[0002] In databases, it's often necessary to change the storage format of database tables. Changing the storage format requires data reorganization of the database table, meaning the data in the table is rewritten according to the new storage format. Related technologies typically use ordinary transaction insertion for data reorganization, but this method is costly. Summary of the Invention

[0003] In a first aspect, embodiments of this disclosure provide a data restructuring method for a database table, the method comprising: reading data from the source table in response to a DDL operation on the source table; the DDL operation being capable of changing the storage structure of the source table; sorting the read data; writing the sorted data into data blocks and storing the data blocks in the disk of the database node where the target table is located; and recording the storage address of each data block in the disk of the database node where the target table is located; and generating an SSTable of the target table based on the storage address of each data block after all data blocks have been written, wherein the SSTable of the target table includes the data blocks.

[0004] In some embodiments, the source table and the target table are database tables in different namespaces, and the namespace where the source table is located is isolated from the namespace where the target table is located; the method further includes: after generating the SSTable of the target table based on the storage address of each data block, moving the target table to the namespace where the source table is located.

[0005] In some embodiments, each namespace includes a flag bit, and namespaces with different flag bits are isolated from each other; moving the target table to the namespace where the source table is located includes: changing the name of the target table to the name of the source table; and changing the flag bit of the namespace where the target table is located to the same flag bit as the flag bit of the namespace where the source table is located.

[0006] In some embodiments, tables in the namespace where the target table resides are invisible.

[0007] In some embodiments, the method further includes: deleting the source table after moving the target table to the namespace where the source table is located.

[0008] In some embodiments, the target table includes multiple target table replicas, which reside on multiple database nodes in a distributed database. The multiple database nodes include a primary node and at least one backup node. Writing the sorted data to the disk of the database node containing the target table in data blocks includes: writing the sorted data to the target table replica of the primary node in data blocks, and writing the data blocks written to the target table replica of the primary node to the redo log of the primary node; synchronizing the redo log of the primary node to the redo logs of each backup node, so that each backup node writes the sorted data to its own disk in data blocks based on its own redo log.

[0009] In some embodiments, the source table includes multiple first partitions, and different first partitions are stored on different first database nodes in a distributed database; sorting the read data includes: reading the data of the first partitions stored in each first database node respectively; and sorting the data of the first partitions read from each first database node.

[0010] In some embodiments, the target table includes multiple second partitions, and different second partitions are stored on different second database nodes in a distributed database; writing the sorted data into data blocks to the disk of the database node where the target table is located includes: writing the sorted data into data blocks to the disk of the second database node where the target table is located, based on the second partition to which the sorted data belongs.

[0011] In some embodiments, the source table includes a first target item, the first target item including constraint information and / or foreign keys of the source table; the method further includes: batch reading second target items corresponding to the first target item in the target table based on the SSTable of the target table; each second target item read corresponds to at least one type of data in the target table; generating a verification vector based on the data corresponding to the read second target item; each element in a verification vector is data of the same type; verifying the verification vector of the corresponding type of data in the read second target item based on the constraint information corresponding to each type of data.

[0012] In some embodiments, the source table includes a trigger; changing the name of the target table to the name of the source table includes: setting a write lock on the source table and the target table, and changing the name of the target table to the name of the source table after both the source table and the target table have been successfully set; the method further includes: creating a trigger in the target table; wherein the operations of creating a trigger in the target table, setting a write lock on the source table, changing the name of the target table to the name of the source table, and changing the flag of the namespace where the target table is located to the same flag as the flag of the namespace where the source table is located are included in the same transaction.

[0013] In some embodiments, the target table includes multiple target table replicas, which are located on multiple database nodes in a distributed database, including a primary node and at least one backup node; the method further includes: creating an index of the target table replicas on the primary node; writing the index of the target table replicas into the redo log of the primary node; and synchronizing the redo log of the primary node to each backup node, so that each backup node creates an index of its own target table replica based on its own redo log.

[0014] In some embodiments, the storage address of each data block on the disk of the database node where the target table resides is recorded in the data storage structure in the memory of the database node where the target table resides.

[0015] Secondly, embodiments of this disclosure provide a data reorganization apparatus for a database table. The apparatus includes: a reading module, configured to read data from the source table in response to a DDL operation on the source table; the DDL operation is capable of changing the storage structure of the source table; a recording module, configured to sort the read data, write the sorted data into data blocks to the disk of the database node where the target table is located, and record the storage address of each data block on the disk of the database node where the target table is located; and a generation module, configured to generate an SSTable of the target table based on the storage address of each data block after all data blocks have been written, the SSTable of the target table including the data blocks.

[0016] Thirdly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the methods described in any of the embodiments.

[0017] Fourthly, embodiments of this disclosure provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any embodiment.

[0018] In this embodiment of the disclosure, when the storage structure of the source table changes, the data in the source table is sorted by sorting method, and the sorted data is written to the disk of the database node where the target table is located by data blocks. Since the storage address of each data block in the disk is recorded, after each data block is written, the SSTable of the target table can be directly generated based on the storage address of each data block. There is no need to use the ordinary transaction method to write the data to memory and then write the data in memory to disk, thereby saving the overhead of the data reorganization process and improving the data reorganization efficiency.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0021] Figure 1 This is a flowchart of a database table data reorganization method according to an embodiment of this disclosure.

[0022] Figure 2 This is a schematic diagram of the data storage structure of the database node in an embodiment of this disclosure.

[0023] Figure 3 This is a schematic diagram of the data storage structure of a database node according to another embodiment of the present disclosure.

[0024] Figure 4 This is a schematic diagram of a distributed database according to an embodiment of this disclosure.

[0025] Figure 5 This is a block diagram of a data reorganization apparatus for a database table according to an embodiment of the present disclosure.

[0026] Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present disclosure. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0028] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. Additionally, the term “at least one” herein means any combination of at least two of any one or more of a plurality.

[0029] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0030] To enable those skilled in the art to better understand the technical solutions in the embodiments of this disclosure, and to make the above-mentioned objectives, features and advantages of the embodiments of this disclosure more apparent and understandable, the technical solutions in the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0031] In databases, it's often necessary to change the storage format of database tables. Scenarios requiring changes to storage format include, but are not limited to, receiving Data Definition Language (DDL) operations such as modifying primary keys, column types, or character sets. When changing the storage format, the database table needs to be reorganized; that is, the data in the database table is rewritten according to the new storage format.

[0032] In related technologies, data reorganization is generally performed using ordinary transaction insertion. This ordinary transaction approach requires sequentially executing processes such as recording data from the source table into memory, obtaining the transaction version number, concurrency control, and writing the data in memory to disk. This process involves multiple compactions between the data in memory and on disk. Furthermore, ordinary transactions process dependent objects row by row, which is time-consuming. In summary, data reorganization using ordinary transactions incurs significant overhead.

[0033] Based on this, this disclosure provides a method for data reorganization of a database table, see [link to relevant documentation]. Figure 1 The method includes:

[0034] Step 102: In response to a DDL operation on the source table, read the data in the source table; the DDL operation can change the storage structure of the source table;

[0035] Step 104: Sort the read data, write the sorted data into data blocks to the disk of the database node where the target table is located, and record the storage address of each data block on the disk of the database node where the target table is located.

[0036] Step 106: After all data blocks have been written, generate the SSTable of the target table based on the storage address of each data block. The SSTable of the target table includes each data block.

[0037] In this embodiment, when the storage structure of the source table changes, the data in the source table is sorted using a sorting method, and the sorted data is written to the disk of the database node where the target table resides, in data blocks. Since the storage address of each data block on the disk is recorded, after all data blocks have been written, the SSTable of the target table can be directly generated based on the storage addresses of each data block. This eliminates the need to write data to memory using a normal transaction method and then write the data in memory to disk, thereby saving the overhead of the data reorganization process and improving the data reorganization efficiency. The solution of this embodiment will be described in detail below.

[0038] In step 102, the source table can be any database table in the database, such as an order table, which can be used to record information such as the time, order number, and order amount of a user's purchase through an online platform. In some embodiments, the database is a distributed database, and the source table can be a database table on any database node of the distributed database. As one implementation, the source table can include multiple partitions (called first partitions), and different first partitions can be stored on different database nodes of the distributed database. The first partitions included in the source table can be determined based on the DDL operation that creates the source table. Assuming that the source table includes M rows and N columns, it can be divided into k first partitions, each first partition including M / k rows and N / k columns from the source table. In addition to uniformly dividing the first partitions, the different first partitions can also be of unequal size, and this disclosure does not limit this.

[0039] DDL operations can be performed on the source table. These DDL operations can be any operations that change the storage structure of the source table, such as modifying the primary key, column types, or character set. Data in the source table can be read in response to DDL operations. When the source table is stored on a single database node, data can be read from that database node. When the source table comprises multiple first partitions, and these first partitions are stored on different database nodes, the data from each first partition can be read separately, and then combined to retrieve the data from the source table.

[0040] In step 104, the data read from the source table can be sorted based on DDL operations on the source table. If the source table includes multiple first partitions, and different first partitions are stored on different first database nodes in the distributed database, the data of the first partitions stored in each first database node can be read separately, and the data of the first partitions read from each first database node can be sorted. This DDL operation can change the storage structure of the source table, for example, changing the storage order of at least two rows and / or at least two columns in the source table. Taking an order table as an example, assuming that the orders in the order table are sorted in descending order of order amount, a DDL operation could be to sort the orders in the order table in ascending order of order amount. Alternatively, assuming that the primary key of the order table is the order number, and the fields in the order table from left to right are order number, order amount, and order time, a DDL operation could be to use the order time as the primary key, and the sorted fields from left to right would be order time, order number, and order amount. Alternatively, a DDL operation could be to change the number of first partitions included in the source table. It is understood that the above is merely an illustrative example, and the sorting method of the embodiments disclosed herein is not limited thereto.

[0041] After sorting the read data, the sorted data can be divided into one or more data blocks, each of which may include at least one row of sorted data. The size of each data block can be fixed, for example, 16KB or 2MB. In some embodiments, the sorted data can be divided into one or more macroblocks, each macroblock being 2MB in size; further, each macroblock can be divided into multiple microblocks, each microblock being 16KB in size. In this case, the sorted data can be written to the disk of the database node where the target table resides, in units of macroblocks. That is, a 2MB macroblock is written to the disk of the database node where the target table resides each time. In some embodiments, the microblocks can also be compressed and organized into macroblocks, and the number of microblocks included in each macroblock can be variable. Besides the above-described methods of dividing data blocks, other methods can also be used to divide the sorted data into data blocks, for example, the sorted data can be divided into multiple data blocks of unequal sizes, which will not be described in detail in this disclosure.

[0042] For each data block written, the storage address of that data block on the disk of the database node where the target table resides can be recorded. The target table can be a pre-created database table or a database table created in response to a DDL operation on the source table. For simplicity, the disk of the database node where the target table resides will be referred to as the target disk. Each data block can be written to contiguous or non-contiguous storage addresses on the target disk. The recorded storage address can be either the offset of the data block's storage address on the target disk or the actual storage address of the data block on the target disk. When the data block size is not fixed, the size of each data block can also be recorded, and the data block size can be associated with its storage address on the target disk.

[0043] In some embodiments, the target table includes multiple second partitions, with different second partitions stored on different second database nodes in a distributed database. The number of second partitions in the target table can be determined based on DDL operations performed on the source table. When the target table includes multiple second partitions, the sorted data can be written to the disk of the second database node where the target table resides, block by block, based on the second partition to which the sorted data belongs. For example, assuming that the second partition to which a certain column of the sorted data belongs is located is database node A, then that column of data can be written to the disk of database node A.

[0044] In step 106, after writing the last data block, an SSTable including all the data blocks can be generated. In some embodiments, the data storage structure of the database node can be implemented based on an LSM Tree; for details, see [link to relevant documentation]. Figure 2The data storage structure of a database node, from top to bottom, includes three layers: a memtable, a Minor SSTable, and a Major SSTable. The memtable is an in-memory data structure within the LSM Tree, stored in the database node's memory. The Minor SSTable is a data structure within the LSM Tree persisted to disk; it can consist of multiple macroblocks and has a relatively small data volume. There may be multiple Minor SSTables. The Major SSTable is a data structure within the LSM Tree persisted to disk, located at the bottom layer of the LSM Tree, and has a larger data volume. There is generally only one Major SSTable. Based on the above data storage structures, the SSTable generated in this step can be the Major SSTable of the target table. Because this embodiment directly generates the Major SSTable at the bottom layer of the LSM Tree, the merging overhead of the LSM Tree is reduced, thereby reducing CPU and I / O overhead.

[0045] Data stored in a database node can be queried in a top-down order according to the data storage structure of that database node. For example, when querying target data in a target table of a database node, the target data can be queried first from the database node's in-memory table. If the target data is not found in the in-memory table, it can then be queried from the database node's MinorSSTable. If the target data is not found in either the database node's in-memory table or the minor SSTable, it can then be queried from the database node's major SSTable.

[0046] In some embodiments, the data blocks of the Major SSTable used to generate the target table can be obtained through compression. For example, in the aforementioned embodiments, the data blocks can be macroblocks, which may include multiple compressed microblocks. In this case, on the one hand, data compression can reduce the disk space occupied by the Major SSTable; on the other hand, due to the adoption of a three-layer data storage structure, the data query process is performed in a top-down order according to the data storage structure of the database nodes. Therefore, during the query process, queries are first performed in the uncompressed memory table and minor SSTable. Only when no relevant data is found in the memory table and minor SSTable is it necessary to query data from the major SSTable. Therefore, even if the data in the major SSTable has been compressed, the impact on the data query process is relatively small.

[0047] Furthermore, to facilitate recording storage addresses, an additional layer of data storage structure can be added beyond the aforementioned three-layer data storage structure to record the storage addresses of each data block, such as... Figure 3 As shown. The newly added data storage structure can be an in-memory data storage structure. To easily distinguish it from the existing memtable in memory, the new data storage structure can be called a DDL memtable. In this way, only the storage address of each data block needs to be recorded in memory, instead of recording the data blocks themselves, thereby reducing memory usage.

[0048] In the above embodiments, the data read from the source table can be existing data in the source table, that is, data stored in the source table before performing DDL operations on the source table. This existing data can eventually be written to the majorSSTable of the target table. Furthermore, in addition to existing data, the source table can also include incremental data, that is, data stored in the source table after performing DDL operations on the source table. Specifically, the incremental data in the source table can be written to the in-memory table and minor SSTable of the database node where the target table resides, without needing to be written to the major SSTable of each database node. Specifically, the incremental data in the source table can be written to the in-memory table of the database node where the target table resides, and the in-memory table can be dumped into a minor SSTable. In this way, if the data blocks included in the major SSTable are compressed, decompression processing is not required for each write, thereby improving data write performance.

[0049] In some embodiments, the source table and the target table are database tables in different namespaces, and the namespaces of the source table and the target table are isolated from each other. Optionally, a flag bit can be set for each namespace, wherein namespaces with different flag bits are isolated from each other, and namespaces with the same flag bit are not isolated from each other. The length of the flag bit can be determined based on the number of namespaces. For example, assuming the number of namespaces is 2, a 1-bit flag bit can be used, such as "0" and "1". In practical applications, the namespace of the source table can be a namespace visible to the user, and the namespace of the target table can be a namespace invisible to the user, that is, the table in the namespace of the target table is invisible. Different flag bits can represent the visibility state of a namespace. For example, a flag bit "0" indicates that it is invisible to the user, and a flag bit "1" indicates that it is visible to the user.

[0050] By setting the source table and the target table in different namespaces, on the one hand, the data restructuring process can be carried out without the user's awareness; on the other hand, the target table will not occupy the name in the namespace of the source table. That is, the namespace where the source table is located can include database tables with the same naming convention as the namespace where the target table is located.

[0051] If the source table and the target table are database tables in different namespaces, after executing step 106, the target table can be moved to the namespace where the source table is located. Specifically, the name of the target table can be changed to the name of the source table, and the flag of the namespace where the target table is located can be changed to the same flag as the flag of the namespace where the source table is located.

[0052] For ease of processing, when changing the name of the target table, the name of the target table can be directly swapped with the name of the source table. It's understood that name swapping is only one optional implementation. In other implementations, the name of the target table can be changed to the name of the source table, and the name of the source table can be changed to another name to prevent name duplication. For example, assuming the source table name was 'x' and the target table name was 'y' before the name change, the names of the target table and source table can be swapped, resulting in 'y' for the source table and 'x' for the target table. Alternatively, the name of the target table can be changed to 'x', and the name of the source table can be changed to 'z'. Furthermore, before changing the name of the target table to the name of the source table, write locks can be set for both the target table and the source table. After both the source table and the target table are successfully set, the name of the target table can be changed to the name of the source table.

[0053] After moving the target table to the namespace where the source table is located, the source table can also be deleted.

[0054] In some embodiments, the target table includes multiple target table replicas, which reside on multiple database nodes in a distributed database. See [link to relevant documentation]. Figure 4 A distributed database consists of multiple database nodes, including a master node and at least one backup node. The diagram illustrates a distributed database with three database nodes, where node 2 is the master node and nodes 1 and 3 are the backup nodes. It is understood that this is merely an illustrative example, and in other embodiments, the number of database nodes in the distributed database is not limited to this.

[0055] To achieve purposes such as redundancy backup, multiple database nodes can include replicas of the same target table. When writing sorted data to the disk of the database node containing the target table in data blocks, the sorted data needs to be synchronized to the disks of each database node containing a replica of the target table in data blocks to maintain consistency among the replicas of the target table.

[0056] In some embodiments, the sorted data can be written to the target table replica of the master node in data blocks, and the data blocks written to the target table replica of the master node can be written to the redo log of the master node. Then, the redo log of the master node is synchronized to the redo logs of each backup node, so that each backup node writes the sorted data to its own disk in data blocks based on its own redo log.

[0057] The redo log records every write operation to the database table. By replaying the redo log, the database can reconstruct complete data. This embodiment uses the redo log to synchronize data between the target table replicas on each standby node and the target table replica on the primary node. The synchronization process does not require re-running transaction logic; therefore, the redo log does not need to record information such as transaction version numbers, reducing the amount of data in the redo log and thus reducing bandwidth consumption during target table replica synchronization. Furthermore, data synchronization using the redo log can reuse the existing redo log transmission capabilities, improving development efficiency.

[0058] In some embodiments, data blocks can be written to the redo log of the primary node in microblocks, and the redo log of the primary node can be synchronized to the standby node. When the data blocks in the redo log of the standby node accumulate to the size of a macroblock, the macroblock in the standby node is written to the disk of the standby node.

[0059] The above embodiments describe methods for migrating data from a source table to a target table. In addition, dependent objects from the source table can also be migrated to the target table. These dependent objects include, but are not limited to, at least one of indexes, constraint information, foreign keys, and triggers.

[0060] If the dependent objects include indexes, an index for the target table can be generated. For example, an index for the target table can be generated based on the SSTable generated in step 106. As another example, if the target table includes incremental data from the source table, and this incremental data is written to an in-memory table on the database node where the target table resides, an index for the target table can be generated based on both the SSTable and the in-memory table.

[0061] Furthermore, when the target table comprises multiple target table replicas, and these replicas reside on multiple database nodes in a distributed database, an index for the target table replicas of the master node can be created based on the SSTable of the target table on the master node among the multiple database nodes. The index for the target table replicas is written to the redo log of the master node, and the redo log of the master node is synchronized to each standby node among the multiple database nodes, so that each standby node creates its own index for the target table replica based on its own redo log. The method of synchronizing the index using the redo log is similar to the method of synchronizing the target table replicas using the redo log, and will not be elaborated here. The redo log used for synchronizing the index and the redo log used for synchronizing the target table replicas can be the same or different; this disclosure does not impose any restrictions on this.

[0062] When the dependent objects include target items such as constraint information and / or foreign keys, second target items corresponding to the first target item in the target table can be read in batches based on the SSTable of the target table. Each time a second target item is read, it corresponds to at least one type of data in the target table. The fields included in the first target item and the corresponding second target item are the same. Taking an order table including order number, order amount, and order time as an example, the order number, order amount, and order time are data of different types. For example, assuming the first target item in the source table is the order number, the second target item in the target table is also the order number.

[0063] When batch reading the second target item, you can read the order number in the first batch read, the order amount in the second batch read, and the order time in the third batch read. Alternatively, you can read both the order number and the order amount in the first batch read, and then read the order time in the second batch read.

[0064] Then, a verification vector can be generated based on the data corresponding to the read second target item, and the verification vector for each type of data in the read second target item can be verified. Each element in a verification vector is of the same data type. For example, when the data read includes order number and order amount data, a first verification vector can be generated based on the order number data, and a second verification vector can be generated based on the order amount data. The elements in the first verification vector represent the order numbers of different orders, and the elements in the second verification vector represent the order amounts of different orders. Then, the first verification vector can be verified based on the verification logic corresponding to the first verification vector in the second target item, and the second verification vector can be verified based on the verification logic corresponding to the second verification vector in the second target item. For example, the verification logic for the order number could be that the length of the order number is equal to a preset length, and the verification logic for the order amount could be that the numerical range of the order amount is within a preset range. Then, each order number in the first verification vector can be verified based on the length of the order number, and each order amount in the second verification vector can be verified based on the numerical range of the order amount. Taking the first verification vector as an example, if the length of each order number in the first verification vector is equal to the preset length, then the verification passes; if the length of any order number in the first verification vector is not equal to the preset length, then the verification of that order number fails.

[0065] For example, assuming the target table includes a foreign key, and that foreign key is the primary key in the table referenced by the target table, a third check vector can be generated based on the foreign key in the target table, and a fourth check vector can be generated based on the primary key in the table referenced by the target table. Then, the third and fourth check vectors are validated based on the validation logic between them. This validation logic can be that the difference between the third and fourth check vectors is empty. If the difference between the third and fourth check vectors is not empty, the validation fails; otherwise, the validation succeeds.

[0066] In the above embodiments, after generating the SSTable of the target table, the second target items of the target table are processed in batches directly. Therefore, it is not necessary to process the second target items row by row, thus improving the processing efficiency of the second target items. It is understood that the above is only an illustrative example. In other embodiments, other validation logic can be used to validate the second target items, and the number of validation logic types corresponding to the same type of data can be greater than 1.

[0067] Compared to the method of processing dependent objects row by row in related technologies, this embodiment reads constraint information in batches and performs batch verification according to the data type. On the one hand, batch verification can utilize the principle of locality in computers to increase the hit rate of instruction and data caches; on the other hand, it can utilize vectorization technology to improve verification efficiency, thereby improving verification performance.

[0068] When the dependent objects include triggers, triggers can be created in the target table. The operation of creating a trigger in the target table can be included in the same transaction as various operations performed after writing the sorted data into data blocks to the disk of the database node where the target table resides. This ensures that the trigger is fired on the target table only after data reorganization is complete, avoiding performance issues caused by triggers firing on the target table during the data reorganization process.

[0069] For example, the operations of creating a trigger in the target table, setting a write lock on the source table, changing the name of the target table to the name of the source table, and changing the flag of the namespace containing the target table to the same flag as the flag of the namespace containing the source table can be included in the same transaction. This transaction can be denoted as:

[0070] Transaction 1:

[0071] Set a write lock on the source table;

[0072] Change the name of the target table to the name of the source table;

[0073] Change the flag bit of the namespace where the target table is located to the same flag bit as the flag bit of the namespace where the source table is located;

[0074] Create a trigger in the target table.

[0075] Transaction 1 will only complete if all operations within it are executed. If any operation in Transaction 1 fails, then Transaction 1 will fail. This ensures the atomicity of transactions.

[0076] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0077] See Figure 5 This disclosure also provides a data reorganization apparatus for a database table, the apparatus comprising:

[0078] The reading module 502 is used to read data from the source table in response to a DDL operation on the source table; the DDL operation can change the storage structure of the source table.

[0079] The recording module 504 is used to sort the read data, write the sorted data into data blocks to the disk of the database node where the target table is located, and record the storage address of each data block on the disk of the database node where the target table is located.

[0080] The generation module 506 is used to generate the SSTable of the target table based on the storage address of each data block after all data blocks have been written. The SSTable of the target table includes the data blocks.

[0081] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0082] This specification also provides a computer device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described in any of the foregoing embodiments.

[0083] Figure 6 This diagram illustrates a more specific hardware structure of a computing device provided in an embodiment of this specification. The device may include a processor 602, a memory 604, an input / output interface 606, a communication interface 608, and a bus 610. The processor 602, memory 604, input / output interface 606, and communication interface 608 are interconnected internally via the bus 610.

[0084] The processor 602 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor 602 may also include a graphics card, such as an Nvidia Titan X graphics card or a 1080Ti graphics card.

[0085] The memory 604 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 604 can store the operating system and other service programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 604 and is called and executed by the processor 602.

[0086] The input / output interface 606 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0087] The communication interface 608 is used to connect the communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0088] Bus 610 includes a pathway for transmitting information between various components of the device, such as processor 602, memory 604, input / output interface 606, and communication interface 608.

[0089] It should be noted that although the above-described device only shows the processor 602, memory 604, input / output interface 606, communication interface 608, and bus 610, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0090] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0091] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0092] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this specification can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this specification, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this specification.

[0093] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can be a personal computer, laptop computer, cellular phone, image acquisition device phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0094] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. When implementing the embodiments of this specification, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0095] The above description is merely a specific implementation of the embodiments of this specification. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles of the embodiments of this specification, and these improvements and modifications should also be considered within the protection scope of the embodiments of this specification.

Claims

1. A method for data reorganization of a database table, the method comprising: In response to a DDL operation on the source table, read the data from the source table; The DDL operation can change the storage structure of the source table; The read data is sorted, and the sorted data is written to the disk of the database node where the target table is located in data blocks. The storage address of each data block on the disk of the database node where the target table is located is recorded. After all data blocks have been written, the SSTable of the target table is generated based on the storage address of each data block. The SSTable of the target table includes each data block and is the Major SSTable in the data storage structure of the database node where the target table is located.

2. The method according to claim 1, wherein the source table and the target table are database tables in different namespaces, and the namespace where the source table is located is isolated from the namespace where the target table is located; the method further includes: After generating the SSTable of the target table based on the storage address of each data block, the target table is moved to the namespace where the source table is located.

3. The method according to claim 2, wherein each namespace includes a flag bit, and namespaces with different flag bits are isolated from each other; the step of moving the target table to the namespace where the source table resides includes: Change the name of the target table to the name of the source table; as well as Change the flag of the namespace containing the target table to the same flag as the flag of the namespace containing the source table.

4. According to the method of claim 2, the tables in the namespace where the target table is located are in an invisible state.

5. The method according to claim 2, further comprising: After moving the target table to the namespace where the source table is located, delete the source table.

6. The method according to claim 1, wherein the target table comprises multiple target table replicas, the multiple target table replicas are located on multiple database nodes in a distributed database, the multiple database nodes including a master node and at least one backup node; the step of writing the sorted data into data blocks to the disk of the database node where the target table is located comprises: The sorted data is written to the target table replica of the master node in data blocks, and the data blocks written to the target table replica of the master node are written to the redo log of the master node. The redo log of the master node is synchronized to the redo log of each backup node, so that each backup node writes the sorted data into its own disk in data blocks based on its own redo log.

7. The method according to claim 1, wherein the source table comprises multiple first partitions, and different first partitions are stored on different first database nodes in a distributed database; the sorting of the read data includes: Read the data of the first partition stored in each of the first database nodes respectively; Sort the data of the first partition read from each first database node.

8. The method according to claim 7, wherein the target table comprises multiple second partitions, and different second partitions are stored on different second database nodes in a distributed database; the step of writing the sorted data into data blocks to the disk of the database node where the target table resides comprises: Based on the second partition to which the sorted data belongs, the sorted data is written to the disk of the second database node where the target table is located, in data blocks.

9. The method according to claim 1, wherein the source table includes a first target item, the first target item including constraint information and / or foreign keys of the source table; the method further includes: Based on the target table, the second target item corresponding to the first target item in the target table is read in batches using SSTable; Each time a second target item is read, it corresponds to at least one type of data in the target table; A verification vector is generated based on the data corresponding to the second target item read; each element in a verification vector is of the same type. Verify the check vector for each type of data in the second target item being read.

10. The method of claim 3, wherein the source table includes a trigger; and the step of changing the name of the target table to the name of the source table includes: Set write locks for the source table and the target table, and after both the source table and the target table are successfully set, change the name of the target table to the name of the source table; The method further includes: Create a trigger in the target table; The operations of creating a trigger in the target table, setting a write lock on the source table, changing the name of the target table to the name of the source table, and changing the flag of the namespace where the target table is located to the same flag as the flag of the namespace where the source table is located are all included in the same transaction.

11. The method according to claim 1, wherein the target table comprises multiple target table replicas, the multiple target table replicas are located on multiple database nodes in a distributed database, the multiple database nodes including a master node and at least one backup node; the method further comprises: Create an index for the target table replica of the primary node; Write the index of the target table replica into the redo log of the master node; The redo logs of the primary node are synchronized to each standby node, so that each standby node can create an index of its target table replica based on its own redo logs.

12. The method according to claim 1, wherein the storage address of each data block on the disk of the database node where the target table is located is recorded in the data storage structure in the memory of the database node where the target table is located.

13. A data reorganization apparatus for a database table, the apparatus comprising: The read module is used to read data from the source table in response to DDL operations on the source table; The DDL operation can change the storage structure of the source table; The recording module is used to sort the read data, write the sorted data into data blocks to the disk of the database node where the target table is located, and record the storage address of each data block on the disk of the database node where the target table is located. The generation module is used to generate the SSTable of the target table based on the storage address of each data block after all data blocks have been written. The SSTable of the target table includes each data block and is the Major SSTable in the data storage structure of the database node where the target table is located.

14. A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method of any one of claims 1 to 12.

15. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method of any one of claims 1 to 12.

Images