Data table merging method, apparatus, device, and medium

Abstract

One or more embodiments of the specification provide a data table merging method, device, equipment and medium. The specification generates a plurality of first merging tasks based on a plurality of column groups to be merged, so that the plurality of column groups can be merged based on the plurality of first merging tasks to obtain a merged first data table, thereby the merging task of the plurality of column groups can be split into a plurality of first merging tasks to avoid data table merging failure caused by too large memory occupation of a single merging task, and improve the success rate of data table merging.

Description

Technical Field

[0001] This specification relates to the field of database technology, and more particularly to a data table merging method, apparatus, device, and medium. Background Technology

[0002] Today, column storage has become a key technology in scenarios such as big data analytics, data warehousing, and real-time data analysis. Column storage has advantages such as low input / output (IO) overhead, high compression ratio, support for efficient queries, and memory saving, and can provide high-performance data processing and query capabilities to meet the ever-increasing data volume and query demands.

[0003] Columnar data is usually static and difficult to update in place. The Sorted String Table (SSTable) in the Log Structured Merge Tree (LSM-Tree) is also static, making SSTable a natural fit for implementing columnar storage.

[0004] When using SSTable to implement column storage, column groups can be set as needed. Each column group can include multiple columns, and each column group corresponds to an SSTable. In this case, how to merge column storage SSTables becomes an urgent problem to be solved. Summary of the Invention

[0005] In view of the above, one or more embodiments of this specification provide a data table merging method, apparatus, device and medium.

[0006] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0007] According to a first aspect of one or more embodiments of this specification, a data table merging method is proposed, comprising:

[0008] Based on the multiple column groups to be merged, multiple first merge tasks are generated. For any first merge task, the first merge task is used to merge some column groups in the multiple column groups. The column groups corresponding to different first merge tasks are different.

[0009] Based on the multiple first merging tasks, the multiple column groups are merged to obtain the merged first data table.

[0010] According to a second aspect of one or more embodiments of this specification, a data table merging apparatus is provided, comprising:

[0011] The generation module is used to generate multiple first merge tasks based on multiple column groups to be merged. For any first merge task, the first merge task is used to merge a portion of the multiple column groups. Different first merge tasks correspond to different column groups.

[0012] The merging module is used to merge the multiple column groups based on the multiple first merging tasks to obtain a merged first data table.

[0013] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising:

[0014] processor;

[0015] Memory used to store processor-executable instructions;

[0016] The processor implements the method as described in the first aspect by running the executable instructions.

[0017] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the method as described in the first aspect.

[0018] This manual generates multiple first merge tasks based on multiple column groups to be merged. This allows multiple column groups to be merged based on multiple first merge tasks to obtain a merged first data table. This splits the merge task of multiple column groups into multiple first merge tasks, thereby avoiding data table merge failure due to excessive memory consumption of a single merge task and improving the success rate of data table merge. Attached Figure Description

[0019] Figure 1 This is a flowchart of a data table merging method provided in an exemplary embodiment.

[0020] Figure 2 This is a schematic diagram of a table structure provided in an exemplary embodiment.

[0021] Figure 3 This is a schematic diagram of the structure of a data table provided in an exemplary embodiment.

[0022] Figure 4 This is a schematic structural diagram of a computing device provided in an exemplary embodiment.

[0023] Figure 5 This is a block diagram of a data table merging apparatus provided in an exemplary embodiment. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0025] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0026] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0027] To facilitate understanding, the technical terms used in this manual will be introduced below.

[0028] Memtable: When writing data to a database, it is first written to memory, and the corresponding data structure is called a Memtable.

[0029] Sorted String Table (SSTable): A data structure that persists data to disk, providing read-only access. SSTables can be divided into two levels: Minor SSTables and Major SSTables. Minor SSTables contain frozen data from MemTables, while Major SSTables generate baseline data through a major merge of MemTables and Minor SSTables. The baseline data contains complete row information.

[0030] Merging and dumping: In the LSM-Tree architecture, data is generally divided into two parts: MemTable and SSTable. When the amount of data in MemTable exceeds a certain threshold, the data in MemTable needs to be dumped to SSTable to free up memory. This process is called dumping. Dumping will generate a new SSTable. When the number of dumps exceeds a certain threshold, or during the daily business off-peak period, the baseline SSTable and the incremental SSTable of subsequent dumps will be merged into one SSTable. This process is called merging.

[0031] Column Group: A Column Group consists of several columns, representing a set of columns. If a Column Group contains all the columns of a table, it is a row-based storage; if each column of a table is a separate Column Group, then it is the "purest" column-based storage.

[0032] After introducing the relevant technical terms in this manual, the solution described in this manual will be described in detail below.

[0033] In related technologies, each Column Group exists as a separate SSTable in the storage layer. That is, for each independent Column Group, an independent SSTable can be built for it. The SSTable corresponding to a Column Group will only contain the data of the corresponding Column Group.

[0034] When merging columnar data, the merging process is not simply a matter of physically splicing the data from several SSTables together. Instead, it requires sorting the data in the SSTables according to the key field (Rowkey), and then writing other non-primary key column groups into a new SSTable based on the sorting results. This process merges the column groups and ensures that the merged SSTable meets the sorting requirements of the Rowkey.

[0035] However, when there are many Column Groups, the number of SSTables that need to be written at the same time will be large, resulting in excessive memory consumption. This may lead to SSTable merging failure due to insufficient memory.

[0036] In view of this, this specification aims to provide a data table merging method to combine multiple SSTables corresponding to Column Groups into a single SSTable. The solution provided in this specification allows the merging task of the first data table to be split into multiple first merging tasks, thereby avoiding data table merging failures due to excessive memory consumption by a single merging task and improving the success rate of data table merging.

[0037] The above database operation methods can be executed by computing devices, which can be servers, such as a single server, multiple servers, a server cluster, a cloud computing platform, etc. This manual does not limit the type or number of computing devices.

[0038] Optionally, the data table merging method provided in this specification can be used to merge data tables in various types of databases. For example, the data table merging method provided in this specification can be applied to distributed databases (such as distributed relational databases, columnar storage databases, etc.), key-value databases, cloud data warehouses, etc. This specification does not limit the specific type of database to which it applies.

[0039] The above is merely an illustrative description of the application scenarios of this specification and does not constitute a limitation on the application scenarios of this specification. In many possible implementations, the solutions provided in this specification can be applied to more databases that use columnar storage as the storage engine. This specification does not limit the specific application scenarios.

[0040] After introducing the application scenarios of this manual, the specific implementation process of this manual will be described next.

[0041] See Figure 1 , Figure 1 This is a flowchart of a data table merging method provided in an exemplary embodiment, such as... Figure 1 As shown, the method includes:

[0042] Step 101: Based on the multiple column groups to be merged, generate multiple first merge tasks. For any first merge task, the first merge task is used to merge some column groups in the multiple column groups. The column groups corresponding to different first merge tasks are different.

[0043] Optionally, the multiple column groups to be merged can be split into multiple batches, thereby generating a first merge task based on the column groups in the same batch. Similarly, each batch of column groups can generate a corresponding first merge task, thus obtaining multiple first merge tasks.

[0044] Step 102: Based on multiple first merge tasks, merge multiple column groups to obtain the merged first data table.

[0045] The solution provided in this manual can split the merging task of multiple column groups into multiple first merging tasks, thereby avoiding the failure of data table merging due to excessive memory consumption of a single merging task and improving the success rate of data table merging.

[0046] After introducing the basic implementation process of this manual, the optional implementation methods of this manual will be introduced below.

[0047] In some embodiments, when generating multiple first merge tasks based on multiple column groups to be merged in step 101, it can be implemented in the following way:

[0048] A first merge task is generated based on a first set number of column groups from multiple column groups, resulting in multiple first merge tasks.

[0049] Optionally, a first set number of column groups can be divided into a batch to divide multiple column groups into multiple batches, thereby generating a first merge task based on the column groups in each batch, generating a first merge task for the first set number of column groups, and so on, to generate multiple first merge tasks.

[0050] The first set quantity can be any value, and this specification does not limit the specific value of the first set quantity.

[0051] In some embodiments, when merging multiple column groups based on multiple first merge tasks to obtain a merged first data table, step 102 can be achieved through the following steps:

[0052] Step 1021: Merge the column groups corresponding to each first merge task.

[0053] In one possible implementation, multiple first merge tasks can be executed sequentially to merge the column groups corresponding to each first merge task.

[0054] In another possible implementation, some of the first merge tasks in multiple first merge tasks can be executed in parallel to merge the column groups corresponding to each first merge task separately.

[0055] It should be noted that the column groups to be merged may include a first column group and a second column group. The first column group is the column group that includes the key field (Rowkey), and the second column group is the column group that does not include the key field.

[0056] For example, for a table with five columns (Rowkey1, Rowkey2, Col1, Col2, Col3), where Rowkey1 and Rowkey2 together form the primary key, (Rowkey1, Rowkey2), (Col1, Col2), and (Col2, Col3) can be set as a Column Group. The table structure can be found in the following diagram. Figure 2 , Figure 2 This is a schematic diagram of a table structure provided in an exemplary embodiment, such as... Figure 2 As shown, Rowkey1 and Rowkey2 form a Column Group (denoted as Column GroupRowkey), Col1 and Col2 form a Column Group (denoted as Column Group 1), and Col2 and Col3 form a Column Group (denoted as Column Group 2).

[0057] like Figure 2 The SSTable corresponding to the column group shown can be as follows: Figure 3 As shown, see Figure 3 , Figure 3 This is a schematic diagram of the structure of a data table provided in an exemplary embodiment, wherein Major SSTable Column Group Rowkey is the SSTable corresponding to ColumnGroup Rowkey, Major SSTable Column Group 1 is the SSTable corresponding to Column Group 1, and Major SSTable Column Group 2 is the SSTable corresponding to Column Group 2.

[0058] Optionally, regardless of whether the multiple first merge tasks are executed serially or in parallel, when merging the column groups corresponding to each first merge task, the first merge task corresponding to the first column group can be used first for merging and log data can be generated; then, based on the log data, the first merge task corresponding to the second column group can be used for merging.

[0059] By generating log data through the merging process based on key field column groups, the origin of the data in the merge result can be recorded, so that the merging of non-key field column groups can be carried out subsequently based on the log data.

[0060] Log data can be used to record the operation type, scope, and content of the merging process. For example, the format of log data can be shown in Table 1 below:

[0061] Table 1

[0062] Operation type (Log Type) Operating Scope (Parameters) Operation content (MergeLog) INSERT_ROW row_index Insert a row at the specified row offset. UPDATE_ROW row_index Update the row at the specified row offset. DELETE_ROW row_index Delete the line at the specified line offset.

[0063] Optionally, the first column group may include non-key fields in addition to key fields. When merging the first column group, in addition to merging the key fields to generate SSTable, other non-key fields should also be written into the new SSTable to directly complete the merging of the first column group.

[0064] Step 1022: Merge multiple second data tables to obtain the first data table.

[0065] The above embodiment illustrates the process of splitting the merging task of multiple column groups into multiple first merging tasks, and processing each of the multiple first merging tasks separately to achieve data table merging. In more possible implementations, if the number of multiple column groups is less than or equal to a first set number, the multiple column groups can be directly merged to obtain a first data table.

[0066] In other words, for multiple column groups to be merged, the number of column groups to be merged can be determined first. If the number of column groups is less than or equal to a first set number, they can be written simultaneously in one merge process. Since the number of column groups is small, there will be no situation where too many columns are written to the SSTable at the same time, resulting in excessive data usage and causing the data table to fail to merge. Furthermore, in this case, since only one merge process is involved, there is no need to generate log data. If the number of column groups is greater than the first set number, the merging task of multiple column groups can be split into multiple first merge tasks using the scheme provided in the above embodiment. These first merge tasks are then processed separately to achieve data table merging. That is, the merging task of multiple column groups is split into several batches, and each batch is responsible for merging several column groups. First, the batch containing the Rowkey is merged to generate log data, and the remaining batches process the non-primary key data by reading the log data. Using batches to execute the merge task can reduce the consumption of reading log data compared to merging each column group separately.

[0067] In some embodiments, if any first merge task fails, multiple second merge tasks are regenerated based on the multiple column groups to be merged.

[0068] Alternatively, when regenerating multiple second merge tasks based on multiple column groups to be merged, this can be achieved in the following way:

[0069] A second merge task is generated based on a second set number of column groups from multiple column groups, resulting in multiple second merge tasks. The second set number is less than the first set number.

[0070] Optionally, the second set number of column groups can be divided into a batch to divide multiple column groups into multiple batches, so that a second merge task can be generated based on the column groups in each batch to generate a second merge task for the second set number of column groups, and so on, to generate multiple second merge tasks.

[0071] The second set quantity can be any value, and this specification does not limit the specific value of the second set quantity.

[0072] Through the above embodiments, when the first merge task fails due to memory issues, the number of column groups scheduled in each batch can be reduced, so that sufficient memory can be guaranteed during the execution of each second merge task, thereby improving the success rate of data table merging.

[0073] Corresponding to the embodiments of the methods described above, this specification also provides corresponding device embodiments.

[0074] See Figure 4 , Figure 4 This is a schematic structural diagram of a computing device provided in an exemplary embodiment. Please refer to... Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, memory 408, and non-volatile memory 410, and may also include other hardware required for tasks. One or more embodiments of this specification can be implemented in software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into memory 408 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0075] This manual also provides a data table merging device; please refer to it. Figure 5 , Figure 5 This is a block diagram of a data table merging apparatus provided in an exemplary embodiment. This data table merging apparatus can be applied to, for example... Figure 4 The computing device shown is used to implement the technical solution of this specification. The data table merging device may include:

[0076] The generation module 501 is used to generate multiple first merge tasks based on multiple column groups to be merged. For any first merge task, the first merge task is used to merge some column groups in the multiple column groups. The column groups corresponding to different first merge tasks are different.

[0077] The merging module 502 is used to merge multiple column groups based on multiple first merging tasks to obtain a merged first data table.

[0078] In some embodiments, the generation module 501, when generating multiple first merge tasks based on multiple column groups to be merged, is used to:

[0079] A first merge task is generated based on a first set number of column groups from multiple column groups, resulting in multiple first merge tasks.

[0080] In some embodiments, the merging module 502 is further configured to directly merge the multiple column groups to obtain a first data table if the number of multiple column groups is less than or equal to a first preset number.

[0081] In some embodiments, the merging module 502, when used to merge multiple column groups based on multiple first merging tasks to obtain a merged first data table, is configured to:

[0082] Merge the column groups corresponding to each first merge task separately;

[0083] Multiple second data tables are merged to obtain the first data table.

[0084] In some embodiments, the merging module 502, when merging the column groups corresponding to each first merging task, is used for any of the following:

[0085] Multiple first merge tasks are executed sequentially to merge the column groups corresponding to each first merge task.

[0086] A portion of the first merge tasks are executed in parallel to merge the column groups corresponding to each first merge task.

[0087] In some embodiments, the plurality of column groups include a first column group and a second column group, wherein the first column group is a column group that includes a key field and the second column group is a column group that does not include a key field;

[0088] The merge module 502, when merging the column groups corresponding to each first merge task, is used to:

[0089] The first merge task corresponding to the first column group is used for merging, and log data is generated.

[0090] Based on the log data, the first merge task corresponding to the second column group is used for merging.

[0091] In some embodiments, the generation module 501 is further configured to regenerate multiple second merge tasks based on the multiple column groups to be merged if any first merge task fails.

[0092] In some embodiments, the generation module 501, when regenerating multiple second merge tasks based on multiple column groups to be merged, is used to:

[0093] A second merge task is generated based on a second set number of column groups in multiple column groups, so as to obtain multiple second merge tasks;

[0094] The second set quantity is less than the first set quantity, and the first set quantity is the number of column groups included in each first merge task.

[0095] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0096] In a typical configuration, a computer includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0097] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0098] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital video disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined in this article, computer-readable media do not include transient computer-readable media, such as modulated data signals and carrier waves.

[0099] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0100] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0101] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0102] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0103] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A method for merging data tables, wherein the data table is divided into multiple column groups to be merged, each column group contains at least one column of the data table, and each column group is constructed as a string table (SSTable) containing only the data of the column group; wherein, The plurality of column groups includes a first column group and a second column group, wherein the first column group includes a primary key field, and the second column group does not include a primary key field; the method includes: Based on the multiple column groups to be merged, multiple first merge tasks are generated. For any first merge task, the first merge task is used to merge some column groups in the multiple column groups. The column groups corresponding to different first merge tasks are different. Based on the multiple first merging tasks, the multiple column groups are merged to obtain the merged first data table; The step of merging the multiple column groups based on the multiple first merging tasks to obtain a merged first data table includes: Merging is performed based on the first merge task corresponding to the first column group; wherein, the merging process includes: sorting the data in the first column group according to the primary key field contained in the first column group; Based on the sorting results of the first column group, the merging process is performed based on the first merging task corresponding to the second column group.

2. The method according to claim 1, wherein generating multiple first merge tasks based on multiple column groups to be merged includes: The multiple column groups are divided into multiple batches, each batch containing a first set number of column groups, and each batch of column groups is generated into a first merge task to obtain the multiple first merge tasks.

3. The method according to claim 2, further comprising: If the number of the multiple column groups is less than or equal to the first set number, the multiple column groups are directly merged to obtain the first data table.

4. The method according to claim 1, wherein merging the plurality of column groups based on the plurality of first merging tasks to obtain a merged first data table includes: Merge the column groups corresponding to each first merge task separately; The first data table is obtained by merging the column combinations and results corresponding to each first merging task again.

5. The method according to claim 4, wherein merging the column groups corresponding to each first merging task includes any one of the following: The plurality of first merge tasks are executed sequentially to merge the column groups corresponding to each first merge task respectively; A portion of the multiple first merge tasks are executed in parallel to merge the column groups corresponding to each first merge task.

6. The method according to claim 1, wherein the merging process based on the sorting result of the first column group and the first merging task corresponding to the second column group includes: Based on the log data generated by the first merge task corresponding to the first column group, the second column group is merged.

7. The method according to claim 1, further comprising: If any of the first merge tasks fails, multiple second merge tasks are generated based on the multiple column groups to be merged.

8. The method according to claim 7, wherein regenerating multiple second merge tasks based on multiple column groups to be merged includes: A second merge task is generated based on a second predetermined number of column groups among the plurality of column groups, so as to obtain the plurality of second merge tasks; Wherein, the second set quantity is less than the first set quantity, and the first set quantity is the number of column groups included in each first merge task.

9. A data table merging apparatus, wherein the data table is divided into multiple column groups to be merged, each column group contains at least one column of the data table, and each column group is constructed as a string table SSTable containing only the data of the column group; wherein, The plurality of column groups includes a first column group and a second column group, wherein the first column group includes a primary key field, and the second column group does not include a primary key field; the device includes: The generation module is used to generate multiple first merge tasks based on multiple column groups to be merged. For any first merge task, the first merge task is used to merge a portion of the multiple column groups. Different first merge tasks correspond to different column groups. The merging module is used to merge the multiple column groups based on the multiple first merging tasks to obtain a merged first data table; The merged modules are specifically used for: Merging is performed based on the first merge task corresponding to the first column group; wherein, the merging process includes: sorting the data in the first column group according to the primary key field contained in the first column group; Based on the sorting results of the first column group, the merging process is performed based on the first merging task corresponding to the second column group.

10. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-8 by executing the executable instructions.

11. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-8.

Images