Combining method and related apparatus

Abstract

This application provides a merging method and related equipment. The method includes: when a trigger condition L is met, the computing node merges the SSTable of the (L-1)th layer. [L‑1，i] Move to the protected area; the compute node, based on the n SSTables of layer L, moves the SSTables of layer L-1. [L‑1，i] Divided into n partial files, the SSTable at level L-1 [L‑1，i] The key value ranges overlap with each SSTable in the Lth layer; establish the association between each partial file and each SSTable in the Lth layer, and any association is used to allocate data from the compute node to the SSTable. [L，N] Reading the key value as key K N When processing KV data, guide the compute nodes to SSTable. [L‑1，i] Reading the key value as key K N KV data, K N Belongs to the key value range [K] uN K dN ], 1≤N≤n.

Description

Technical Field

[0001] This application relates to the field of databases, and more particularly to a merging method and related equipment. Background Technology

[0002] A key-value database is a type of non-relational database that uses a simple key-value approach to store data. That is, a key-value database consists of multiple records, each containing a "key" and a "value". The "key" serves as a unique identifier for the record, allowing the database to find the corresponding "value" based on the "key". Both the "key" and "value" can be anything, ranging from simple objects to complex composite objects.

[0003] However, as the content stored in key-value databases increases, the existing key-value databases involve a very large number of records. Therefore, improving the efficiency of the database is very important. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a merging method and related equipment that can effectively reduce read / write amplification and improve database efficiency.

[0005] Firstly, a merging method is provided, including:

[0006] When trigger condition L is met, the computing node executes the task L to establish associations, and the SSTable at level L-1 is... [L-1，i] Move to the protected area, where L is a positive integer greater than 1;

[0007] The computing node is based on n SSTables in the Lth layer. [L，1] SSTable [L，2] , ...,SSTable [L，n] , the SSTable of the (L-1)th layer [L-1，i] The file is divided into n parts, where n is a positive integer, and the SSTable of the (L-1)th layer is... [L-1，i] The key value ranges overlap with each SSTable in the Lth layer;

[0008] The compute node establishes an association between each partial file and each SSTable of the Lth layer, wherein any association is used by the compute node to access the SSTable. [L，N] Reading the key value as key K N When processing KV data, the compute node is guided to the SSTable. [L-1，i] Reading the key value as key K N KV data, K NBelongs to the key value range [K] uN K dN ], 1≤N≤n.

[0009] In some possible designs, the method further includes:

[0010] If the conditions for true merging are met, the compute node can select the first target SSTable from the SSTable of level L and merge it with the corresponding part of the file in the protected area according to the selection strategy.

[0011] In some possible designs, the actual merging conditions include the number of SSTables in the protected area exceeding a first preset threshold and the number of SSTables in the Lth layer that establish associations with corresponding partial files in the protected area exceeding one or more of a second preset threshold.

[0012] In some possible designs, the factors influencing the selection strategy include: whether the first target SSTabel is read into the cache, and the number of associations established between the first target SSTabel and a portion of files in the protected area.

[0013] In some possible designs, the method further includes:

[0014] If the second target SSTable in the protected area meets the deletion condition, the second target SSTable is deleted, wherein the deletion condition may be that the reference count of the second target SSTable is equal to zero.

[0015] Secondly, a computing node is provided, comprising: a protection module, a partitioning module, and an association module. The protection module is used to execute an association establishment task L when a triggering condition L is met, thereby establishing an association relationship within the (L-1)th layer of the SSTable. [L-1，i] Move to the protected area, where L is a positive integer greater than 1;

[0016] The partitioning module is used to partition the data according to the n SSTables in the Lth layer. [L，1] SSTable [L，2] , ...,SSTable [L，n] , the SSTable of the (L-1)th layer [L-1，i] The file is divided into n parts, where n is a positive integer, and the SSTable of the (L-1)th layer is... [L-1，i] The key value ranges overlap with each SSTable in the Lth layer;

[0017] The association module is used to establish the association relationship between each partial file and each SSTable of the Lth layer, wherein any association relationship is used by the computing node to direct the data to the SSTable. [L，N] Reading the key value as key K N When processing KV data, the compute node is guided to the SSTable. [L-1，i] Reading the key value as key K N KV data, K N Belongs to the key value range [K] uN K dN ], 1≤N≤n.

[0018] In some possible designs, the node also includes a merging module.

[0019] The merging module is used to select the first target SSTable from the SSTable of layer L and merge it with the corresponding part of the files in the protected area according to the selection strategy, provided that the actual merging conditions are met.

[0020] In some possible designs, the actual merging conditions include the number of SSTables in the protected area exceeding a first preset threshold and the number of SSTables in the Lth layer that establish associations with corresponding partial files in the protected area exceeding one or more of a second preset threshold.

[0021] In some possible designs, the factors influencing the selection strategy include: whether the first target SSTabel is read into the cache, and the number of associations established between the first target SSTabel and a portion of files in the protected area.

[0022] In some possible designs, the node also includes a deletion module.

[0023] The deletion module is used to delete the second target SSTable in the protected area if the deletion condition is met, wherein the deletion condition may be that the reference count of the second target SSTable is equal to zero.

[0024] Thirdly, a non-transient readable storage medium is provided, including instructions that, when executed on a computing node, cause the computing node to perform the method as described in any of the first aspects.

[0025] Fourthly, a program product is provided, which, when read and executed by a merging node, will execute the method described in any of the first aspects.

[0026] Fifthly, a computing node is provided, including a processor and a memory, wherein the processor executes code in the memory to perform the method as described in any of the first aspects.

[0027] In the above scheme, when the trigger condition L is met, the computing node does not execute the merge task L and will not sink the KV data of layer L-1 to the KV data of layer L. Instead, it modifies the metadata to record the association between the KV data of layer L-1 and the KV data of layer L. Only when the actual merge condition is met will the KV data of layer L-1 be sinked to the KV data of layer L. This can effectively reduce the read and write amplification caused by merging. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0029] Figure 1 This is a schematic diagram of a key-value pair storage database involved in this application;

[0030] Figure 2 This is a schematic diagram of the execution of the merging task L involved in this application;

[0031] Figure 3 This is a schematic diagram illustrating the read / write amplification caused by the multiple executions of the merge task involved in this application;

[0032] Figure 4 This is a flowchart illustrating a merging method provided in this application;

[0033] Figure 5 This is a schematic diagram illustrating the relationship between a portion of the files at level L and the SSTable at level L in this application.

[0034] Figure 6 This is a schematic diagram showing that the protected area in this application contains multiple SSTables;

[0035] Figure 7 This is a schematic diagram of multiple SSTables at level L-1 in the same protected area of ​​the same SSTable at level L in this application;

[0036] Figure 8 This is a diagram illustrating the multiple triggering of an SSTable within the same protected area in this application;

[0037] Figure 9 This is a schematic diagram of the structure of a computing node provided in this application;

[0038] Figure 10 This is a schematic diagram of another computing node structure provided in this application. Detailed Implementation

[0039] See Figure 1 , Figure 1 This is a schematic diagram of a key-value pair storage database involved in this application. The key-value pair storage (KV) database of this embodiment includes two main parts: a memory part and an external storage part.

[0040] The in-memory portion includes readable and writable tables as well as non-writable tables. Readable and writable tables are for users to read or write key-value data, while non-writable tables are obtained by freezing the readable and writable tables. Both types of tables are in-memory data structures. Readable and writable tables allow both read and write operations, while non-writable tables only allow read operations. In HBase databases, readable and writable tables are referred to as Memstore, in Kudu databases as MemRowSets, and in Cassandra databases as Memtables, etc. Similarly, non-writable tables are referred to as immemstore in HBase databases, imMemRowSets in Kudu databases, and imMemtables in Cassandra databases, etc.

[0041] The external storage portion can include multiple persistent data files. Here, persistent data files refer to data structures in external storage. In HBase databases, persistent data files may be called HFiles; in Kudu databases, they may be called DiskRowSets; in Cassandra databases, they may be called sorted string tables (SSTables), and so on. Taking SSTables as an example, the external storage portion includes multiple levels, each containing one or more SSTables. Typically, the number of SSTables in lower levels is greater than the number in upper levels. SSTables are used to store a series of ordered key-value pairs (KV). Both the key and value are byte strings of arbitrary length. The KV data is written into the SSTable in an ordered manner according to configured comparison rules. The SSTable is internally divided into a series of blocks and contains necessary index information. This allows for both sequential reading of internal key-value records and fast location based on a specific key.

[0042] In addition, the external storage also stores metadata files. These metadata files continuously record updated metadata using an append-only approach, such as file additions and deletions, and column family changes. In HBase databases, this metadata file can be called the persistent database state change log file (manifest); in Kudu databases, it can be called the Kudumanifest (a persistent database state change log file); and in Cassandra databases, it can be called the lightweight persistent database state change log file (manifest.json), and so on. Taking the persistent data file as the Manifest file as an example, the Manifest file records the distribution of SSTables at different levels, the maximum and minimum keys of a single SSTable, and other necessary metadata.

[0043] In addition, the external storage section also stores log files and current files, among others. The log files ensure that key-value data is not lost during system failure recovery. Because records are written to the log file before being written to the in-memory Memtable, even if a system failure occurs and the data in the Memtable hasn't yet been migrated to the SSTable in the external storage section, the in-memory Memtable data structure can be recovered from the log file, preventing data loss. The current file is a text file used to store the filename of the most recent Manifest file.

[0044] The following introduction will use Memtable (a readable and writable table), imMemtable (a non-writable table), and SSTable (a persistent data file) as examples to illustrate the merging process in a key-value database.

[0045] When a write operation is required to write the key-value data to the key-value database, the compute node can first write the key-value data to the memtable. When the memtable is full, the readable and writable memtable is converted into a read-only immutable memtable.

[0046] When trigger condition 1 is met, the compute node executes merge task 1 to sink the KV data in the immutable Memtable to the first-level SSTable. After merge task 1 is completed, the KV database generates the first incremental Manifest file. This first incremental Manifest file records the distribution of the incremental SSTables at different levels after merge task 1, the maximum and minimum keys of a single SSTable, and other necessary metadata.

[0047] ...;

[0048] When trigger condition L is met, the compute node executes merge task L to merge the SSTables at level L-1 in the external storage and level L in the external storage, thereby sinking the key-value data from the SSTables at level L-1 in the external storage to the SSTables at level L. After merge task L is completed, the KV database generates an incremental Manifest file of level L. This Manifest file records the distribution of the incremental SSTables at different levels after merge task L, the maximum and minimum keys of a single SSTable, and other necessary metadata.

[0049] ...;

[0050] When trigger condition n is met, the compute node executes merge task n, merging the SSTables at level n-1 in the external storage and the SSTable at level n in the external storage, thus sinking the KV data from the SSTable at level n-1 in the external storage to the SSTable at level n. After merge task n is completed, the KV database generates an incremental n-th Manifest file. This n-th Manifest file records the distribution of the incremental SSTables at different levels after merge task n, the maximum and minimum keys of a single SSTable, and other necessary metadata.

[0051] Where n is the total number of levels in the external storage part, L is a certain level in the external storage part, and 1≤L≤n.

[0052] It can be understood that trigger condition 1, ..., trigger condition L, ..., trigger condition n are the conditions for triggering the execution of merge task 1, ..., merge task L, ..., merge task n, respectively. Trigger conditions 1, ..., trigger condition L, ..., trigger condition n can all be set according to actual needs. For example, trigger condition 1 could be that the data volume of the first-level SSTable reaches 10M, ..., trigger condition L could be that the data volume of the Lth-level SSTable reaches 1G, ..., trigger condition n could be that the data volume of the nth-level SSTable reaches 10G; no specific limitations are made here.

[0053] The following will use merge task L as an example to explain in detail how each merge task is completed. (Combined with...) Figure 2 As shown, the merge task L includes the following steps: the compute node obtains the first SSTable 110 of the (L-1)th layer, and the compute node obtains the second SSTable 120 of the Lth layer, wherein the key value ranges of the first SSTable 110 and the second SSTable overlap; the compute node merges the first SSTable 110 and the second SSTable 120 to generate the third SSTable 130 of the Lth layer; the first SSTable 110 is deleted from the SSTables of the (L-1)th layer, and the second SSTable 120 is deleted from the SSTables of the Lth layer.

[0054] After the compute node sinks the first SSTable of layer L-1 into the third SSTable of layer L, the compute node will delete the first SSTable. Therefore, when the compute node needs to read the key-value range of the original first SSTable, it will change from reading from the first SSTable to obtaining it from the third SSTable.

[0055] However, the above merging process can lead to read / write amplification, which will be explained in detail from two perspectives below.

[0056] The first type is read / write amplification caused by multiple merging operations. For clarity, a specific example will be used below.

[0057] like Figure 3 As shown, it is assumed that before the merger, the SSTables in the second layer include SSTable T21 and SSTable T22, the SSTables in the third layer include SSTable T31, SSTable T32 and SSTable T33, and the SSTables in the fourth layer include SSTable T41, SSTable T42 and SSTable T43.

[0058] like Figure 3 As shown in the dashed box in the upper left corner, during the first merge, SSTable T22 (level 2), SSTable T32 (level 3), and SSTable T33 (level 3) need to be merged to generate SSTable T34, SSTable T35, and SSTable T36 (level 3). The key values ​​of SSTable T22, SSTable T32, and SSTable T33 overlap. SSTable T22 (level 2) is then deleted from the level 2 SSTables, and SSTable T32 and SSTable T33 (level 3) are deleted from the level 3 SSTables. Therefore, after the first merge, the level 2 SSTables include SSTable T21, the level 3 SSTables include SSTable T31, SSTable T34, SSTable T35, and SSTable T36, and the level 4 SSTables include SSTable T41, SSTable T42, and SSTable T43.

[0059] like Figure 3As shown in the dashed box in the lower right corner, during the second merge, it is necessary to merge SSTables T34, T35, and T36 from the third level, and SSTables T42 and T43 from the fourth level, thereby generating SSTables T44, T45, T46, T47, and T48 from the fourth level. The keys of SSTables T34, T35, T36, T42, and T43 overlap. Then, SSTables T34, T35, and T36 from the third level are deleted from the third-level SSTables, and SSTables T42 and T43 from the fourth level are deleted from the fourth-level SSTables. Therefore, after the second merge, the SSTables at level 2 include SSTable T21, the SSTables at level 3 include SSTable T31, and the SSTables at level 4 include SSTable T41, SSTable T44, SSTable T45, SSTable T46, SSTable T47, and SSTable T48.

[0060] However, the two merges involved reading a total of eight SSTables: SSTable T22, SSTable T32, SSTable T33, SSTable T34, SSTable T35, SSTable T36, SSTable T42, and SSTable T43, and writing to eight SSTables: SSTable T34, SSTable T35, SSTable T36, SSTable T44, SSTable T45, SSTable T46, SSTable T47, and SSTable T48. In other words, moving SSTable T22 from the second-level SSTable to the fourth-level SSTable resulted in read / write amplification of eight SSTables.

[0061] The second type is read / write amplification caused by merging at the SSTable level. For clarity, a specific example will be used below. Assume the first SSTable 110 contains five key-value pairs (KV) with key values ​​of 10, 15, 27, 190, and 210, and the second SSTable 120 contains six key-value pairs (KV) with key values ​​of 1, 15, 27, 30, 50, and 90. In the second SSTable 120, only the key-value pairs with key values ​​of 15 and 27 need to be overwritten. However, during merging, the key-value pairs with key values ​​of 1, 30, 50, and 90 are also read and written, resulting in significant read / write amplification.

[0062] To address the aforementioned issues, this application provides a merging method and related equipment that can effectively reduce read / write amplification and resource consumption.

[0063] See Figure 4 , Figure 4 This is a flowchart illustrating a merging method provided in this application. Taking the merging of the (L-1)th level SSTable and the Lth level SSTable as an example, the merging method of this embodiment includes the following steps:

[0064] S101: When triggering condition L is met, the computing node executes the task L to establish association relationships, and the SSTable of layer L-1 is... [L-1，i] Move to the protected area. Where L is a positive integer greater than 1, and the SSTable in the protected area cannot be used by other merge tasks.

[0065] In a specific embodiment of this application, the triggering condition L can be... Figure 1 The triggering condition L in the illustrated embodiment, however, in Figure 1 In the illustrated embodiment, triggering condition L triggers the merging task L, while in this embodiment, triggering condition L triggers the establishing association task L.

[0066] In specific embodiments of this application, "moving to the protected area" can be a figurative expression. In practical applications, SSTable [L-1，i] Alternatively, it may not require an actual move, but rather an action on the SSTable. [L-1，i] Mark it to avoid SSTable [L-1，i] Used by other merge tasks.

[0067] S102: The compute node is based on the SSTable of the Lth layer. [L，1] SSTable [L，2] , ...,SSTable [L，n] The SSTable of layer L-1[L-1，i] The file is divided into n parts, where n is a positive integer, and the SSTable is at level L-1. [L-1，i] The key value ranges are respectively related to the SSTable of the Lth level. [L，1] SSTable [L，2] , ...,SSTable [L，n] There are overlapping parts.

[0068] In a specific embodiment of this application, the key-value range of the KV data in the first partial file is [K u1 K d1 In the second part of the file, the key-value range of the KV data is [K u2 K d2 The key-value range of the KV data in the nth partial file is [K...]. un K dn ]. Among them, K u1 K represents the upper limit of the key values ​​of the KV data in the first part of the file. u1 The value is equal to SSTable [L-1，i] The smallest key value in K d1 The lower bound of the key-value pairs in the KV data of the first part of the file, K d1 The value is equal to SSTable [L-1，i] SSTable belonging to the Lth layer [L，1] The largest key value; K u2 K represents the upper limit of the key values ​​for the KV data in the second part of the file. u2 The value is equal to SSTable [L-1，i] The smallest key value among the remaining key values, K d2 The lower bound of the key-value pairs in the KV data of the second part of the file, K d2 The value is equal to SSTable [L-1，i] SSTable belonging to the Lth layer [L，2] The largest key value, ..., K un K represents the upper limit of the key values ​​of the KV data in the nth partial file. un The value is equal to SSTable [L-1，i] The smallest key value among the last remaining key values ​​in the array, K dn K is the lower bound of the key value of the KV data in the nth partial file. dn The value is equal to SSTable [L-1，i] SSTable belonging to the Lth layer [L，n] The largest key value.

[0069] In a specific embodiment of this application, the first partial file, the second partial file, ..., the nth partial file are all stored in a protected area.

[0070] The following will illustrate, with specific examples, how to use the SSTable of the Lth layer. [L，1] SSTable [L，2] , ...,SSTable [L，n] The SSTable of layer L-1 [L-1，i] Divide the file into n parts.

[0071] Assume the SSTable of the (L-1)th layer [L-1，i] Includes five key-value pairs with key values ​​of 10, 15, 27, 190, and 210, in the L-th level SSTable. [L，1] Includes six key-value pairs with key values ​​of 1, 15, 27, 30, 50, and 90, in the L-th level SSTable. [L，2] This includes four key-value pairs with key values ​​of 200, 220, 300, and 390. Therefore, the SSTable at level L-1 can be... [L-1，i] The file is divided into two parts. In the first part, the key-value range of the KV data is [10, 27], and in the second part, the key-value range is [190, 210]. Here, the upper limit of 10 in the key-value range [10, 27] of the KV data in the first part is SSTable. [L-1，i] The minimum key value in the first part of the file, the lower limit of the key value range [10, 27] of the KV data, is 27, which is the SSTable of the (L-1)th level. [L-1，i] SSTable belonging to the Lth layer [L，1] The largest key value, 190, is the upper limit of the key value range [190, 210] in the second part of the file's KV data. [L-1，i] The minimum key value of the remaining key values ​​190 and 210 in the second part of the file, the lower limit of key value 210 in the key value range [190, 210] of the KV data is the SSTable of level L-1. [L-1，i] SSTable belonging to the Lth layer [L，2] The largest key value.

[0072] It should be understood that the above examples are merely to illustrate how to use the SSTable of the Lth level. [L，1] SSTable [L，2] , ...,SSTable [L，n] The SSTable of layer L-1 [L-1，i] In practical applications, the SSTable is divided into n parts. [L-1，i] SSTable of Level L [L，1] and the Lth layer SSTable [L，2]It can be any other SSTable. The number of SSTables in the Lth layer can be more than 2, it can be 3, 4 or more, there is no specific limitation here.

[0073] S103: The compute node creates n partial files, which are respectively connected to the SSTable of the Lth layer. [L，1] SSTable [L，2] , ...,SSTable [L，n] The relationship between them.

[0074] In specific embodiments of this application, such as Figure 5 As shown, the compute node creates the first partial file and SSTable. [L，1] The first relationship between them, the second part of the file and SSTable [L，2] The second association between them, ..., the nth part file and SSTable [L，n] The nth association between them, wherein the first association is used to transfer data from the compute node to the SSTable. [L，1] When reading KV data with key value K1, the compute node is guided to access the SSTable. [L-1，i] Read key-value data with key value K1, where K1 belongs to the key value range [K]. u1 K d1 The second association is used to link the compute node to the SSTable. [L，2] When reading KV data with key value K2, the compute node is guided to access the SSTable. [L-1，i] Read KV data with key value K2, where K2 belongs to the key value range [K u2 K d2 The nth association is used to link the compute nodes to the SSTable. [L，n] Reading the key value as key K n When processing KV data, the compute node is guided to the SSTable. [L-1，i] Reading the key value as key K n KV data, K n Belongs to the key value range [K] un K dn ].

[0075] In specific embodiments of this application, the first association, second association, ..., nth association are recorded in metadata files, such as the manifest file in the H base database (HBASE), the kudumanifest file for database state changes in the Kudu database, the manifest.json file for database state changes in the Cassandra database, etc., without specific limitations. In one specific embodiment, taking the manifest file as an example, the first association, second association, ..., nth association can be recorded in the Lth manifest file.

[0076] After completing the above tasks, the computing node assumes that it has completed the merging task L, that is, it assumes that the SSTable of the (L-1)th layer has been completed. [L-1，i] Already connected with the SSTable of the Lth layer [L，1] SSTable [L，2] , ...,SSTable [L，n] The SSTables of level L-1 were merged. [L-1，i] The key-value data is pushed down to the SSTable at level L. Therefore, the compute nodes will consider the SSTable at level L-1 to be... [L-1，i] It has been deleted. When you need to read the original SSTable at level L-1, [L-1，i] When processing key-value (KV) data, the corresponding KV data will be retrieved from the SSTable at level L.

[0077] The following section will provide a detailed explanation of how to use the first association, the second association, ..., the nth association, using specific real examples.

[0078] Assume the SSTable of the (L-1)th layer [L-1，i] It includes five key-value pairs with key values ​​of 10, 15, 27, 190, and 210, and also contains an SSTable at level L-1. [L-1，i] It is divided into a first part file and a second part file. The key-value range of the KV data in the first part file is [10, 27], and the key-value range of the KV data in the second part file is [190, 210]. The SSTable at level L. [L，1] Includes six key-value pairs with key values ​​of 1, 15, 27, 30, 50, and 90, in the L-th level SSTable. [L，2] It includes four key-value pairs with key values ​​of 200, 220, 300, and 390.

[0079] When a compute node needs to read KV data with a key value of 10, the compute node uses the SSTable at level L-1. [L-1，i] The key-value data in the table has been moved down to the next level; therefore, the compute nodes will move the data to the SSTable at level L. [L，1] Read the key-value data with a key value of 10. However, the key-value data with a key value of 10 has not actually been pushed down to the L-th level SSTable. [L，1] Therefore, the computing node can read KV data with a key value of 10 from the first partial file in the protected area based on the first association relationship.

[0080] When a compute node needs to read KV data with a key value of 15, the compute node uses the SSTable at level L-1. [L-1，i] The key-value data in the table has been moved down to the next level; therefore, the compute nodes will move the data to the SSTable at level L. [L，1] The key-value pair with key value 15 is read. However, the key-value pair with key value 15 is not actually pushed down to the L-th level SSTable. [L，1] Therefore, the computing node can read KV data with a key value of 15 from the first partial file in the protected area based on the first association relationship.

[0081] When a compute node needs to read KV data with a key value of 27, the compute node uses the SSTable at level L-1. [L-1，i] The key-value data in the table has been moved down to the next level; therefore, the compute nodes will move the data to the SSTable at level L. [L，1] The key-value pair with key value 27 is read. However, the key-value pair with key value 27 has not actually been pushed down to the L-th level SSTable. [L，1] Therefore, the computing node can read KV data with key value 27 from the first partial file in the protected area based on the first association relationship.

[0082] When a compute node needs to read KV data with a key value of 190, the compute node uses the SSTable at level L-1. [L-1，i] The key-value data in the table has been moved down to the next level; therefore, the compute nodes will move the data to the SSTable at level L. [L，2] Read the key-value data with key value 190. However, the key-value data with key value 90 did not actually sink down to the L-th level SSTable. [L，2] Therefore, the computing node can read KV data with a key value of 90 from the second part of the file in the protected area based on the first association.

[0083] When a compute node needs to read KV data with key value 210, the compute node uses the SSTable at level L-1. [L-1，i] The key-value data in the table has been moved down to the next level; therefore, the compute nodes will move the data to the SSTable at level L.[L，2] Read the key-value data with key value 210. However, the key-value data with key value 210 has not actually been pushed down to the L-th level SSTable. [L，2] Therefore, the computing node can read KV data with key value 210 from the second part of the file in the protected area based on the first association.

[0084] It should be understood that the above examples are merely to illustrate how to use the first association, the second association, ..., the nth association. In practical applications, the L-1 level SSTable... [L-1，i] SSTable of Level L [L，1] and the Lth layer SSTable [L，2] It can be any other SSTable. The number of SSTables in the Lth layer can be more than 2, it can be 3, 4 or more, there is no specific limitation here.

[0085] However, as the number of times the relationship establishment task is triggered increases, the read and write performance of the entire KV database will decrease. The specific reasons include the following two aspects:

[0086] (1) After multiple triggers of the association establishment task, the number of SSTables in the protected area will increase. Figure 6 As shown in the example, assuming that after two triggers to establish the association task, an SSTable can exist in the protected area. [L-1，1] and SSTable [L-1，2] The specific process and Figure 3 The methods shown are similar, and no specific limitations are specified here. Here, the SSTable of the (L-1)th layer... [L-1，1] It can be divided into a first part file, a second part file, ..., an nth part file, where the first part file and the Lth level SSTable [L，1] Establish the relationship between the second part of the file and the SSTable at level L. [L，2] Establish relationships, ..., the nth part of the file and the Lth level SSTable [L，n] Establish relationships. SSTable at level L-1 [L-1，2] It can be divided into a first part file, a second part file, ..., an m-th part file, where the first part file and the L-th level SSTable [L，n+1] Establish the relationship between the second part of the file and the SSTable at level L. [L，n+2] Establish the relationship between the m-th part of the file and the L-th level SSTable. [L，n+m] Establish the association. It should be understood that the number of SSTables in a protected area can be greater; no specific limit is set here.

[0087] (2) After multiple triggers of the association establishment task, the number of files with established associations in the SSTable and protected area of ​​layer L increases. Figure 7 As shown in the example, assuming that after two triggers to establish the association task, an SSTable can exist in the protected area. [L-1，1] and SSTable [L-1，2] The specific process and Figure 3 The methods shown are similar, and specific limitations are not specified here. Specifically, the SSTable of the (L-1)th layer... [L-1，1] The SSTable can be divided into a first part file, a second part file, ..., an nth part file, and an L-1th level file. [L-1，2] It can be divided into a first part file, a second part file, ..., an m-th part file. Here, the L-th level SSTable... [L，1] and SSTable [L-1，1] The first part establishes the relationship between the files, the Lth level SSTable [L，1] and the SSTable of layer L-1 [L-1，2] The second part of the file establishes the association relationship. It should be understood that the SSTable at level L can be associated with more files, but this is not specifically limited here.

[0088] Besides the reasons mentioned above, the read / write performance of a KV database can also be assessed by considering whether the SSTable has been cached. It's understandable that reading data from the cache is much faster than reading data from disk; therefore, if the SSTable is cached, its read / write speed will be relatively fast.

[0089] After multiple triggers of the association establishment task, an SSTable within a protected area may be triggered multiple times. Figure 8 As shown in the example, assuming that after the first triggering of the association establishment task, the SSTable at layer L-1 in the protected area... [L-1，1] It can be divided into a first part file, a second part file, and a third part file, where the first part file and the SSTable of the Lth level are... [L，1] Establish the relationship between the second part of the file and the SSTable at level L. [L，2] Establish the relationship between the third part of the file and the SSTable at level L. [L，3] Establish association; after the second triggering of the association establishment task, the SSTable at level L-1 in the protected area. [L-1，1] It can be divided into a fourth part file, a fifth part file, and a sixth part file, where the fourth part file and the SSTable of the Lth level are... [L，2]Establish the relationship, Part 5 file and SSTable of Level L. [L，4] Establish the relationship, Part 6 file and SSTable of Level L. [L，5] Establish the relationship. At this point, you can use SSTable. [L-1，1] The reference count is set to 2. It should be understood that the number of times an SSTable in a protected zone is triggered can be less or more, without being specifically limited here.

[0090] Under the condition of a true merge, the compute node can select the first target SSTable from the SSTable at level L and merge it with the corresponding portion of files in the protected area according to the selection strategy. It is understandable that the read / write performance of the KV database has degraded under the condition of a true merge; therefore, it is necessary to push the SSTable in the protected area down to the SSTable at level L to improve the read / write performance of the KV database.

[0091] In a specific implementation of this application, the actual merging conditions may include one or more of the following: the number of SSTables in the protected area exceeds a first preset threshold, and the number of SSTables in the L-th layer that establish associations with some files in the protected area exceeds a second preset threshold. Here, the first preset threshold may be set according to the size of the protected area and the user's requirements for read / write performance, and the second preset threshold may be set according to the user's requirements for read / write performance.

[0092] In the specific implementation of this application, the selection strategy is affected by factors such as whether the SSTable of the Lth layer is read into the cache, and the number of associations established between the SSTable of the Lth layer and some files in the protected area. Specifically, if a certain SSTable of the Lth layer is not read into the cache, it can be selected for merging first. If a certain SSTable of the Lth layer has more associations established with some files in the protected area, it can be selected for merging first.

[0093] In a specific embodiment of this application, merging the target area and the corresponding part of the file can be as follows: assuming the first target SSTable is merged with the SSTable in the protected area respectively. [L-1，1] The first part of the file and the SSTable in the protected area [L-1，2] The second part of the file establishes a relationship, therefore, the first target SSTable and SSTable can be linked. [L-1，1] The first part of the file and SSTable [L-1，2]The second part of the file is merged to obtain a new SSTable for level L. Then, the SSTable is... [L-1，1] and SSTable [L-1，2] Subtract one from the number of citations for each.

[0094] If a second target SSTable in the protected area meets the deletion criteria, it can be deleted, thereby reducing the resource occupation of the protected area. The deletion criteria could be that the reference count of the second target SSTable is zero.

[0095] See Figure 9 , Figure 9 This is a schematic diagram of the structure of a computing node provided in this application. The computing node in this embodiment includes: a protection module 210, a partitioning module 220, and an association module 230.

[0096] The protection module 210 is used to execute the task of establishing an association relationship L when the trigger condition L is met, and to move the SSTable[L-1, i] of the L-1 layer to the protection area, where L is a positive integer greater than 1.

[0097] The partitioning module 220 is used to divide the SSTable[L-1, i] of the (L-1)th layer into n partial files according to the n SSTables of the Lth layer, SSTable[L, 1], SSTable[L, 2], ..., SSTable[L, n], where n is a positive integer, and the key value range of the SSTable[L-1, i] of the (L-1)th layer overlaps with each SSTable of the Lth layer.

[0098] The association module 230 is used to establish an association relationship between each partial file and each SSTable of the Lth layer. Any association relationship is used to guide the computing node to read KV data with key value KN from the SSTable [L-1, i] when the computing node reads KV data with key value KN from the SSTable [L, N]. KN belongs to the key value range [KuN, KdN], 1≤N≤n.

[0099] Optionally, the node further includes a merging module 240, which is used to select a first target SSTable from the SSTable of the Lth layer and merge it with the corresponding partial files in the protected area according to the selection strategy, provided that the actual merging conditions are met.

[0100] Optionally, the true merging conditions include the number of SSTables in the protected area exceeding a first preset threshold and the number of SSTables in the Lth layer that establish association relationships with corresponding partial files in the protected area exceeding one or more of a second preset threshold.

[0101] Optionally, the factors influencing the selection strategy include: whether the first target SSTabel is read into the cache, and the number of files associated with the first target SSTabel in the protected area.

[0102] Optionally, the node further includes a deletion module 250, which is used to delete the second target SSTable if the second target SSTable in the protected area meets the deletion condition, wherein the deletion condition may be that the reference count of the second target SSTable is equal to zero.

[0103] The above computing nodes can complete Figure 9 For details on the merging method shown, please refer to [link / reference]. Figure 9 And related descriptions, which will not be elaborated here.

[0104] See Figure 10 , Figure 10 This is a schematic diagram of the structure of a computing node provided in this application. The computing node in this embodiment can be a single node or a cluster, and typically includes a main processor 310, a main memory 320, a smart network interface card 330, and a bus 340.

[0105] The main processor 310 can be one or more general-purpose processors. These general-purpose processors can be any type of device capable of processing electronic instructions, including a central processing unit (CPU), microprocessor, microcontroller, main processor, controller, and application-specific integrated circuit (ASIC), etc. The main processor 310 executes various types of digital storage instructions, such as software or firmware programs stored in the main memory 320. In a specific embodiment, the main processor 310 can be an x86 processor, etc. The main processor 310 sends commands to the main memory 320 through a physical interface to complete storage-related tasks. For example, the commands that the main processor 310 can provide include read commands, write commands, copy commands, and erase commands, etc. The commands can specify operations related to specific pages and blocks of the main memory 320. The main processor 310 can perform multiple tasks.

[0106] The main memory 320 may include random access memory (RAM), flash memory, or other similar formats. It can also be RAM, read-only memory (ROM), hard disk drive (HDD), or solid-state drive (SSD). The RAM may contain readable and writable tables as well as non-readable and writable tables. The ROM may contain protected areas containing SSTables, which are divided into multiple partial files. Multiple SSTables can also be stored in the ROM.

[0107] The smart network interface card 330 is also known as a network interface controller, network interface card, or local area network (LAN) adapter. Each smart network interface card 330 has a unique MAC address, which is burned into a read-only memory chip by the smart network interface card manufacturer during production. The smart network interface card 330 includes a secondary processor 331, a secondary memory 332, and a transceiver 333. The secondary processor 331 is similar to the main processor 320, but the performance requirements of the secondary memory 332 can be lower than those of the main processor 310. In a specific embodiment, the secondary memory 332 can be an ARM processor, etc. The secondary memory 332 can also be flash memory, HDD, or SDD, and its storage capacity can be smaller than that of the main memory 320. The transceiver 333 can be used to receive and send messages, and upload the received messages to the secondary memory 332 for processing. The smart network interface card 330 may also include multiple ports, which can be any one or more of the following three interface types: thick cable interface, thin cable interface, and twisted pair interface.

[0108] When trigger condition L is met, the main processor 310 executes the task L to establish the association relationship, and transfers the SSTable of layer L-1. [L-1，i] Move to the protected area, where L is a positive integer greater than 1;

[0109] The main processor 310 uses n SSTables from the Lth layer. [L，1] SSTable [L，2] , ...,SSTable [L，n] , the SSTable of the (L-1)th layer [L-1，i] The file is divided into n parts, where n is a positive integer, and the SSTable of the (L-1)th layer is... [L-1，i] The key value ranges overlap with each SSTable in the Lth layer;

[0110] The main processor 310 establishes an association between each partial file and each SSTable of the Lth layer, wherein any association is used by the computing node to access the SSTable. [L，N] Reading the key value as key K N When processing KV data, the compute node is guided to the SSTable. [L-1，i] Reading the key value as key K N KV data, K N Belongs to the key value range [K] uN K dN ], 1≤N≤n.

[0111] Optionally, if the conditions for a true merge are met, the main processor 310 may select the first target SSTable from the SSTable of the Lth layer and merge it with the corresponding portion of the files in the protected area according to the selection strategy.

[0112] Optionally, the true merging conditions include the number of SSTables in the protected area exceeding a first preset threshold and the number of SSTables in the Lth layer that establish association relationships with corresponding partial files in the protected area exceeding one or more of a second preset threshold.

[0113] Optionally, the factors influencing the selection strategy include: whether the first target SSTabel is read into the cache, and the number of files associated with the first target SSTabel in the protected area.

[0114] Optionally, if the second target SSTable in the protected area meets the deletion condition, the second target SSTable is deleted, wherein the deletion condition may be that the reference count of the second target SSTable is equal to zero.

[0115] The above computing nodes can complete Figure 10 For details on the merging method shown, please refer to [link / reference]. Figure 10 And related descriptions, which will not be elaborated here.

[0116] In the above scheme, when the trigger condition L is met, the computing node does not execute the merge task L and will not sink the KV data of layer L-1 to the KV data of layer L. Instead, it modifies the metadata to record the association between the KV data of layer L-1 and the KV data of layer L. Only when the actual merge condition is met will the KV data of layer L-1 be sinked to the KV data of layer L. This can effectively reduce the read and write amplification caused by merging.

[0117] In addition, during the actual merging process, only files with overlapping key values ​​are merged with the SSTable of the Lth level, avoiding merging files that do not involve overlapping key values, thus further reducing read / write amplification caused by merging.

[0118] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, storage disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium.

Claims

1. A merging method, characterized in that, include: In the case of meeting the triggering condition L, the computing node performs the association relationship establishing task L, and moves the SSTable of the L-1th layer to the protection area, where L is a positive integer greater than 1, and the triggering condition L triggers the association relationship establishing task. [L-1，i] moving to a protection area, where L is a positive integer greater than 1, and the triggering condition L triggers the association relationship establishing task. The computing node is based on n SSTables in the Lth layer. [L，1] SSTable [L，2] , ...,SSTable [L，n] , the SSTable of the (L-1)th layer [L-1，i] The file is divided into n parts, where n is a positive integer, and the SSTable of the (L-1)th layer is... [L-1，i] The key value ranges overlap with each SSTable in the Lth layer; The compute node establishes an association between each partial file and each SSTable of the Lth layer, wherein any association is used by the compute node to access the SSTable. [L，N] Reading the key value as key K N When processing KV data, the compute node is guided to the SSTable. [L-1，i] Reading the key value as key K N KV data, K N Belongs to the key value range [K] uN K dN ], 1≤N≤n, K uN The value is equal to SSTable [L-1，i] The smallest key value among the last remaining key values ​​in the array, K dn The value is equal to SSTable [L-1，i] SSTable belonging to the Lth layer [L，n] The largest key value; If the conditions for true merging are met, the compute node selects the first target SSTable from the SSTable of layer L and merges it with the corresponding part of the files in the protected area according to the selection strategy.

2. The method according to claim 1, characterized in that, The true merging conditions include the number of SSTables in the protected area exceeding a first preset threshold and the number of SSTables in the Lth layer that establish association relationships with corresponding files in the protected area exceeding one or more of a second preset threshold.

3. The method according to claim 1 or 2, characterized in that, The factors influencing the selection strategy include: whether the first target SSTabel is read into the cache, and the number of associations established between the first target SSTabel and some files in the protected area.

4. The method according to claim 1 or 2, characterized in that, The method further includes: If the second target SSTable in the protected area meets the deletion condition, the second target SSTable is deleted, wherein the deletion condition is that the reference count of the second target SSTable is equal to zero.

5. A computing node, characterized in that, include: Protection module, partitioning module, and association module. The protection module is used to execute the association establishment task L when the trigger condition L is met, and to connect the SSTable of the (L-1)th layer. [L-1，i] Move to the protected area, where L is a positive integer greater than 1, and the triggering condition L triggers the task of establishing an association relationship; The partitioning module is used to partition the data according to the n SSTables in the Lth layer. [L，1] SSTable [L，2] , ...,SSTable [L，n] , the SSTable of the (L-1)th layer [L-1，i] The file is divided into n parts, where n is a positive integer, and the SSTable of the (L-1)th layer is... [L-1，i] The key value ranges overlap with each SSTable in the Lth layer; The association module is used to establish the association relationship between each partial file and each SSTable of the Lth layer, wherein any association relationship is used by the computing node to direct the data to the SSTable. [L，N] Reading the key value as key K N When processing KV data, the compute node is guided to the SSTable. [L-1，i] Reading the key value as key K N KV data, K N Belongs to the key value range [K] uN K dN ], 1≤N≤n, where K uN The value is equal to SSTable [L-1，i] The smallest key value among the last remaining key values ​​in the array, K dn The value is equal to SSTable [L-1，i] SSTable belonging to the Lth layer [L，n] The largest key value; The merging module is used to select the first target SSTable from the SSTable of the Lth layer and merge it with the corresponding part of the file in the protected area, according to the selection strategy, when the actual merging conditions are met.

6. The node according to claim 5, characterized in that, The true merging conditions include the number of SSTables in the protected area exceeding a first preset threshold and the number of SSTables in the Lth layer that establish association relationships with corresponding files in the protected area exceeding one or more of a second preset threshold.

7. The node according to claim 5 or 6, characterized in that, The factors influencing the selection strategy include: whether the first target SSTabel is read into the cache, and the number of associations established between the first target SSTabel and some files in the protected area.

8. The node according to claim 5 or 6, characterized in that, The node also includes a deletion module. The deletion module is used to delete the second target SSTable in the protected area if the second target SSTable meets the deletion condition, wherein the deletion condition is that the reference count of the second target SSTable is equal to zero.

9. A non-transient readable storage medium comprising instructions that, when executed on a computing node, cause the computing node to perform the method as claimed in any one of claims 1 to 4.

10. A computing node, characterized in that, include: It includes a processor and a memory, wherein the processor executes code in the memory to perform the method as claimed in any one of claims 1 to 4.

Images