A data storage method and device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of computer data storage, and particularly provides a data storage method and device, electronic equipment and storage medium, aiming at solving the problem of consuming a large amount of cpu and disk io when using the existing B+ tree to flush. To this end, the data storage method comprises: constructing a database data layer, containing a plurality of shard pages, and storing complete KV key-value pair data on each shard page; constructing a database index layer, storing index information of the KV key-value pair data stored in the database data layer, the index information being stored in a tree structure, including a split key of each shard page as an index key, and ID information of the shard page corresponding to each split key as an index value. The present application stores data and index separately by constructing the database data layer and the index layer, so as to improve performance and reduce resource consumption.

Description

Technical Field

[0001] This invention relates to the field of computer data storage technology, and specifically provides a data storage method and apparatus, electronic device and storage medium. Background Technology

[0002] B+ trees are tree data structures commonly used in file indexing systems of databases and operating systems. File systems such as NTFS, ReiserFS, NSS, XFS, JFS, ReFS, and BFS all use B+ trees as metadata indexes. A key characteristic of B+ trees is their ability to maintain stable and ordered data, with insertion and modification operations exhibiting relatively stable logarithmic time complexity. A B+ tree is a balanced search tree where all records are stored in key-value order within leaf nodes of the same level, connected by pointers between leaf nodes.

[0003] In existing technologies, classic B+ trees are inherently compactly sorted. Therefore, deleting, updating, and adding keys all involve changes to leaf nodes. This creates a problem: whenever a leaf node is involved, even if some keys don't need updating, their data must be copied to a new leaf node to ensure compact data storage. This results in memory allocation for copying and disk I / O operations. Because B+ trees are balanced, adding and deleting nodes causes changes to the parent node's index information. Balancing calculations are CPU-intensive, and index updates consume memory and I / O. Therefore, during a flush operation, a classic B+ tree writes and deletes a large number of keys, consuming significant CPU and disk I / O. This limits the efficiency of classic B+ trees during flush operations, resulting in high consumption of both necessary and unnecessary resources. Furthermore, while the flush time represents a relatively small percentage of the overall database runtime, CPU usage is uneven and spikes.

[0004] Accordingly, there is a need in this field for a new data storage solution to address the aforementioned problems. Summary of the Invention

[0005] To overcome the above-mentioned defects, the present invention is proposed to provide a data storage method, apparatus, electronic device, and storage medium that solves or at least partially solves the technical problem of consuming a large amount of CPU and disk I / O when flushing B+ trees using existing methods.

[0006] In a first aspect, the present invention provides a data storage method, the data storage method comprising:

[0007] Construct a database data layer containing several sharded pages, each storing complete key-value pair data;

[0008] A database index layer is constructed to store the index information of the key-value pairs stored in the database data layer. The index information is stored in a tree structure, including the split key that generates each shard page as the index key, and the ID information of the shard page corresponding to each split key as the index value.

[0009] In one technical solution of the aforementioned data storage method, each shard page in the database data layer contains an incremental update file and a storage file:

[0010] Both the incremental update file and the storage file store complete key-value pair data, and the key values ​​in the key-value pair data are arranged in an ordered manner;

[0011] The key values ​​in the incremental update file and the storage file overlap. When the same key value is stored in both the incremental update file and the storage file, the incremental update file is read first, and then the storage file is searched.

[0012] In one technical solution of the above-mentioned data storage method, the database data layer includes shards p1, p2, ..., pn. Shard p1 stores KV key-value pair data including (k11, v11), (k12, v12), ..., (k1r, vlr), with key values ​​increasing in order from k11 to k1r. Shard p2 stores KV key-value pair data including (k21, v21), (k22, v22), ..., (k2s, v2s), with key values ​​increasing in order from k21 to k2s. Shard pn stores KV key-value pair data including (knl, vn1), (kn2, vn2), ..., (knt, vnt), with key values ​​increasing in order from kn1 to knt. The minimum key in the next shard from shard p1 to shard pn is greater than the maximum key in the previous shard.

[0013] The index keys storing index information in the database index layer include (k1r, k2s, ..., knt), and the index values ​​include (ID of shard p1, ID of shard p2, ..., ID of shard pn).

[0014] The key is searched in the index information of the database index layer to determine the position range of the key in the index key, and the ID of the shard pn corresponding to the index value is determined based on the position range.

[0015] In one technical solution of the aforementioned data storage method, the data storage method includes a data writing process:

[0016] The index value is obtained by searching the index information in the tree structure of the database index layer for the key value to be written in the memory table;

[0017] Write the complete key-value pair data corresponding to the key value in the memory table to the incremental update file in the shard page corresponding to the index value.

[0018] In one technical solution of the above-mentioned data storage method, the incremental update file stores KV key-value pair data, as well as forward and backward pointers of the key values. The forward pointer is the offset in the file where the forward key value is located, and the backward pointer is the offset in the file where the backward key value is located.

[0019] When writing complete key-value pairs corresponding to a key, the corresponding position is found through binary search. The pointers of the preceding nodes, the writing pointers, and the following nodes are then updated. An ordered list of key values ​​can be obtained through these pointers.

[0020] In one technical solution of the aforementioned data storage method, the data storage method includes:

[0021] During the process of writing data from the memory table to the incremental update file of the corresponding shard in the database data layer;

[0022] Once the data in the memory table is written, if the size of the incremental update file exceeds the preset flash threshold, the internal data merging process of the paging will be triggered.

[0023] The pagination-internal data merging process merges the incremental update file and the storage file of the same page. All data in the incremental update file is written to the storage file, and the incremental update file is empty after the data merging.

[0024] In one technical solution of the aforementioned data storage method, the data storage method includes:

[0025] After the pagination internal data merging process is completed, determine whether the size of the storage file exceeds the preset storage threshold;

[0026] If the judgment result is yes, the pagination splitting process is triggered: a new first page and a new second page are created, and the pages that need to be split are traversed in an ordered manner. First, they are written to the storage file of the newly created first page. After the first page is full, they are written to the storage file of the newly created second page until the end. At this time, the maximum key value corresponding to the first page / second page is obtained, and then it is updated to the index information of the database index layer. The index information corresponding to the pages that need to be split is released at an appropriate time.

[0027] In a second aspect, the present invention provides a data storage device, the data storage device comprising:

[0028] The database data layer module contains several sharded pages, each storing complete key-value pair data.

[0029] The database index layer module stores the index information of the key-value pairs stored in the database data layer. The index information is stored in a tree structure, including the split key that generates each shard page as the index key, and the ID information of the shard page corresponding to each split key as the index value.

[0030] In a third aspect, the present invention provides an electronic device including a processor and a memory, the memory being used to store a computer-executable program, wherein when the computer-executable program is executed by the processor, the processor performs a data storage method as described in any one of claims 1-7.

[0031] In a fourth aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the data storage method described in any of the above-described technical solutions.

[0032] The above-described technical solutions of the present invention have at least one or more of the following beneficial effects:

[0033] In implementing the technical solution of this invention, a data storage method is proposed. Addressing the issue of high resource consumption during flush operations in classic B+ trees, which negatively impacts system stability, this invention proposes separating data and indexes in the database data layer and index layer. This allows new and old data to be stored independently during flush operations, preventing interference and avoiding the significant resource consumption caused by merging old and new data in classic B+ tree implementations. Furthermore, during non-flush phases, data cleanup can be performed when conditions are met, resulting in smoother overall resource consumption. This data storage method is more flush-friendly, promotes stable system updates, and demonstrates high innovation. Attached Figure Description

[0034] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Furthermore, similar numbers in the drawings are used to denote similar components, wherein:

[0035] Figure 1 This is a schematic flowchart of the main steps of a data storage method according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic flowchart of the main steps of step S103 according to an embodiment of the present invention;

[0037] Figure 3 This is a schematic flowchart of the main steps of step S104 according to an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the internal data merging process of pagination according to an embodiment of the present invention;

[0039] Figure 5 This is a schematic flowchart of the main steps of step S105 according to an embodiment of the present invention;

[0040] Figure 6 This is a schematic diagram of the triggering pagination splitting process according to an embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram of the main structure of a data storage device according to an embodiment of the present invention;

[0042] Figure 8 This is a schematic diagram of the data storage structure of a data storage device according to an embodiment of the present invention. Detailed Implementation

[0043] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0044] In the description of this invention, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" and "this" can also include plural forms.

[0045] Operations on a classic B+ tree include adding, deleting, and updating keys. The specific operations for adding, deleting, and updating keys in a B+ tree are explained below.

[0046] The specific operation of adding a key is as follows: After adding a key to the B+ tree, if it is only adding one element to the corresponding leaf node, the B+ tree will remain unchanged; or, it may cause too many elements under the same node, so one node needs to be split into two and a new parent element needs to be added; or the node where the newly added parent element is located may have too many elements, which may also cause the parent node to split, recursively going up until the number of elements under a certain node is a reasonable value.

[0047] The specific operation of updating the key is as follows: updating the key will not change the number of elements in the node; it will only update the element data on the page node.

[0048] The specific operation of deleting a key is as follows: Deleting a key is similar to adding a key. If only one element is deleted from the corresponding leaf node, the B+ tree remains unchanged. Alternatively, it may result in too few elements under the same node, requiring node balancing. In this case, the index key elements will be redistributed according to the element values ​​of the parent node. When only one key is deleted, it usually does not cause changes to multiple parent nodes. However, if a large number of keys are deleted, it will affect the changes and balancing of a large number of parent nodes corresponding to these keys.

[0049] Therefore, while the time complexity of a classic B+ tree for a single insertion, update, or deletion is not very high, when writing a large amount of data at once (e.g., batch writing 100,000 records), involving numerous update, deletion, and insertion operations, several issues arise. Firstly, updates and additions cause many leaf nodes to be rewritten, including those containing unchanged keys. If these unchanged keys reside in the same node as changed keys, their values ​​also need to be rewritten. Secondly, deletions can lead to two outcomes: one, when the parent node doesn't need balancing, deleting a key from a corresponding leaf node will rewrite the leaf node; two, when the parent node finds its child nodes are too few and can be merged with adjacent child nodes, node balancing is required. In this case, the two adjacent leaf nodes are reallocated, two new leaf nodes are created and rewritten, and the parent node's element values ​​are also rewritten. In short, during a flush, the B+ tree undergoes a significant amount of memory allocation and copying, CPU computation, and disk I / O operations, resulting in substantial resource usage.

[0050] Based on this, the present invention proposes a data storage method and apparatus to avoid the large amount of CPU and disk I / O consumed by batch insertion, update and deletion of nodes when implementing B+ tree flush (flush is used to refresh the buffer, that is, to immediately write the data in the buffer to the file and clear the buffer at the same time).

[0051] The data storage method provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. (See attached drawings.) Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a data storage method according to an embodiment of the present invention. Figure 1 As shown, the data storage method in this embodiment of the invention mainly includes the following steps S101-S102.

[0052] Step S101: Construct the database data layer, which contains several shard pages, and each shard page stores complete KV key-value pair data.

[0053] In this embodiment, a KV (Key-Value) pair includes an index value (Key) and a storage value (Value) for the inserted data. The index value can be a number, such as 3, 5.5, π, etc., or a text type, such as a string, or a byte type. The index value can be used to sort the key-value pairs. The storage value can be a string, linked list, set, ordered set, etc., and contains data information.

[0054] In one embodiment of the present invention, each shard page in the database data layer includes an incremental update file and a storage file:

[0055] Both the incremental update file and the storage file store complete key-value pair data, and the key values ​​in the key-value pair data are arranged in an ordered manner;

[0056] The key values ​​in the incremental update file and the storage file overlap. When the same key value is stored in both the incremental update file and the storage file, the incremental update file is read first, and then the storage file is searched.

[0057] In this implementation, each shard page in the database data layer contains two files: an incremental update file and a storage file. Both the incremental update file and the storage file can be used to store complete key-value pair data. The keys in both files are ordered, but there may be overlap between them. When this happens—meaning the same key is stored in both files—the incremental update file is read first, followed by the storage file when writing new key-value pair data. Therefore, data in the incremental update file is always written first, and only after certain conditions are met is it written to the storage file, ensuring that the data in the incremental update file is always up-to-date.

[0058] Step S102: Construct a database index layer to store the index information of the KV key-value pair data stored in the database data layer. The index information is stored in a tree structure, including the split key that generates each shard page as the index key, and the ID information of the shard page corresponding to each split key as the index value.

[0059] The data storage method in this invention addresses the high resource consumption of classic B+ trees during flush operations, which negatively impacts system stability. It proposes constructing a database index layer and a database data layer, storing the index and data separately. This allows new and old data to be stored independently during flush operations, preventing interference and avoiding the significant resource consumption caused by merging old and new data in classic B+ tree implementations. Furthermore, during non-flush phases, data cleanup can be performed when conditions are met, resulting in smoother overall resource consumption. This data storage method is more flush-friendly, promotes stable system updates, and demonstrates high innovation.

[0060] In this embodiment, the database index layer is used to store index information for key-value pairs (KV). The index information includes an index key and its corresponding index value. In a specific example, if the database data layer has two shards, p1 and p2, with shard p1 having a split key of 'a' and shard p2 having a split key of 'c', then the database index layer's index information includes two index keys: 'a' and 'c'. The index value corresponding to 'a' is the ID of shard p1, and the index value corresponding to 'c' is the ID of shard p2.

[0061] In one embodiment of the present invention, the database data layer includes shards p1, p2, ..., pn. Shard p1 stores key-value pairs including (k11, v11), (k12, v12), ..., (k1r, v1r), with key values ​​increasing sequentially from k11 to k1r. Shard p2 stores key-value pairs including (k21, v21), (k22, v22), ..., (k2s, v2s), with key values ​​increasing sequentially from k21 to k2s. Shard pn stores key-value pairs including (kn1, vn1), (kn2, vn2), ..., (knt, vnt), with key values ​​increasing sequentially from kn1 to knt. The smallest key in the next shard from shard p1 to shard pn is greater than the largest key in the previous shard.

[0062] The index keys storing index information in the database index layer include (klr, k2s, ..., knt), and the index values ​​include (ID of shard p1, ID of shard p2, ..., ID of shard pn).

[0063] The key is searched in the index information of the database index layer to determine the position range of the key in the index key, and the ID of the shard pn corresponding to the index value is determined based on the position range.

[0064] In one implementation, the database data layer contains four shard pages, namely shard p1, shard p2, shard p3 and shard p4, wherein shard p1, shard p2, shard p3 and shard p4 can store the corresponding complete key-value pair data. For example, shard p1 can store 1024 pairs of KV key-value pairs, namely (k11, v11), (k12, v12), ..., (k1(1024), v1(1024)); shard p2 can store 521 pairs of KV key-value pairs, namely (k21, v21), (k22, v22), ..., (k2(521), v2(521)); shard p3 can store 2048 pairs of KV key-value pairs, namely (k31, v31), (k32, v32), ..., (k3(2048), v3(2048)); shard p4 can store 4096 pairs of KV key-value pairs, namely (k41, v41), (k42, v42), ..., (k4(4096), v4(4096)).

[0065] Within each shard, the keys are stored in ascending order. Therefore, the index values ​​in shard p1 are sorted as follows: k11 < k12 < ... < k1r, where r = 1024; the index values ​​in shard p2 are sorted as follows: k21 < k22 < ... < k2s, where s = 512; the index values ​​in shard p3 are sorted as follows: k31 < k32 < ... < k3m, where m = 2048; and the index values ​​in shard p4 are sorted as follows: k41 < k42 < ... < k4t, where t = 4096. The k1r corresponding to the largest key in shard p1 is less than the k21 corresponding to the smallest key in shard p2; the k2s corresponding to the largest key in shard p2 is less than the k31 corresponding to the smallest key in shard p3; and the k3m corresponding to the largest key in shard p3 is less than the k41 corresponding to the smallest key in shard p4.

[0066] In this implementation, the largest key in each shard page is used as the split key for that shard page. Therefore, the split key for shard p1 is k1 (1024); the split key for shard p2 is k2 (512); the split key for shard p3 is k3 (2048); and the split key for shard p4 is k4 (4096). In a specific example, if k1 (1024) = a; k2 (512) = c; k3 (2048) = f; and k4 (4096) = z, then the database index layer stores four index keys in the index information: a, c, f, and z. The index value corresponding to a is shard p1; the index value corresponding to c is shard p2; the index value corresponding to f is shard p3; and the index value corresponding to z is shard p4.

[0067] When writing cached key-value pairs to the database data layer in batches, the key of the key-value pair to be written is first searched in the index information of the database index layer to determine the position range of the key in the index. Based on the position range, the ID of the shard pn corresponding to the index value is determined. Specifically, if the key to be written is less than or equal to k1r, the key-value pair to be written is written to shard p1 in the database data layer; if the key to be written is less than or equal to k2s, the key-value pair to be written is written to shard p2 in the database data layer; if the key to be written is less than or equal to k3m, the key-value pair to be written is written to shard p3 in the database data layer; if the key to be written is less than or equal to k4t, the key-value pair to be written is written to shard p4 in the database data layer.

[0068] In one embodiment of the present invention, the data storage method further includes step S103: data writing process. For example... Figure 2 As shown, the main steps of step S103 are as follows:

[0069] Step S301: Search for the write key value in the memory table in the tree structure index information of the database index layer to obtain the index value;

[0070] Step S302: Write the complete key-value pair data corresponding to the key value in the memory table into the incremental update file in the shard page corresponding to the index value.

[0071] In this embodiment, when writing KV key-value pair data, the index is first searched in the tree structure index information of the database index layer to find the index key corresponding to the key value in the KV key-value pair data. The ID of the shard pn corresponding to the index value is determined according to the index key. Then, the KV key-value pair data is written to the incremental update file in the ID of the corresponding shard pn.

[0072] In one embodiment of the present invention, the incremental update file stores KV key-value pair data, and forward and backward pointers of the key values. The forward pointer is the offset in the file where the forward key value is located, and the backward pointer is the offset in the file where the backward key value is located.

[0073] When writing complete key-value pairs corresponding to a key, the corresponding position is found through binary search. The pointers of the preceding nodes, the writing pointers, and the following nodes are then updated. An ordered list of key values ​​can be obtained through these pointers.

[0074] In one implementation, after determining the incremental update file in the ID of the corresponding shard pn where the KV key-value pair data needs to be written, it is also necessary to determine the specific location of the KV key-value pair data in the incremental update file. For example, after the newly arrived 5 pairs of KV key-value pairs are searched by index, it is determined that they need to be written to the incremental update file of shard p1. At this time, the incremental update file of shard p1 stores 7 pairs of KV key-value pairs and forward and backward pointers for each key value of the 7 pairs of KV key-value pairs. The forward pointer is the offset in the file where the forward key value is located. The quantity, the backward pointer is the offset in the file where the backward key value is located. When writing 5 new KV key-value pairs to the incremental update file of shard p1, according to the writing order of the 5 KV key-value pairs, a binary search is used to first find the corresponding position in the incremental update file where the first KV key-value pair needs to be written. Then the pointer of the forward node for writing the key value is updated, the pointer of writing the key value is updated, the pointer of the backward node for writing the key value is updated, and so on. Finally, an ordered list of 12 KV key-value pairs is obtained through the pointers.

[0075] When new key-value pairs are continuously written to the incremental update file of a certain shard ID, the amount of data that needs to be stored in the incremental update file may exceed the preset flash memory data volume threshold. In one embodiment of this invention, the data storage method further includes step S104, such as... Figure 3 As shown, the main steps of step S104 are as follows:

[0076] Step S1041: During the process of writing data from the memory table to the incremental update file of the corresponding shard in the database data layer;

[0077] Step S1042: After the data in the memory table is written, if the size of the incremental update file exceeds the preset flash threshold, the internal data merging process of the paging is triggered.

[0078] Step S1043: The pagination internal data merging process merges the incremental update file and the storage file of the same page. All data in the incremental update file is written to the storage file. After the data merging, the incremental update file is empty.

[0079] In a specific example, such as Figure 4 As shown, Figure 4The maximum storage capacity of the incremental update file flush is 300MB. When the size of the written key-value pair data exceeds 300MB, the internal data merging process of the page is triggered. The incremental update file flush (300MB) of the same page, i.e., page p2, is merged with the storage file storage (100MB). All the data in the incremental update file flush is written to the storage file storage. After the data merging, the incremental update file is empty. At this time, the storage capacity of the incremental update file becomes 0MB, while the storage capacity of the storage file becomes 400MB.

[0080] In one embodiment of the present invention, the data storage method further includes step S105. For example... Figure 5 As shown, the main steps of step S105 are as follows:

[0081] Step S1051: After the pagination internal data merging process is completed, determine whether the size of the storage file exceeds the preset storage threshold;

[0082] Step S1052: If the judgment result is yes, the pagination splitting process is triggered: a new first page and a new second page are created, and the pages that need to be split are traversed in an orderly manner. First, they are written to the storage file of the newly created first page. After the first page is full, they are written to the storage file of the newly created second page until the end. At this time, the maximum key value corresponding to the first page / second page is obtained, and then it is updated to the index information of the database index layer. The index information corresponding to the pages that need to be split is released at an appropriate time.

[0083] In a specific example, a storage threshold is also set for the size of the storage file. After the internal data merging process of paging is completed, it is determined whether the size of the storage file exceeds the preset storage threshold. In this example, the storage threshold for the storage file is set to 256MB. After the aforementioned internal data merging process of paging, the storage capacity of the storage file becomes 400MB, exceeding the storage threshold, thus triggering the paging splitting process. Figure 6 As shown, page p2 is re-split into two pages, p21 and p22. Page p21 is 256MB in size, and page p22 is 144MB in size. The maximum key of page p21 is b, and the maximum key of page p22 is c. At this point, the B+ tree index keys need to be updated to include (a, b, c, f, z), and the index values ​​need to include (shard p1, shard p21, shard p22, shard p3, shard p4).

[0084] Based on the above steps S101-S102, S101-S103, S101-S104, or S101-S105, this invention proposes a data storage method. This method separates and stores the index information of the index layer and the key-value pair data of the data layer by constructing a database data layer and a database index layer. The database data layer contains several shard pages, and each shard page stores complete key-value pair data. The database index layer stores the index information of the key-value pair data stored in the database data layer. The index information is stored in a tree structure, including the split key that generates each shard page as the index key, and the ID information of the shard page corresponding to each split key as the index value. This ultimately improves performance and reduces resource consumption.

[0085] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of the present invention.

[0086] Furthermore, the present invention also provides a data storage device.

[0087] See appendix Figure 7 , Figure 7 This is a main structural block diagram of a data storage device according to an embodiment of the present invention. Figure 7 As shown, the data storage device in this embodiment of the invention mainly includes a database data layer module 11 and a database index layer module 12. In some embodiments, the database data layer module 11 can be configured to contain a plurality of shard pages, each shard page storing complete key-value pair data. The database index layer module 12 can be configured to store index information of the key-value pair data stored in the database data layer. The index information is stored in a tree structure, including the split key that generates each shard page as the index key, and the ID information of the shard page corresponding to each split key as the index value.

[0088] In one implementation, the specific function can be described in steps S101-S102, S101-S103, S101-S104, or S101-S105.

[0089] This data storage device decouples the index layer and the data layer, making the index layer smaller. A large number of real-time disk write requirements are transferred to the flush file in the data layer, thereby reducing the overall amount of data written. At the same time, it uses idle time to perform data paging, internal data merging and paging splitting, making the flush operation asynchronous and making the consumption of CPU and other resources smoother and more controllable.

[0090] See Figure 8 As shown, the data storage device of this embodiment includes two layers: a database index layer module (db index layer) and a database data layer module (db data layer). The db cache layer in the figure is memory-based and is the input layer of this device.

[0091] The data storage device in this embodiment separates the index and data through a database index layer module (db index layer) and a database data layer module (db data layer). By reducing the capacity of the B+ tree, the overhead of B+ tree node rewriting and balancing is significantly reduced. Synchronous writing to the flush file greatly reduces the amount of data rewritten in a single I / O operation, and asynchronous data merging and shard splitting alleviate the overall write pressure.

[0092] The database index layer module (db index layer) stores index information for only a limited number of keys, not all keys. These limited keys are the keys that generate page splits, and the values ​​are the IDs of the data layer pages. Assuming the database is as shown above, its index layer contains four elements (a, c, f, z), with corresponding values: a = p1, c = p2, f = p3, z = p4 (actually, p is not stored; only the IDs of the split pages, 1-4, are stored).

[0093] Index lookup process: If key <= a, search on p1; if key falls between (a, c), search on p2, and so on.

[0094] The database data layer module (db data layer) contains several shard pages. Each shard page stores the complete data of key-value pairs, and each shard page contains two files: an incremental update file (flush file) and a storage file (storage file).

[0095] The characteristics of the two types of files are as follows:

[0096] Incremental update files (flush files) and storage files (storage files) both store key-value pairs, and these keys are arranged in an ordered manner;

[0097] The keys in the incremental update file (flush file) and the storage file (storage file) overlap, meaning that the same key can be stored in both the incremental update file (flush file) and the storage file (storage file). When both exist, the incremental update file (flush file) is read first, and then the storage file (storage file) is searched.

[0098] This high-performance tree-structured storage device includes three operations: data writing, page merging, and page splitting.

[0099] The aforementioned data storage device is used for execution Figure 1 The data storage method embodiments shown are similar in technical principle, the technical problems they solve, and the technical effects they produce. Those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the data storage device can be referred to the content described in the embodiments of the data storage method, and will not be repeated here.

[0100] Those skilled in the art will understand that all or part of the processes in the method of the above embodiment of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0101] Furthermore, the present invention also provides a control device. In one embodiment of the control device according to the present invention, the control device includes a processor and a storage device. The storage device can be configured to store a program for executing the data storage method of the above-described method embodiments, and the processor can be configured to execute the program in the storage device. The program includes, but is not limited to, the program for executing the data storage method of the above-described method embodiments. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. This control device can be a control device device comprising various electronic devices.

[0102] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program that performs the data storage method of the above-described method embodiments, the program being loaded and run by a processor to implement the above-described data storage method. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device comprising various electronic devices; optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0103] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, hardware, or a combination of software and hardware. Therefore, the number of modules shown in the figures is merely illustrative.

[0104] Those skilled in the art will understand that the various modules in the device can be adaptively split or combined. Such splitting or combining of specific modules will not cause the technical solution to deviate from the principles of the present invention; therefore, the technical solutions after splitting or combining will fall within the protection scope of the present invention.

[0105] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A data storage method, characterized in that, include: Construct a database data layer containing several sharded pages, each storing complete key-value pair data; A database index layer is constructed to store the index information of the key-value pairs stored in the database data layer. The index information is stored in a tree structure, including the split key that generates each shard page as the index key and the ID information of the shard page corresponding to each split key as the index value. Each shard page in the database data layer contains incremental update files and storage files: Both the incremental update file and the storage file store complete key-value pair data, and the key values ​​in the key-value pair data are arranged in an ordered manner; The key values ​​in the incremental update file and the storage file overlap. When the same key value is stored in both the incremental update file and the storage file, the incremental update file is read first, and then the storage file is searched.

2. The data storage method according to claim 1, characterized in that, The database data layer includes shards p1, p2, ..., pn. Shard p1 stores key-value pairs including (k11, v11), (k12, v12), ..., (k1r, v1r), with key values ​​increasing in order from k11 to k1r. Shard p2 stores key-value pairs including (k21, v21), (k22, v22), ..., (k2s, v2s), with key values ​​increasing in order from k21 to k2s. Shard pn stores key-value pairs including (kn1, vn1), (kn2, vn2), ..., (knt, vnt), with key values ​​increasing in order from kn1 to knt. The smallest key in the next shard from shard p1 to shard pn is greater than the largest key in the previous shard. The index keys storing index information in the database index layer include (k1r, k2s, ..., knt), and the index values ​​include (ID of shard p1, ID of shard p2, ..., ID of shard pn). The key is searched in the index information of the database index layer to determine the position range of the key in the index key, and the ID of the shard pn corresponding to the index value is determined based on the position range.

3. The data storage method according to claim 2, characterized in that, Including the data writing process: The index value is obtained by searching the index information in the tree structure of the database index layer for the key value to be written in the memory table; Write the complete key-value pair data corresponding to the key value in the memory table to the incremental update file in the shard page corresponding to the index value.

4. The data storage method according to claim 3, characterized in that, The incremental update file stores key-value pair data, as well as forward and backward pointers to the key values. The forward pointer is the offset in the file where the forward key value is located, and the backward pointer is the offset in the file where the backward key value is located. When writing complete key-value pairs corresponding to a key, the corresponding position is found through binary search. The pointers of the preceding nodes, the writing pointers, and the following nodes are then updated. An ordered list of key values ​​can be obtained through these pointers.

5. A data storage method according to claim 3, characterized in that, include: During the process of writing data from the memory table to the incremental update file of the corresponding shard in the database data layer; Once the data in the memory table is written, if the size of the incremental update file exceeds the preset flash threshold, the internal data merging process of the paging will be triggered. The pagination-internal data merging process merges the incremental update file and the storage file of the same page. All data in the incremental update file is written to the storage file, and the incremental update file is empty after the data merging.

6. A data storage method according to claim 5, characterized in that, include: After the pagination internal data merging process is completed, determine whether the size of the storage file exceeds the preset storage threshold; If the judgment result is yes, the pagination splitting process is triggered: a new first page and a new second page are created, and the pages that need to be split are traversed in an ordered manner. First, they are written to the storage file of the newly created first page. After the first page is full, they are written to the storage file of the newly created second page until the end. At this time, the maximum key value corresponding to the first page / second page is obtained, and then it is updated to the index information of the database index layer. The index information corresponding to the pages that need to be split is released at an appropriate time.

7. A data storage device, characterized in that, include: The database data layer module contains several sharded pages, each storing complete key-value pair data. The database index layer module stores the index information of the key-value pair data stored in the database data layer. The index information is stored in a tree structure, including the split key that generates each shard page as the index key, and the ID information of the shard page corresponding to each split key as the index value. Each shard page in the database data layer contains incremental update files and storage files: Both the incremental update file and the storage file store complete key-value pair data, and the key values ​​in the key-value pair data are arranged in an ordered manner; The key values ​​in the incremental update file and the storage file overlap. When the same key value is stored in both the incremental update file and the storage file, the incremental update file is read first, and then the storage file is searched.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store a computer-executable program, and when the computer-executable program is executed by the processor, the processor performs a data storage method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The device stores a computer-executable program, which, when executed, implements a data storage method as described in any one of claims 1-6.

Images