A data storage method, system, electronic device and storage medium

Abstract

The present application discloses a data storage method, system, electronic device, and storage medium, relating to the technical field of data storage. Since the entire data storage system is split into multiple storage units, each storage unit includes a buffer and a storage tree. Newly written data is first stored in the buffer, and then the data is flushed to the storage tree by flushing the buffer subsequently, so as to achieve batch writing of data in the storage tree, reduce the write amplification of the storage tree. Moreover, when the level of the storage tree is relatively high, the high-level storage tree is split into multiple storage sub-trees, realizing the decentralized storage of data, avoiding the situation of too high a level of the storage tree, reducing the data access latency of the storage tree, and laying a foundation for improving the concurrent access volume and access efficiency of data.

Description

Technical Field

[0001] This application relates to the technical field of data storage, and particularly relates to a data storage method, system, electronic device, and storage medium. Background Art

[0002] With the development of information technology, more and more digital data needs to be stored. During the process of data storage, the concurrent access volume and access efficiency of data need to be considered. Therefore, how to store data efficiently has become the focus of research.

[0003] In related technologies, B+ trees are usually used for data storage. When reading and writing data, it searches down level by level in the B+ tree until the leaf nodes. However, when the amount of data to be stored is large, since the capacity of a single leaf node of the B+ tree is limited, node splitting is required, generating new leaf nodes and non-leaf nodes. The newly generated nodes will increase the level number of the B+ tree. When the level number of the B+ tree is large, it will increase the data access latency of the B+ tree, which is not conducive to ensuring the concurrent access volume and access efficiency of data. Summary of the Invention

[0004] This application provides a data storage method, system, electronic device, and storage medium to at least solve the problem in related technologies that when the level number of the B+ tree is large, it will increase the data access latency of the B+ tree, which is not conducive to ensuring the concurrent access volume and access efficiency of data.

[0005] This application provides a data storage method, including: 0]

[0006] Obtain a data storage request;

[0007] According to the target storage address represented by the data storage request, screen the target storage unit from multiple storage units; wherein, each storage unit includes a buffer and a storage tree;

[0008] Write the data to be stored indicated by the data storage request into the buffer of the target storage unit;

[0009] When the buffer meets the preset write-down condition, write down the data to be written down in the buffer to the storage tree;

[0010] When the current level number of the storage tree exceeds the storage tree level number threshold, perform split processing on the storage tree to split the storage tree into multiple storage sub-trees; wherein, the level number of the storage sub-tree is equal to the storage tree level number threshold.

[0011] This application also provides a data storage device, including:

[0012] An obtaining module, configured to obtain a data storage request;

[0013] A screening module, configured to screen a target storage unit from multiple storage units according to a target storage address represented by a data storage request; wherein each storage unit includes a buffer and a storage tree;

[0014] A writing module, configured to write data to be stored indicated by the data storage request into the buffer of the target storage unit;

[0015] A down - brushing module, configured to down - brush data to be down - brushed in the buffer to the storage tree when the buffer meets a preset down - brushing condition;

[0016] A splitting module, configured to perform splitting processing on the storage tree when the current level number of the storage tree exceeds the storage tree level number threshold, so as to split the storage tree into multiple storage sub - trees; wherein the level number of the storage sub - tree is equal to the storage tree level number threshold.

[0017] This application also provides a data storage system, including: multiple storage units and a data storage device; wherein each storage unit includes a buffer and a storage tree;

[0018] The data storage device uses any of the above - mentioned data storage methods to perform data storage processing on multiple storage units.

[0019] This application also provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any of the above - mentioned data storage methods when executing the computer program.

[0020] This application also provides a computer - readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above - mentioned data storage methods are implemented.

[0021] This application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of any of the above - mentioned data storage methods are implemented.

[0022] Through this application, since the entire data storage system is split into multiple storage units, each storage unit includes a buffer and a storage tree, newly written data is first stored in the buffer, and then the data in the buffer is down - brushed to the storage tree through data down - brushing to achieve batch writing of data to the storage tree, reducing the write amplification of the storage tree. Moreover, when the storage tree level is relatively high, the high - level storage tree is split into multiple storage sub - trees, realizing decentralized storage of data, avoiding the situation of too high a storage tree level, reducing the data access latency of the storage tree, and laying a foundation for improving the concurrent access volume and access efficiency of data. Description of the Drawings

[0023] To more clearly illustrate the embodiments of the present application, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0024] Figure 1 Schematic diagram of the network structure based on the embodiments of the present application;

[0025] Figure 2 Schematic flow diagram of the data storage method provided by the embodiments of the present application;

[0026] Figure 3 Schematic diagram of the structure of an exemplary data storage system provided by the embodiments of the present application;

[0027] Figure 4 Schematic diagram of the structure of the storage unit provided by the embodiments of the present application;

[0028] Figure 5 Schematic diagram of the storage tree splitting process provided by the embodiments of the present application;

[0029] Figure 6 Schematic diagram of a storage tree merging process provided by the embodiments of the present application;

[0030] Figure 7 Schematic diagram of another storage tree merging process provided by the embodiments of the present application;

[0031] Figure 8 Schematic flow diagram of the request response provided by the embodiments of the present application;

[0032] Figure 9 Schematic diagram of the structure of the data storage device provided by the embodiments of the present application;

[0033] Figure 10 Schematic diagram of the structure of the data storage system provided by the embodiments of the present application;

[0034] Figure 11 Schematic diagram of the structure of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0036] It should be noted that in the description of this application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0037] In a data storage system, data Grain (data granularity, the smallest data unit) is the basis for data information storage and the smallest unit of data. In recent years, with the development of information technology, a large amount of data has been generated. However, how to effectively manage and organize this large amount of data has become a prominent problem. For the large amount of stored data, querying and analyzing the data content and meaning therein can make more effective use of the data. The efficient organization and management of metadata in the storage system is an effective means to solve this problem and can support the system's management and maintenance of data.

[0038] In all-flash data storage, there will inevitably be a large number of and high-concurrency data access and query problems. Only by effectively managing metadata can the concurrent access volume and access efficiency be increased. Therefore, the method for effectively managing multi-concurrent read and write of metadata in all-flash is crucial, which can enable large-scale concurrent random access of metadata to have higher throughput and smaller latency.

[0039] In the related art, a B+ tree is usually used for data storage. When reading and writing data, it searches down level by level from the B+ tree until the leaf node. However, when the amount of data to be stored is large, due to the limited capacity of a single leaf node of the B+ tree, node splitting is required, generating new leaf nodes and non-leaf nodes. The newly generated nodes will increase the level number of the B+ tree. When the level number of the B+ tree is large, it will increase the data access latency of the B+ tree. Especially in the write process, the newly written metadata can only be returned after operating on the B+ tree. In the operations of insertion and deletion, KV insertion (key-value pair insertion), node splitting, and KV deletion (key-value pair deletion) all involve data movement and copying, etc., which are all in the entire I / O processing path, resulting in high CPU consumption and low performance, and being unfavorable for ensuring the concurrent access volume and access efficiency of data.

[0040] Embodiments of the present application are provided to solve the above technical problems, and a data storage method, system, electronic device, and storage medium are provided. The method includes: obtaining a data storage request; screening a target storage unit from multiple storage units according to the target storage address represented by the data storage request; where each storage unit includes a buffer and a storage tree; writing the data to be stored indicated by the data storage request into the buffer of the target storage unit; when the buffer meets a preset write-down condition, writing down the data to be written down in the buffer to the storage tree; when the current level number of the storage tree exceeds the storage tree level number threshold, performing a splitting process on the storage tree to split the storage tree into multiple storage sub-trees; where the level number of the storage sub-tree is equal to the storage tree level number threshold. The method provided by the above solution splits the entire data storage system into multiple storage units, each storage unit includes a buffer and a storage tree, newly written data is first stored in the buffer, and then the data in the buffer is written down to the storage tree by writing down the data in the buffer, so as to achieve batch writing of data in the storage tree, reduce the write amplification of the storage tree, and moreover, when the storage tree level is relatively high, the high-level storage tree is split into multiple storage sub-trees, realizing the dispersed storage of data, avoiding the situation of too high a storage tree level, and reducing the data access delay of the storage tree, laying a foundation for improving the concurrent access volume and access efficiency of data.

[0041] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0042] Combined with the specific application environment architecture or specific hardware architecture on which the execution of the data storage method depends, the specific application environment architecture or specific hardware architecture is described herein.

[0043] First, the network structure on which the present application is based is described:

[0044] The data storage method, system, electronic device, and storage medium provided by the embodiments of the present application are applicable to data storage processing of metadata waiting to be stored. As Figure 1 shown, it is a schematic diagram of the network structure based on the embodiments of the present application, mainly including a client and a server. The client is used to initiate a data storage request or a data reading request to the server, and the server is used to respond to the data storage request or data reading request initiated by the client, and perform corresponding data processing based on the data storage method provided by the embodiments of the present application, so as to achieve the dispersed storage of data, avoid the situation of too high a storage tree level, and reduce the data access delay of the storage tree.

[0045] An embodiment of the present application provides a data storage method for performing data storage processing on metadata waiting for stored data. The execution subject of the embodiment of the present application is an electronic device, such as a server, a desktop computer, a laptop computer, a tablet computer, and other electronic devices that can be used for data storage.

[0046] As Figure 2 shown, it is a schematic flowchart of the data storage method provided by the embodiment of the present application. The method includes:

[0047] Step 201, obtain a data storage request.

[0048] Among them, the data storage request is a write request initiated by the client and the upper-layer application. The request at least includes the data to be stored and the target storage address. The target storage address at least includes a logical block address (Logical Block Address, abbreviated as: LBA).

[0049] Step 202, screen the target storage unit from multiple storage units according to the target storage address characterized by the data storage request.

[0050] Among them, each storage unit includes a buffer and a storage tree. The buffer is a temporary storage area in the memory, and the storage tree adopts a tree-shaped index structure such as a B+ tree.

[0051] Step 203, write the data to be stored indicated by the data storage request into the buffer of the target storage unit.

[0052] It should be noted that the buffer is located in the memory area, and the leaf nodes for storing data in the storage tree are located in the external storage area such as a disk. Since the speed of writing data into the memory is much higher than that of writing data into the external storage, by writing the data to be stored indicated by the data storage request into the buffer of the target storage unit, the data storage efficiency can be improved. And, since the data to be stored may be modified within a certain period, the buffer is more convenient for data modification, and at the same time, it also avoids the situation of write amplification to the storage tree caused by frequent data modification.

[0053] Step 204, when the buffer meets the preset flushing condition, flush the data to be flushed in the buffer to the storage tree.

[0054] Specifically, when the buffer utilization rate (storage space occupancy rate) reaches the preset threshold or reaches the timing refresh period, it is determined that the buffer meets the preset flushing condition. By flushing the data to be flushed in the buffer to the storage tree, the storage space of the buffer can be released for storing subsequent data.

[0055] Step 205, when the current level number of the storage tree exceeds the storage tree level number threshold, perform split processing on the storage tree to split the storage tree into multiple storage subtrees.

[0056] Among them, the number of levels of the storage subtree is equal to the threshold of the number of levels of the storage tree.

[0057] Specifically, when the number of levels of the storage tree is relatively high (the current number of levels exceeds the threshold of the number of levels of the storage tree), the storage tree is split into multiple short trees (storage subtrees), reducing the number of non-leaf nodes, achieving control over the height of the storage tree, and preventing the query performance from decreasing due to the excessive height of the storage tree.

[0058] Based on the above embodiments, as an implementable manner, in one embodiment, screening target storage units from multiple storage units according to the target storage address represented by the data storage request includes:

[0059] Step 2021, in the preset static index layer, search layer by layer through multiple routing layers according to the target storage address represented by the data storage request to determine the target location layer entry corresponding to the target storage address;

[0060] Step 2022, screen the target storage units from multiple storage units according to the target location layer entry.

[0061] Among them, the preset static index layer includes multiple routing layers and location layers, the location layer includes multiple location layer entries, and each location layer entry corresponds to a storage unit one by one.

[0062] Exemplarily, such as Figure 3As shown in the figure, it is a schematic structural diagram of an exemplary data storage system provided by an embodiment of the present application. The system includes a preset static index layer and a slot layer. The slot layer includes multiple storage units. The routing layer includes multiple routing layer entries. Each routing layer entry records the entry information in the form of key-value pairs. The starting logical address is used as the Key value (K), and the offset is used as the Value value (V). The logical address range corresponding to the routing layer entry is [Key, Key + Value]. The offsets of the routing layer entries in adjacent routing layers differ by a factor of 10, and the number of routing layer entries in adjacent routing layers also differs by a factor of 10, that is, the logical address ranges recorded by the routing layer entries decrease gradually. By searching through multiple routing layers layer by layer, the logical address range (routing layer entry) corresponding to the target storage address can be accurately located in the last routing layer. Finally, according to the correspondence between the routing layer entry in the last routing layer and each positioning layer entry in the positioning layer, the target positioning layer entry corresponding to the target storage address is determined. The positioning layer entry also records the entry information in the form of key-value pairs. In the positioning layer entry, the starting logical address is used as the Key value, and the pointer Ptr is used as the Value value. The target storage unit is screened in the slot layer according to the target pointer represented by the Value value of the target positioning layer entry. The dynamic index layer (slot layer) is composed of multiple storage slots (storage units). Each storage slot stores a certain range of key-value pair data and is pointed to by an index entry in a positioning layer. The data ranges between the storage slots do not intersect and are arranged in ascending order according to the array subscript order of the positioning layer.

[0063] Among them, the static index layer is used to accelerate positioning and is composed of a routing layer and a positioning layer. Each structural expansion generates a new static index layer (routing layer and positioning layer), but once generated, it will not change. The positioning layer is an index entry array large enough. Each array element stores a Key and a pointer to the corresponding storage slot, which is used to index the storage slots in the dynamic index layer. The index entry Key indicates the maximum Key stored in the storage slot, that is, the maximum Key of the root node of the storage tree in the storage slot. The routing layer is a multi-dimensional array based on the positioning layer and is a skip list-like structure. Each element is the array subscript of the positioning layer, and the specific slot (target storage unit) in the dynamic index layer is located through multi-level binary search.

[0064] In the data storage system provided by the embodiments of the present application, a hierarchical index organization method is proposed, which combines static index and dynamic index (slot layer). The upper-layer static index is used to accelerate positioning, and the lower-layer dynamic index layer consists of multiple storage units each containing an independent buffer and a storage tree, which is responsible for persisting data. The hierarchical structure reduces the number of some non-leaf nodes. At the same time, the amount of non-leaf nodes of the storage tree with high access frequency is much smaller than that of leaf nodes. Therefore, all non-leaf nodes are cached in memory to ensure that only one external memory access is required to obtain data when the read request misses the cache, thereby improving the read performance. At the same time, using multiple short trees and independent and scattered buffers will greatly reduce the impact of disk flushing, and the background regularly expands and maintains the index structure, which always consists of short trees. When the system is initialized, it only contains a single storage unit and gradually expands as the data volume grows.

[0065] Correspondingly, in an embodiment, after the storage tree is to be split to split the storage tree into multiple storage sub-trees, corresponding numbers of new storage units can be generated according to the multiple storage sub-trees; allocate location layer entries for the new storage units; and modify the preset static index layer according to the allocation result of the location layer entries.

[0066] Specifically, while splitting the storage tree, the data pages cached in the buffer corresponding to the storage tree are also correspondingly divided to generate corresponding new storage units for each storage sub-tree, allocate location layer entries for the new storage units, and adaptively modify the preset static index layer, so that when accessing the data stored in the storage sub-tree later, the corresponding new storage unit can be accurately located through the preset static index layer to achieve fast data access.

[0067] Based on the above embodiments, as an implementable manner, in an embodiment, when the buffer meets the preset down-flush condition, flushing the data to be down-flushed in the buffer to the storage tree includes:

[0068] Step 2041, when the buffer meets the preset down-flush condition, convert the buffer to a read-only state and create a new buffer, where the new buffer is used to cache the data to be stored written by the superior during the down-flush process;

[0069] Step 2042, flush the data to be down-flushed in the read-only state buffer to the storage tree.

[0070] It should be noted that the buffer stores the incremental key-value pair data of the logical address range of the storage slot in an ordered structure. The incremental key-value pair data includes the key-value pair data of insertions, modifications, and deletions, that is, the data to be flushed down is the incremental key-value pair data. These complete incremental key-value pair data in the buffer will be saved when the power is off to prevent data loss. The flushing down of the buffer data is to check whether there is a buffer cache refresh for a single storage slot that meets the preset flushing-down condition for all storage slots (storage units) of the dynamic index layer (slot layer) from left to right. If the buffer flushes down data because the timing refresh period is reached, the amount of data flushed down for dirty data (data to be flushed down) can be reduced; if the buffer flushes down data because the storage space occupancy rate reaches the preset threshold, the flushing down of a large amount of data can be dispersed over multiple times, thereby reducing the structural adjustment and write amplification overhead caused by writing dirty data to disk each time, alleviating performance jitter, and reducing the tail latency.

[0071] Specifically, when the buffer meets the preset flushing-down condition, first lock the storage unit, set the buffer to read-only status, and prevent new data from being written into the buffer to avoid new data mixing in during the data flushing-down process and ensure that the data flushed down to the storage tree is accurate. At the same time, create a new buffer to receive the data to be stored written by the upper level during the data flushing-down process to prevent the write operation from being blocked due to the flushing down of the old buffer. After creating the new buffer, unlock the storage unit to enable the storage unit to provide services normally.

[0072] Among them, by flushing down the data to be flushed down in the read-only status buffer to the storage tree, the transformation of data from temporary storage in memory to persistent storage in external storage is realized, ensuring the reliability and persistence of the data.

[0073] Specifically, in one embodiment, for any data to be flushed down, the target leaf node can be determined in the storage tree belonging to the same storage unit as the read-only status buffer according to the target storage address of the data to be flushed down; perform a copy process on the target leaf node to obtain a copy leaf node; flush down the data to be flushed down to the copy leaf node and temporarily store the copy leaf node in memory; when all the data to be flushed down in the read-only status buffer have been flushed down to the corresponding copy leaf nodes, perform a merge process on the copy leaf nodes generated by the same target node to obtain at least one leaf node to be persisted; insert at least one leaf node to be persisted into the storage tree to replace the original target leaf node in the storage tree.

[0074] Specifically, for each piece of data to be flushed, first find the corresponding target leaf node in the storage tree that belongs to the same storage unit as the read-only status buffer. After finding the target leaf node, perform a copy operation on it to generate a copy leaf node. During the copying process, the copy leaf node can inherit the attributes and data content of the original target leaf node, that is, the data is flushed in a copy-on-write manner. In a multi-threaded concurrent access scenario, read and write operations can still access the original target leaf node, and write operations will not be blocked, ensuring the concurrent performance of leaf node data access and preventing direct modification of the original node from affecting other ongoing read and write operations.

[0075] Specifically, write the data to be flushed into the copy leaf node. The copy leaf node becomes a dirty node containing new data and is temporarily stored in memory. After all the data to be flushed in the read-only status buffer has been flushed to the corresponding copy leaf nodes, perform a merge operation on the copy leaf nodes generated from the same target node. During the merge process, integrate the data in the copy leaf nodes, remove duplicate or redundant parts, and form at least one leaf node to be persisted. Finally, insert the at least one leaf node to be persisted obtained after the merge process into the storage tree to replace the original target leaf node, completing the update and persistent storage of the data in the storage tree, enabling the storage tree to reflect the latest data state and ensuring the consistency and integrity of the data.

[0076] Specifically, in one embodiment, for screening the data to be flushed from the buffer, the value of the data in the buffer can be defined. The value of each piece of data is proportional to the data importance. When the buffer meets the preset flush condition, sort the data in descending order of value and select the part of the data with the highest value as the data to be flushed for flushing, so as to ensure that critical data (data with higher value) is preferentially persisted to the storage tree.

[0077] Specifically, in one embodiment, the data in the buffer can also be classified into hot and cold data according to the access frequency of each piece of data in the buffer. The cold data is used as the data to be flushed, so that the hot data is retained in the buffer. When the user subsequently initiates a data read request for the hot data, the data can be directly read from the buffer, thereby improving the user's access efficiency to the storage unit.

[0078] Further, in one embodiment, a new non-leaf node can be generated according to the index information of the data to be flushed stored in the leaf node to be persisted; insert the new non-leaf node into the storage tree to replace the original non-leaf node in the storage tree.

[0079] Among them, the index information of the data to be flushed down includes a key range (logical address range) and a pointer. Non-leaf nodes mainly play the role of indexing and guiding searches in storage tree structures such as B+ trees. They do not store actual data, but instead use key-value pairs and pointers to indicate the positions of the leaf nodes where the data is located.

[0080] Specifically, the data to be persisted in the leaf nodes stores the data flushed down to the storage tree. Based on the index information of these data, new non-leaf nodes are constructed. The construction of the new non-leaf nodes is based on the data in the leaf nodes to be persisted, and updates and optimizes the index structure of the storage tree. For example, if the leaf nodes to be persisted contain a series of consecutive key ranges, the new non-leaf nodes can be reasonably divided and organized according to these ranges to more efficiently locate data in subsequent queries. Finally, the generated new non-leaf nodes are inserted into the storage tree to replace the original non-leaf nodes.

[0081] Furthermore, synchronize the information of the new root node of the storage tree to the system metadata so that the preset static index layer can be adaptively modified according to the system metadata later. Among them, the root node information includes the address information of the root node of the storage tree.

[0082] Among them, as Figure 4 shown, it is a schematic structural diagram of the storage unit provided by the embodiment of the present application. During the data process of the storage unit, the storage nodes include a new buffer (Buffer_cache), a read-only status buffer (Buffer_commit), and a storage tree. After completing the above data flushing operation, lock the storage unit again, delete the read-only status buffer in the storage nodes, and switch the new root node of the storage tree according to the new tree node information represented by the system metadata. Finally, unlock the storage unit so that the storage unit can be normally applied.

[0083] Based on the above embodiments, as an implementable manner, in one embodiment, the method further includes:

[0084] Step 301, obtain the number of data volumes, the total data volume, and the written data volume of the data storage system;

[0085] Step 302, determine the structural constraint conditions of the storage unit according to the number of data volumes, the total data volume, and the written data volume of the data storage system;

[0086] Step 303, according to the current number of storage units and the structural constraint conditions of the storage unit;

[0087] Step 304, determine the threshold of the storage tree level according to the current number of storage units and the structural constraint conditions of the storage unit.

[0088] Among them, the structural constraint conditions of the storage unit include the minimum height of the storage tree, the maximum height of the storage tree, the expected number of storage units, and the current number of storage units.

[0089] It should be noted that the number of data volumes is the total number of logical volumes in the system; the total capacity of the data volumes is the total storage space of all logical volumes; the written data capacity is the actual amount of data that has been written into the system currently. The minimum height of the storage tree is the lowest level number that should be maintained to ensure query efficiency; the maximum height of the storage tree is the highest level number allowed to avoid the decline of query performance due to the excessive height of the tree; the expected number of storage units is the optimal number of storage units calculated according to the data volume and load. Among them, as the number of storage units increases, the threshold of the storage tree level number will gradually increase and finally become the maximum height of the storage tree.

[0090] Specifically, in one embodiment, the threshold of the storage tree level number can be determined based on the following formula:

[0091]

[0092] Among them, represents the threshold of the storage tree level number, represents the minimum height of the storage tree, represents the maximum height of the storage tree, represents the current number of storage units, represents the expected number of storage units.

[0093] Specifically, as the number of data volumes, the total capacity of the data volumes, the written data capacity in the data storage system change, and as the current number of storage units after the splitting and merging of the storage tree changes, the threshold of the storage tree level number is also adaptively adjusted to ensure that the threshold of the storage tree level number matches the actual business scenario of the data storage system, thereby ensuring the reliability of subsequent storage tree splitting processing and merging processing.

[0094] Based on the above embodiments, as an implementable method, in one embodiment, the storage tree will be split to split the storage tree into multiple storage sub-trees, including:

[0095] Step 2051, determine the splitting strategy of the storage tree according to the difference between the current level number of the storage tree and the threshold of the storage tree level number;

[0096] Step 2052, according to the splitting strategy of the storage tree, remove at least one upper-level node of the storage tree to make multiple non-leaf nodes become root nodes, and the storage tree is split into multiple storage sub-trees.

[0097] It should be noted that the system needs to ensure that the storage tree of each storage unit always maintains a relatively low tree height (level) state. Therefore, the average number of leaf nodes of the storage tree of each storage unit can be used as a measurement basis to design background expansion operations to balance the storage tree structures of all storage slots. The expansion operation is an operation friendly to hot loads. When the written data is mainly concentrated in several storage units, the growth of the storage tree structure will reduce the access efficiency of hot data. At this time, through the expansion operation (storage tree splitting), the height of the storage tree in the storage unit with concentrated hot spots can be reduced, and the hot spots can be dispersed to more storage units, so as to maintain stable performance.

[0098] Among them, the segmented expansion method adopted in the embodiments of the present application divides the hierarchical index structure into three segments during the expansion process, namely the expanded area, the area being expanded, and the unexpanded area. The system maintains the Key range (logical address range) of the part being expanded. When the Key of a new request (destination address) is within this range, it cannot be processed immediately and needs to wait for the expansion of this range to complete; if the Key of the new request is less than or equal to the system expansion Key, it will go to the expanded area part, that is, the new version of the index for processing; if the Key of the new request is greater than the system expansion Key, it will go to the unexpanded part, that is, the old version of the index for processing.

[0099] Specifically, during the expansion process of the storage tree, first create a new empty positioning layer large enough, and then sequentially traverse each storage unit of the original dynamic index layer, and perform balance processing according to the specific situation of the storage tree in the storage unit. When the height of the storage tree is close to the height threshold t, no processing is performed, and only the index items of the new positioning layer are pointed to the original storage slot. When the height of the storage tree exceeds the height threshold, a split operation is performed to split the buffer and the storage tree into multiple parts to form multiple new storage units, and new positioning layer index items are created in sequence to point to the corresponding storage units. When the height of the storage tree is less than the height threshold, a merge operation is performed, and continue to traverse the subsequent storage units until a storage tree with a height not less than the threshold is found, and then all the storage trees with a height less than the threshold that have been traversed are merged, and it is ensured that the height of the merged storage tree does not exceed the threshold t, and at the same time, the cache buffer is merged to form a new storage unit. Finally, new positioning layer index items (positioning layer table items) are created. After each new positioning layer index item is generated, this information is written into the system metadata file, and a new routing layer index is constructed during the continuous generation of new positioning layer index items. When all the storage units of the dynamic index layer are expanded, the old version of the routing layer and positioning layer are deleted, and the new version of the index is used as the new core index to achieve adaptive update of the preset static index layer.

[0100] Exemplarily, such as Figure 5As shown in the figure, it is a schematic diagram of the storage tree splitting process provided by the embodiment of the present application. The splitting operation of the storage tree takes the storage tree level number threshold t as the dividing line, and the process of splitting the high-level storage tree into multiple sub-storage trees. The splitting operation does not require updating the B+Tree, only updating the memory pointer in the index area, so it will not bring additional node data persistence. Taking t = 2 and n = 4 as an example, the difference between the two is 2, and then it is determined that the splitting strategy of the storage tree is to remove the first-level upper node (root node 1) and the second-level upper node (sub-nodes 2 and 3 of the root node) in the storage, so that the non-leaf nodes 4, 5, and 6 become the root nodes, that is, the splitting expands 3 storage sub-trees (storage units).

[0101] Correspondingly, in an embodiment, when the current level number of the storage tree is lower than the storage tree level number threshold, the storage tree is used as the storage tree to be merged; multiple storage trees to be merged are merged to obtain the merged storage tree, and at the same time, the storage units of the storage trees to be merged are merged to obtain the merged storage units.

[0102] Exemplarily, as Figure 6 shown, it is a schematic diagram of a storage tree merging process provided by the embodiment of the present application. As Figure 7 shown, it is another schematic diagram of the storage tree merging process provided by the embodiment of the present application. The merging operation is divided into two types. One is the merging of multiple storage trees with the same level number as Figure 6 shown. Since the logical address ranges between the storage units are arranged in an increasing order and the root nodes between the storage trees are in order, in this case, only a new root node needs to be generated to organize the multiple trees. The other is as Figure 7 shown, the merging of multiple storage trees with different levels. Similarly, due to the orderliness of the Key (logical address) between the trees, the merging process only needs to insert the root node of the shorter tree as a sub-node into the root node of the taller tree. Both merging operations will generate new dirty root nodes, bringing a small amount of additional node data persistence.

[0103] Based on the above embodiments, as an implementable manner, in an embodiment, the method further includes:

[0104] Step 401, obtaining a data reading request;

[0105] Step 402, screening the storage units to be accessed from multiple storage units according to the target reading address represented by the data storage request;

[0106] Step 403, traversing the buffer of the storage unit to be accessed to determine whether the buffer stores the data to be read;

[0107] Step 404, directly reading the data to be read from the buffer when it is determined that the buffer stores the data to be read;

[0108] Step 405: When it is determined that the buffer does not store the data to be read, traverse the storage tree of the storage unit to be accessed to locate the leaf node to be accessed, and read the data to be read from the leaf node to be accessed.

[0109] Among them, as Figure 8 shown, it is a schematic diagram of the request response process provided by the embodiment of the present application. The routing layer and the positioning layer are static structures. No locking is required to handle conflicts during searching, and all user request threads can perform parallel addressing. Each storage slot (storage unit) in the dynamic index layer (slot layer) is independently managed and the key ranges do not intersect. The system allocates threads according to storage units, one thread per storage unit. No locking is required for internal access to storage units. The data dispersion effect of multiple storage units can effectively alleviate the impact of conflicts on performance. As Figure 8 shown in, Write 2 (Write Request 2) needs to wait for Write 1 (Write Request 1) to complete before writing. Read requests for the same storage unit can be responded to in parallel, that is, as Figure 8 shown in, Read 1 (Read Request 1) and Read 2 (Read Request 2) can be performed simultaneously.

[0110] Specifically, when a read request (data read request) is obtained, first locate the corresponding storage slot in the dynamic index layer according to the read key value (target read address) through the routing layer and the positioning layer. First, query the buffer. If the data to be read is not found, continue to search in the B+Tree (storage tree). The search in the storage tree starts from the root node. Binary search is performed within each node and goes down layer by layer until it is determined whether the data exists in the leaf node. If any leaf node cannot be obtained from the node cache, it will be obtained from the disk file according to the node address and temporarily stored in the node cache for the next query.

[0111] The data storage method provided by the embodiments of the present application includes obtaining a data storage request; screening a target storage unit from multiple storage units according to the target storage address represented by the data storage request; wherein each storage unit includes a buffer and a storage tree; writing the data to be stored indicated by the data storage request into the buffer of the target storage unit; when the buffer meets the preset downflush condition, downflushing the data to be downflushed in the buffer to the storage tree; when the current level number of the storage tree exceeds the storage tree level number threshold, performing a splitting process on the storage tree to split the storage tree into multiple storage subtrees; wherein the level number of the storage subtree is equal to the storage tree level number threshold. The method provided by the above solution splits the entire data storage system into multiple storage units, each storage unit includes a buffer and a storage tree, the newly written data is first stored in the buffer, and then the data is downflushed to the storage tree by downflushing the buffer, so as to achieve batch writing of data to the storage tree, reduce the write amplification of the storage tree, and when the storage tree level is relatively high, split the high-level storage tree into multiple storage subtrees, realizing the decentralized storage of data, avoiding the situation of too high storage tree levels, reducing the data access latency of the storage tree, and laying a foundation for improving the concurrent access volume and access efficiency of data.

[0112] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0113] The embodiments of the present application also provide a data storage device for executing the data storage method provided by the above embodiments.

[0114] As Figure 9 shown, it is a schematic structural diagram of the data storage device provided by the embodiments of the present application. The data storage device 90 includes: an acquisition module 901, a screening module 902, a writing module 903, a downflushing module 904, and a splitting module 905.

[0115] Among them, the acquisition module is used to obtain a data storage request; the screening module is used to screen a target storage unit from multiple storage units according to the target storage address represented by the data storage request; wherein each storage unit includes a buffer and a storage tree; the writing module is used to write the data to be stored indicated by the data storage request into the buffer of the target storage unit; the downflushing module is used to downflush the data to be downflushed in the buffer to the storage tree when the buffer meets the preset downflush condition; the splitting module is used to perform a splitting process on the storage tree when the current level number of the storage tree exceeds the storage tree level number threshold, so as to split the storage tree into multiple storage subtrees; wherein the level number of the storage subtree is equal to the storage tree level number threshold.

[0116] For the description of the features in the corresponding embodiments of the data storage device, reference can be made to the relevant descriptions in the corresponding embodiments of the data storage method, which will not be elaborated here one by one.

[0117] An embodiment of the present application further provides a data storage system for executing the data storage method provided in the above embodiment.

[0118] As Figure 10 shown, it is a schematic structural diagram of the data storage system provided by the embodiment of the present application. The data storage system includes: a plurality of storage units and a data storage device.

[0119] Among them, each storage unit includes a buffer and a storage tree.

[0120] The data storage device uses the data storage method provided in the above embodiment to perform data storage processing on a plurality of storage units.

[0121] For the description of the features in the corresponding embodiments of the data storage device, reference can be made to the relevant descriptions in the corresponding embodiments of the data storage method, which will not be elaborated here one by one.

[0122] An embodiment of the present application further provides an electronic device. As Figure 11 shown, it is a schematic structural diagram of the electronic device provided by the embodiment of the present application, including a processor 10 and a memory 20. A computer program is stored in the memory 20, and the processor 10 is configured to run the computer program to execute the steps in any of the above data storage method embodiments.

[0123] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any of the above data storage method embodiments when running.

[0124] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drives, read-only memories (ROM for short), random access memories (RAM for short), mobile hard disks, magnetic disks, or optical discs and other various media that can store computer programs.

[0125] An embodiment of the present application further provides a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above data storage method embodiments.

[0126] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps in any of the above-described data storage method embodiments are implemented.

[0127] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0128] The above has introduced in detail a data storage method, system, electronic device, and storage medium provided by the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A data storage method, characterized in that, Including: Obtain a data storage request; Filter a target storage unit from multiple storage units according to the target storage address characterized by the data storage request; wherein, each of the storage units includes a buffer and a storage tree; Write the data to be stored indicated by the data storage request into the buffer of the target storage unit; When the buffer meets the preset flush condition, flush the data to be flushed in the buffer to the storage tree; When the current level number of the storage tree exceeds the storage tree level number threshold, perform a splitting process on the storage tree to split the storage tree into multiple storage sub-trees; wherein, the level number of the storage sub-tree is equal to the storage tree level number threshold; The step of flushing the data to be flushed in the buffer to the storage tree when the buffer meets the preset flush condition includes: When the buffer meets the preset flush condition, convert the buffer to a read-only state and create a new buffer, which is used to cache the data to be stored written by the upper level during the flush process; Flush the data to be flushed in the read-only state buffer to the storage tree.

2. The data storage method according to claim 1, wherein The step of filtering a target storage unit from multiple storage units according to the target storage address characterized by the data storage request includes: At a preset static index layer, perform a layer-by-layer search on multiple routing layers according to the target storage address characterized by the data storage request to determine the target location layer entry corresponding to the target storage address; Filter a target storage unit from multiple storage units according to the target location layer entry; Wherein, the preset static index layer includes multiple routing layers and a location layer, the location layer includes multiple location layer entries, and each location layer entry corresponds to a storage unit one by one.

3. The data storage method according to claim 2, wherein After performing a splitting process on the storage tree to split the storage tree into multiple storage sub-trees, the method further includes: Generate a corresponding number of new storage units according to the multiple storage sub-trees; Allocate location layer entries for the new storage units; Modify the preset static index layer according to the allocation result of the location layer entries.

4. The data storage method according to claim 3, wherein The step of flushing the data to be flushed in the read-only state buffer to the storage tree includes: For any of the data to be flushed, determine a target leaf node in the storage tree belonging to the same storage unit as the read-only state buffer according to the target storage address of the data to be flushed; Perform a replication process on the target leaf node to obtain a replicated leaf node; Flush the data to be flushed to the replicated leaf node and temporarily store the replicated leaf node in the memory; When all the data to be flushed in the read-only state buffer have been flushed to the corresponding replicated leaf nodes, perform a merging process on the replicated leaf nodes generated by replicating the same target node to obtain at least one leaf node to be persisted; Insert the at least one leaf node to be persisted into the storage tree to replace the original target leaf node in the storage tree.

5. The data storage method according to claim 4, characterized in that The method further includes: Generate a new non-leaf node according to the index information of the data to be flushed stored in the leaf node to be persisted; Insert the new non-leaf node into the storage tree to replace the original non-leaf node in the storage tree.

6. The data storage method according to claim 1, wherein The method further includes: Obtain the number of data volumes, the total data volume capacity, and the written data capacity of the data storage system; Determine the structural constraint conditions of the storage unit according to the number of data volumes, the total data volume capacity, and the written data capacity of the data storage system; According to the current number of the storage units and the structural constraint conditions of the storage unit; Determine the threshold of the storage tree level number according to the current number of the storage units and the structural constraint conditions of the storage unit; Wherein, the structural constraint conditions of the storage unit include the minimum height of the storage tree, the maximum height of the storage tree, the expected number of storage units, and the current number of storage units.

7. The data storage method according to claim 6, wherein The determining the threshold of the storage tree level number according to the current number of the storage units and the structural constraint conditions of the storage unit includes: Determine the threshold of the storage tree level number based on the following formula: wherein, represents the threshold of the number of levels of the storage tree, represents the minimum height of the storage tree, represents the maximum height of the storage tree, represents the current number of the storage units, represents the expected number of the storage units.

8. The data storage method according to claim 1, characterized in that The splitting process to be performed on the storage tree to split the storage tree into multiple storage sub-trees includes: Determine the splitting strategy of the storage tree according to the difference between the current level number of the storage tree and the threshold of the storage tree level number; According to the splitting strategy of the storage tree, remove at least one upper-level node of the storage tree, so that multiple non-leaf nodes become root nodes, and the storage tree is split into multiple storage sub-trees.

9. The data storage method according to claim 1, wherein The method further includes: When the current level number of the storage tree is lower than the threshold of the storage tree level number, regard the storage tree as a storage tree to be merged; Perform a merging process on multiple storage trees to be merged to obtain a merged storage tree, and at the same time perform a merging process on the storage units of the storage trees to be merged to obtain merged storage units.

10. The data storage method according to claim 1, characterized in that The method further includes: Obtain a data reading request; Screen the storage units to be accessed among multiple storage units according to the target reading address represented by the data storage request; Traverse the buffer of the storage unit to be accessed to determine whether the buffer stores the data to be read; When it is determined that the buffer stores the data to be read, directly read the data to be read from the buffer; When it is determined that the buffer does not store the data to be read, traverse the storage tree of the storage unit to be accessed to locate the leaf node to be accessed, and read the data to be read from the leaf node to be accessed.

11. A data storage system, characterized in that, Includes: Multiple storage units and a data storage device; wherein, each storage unit includes a buffer and a storage tree; The data storage device uses the data storage method according to any one of claims 1 to 10 to perform data storage processing on the multiple storage units.

12. An electronic device, characterized in that, Includes: A memory for storing a computer program; A processor for implementing the steps of the data storage method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of the data storage method according to any one of claims 1 to 10 when executed by a processor.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the data storage method according to any one of claims 1 to 10 are implemented.

Images