Data storage method and system, electronic equipment and storage medium

Abstract

The invention discloses a data storage method and system, electronic equipment and a storage medium, and relates to the technical field of data storage.The whole data storage system is divided into a plurality of storage units, each storage unit comprises a buffer and a storage tree, newly-written data are firstly stored in the buffers, and then the newly-written data are stored in the storage tree; according to the method and the device, the buffer is divided into a plurality of storage sub-trees, the buffer is subsequently subjected to data down-brushing to brush the data down to the storage tree, so that batch writing of the data of the storage tree is realized, write amplification of the storage tree is reduced, and when the storage tree level is high, the high-level storage tree is divided into the plurality of storage sub-trees, so that scattered storage of the data is realized, the situation that the storage tree level is too high is avoided, and the storage efficiency is improved. The data access time delay of the storage tree is reduced, and a foundation is laid for improving the concurrent access amount and the access efficiency of the data.

Description

Technical Field

[0001] This application relates to the technical field of data storage, and in particular, to a data storage method, system, electronic device, and storage medium. Background Art

[0002] With the development of information technology, more and more digital data needs to be stored. During the data storage process, the concurrent access volume and access efficiency of the data need to be considered. Therefore, how to store data efficiently has become the focus of research.

[0003] In the related art, a B+ tree is usually used for data storage. When reading and writing data, it searches down level by level in the B+ tree until the leaf nodes. However, when the amount of data to be stored is large, since the capacity of a single leaf node of the B+ tree is limited, node splitting is required to generate new leaf nodes and non-leaf nodes. The newly generated nodes will increase the level number of the B+ tree. When the level number of the B+ tree is large, it will increase the data access latency of the B+ tree, which is not conducive to ensuring the concurrent access volume and access efficiency of the data. Summary of the Invention

[0004] This application provides a data storage method, system, electronic device, and storage medium to at least solve the problem in the related art that when the level number of the B+ tree is large, it will increase the data access latency of the B+ tree, which is not conducive to ensuring the concurrent access volume and access efficiency of the data.

[0005] This application provides a data storage method, including: Obtaining a data storage request; According to the target storage address represented by the data storage request, screening a target storage unit from multiple storage units; wherein, each storage unit includes a buffer and a storage tree; Writing the data to be stored indicated by the data storage request into the buffer of the target storage unit; When the buffer meets the preset down-flush condition, flushing the data to be down-flushed in the buffer to the storage tree; When the current level number of the storage tree exceeds the storage tree level number threshold, performing a splitting process on the storage tree to split the storage tree into multiple storage sub-trees; wherein, the level number of the storage sub-tree is equal to the storage tree level number threshold.

[0006] This application also provides a data storage device, including: An obtaining module, configured to obtain a data storage request; A screening module, configured to screen a target storage unit from multiple storage units according to the target storage address represented by the data storage request; wherein, each storage unit includes a buffer and a storage tree; A writing module, configured to write data to be stored indicated by a data storage request into a buffer of a target storage unit; A flushing module, configured to flush data to be flushed in the buffer to a storage tree when the buffer meets a preset flushing condition; A splitting module, configured to perform a splitting process on the storage tree when the current level number of the storage tree exceeds a storage tree level number threshold, so as to split the storage tree into multiple storage sub-trees; wherein, the level number of the storage sub-tree is equal to the storage tree level number threshold.

[0007] The present application further provides a data storage system, including: a plurality of storage units and a data storage device; wherein, each storage unit includes a buffer and a storage tree; The data storage device uses any of the above data storage methods to perform data storage processing on the plurality of storage units.

[0008] The present application further provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any of the above data storage methods when executing the computer program.

[0009] The present application further provides a computer-readable storage medium, in which a computer program is stored, and wherein the computer program implements the steps of any of the above data storage methods when executed by a processor.

[0010] The present application further provides a computer program product, including a computer program, and the computer program implements the steps of any of the above data storage methods when executed by a processor.

[0011] Through the present application, since the entire data storage system is split into multiple storage units, each storage unit includes a buffer and a storage tree, newly written data is first stored in the buffer, and then the data in the buffer is flushed to the storage tree through data flushing to achieve batch writing of data to the storage tree, reducing the write amplification of the storage tree. Moreover, when the storage tree has a high level, the high-level storage tree is split into multiple storage sub-trees, realizing the dispersed storage of data, avoiding the situation of too high a level of the storage tree, reducing the data access latency of the storage tree, and laying a foundation for improving the concurrent access volume and access efficiency of data. Description of the Drawings

[0012] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0013] Figure 1Schematic diagram of the network structure on which the embodiments of this application are based; Figure 2 Flowchart of the data storage method provided by the embodiments of this application; Figure 3 Schematic diagram of the structure of an exemplary data storage system provided by the embodiments of this application; Figure 4 Schematic diagram of the structure of the storage unit provided by the embodiments of this application; Figure 5 Schematic diagram of the storage tree splitting process provided by the embodiments of this application; Figure 6 Schematic diagram of a storage tree merging process provided by the embodiments of this application; Figure 7 Schematic diagram of another storage tree merging process provided by the embodiments of this application; Figure 8 Schematic diagram of the request response process provided by the embodiments of this application; Figure 9 Schematic diagram of the structure of the data storage device provided by the embodiments of this application; Figure 10 Schematic diagram of the structure of the data storage system provided by the embodiments of this application; Figure 11 Schematic diagram of the structure of the electronic device provided by the embodiments of this application. Detailed implementation manners

[0014] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without making creative efforts belong to the scope of protection of this application.

[0015] It should be noted that in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects and not to describe a specific order or sequence.

[0016] In a data storage system, a data Grain (data granularity, the smallest data unit) is the basis for storing data information and is the smallest unit of data. In recent years, with the development of information technology, a vast amount of data has been generated. However, how to effectively manage and organize this vast amount of data has become a prominent problem. For the large amount of stored data, querying and analyzing the data content and meaning therein can enable more effective utilization of the data. The efficient organization and management of metadata in the storage system is an effective means to solve this problem and can support the system's management and maintenance of data.

[0017] In all-flash data storage, there are inevitably a large number of and high-concurrency data access and query problems. Only by effectively managing metadata can the concurrent access volume and access efficiency be increased. Therefore, an effective management method for multi-concurrent read and write of metadata in all-flash storage is crucial, which can enable large-scale concurrent random access to metadata to have higher throughput and smaller latency.

[0018] In related technologies, B+ trees are usually used for data storage. When reading and writing data, it searches down level by level from the B+ tree until the leaf nodes. However, when the amount of data to be stored is large, since the capacity of a single leaf node of the B+ tree is limited, node splitting is required, generating new leaf nodes and non-leaf nodes. The newly generated nodes will increase the level number of the B+ tree. When the level number of the B+ tree is large, it will increase the data access latency of the B+ tree. Especially in the write process, the newly written metadata can only be returned after operating on the B+ tree. In the operations of insertion and deletion, KV insertion (key-value pair insertion), node splitting, and KV deletion (key-value pair deletion) all involve data movement and copying, etc., which are all in the entire I / O processing path, resulting in high CPU consumption and low performance, and being unfavorable for ensuring the concurrent access volume and access efficiency of data.

[0019] Embodiments of the present application are provided to solve the above technical problems, and a data storage method, system, electronic device, and storage medium are provided. The method includes: obtaining a data storage request; screening a target storage unit from multiple storage units according to the target storage address represented by the data storage request; where each storage unit includes a buffer and a storage tree; writing the data to be stored indicated by the data storage request into the buffer of the target storage unit; when the buffer meets a preset write-down condition, writing down the data to be written down in the buffer to the storage tree; when the current level number of the storage tree exceeds the storage tree level number threshold, performing a splitting process on the storage tree to split the storage tree into multiple storage sub-trees; where the level number of the storage sub-tree is equal to the storage tree level number threshold. The method provided by the above solution splits the entire data storage system into multiple storage units, each storage unit includes a buffer and a storage tree. Newly written data is first stored in the buffer, and then the data in the buffer is written down to the storage tree by writing down the data in the buffer, so as to achieve batch writing of data in the storage tree, reduce the write amplification of the storage tree, and when the storage tree level is relatively high, split the high-level storage tree into multiple storage sub-trees, realizing the decentralized storage of data, avoiding the situation of too high a storage tree level, and reducing the data access latency of the storage tree, laying a foundation for improving the concurrent access volume and access efficiency of data.

[0020] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Combined with the specific application environment architecture or specific hardware architecture on which the execution of the data storage method depends, the specific application environment architecture or specific hardware architecture is described herein.

[0022] First, the network structure on which the present application is based will be described: The data storage method, system, electronic device, and storage medium provided by the embodiments of the present application are applicable to data storage processing of metadata waiting to be stored. As Figure 1 shown, it is a schematic diagram of the network structure based on the embodiments of the present application, mainly including a client and a server. The client is used to initiate a data storage request or a data reading request to the server, and the server is used to respond to the data storage request or the data reading request initiated by the client, and perform corresponding data processing based on the data storage method provided by the embodiments of the present application to achieve the decentralized storage of data, avoid the situation of too high a storage tree level, and reduce the data access latency of the storage tree.

[0023] An embodiment of the present application provides a data storage method for performing data storage processing on metadata waiting to be stored. The execution subject of the embodiment of the present application is an electronic device, such as a server, a desktop computer, a laptop computer, a tablet computer, and other electronic devices that can be used for data storage.

[0024] As Figure 2 shown, it is a schematic flowchart of the data storage method provided by the embodiment of the present application. The method includes: Step 201, obtain a data storage request.

[0025] Among them, the data storage request is a write request initiated by the client and the upper-layer application. The request at least includes the data to be stored and the target storage address. The target storage address at least includes a logical block address (Logical Block Address, abbreviated as: LBA).

[0026] Step 202, screen the target storage unit from multiple storage units according to the target storage address represented by the data storage request.

[0027] Among them, each storage unit includes a buffer and a storage tree. The buffer is a temporary storage area in the memory, and the storage tree adopts a tree index structure such as a B+ tree.

[0028] Step 203, write the data to be stored indicated by the data storage request into the buffer of the target storage unit.

[0029] It should be noted that the buffer is located in the memory area, and the leaf nodes for storing data in the storage tree are located in the external storage area such as a disk. Since the speed of writing data into the memory is much higher than that of writing data into the external storage, by writing the data to be stored indicated by the data storage request into the buffer of the target storage unit, the data storage efficiency can be improved. And, since the data to be stored may be modified within a certain period, the buffer is more convenient for data modification, and at the same time, it avoids the situation of write amplification of the storage tree caused by frequent data modification.

[0030] Step 204, when the buffer meets the preset flushing condition, flush the data to be flushed in the buffer to the storage tree.

[0031] Specifically, when the buffer utilization rate (storage space occupancy rate) reaches the preset threshold or reaches the timing refresh period, it is determined that the buffer meets the preset flushing condition. By flushing the data to be flushed in the buffer to the storage tree, the storage space of the buffer can be released for storing subsequent data.

[0032] Step 205, when the current level number of the storage tree exceeds the storage tree level number threshold, perform a split process on the storage tree to split the storage tree into multiple storage sub-trees.

[0033] Among them, the number of levels of the storage subtree is equal to the threshold of the number of levels of the storage tree.

[0034] Specifically, when the number of levels of the storage tree is relatively high (the current number of levels exceeds the threshold of the number of levels of the storage tree), the storage tree is split into multiple short trees (storage subtrees), reducing the number of non-leaf nodes, achieving control over the height of the storage tree, and preventing the query performance from decreasing due to the excessive height of the storage tree.

[0035] Based on the above embodiments, as an implementable manner, in one embodiment, screening a target storage unit from multiple storage units according to the target storage address represented by the data storage request includes: Step 2021, in a preset static index layer, search layer by layer through multiple routing layers according to the target storage address represented by the data storage request to determine the target location layer entry corresponding to the target storage address; Step 2022, screen the target storage unit from multiple storage units according to the target location layer entry.

[0036] Among them, the preset static index layer includes multiple routing layers and location layers, the location layer includes multiple location layer entries, and each location layer entry corresponds to a storage unit one by one.

[0037] Exemplarily, such as Figure 3As shown in the figure, it is a schematic structural diagram of an exemplary data storage system provided by an embodiment of the present application. The system includes a preset static index layer and a slot layer. The slot layer includes multiple storage units. The routing layer includes multiple routing layer entries. Each routing layer entry records the entry information in the form of key-value pairs. The starting logical address is used as the Key value (K), and the offset is used as the Value value (V). The logical address range corresponding to the routing layer entry is [Key, Key + Value]. The offsets of the routing layer entries in adjacent routing layers differ by 10 times, and the number of routing layer entries in adjacent routing layers also differs by 10 times, that is, the logical address ranges recorded by the routing layer entries decrease gradually. By searching through multiple routing layers layer by layer, the logical address range (routing layer entry) corresponding to the target storage address can be accurately located in the last routing layer. Finally, according to the correspondence between the routing layer entry in the last routing layer and each location layer entry in the location layer, the target location layer entry corresponding to the target storage address is determined. The location layer entry also records the entry information in the form of key-value pairs. In the location layer entry, the starting logical address is used as the Key value, and the pointer Ptr is used as the Value value. According to the target pointer represented by the Value value of the target location layer entry, the target storage unit is screened in the slot layer. The dynamic index layer (slot layer) is composed of multiple storage slots (storage units). Each storage slot stores a certain range of key-value pair data and is pointed to by an index entry of a location layer. The data ranges between the storage slots do not intersect and are arranged in ascending order according to the array subscript order of the location layer.

[0038] Among them, the static index layer is used to accelerate positioning and is composed of a routing layer and a location layer. Each structural expansion generates a new static index layer (routing layer and location layer), but once generated, it will not change. The location layer is an index entry array large enough. Each array element stores a Key and a pointer to the corresponding storage slot, which is used to index the storage slots of the dynamic index layer. The index entry Key indicates the maximum Key stored in this storage slot, that is, the maximum Key of the root node of the storage tree in the storage slot. The routing layer is a multi-dimensional array based on the location layer and is a skip list-like structure. Each element is the array subscript of the location layer, and the specific slot (target storage unit) of the dynamic index layer is located through multi-level binary search.

[0039] In the data storage system provided by the embodiments of the present application, a hierarchical index organization method is proposed, which combines static index and dynamic index (slot layer). The upper-layer static index is used to accelerate positioning, and the lower-layer dynamic index layer consists of multiple storage units each containing an independent buffer and a storage tree, which is responsible for persisting data. The hierarchical structure reduces the number of some non-leaf nodes. At the same time, the amount of non-leaf nodes of the storage tree with high access frequency is much smaller than that of leaf nodes. Therefore, all non-leaf nodes are cached in memory, ensuring that only one external memory access is required to obtain data when the read request misses the cache, thereby improving the read performance. At the same time, using multiple short trees and independent and scattered buffers (buffers) will greatly reduce the impact of disk flushing, and the background regularly expands and maintains the index structure, which always consists of short trees. When the system is initialized, it only contains a single storage unit and gradually expands as the data volume grows.

[0040] Correspondingly, in one embodiment, after the storage tree is to be split to split the storage tree into multiple storage sub-trees, corresponding numbers of new storage units can be generated according to the multiple storage sub-trees; location layer entries are allocated to the new storage units; and the preset static index layer is modified according to the allocation result of the location layer entries.

[0041] Specifically, when the storage tree is split, the data pages cached in the buffer corresponding to the storage tree are also correspondingly divided, so as to generate corresponding new storage units for each storage sub-tree, allocate location layer entries to the new storage units, and adaptively modify the preset static index layer, so that when the data stored in the storage sub-tree is accessed subsequently, the corresponding new storage unit can be accurately located through the preset static index layer to achieve fast access to the data.

[0042] Based on the above embodiments, as an implementable manner, in one embodiment, when the buffer meets the preset disk flushing condition, flushing the data to be flushed in the buffer to the storage tree includes: Step 2041, when the buffer meets the preset disk flushing condition, convert the buffer to a read-only state and create a new buffer, where the new buffer is used to cache the data to be stored written by the upper level during the disk flushing process; Step 2042, flush the data to be flushed in the read-only state buffer to the storage tree.

[0043] It should be noted that the buffer stores the incremental key-value pair data of the logical address range of the storage slot in an ordered structure. The incremental key-value pair data includes the key-value pair data of insertions, modifications, and deletions, that is, the data to be flushed is the incremental key-value pair data. These complete incremental key-value pair data in the buffer will be saved during power failure to prevent data loss. The flushing of the buffer data is to check whether there is a buffer cache refresh of a single storage slot that meets the preset flushing conditions for all storage slots (storage units) in the dynamic index layer (slot layer) from left to right. If the buffer flushes data because it reaches the timing refresh period, it can reduce the amount of data flushed for dirty data (data to be flushed); if the buffer flushes data because the storage space occupancy rate reaches the preset threshold, it can disperse the flushing of a large amount of data into multiple times, thereby reducing the structural adjustment and write amplification overhead caused by writing dirty data to disk each time, alleviating performance jitter, and reducing tail latency.

[0044] Specifically, when the buffer meets the preset flushing conditions, first lock the storage unit, set the buffer to a read-only state, and prevent new data from being written into the buffer to avoid new data mixing in during the data flushing process and ensure that the data flushed to the storage tree is accurate. At the same time, create a new buffer to receive the data to be stored written by the upper layer during the data flushing process to prevent the write operation from being blocked due to the flushing of the old buffer. After creating the new buffer, unlock the storage unit to enable the storage unit to provide services normally.

[0045] Among them, by flushing the data to be flushed in the read-only state buffer to the storage tree, the transformation of data from temporary storage in memory to persistent storage in external storage is realized, ensuring the reliability and persistence of the data.

[0046] Specifically, in one embodiment, for any data to be flushed, the target leaf node can be determined in the storage tree of the same storage unit as the read-only state buffer according to the target storage address of the data to be flushed; perform a copy process on the target leaf node to obtain a copy leaf node; flush the data to be flushed to the copy leaf node and temporarily store the copy leaf node in memory; when all the data to be flushed in the read-only state buffer have been flushed to the corresponding copy leaf nodes, perform a merge process on the copy leaf nodes generated by the same target node to obtain at least one leaf node to be persisted; insert at least one leaf node to be persisted into the storage tree to replace the original target leaf node in the storage tree.

[0047] Specifically, for each piece of data to be flushed, first its target storage address is found in the storage tree of the same storage unit as the read-only status buffer to find the corresponding target leaf node. After finding the target leaf node, a copy operation is performed on it to generate a copy leaf node. During the copying process, the copy leaf node can inherit the attributes and data content of the original target leaf node, that is, the data is flushed using a copy-on-write method. In the case of multi-threaded concurrent access, read and write operations can still access the original target leaf node, and write operation blocking will not occur, which ensures the concurrent performance of leaf node data access and prevents direct modification of the original node from affecting other ongoing read and write operations.

[0048] Specifically, the data to be flushed is written into the replica leaf node, and the replica leaf node becomes a dirty node containing new data, which is temporarily stored in the memory. When all the data to be flushed in the read-only state buffer are flushed to the corresponding replica leaf node, the replica leaf nodes copied from the same target node are merged. During the merging process, the data in the replica leaf nodes are integrated, and the duplicate or redundant parts are removed to form at least one leaf node to be persisted. Finally, at least one leaf node to be persisted obtained after the merge process is inserted into the storage tree, replacing the original target leaf node, completing the update and persistent storage of the data in the storage tree, so that the storage tree can reflect the latest data status and ensure the consistency and integrity of the data.

[0049] Specifically, in one embodiment, in order to filter the data to be flushed from the buffer, the value of the data in the buffer can be defined, and the value of each piece of data is proportional to the importance of the data. When the buffer meets the preset flushing conditions, it is sorted in descending order of value, and the data with the highest value is selected as the data to be flushed, so as to ensure that key data (data with higher value) is persisted to the storage tree first.

[0050] Specifically, in one embodiment, the data in the buffer can also be classified into hot and cold data according to the access frequency of each data in the buffer, and the cold data can be used as data to be flushed, so that the hot data can be retained in the buffer, so that when the user subsequently initiates a data read request for the hot data, the data can be directly read from the buffer, thereby improving the user's access efficiency to the storage unit.

[0051] Furthermore, in one embodiment, a new non-leaf node can be generated based on the index information of the data to be flushed stored in the leaf node to be persisted; the new non-leaf node is inserted into the storage tree to replace the original non-leaf node in the storage tree.

[0052] Among them, the index information of the data to be flushed includes the key value range (logical address range) and pointers. Non-leaf nodes mainly play the role of indexing and guiding searches in storage tree structures such as B+ trees. They do not store actual data but indicate the location of the leaf nodes where the data is located through key-value pairs and pointers.

[0053] Specifically, the data to be persisted in the leaf nodes stores the data flushed to the storage tree. Based on the index information of these data, new non-leaf nodes are constructed. The construction of the new non-leaf nodes is based on the data in the leaf nodes to be persisted, updating and optimizing the index structure of the storage tree. For example, if the leaf nodes to be persisted contain a series of consecutive key value ranges, the new non-leaf nodes can be reasonably divided and organized according to these ranges to more efficiently locate the data in subsequent queries. Finally, the generated new non-leaf nodes are inserted into the storage tree to replace the original non-leaf nodes.

[0054] Furthermore, the information of the new root node of the storage tree is synchronized to the system metadata so that the preset static index layer can be adaptively modified according to the system metadata subsequently. Among them, the root node information includes the address information of the root node of the storage tree.

[0055] Among them, as Figure 4 shown, it is a schematic structural diagram of the storage unit provided by the embodiment of the present application. During the data process of the storage unit, the storage nodes include a new buffer (Buffer_cache), a read-only status buffer (Buffer_commit), and a storage tree. After completing the above data flushing operation, the storage unit is locked again, the read-only status buffer is deleted in the storage nodes, and the new root node of the storage tree is switched according to the new tree node information represented by the system metadata. Finally, the storage unit is unlocked so that the storage unit can be normally applied.

[0056] Based on the above embodiments, as an implementable manner, in one embodiment, the method further includes: Step 301, obtaining the number of data volumes, the total data volume, and the written data volume of the data storage system; Step 302, determining the structural constraint conditions of the storage unit according to the number of data volumes, the total data volume, and the written data volume of the data storage system; Step 303, according to the current number of storage units and the structural constraint conditions of the storage unit; Step 304, determining the threshold of the storage tree level according to the current number of storage units and the structural constraint conditions of the storage unit.

[0057] Among them, the structural constraint conditions of the storage unit include the minimum height of the storage tree, the maximum height of the storage tree, the expected number of storage units, and the current number of storage units.

[0058] It should be noted that the number of data volumes is the total number of logical volumes in the system; the total data volume capacity is the total storage space of all logical volumes; the written data capacity is the actual amount of data that has been written into the system currently. The minimum height of the storage tree is the lowest level number that should be maintained to ensure query efficiency; the maximum height of the storage tree is the highest level number allowed to avoid a decline in query performance due to an overly tall tree; the expected number of storage units is the optimal number of storage units calculated based on the data volume and load. Among them, as the number of storage units increases, the threshold of the storage tree level number will gradually increase and finally become the maximum height of the storage tree.

[0059] Specifically, in one embodiment, the threshold of the storage tree level number can be determined based on the following formula:

[0060] Among them, represents the threshold of the storage tree level number, represents the minimum height of the storage tree, represents the maximum height of the storage tree, represents the current number of storage units, represents the expected number of storage units.

[0061] Specifically, as the number of data volumes, the total data volume capacity, and the written data capacity of the data storage system change, and as the current number of storage units after the split and merge of the storage tree changes, the threshold of the storage tree level number also adapts and adjusts accordingly to ensure that the threshold of the storage tree level number matches the actual business scenario of the data storage system, thereby ensuring the reliability of subsequent storage tree split processing and merge processing.

[0062] Based on the above embodiment, as an implementable method, in one embodiment, the storage tree will be split to split the storage tree into multiple storage sub - trees, including: Step 2051, determine the split strategy of the storage tree according to the difference between the current level number of the storage tree and the threshold of the storage tree level number; Step 2052, according to the split strategy of the storage tree, remove at least one upper - level node of the storage tree to make multiple non - leaf nodes become root nodes, and the storage tree is split into multiple storage sub - trees.

[0063] It should be noted that the system needs to ensure that the storage tree of each storage unit always maintains a relatively low tree height (level) state. Therefore, the average number of leaf nodes of the storage tree of each storage unit can be used as a measurement basis to design background expansion operations to balance the storage tree structures of all storage slots. The expansion operation is an operation that is friendly to hot loads. When the written data is mainly concentrated in a few storage units, the growth of the storage tree structure will reduce the access efficiency of hot data. At this time, through the expansion operation (storage tree splitting), the height of the storage tree in the storage unit with concentrated hot spots can be reduced, and the hot spots can be dispersed to more storage units, so as to maintain stable performance.

[0064] Among them, the segmented expansion method adopted in the embodiments of the present application divides the hierarchical index structure into three segments during the expansion process, namely the expanded area, the area being expanded, and the unexpanded area. The system maintains the Key range (logical address range) of the part being expanded. When the Key of a new request (destination address) is within this range, it cannot be processed immediately and needs to wait for the expansion of this range to complete; if the Key of the new request is less than or equal to the system expansion Key, it will go to the expanded area part, that is, the new version of the index for processing; if the Key of the new request is greater than the system expansion Key, it will go to the unexpanded part, that is, the old version of the index for processing.

[0065] Specifically, during the expansion process of the storage tree, first create a new empty positioning layer large enough, and then sequentially traverse each storage unit of the original dynamic index layer, and perform balancing processing according to the specific situation of the storage tree in the storage unit. When the height of the storage tree approaches the height threshold t, no processing is performed, and only the index items of the new positioning layer are pointed to the original storage slot. When the height of the storage tree exceeds the height threshold, a split operation is performed to split the buffer and the storage tree into multiple parts to form multiple new storage units, and new positioning layer index items are created in sequence to point to the corresponding storage units. When the height of the storage tree is less than the height threshold, a merge operation is performed, and continue to traverse the subsequent storage units until a storage tree with a height not less than the threshold is found, and then all the storage trees with a height less than the threshold that have been traversed are merged, and it is ensured that the height of the merged storage tree does not exceed the threshold t, and at the same time, the cache buffer is merged to form a new storage unit. Finally, new positioning layer index items (positioning layer table items) are created. After each new positioning layer index item is generated, this information is written into the system metadata file, and a new routing layer index is constructed during the continuous generation of new positioning layer index items. When all the storage units of the dynamic index layer are expanded, the old version of the routing layer and positioning layer are deleted, and the new version of the index is used as the new core index to achieve adaptive update of the preset static index layer.

[0066] Exemplarily, such as Figure 5As shown in the figure, it is a schematic diagram of the storage tree splitting process provided by the embodiment of the present application. The splitting operation of the storage tree takes the storage tree level number threshold t as the dividing line, and the process of splitting the high-level storage tree into multiple sub-storage trees. The splitting operation does not require updating the B+Tree, but only needs to update the memory pointer in the index area, so it will not bring additional node data persistence. Taking t = 2 and n = 4 as an example, the difference between the two is 2, and then it is determined that the splitting strategy of the storage tree is to remove the first-level upper node (root node 1) and the second-level upper node (sub-nodes 2 and 3 of the root node) in the storage, so that the non-leaf nodes 4, 5, and 6 become the root nodes, that is, the splitting expands 3 storage sub-trees (storage units).

[0067] Correspondingly, in an embodiment, when the current level number of the storage tree is lower than the storage tree level number threshold, the storage tree is used as the storage tree to be merged; multiple storage trees to be merged are merged to obtain the merged storage tree, and at the same time, the storage units of the storage trees to be merged are merged to obtain the merged storage unit.

[0068] Exemplarily, as Figure 6 shown, it is a schematic diagram of a storage tree merging process provided by the embodiment of the present application. As Figure 7 shown, it is another schematic diagram of a storage tree merging process provided by the embodiment of the present application. The merging operation is divided into two types. One is the merging of multiple storage trees with the same level number as Figure 6 shown. Since the logical address ranges between the storage units are arranged in an increasing order and the root nodes between the storage trees are ordered, in this case, only a new root node needs to be generated to organize the multiple trees. The other is as Figure 7 shown, the merging of multiple storage trees with different numbers of layers. Similarly, due to the orderliness of the Key (logical address) between the trees, the merging process only needs to insert the root node of the shorter tree as a sub-node into the root node of the taller tree. Both merging operations will generate new dirty root nodes, bringing a small amount of additional node data persistence.

[0069] Based on the above embodiments, as an implementable manner, in an embodiment, the method further includes: Step 401, obtaining a data reading request; Step 402, screening the storage units to be accessed from multiple storage units according to the target reading address represented by the data storage request; Step 403, traversing the buffer of the storage unit to be accessed to determine whether the buffer stores the data to be read; Step 404, directly reading the data to be read from the buffer when it is determined that the buffer stores the data to be read; Step 405, in the case where it is determined that the buffer does not store the data to be read, traverse the storage tree of the storage unit to be accessed to locate the leaf node to be accessed, and read the data to be read from the leaf node to be accessed.

[0070] Among them, as Figure 8 shown, it is a schematic diagram of the request response process provided by the embodiment of the present application. The routing layer and the positioning layer are static structures, and there is no need to lock to handle conflicts during searching, and all user request threads can perform parallel addressing. Each storage slot (storage unit) in the dynamic index layer (slot layer) is independently managed and the key ranges do not intersect. The system allocates threads according to storage units, one thread per storage unit, and there is no need to lock for internal access to storage units. The data dispersion effect of multiple storage units can effectively alleviate the impact of conflicts on performance. As Figure 8 in the write 2 (write request 2) needs to wait for the write 1 (write request 1) to complete before writing, and the read requests for the same storage unit can be responded in parallel, that is, as Figure 8 in the read 1 (read request 1) and read 2 (read request 2) in can be performed simultaneously.

[0071] Specifically, when a read request (data read request) is obtained, first locate the corresponding storage slot in the dynamic index layer through the routing layer and the positioning layer according to the read key value (target read address). First, query the buffer. If the data to be read is not found, continue to search in the B+Tree (storage tree). The search in the storage tree starts from the root node, and binary search is performed in each node, layer by layer down until it is determined whether the data exists in the leaf node. If any leaf node cannot be obtained from the node cache, it will be obtained from the disk file according to the node address and temporarily stored in the node cache for the next query.

[0072] The data storage method provided by the embodiments of the present application includes obtaining a data storage request; screening a target storage unit from multiple storage units according to the target storage address represented by the data storage request; wherein each storage unit includes a buffer and a storage tree; writing the data to be stored indicated by the data storage request into the buffer of the target storage unit; when the buffer meets a preset write-down condition, writing down the data to be written down in the buffer to the storage tree; when the current level number of the storage tree exceeds the storage tree level number threshold, performing a splitting process on the storage tree to split the storage tree into multiple storage sub-trees; wherein the level number of the storage sub-tree is equal to the storage tree level number threshold. The method provided by the above solution splits the entire data storage system into multiple storage units, each storage unit includes a buffer and a storage tree. The newly written data is first stored in the buffer, and then the data is written down to the storage tree by writing down the data in the buffer, so as to realize batch writing of data in the storage tree, reduce the write amplification of the storage tree, and when the storage tree level is relatively high, split the high-level storage tree into multiple storage sub-trees, realizing decentralized storage of data, avoiding the situation of too high a storage tree level, reducing the data access latency of the storage tree, and laying a foundation for improving the concurrent access volume and access efficiency of data.

[0073] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0074] The embodiments of the present application further provide a data storage device for executing the data storage method provided by the above embodiments.

[0075] As Figure 9 shown, it is a schematic structural diagram of the data storage device provided by the embodiments of the present application. The data storage device 90 includes: an obtaining module 901, a screening module 902, a writing module 903, a write-down module 904, and a splitting module 905.

[0076] Among them, the obtaining module is used to obtain a data storage request; the screening module is used to screen a target storage unit from multiple storage units according to the target storage address represented by the data storage request; wherein each storage unit includes a buffer and a storage tree; the writing module is used to write the data to be stored indicated by the data storage request into the buffer of the target storage unit; the write-down module is used to write down the data to be written down in the buffer to the storage tree when the buffer meets a preset write-down condition; the splitting module is used to perform a splitting process on the storage tree when the current level number of the storage tree exceeds the storage tree level number threshold, so as to split the storage tree into multiple storage sub-trees; wherein the level number of the storage sub-tree is equal to the storage tree level number threshold.

[0077] For the descriptions of the features in the corresponding embodiments of the data storage device, reference can be made to the relevant descriptions in the corresponding embodiments of the data storage method, which will not be elaborated here one by one.

[0078] An embodiment of the present application further provides a data storage system for executing the data storage method provided in the above embodiment.

[0079] As Figure 10 shown, it is a schematic structural diagram of the data storage system provided by the embodiment of the present application. The data storage system includes: a plurality of storage units and a data storage device.

[0080] Among them, each storage unit includes a buffer and a storage tree.

[0081] The data storage device uses the data storage method provided in the above embodiment to perform data storage processing on a plurality of storage units.

[0082] For the descriptions of the features in the corresponding embodiments of the data storage device, reference can be made to the relevant descriptions in the corresponding embodiments of the data storage method, which will not be elaborated here one by one.

[0083] An embodiment of the present application further provides an electronic device, as Figure 11 shown, it is a schematic structural diagram of the electronic device provided by the embodiment of the present application, including a processor 10 and a memory 20. A computer program is stored in the memory 20, and the processor 10 is configured to run the computer program to execute the steps in any of the above data storage method embodiments.

[0084] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. Among them, the computer program is configured to execute the steps in any of the above data storage method embodiments when running.

[0085] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: USB flash drives, read-only memories (abbreviated as ROM), random access memories (abbreviated as RAM), mobile hard disks, magnetic disks, or optical discs and other various media that can store computer programs.

[0086] An embodiment of the present application further provides a computer program product. The above computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above data storage method embodiments.

[0087] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium storing a computer program, and the computer program, when executed by a processor, implements the steps in any of the above data storage method embodiments.

[0088] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0089] The above has introduced in detail a data storage method, system, electronic device, and storage medium provided by the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A data storage method, characterized in that, including: obtaining a data storage request; screening a target storage unit from multiple storage units according to a target storage address characterized by the data storage request; wherein each of the storage units includes a buffer and a storage tree; writing data to be stored indicated by the data storage request into the buffer of the target storage unit; when the buffer meets a preset write-down condition, writing the data to be written down in the buffer to the storage tree; when the current level number of the storage tree exceeds a storage tree level number threshold, performing a splitting process on the storage tree to split the storage tree into multiple storage sub-trees; wherein the level number of the storage sub-tree is equal to the storage tree level number threshold.

2. The data storage method according to claim 1, wherein The screening of the target storage unit from multiple storage units according to the target storage address characterized by the data storage request includes: at a preset static index layer, performing a layer-by-layer search on multiple routing layers according to the target storage address characterized by the data storage request to determine a target location layer entry corresponding to the target storage address; screening a target storage unit from multiple storage units according to the target location layer entry; wherein the preset static index layer includes multiple routing layers and a location layer, the location layer includes multiple location layer entries, and each of the location layer entries corresponds to a storage unit one by one.

3. The data storage method according to claim 2, wherein After performing the splitting process on the storage tree to split the storage tree into multiple storage sub-trees, the method further includes: generating a corresponding number of new storage units according to the multiple storage sub-trees; allocating location layer entries to the new storage units; modifying the preset static index layer according to the allocation result of the location layer entries.

4. The data storage method according to claim 1, wherein The writing of the data to be written down in the buffer to the storage tree when the buffer meets a preset write-down condition includes: when the buffer meets a preset write-down condition, converting the buffer to a read-only state and creating a new buffer, the new buffer being used to cache data to be stored written by a superior during the write-down process; writing the data to be written down in the read-only state buffer to the storage tree.

5. The data storage method according to claim 4, wherein The writing of the data to be written down in the read-only state buffer to the storage tree includes: for any of the data to be written down, determining a target leaf node in the storage tree of the same storage unit as the read-only state buffer according to the target storage address of the data to be written down; performing a copy process on the target leaf node to obtain a copy leaf node; writing the data to be written down to the copy leaf node and temporarily storing the copy leaf node in a memory; when all the data to be written down in the read-only state buffer have been written down to the corresponding copy leaf nodes, performing a merging process on the copy leaf nodes copied from the same target node to obtain at least one leaf node to be persisted; inserting the at least one leaf node to be persisted into the storage tree to replace the original target leaf node in the storage tree.

6. The data storage method according to claim 5, characterized in that The method further includes: generating a new non-leaf node according to index information of the data to be written down stored in the leaf node to be persisted. Insert the new non-leaf node into the storage tree to replace the original non-leaf node in the storage tree.

7. The data storage method according to claim 1, wherein The method further includes: Obtain the number of data volumes, the total data volume capacity, and the written data capacity of the data storage system; Determine the structural constraint conditions of the storage unit according to the number of data volumes, the total data volume capacity, and the written data capacity of the data storage system; According to the current number of the storage units and the structural constraint conditions of the storage unit; Determine the threshold of the storage tree level number according to the current number of the storage units and the structural constraint conditions of the storage unit; Wherein, the structural constraint conditions of the storage unit include the minimum height of the storage tree, the maximum height of the storage tree, the expected number of storage units, and the current number of storage units.

8. The data storage method according to claim 7, wherein The determining the threshold of the storage tree level number according to the current number of the storage units and the structural constraint conditions of the storage unit includes: Determine the threshold of the storage tree level number based on the following formula: Among them, represents the threshold of the number of levels of the storage tree, represents the minimum height of the storage tree, represents the maximum height of the storage tree, represents the current number of the storage units, represents the expected number of the storage units.

9. The data storage method according to claim 1, wherein The splitting process to be performed on the storage tree to split the storage tree into multiple storage sub-trees includes: Determine the splitting strategy of the storage tree according to the difference between the current level number of the storage tree and the threshold of the storage tree level number; According to the splitting strategy of the storage tree, remove at least one upper-level node of the storage tree, so that multiple non-leaf nodes become root nodes, and the storage tree is split into multiple storage sub-trees.

10. The data storage method according to claim 1, wherein The method further includes: In the case where the current level number of the storage tree is lower than the threshold of the storage tree level number, regard the storage tree as a storage tree to be merged; Perform a merging process on multiple storage trees to be merged to obtain a merged storage tree, and at the same time perform a merging process on the storage units of the storage trees to be merged to obtain merged storage units.

11. The data storage method according to claim 1, wherein The method further includes: Obtain a data reading request; Filter the storage units to be accessed among multiple storage units according to the target reading address represented by the data storage request; Traverse the buffer of the storage unit to be accessed to determine whether the buffer stores the data to be read; In the case where it is determined that the buffer stores the data to be read, directly read the data to be read from the buffer; In the case where it is determined that the buffer does not store the data to be read, traverse the storage tree of the storage unit to be accessed to locate the leaf node to be accessed, and read the data to be read from the leaf node to be accessed.

12. A data storage system, characterized in that, Includes: Multiple storage units and a data storage device; wherein each storage unit includes a buffer and a storage tree; The data storage device performs data storage processing on the multiple storage units by using the data storage method according to any one of claims 1 to 11.

13. An electronic device, characterized in that, Includes: A memory for storing a computer program; A processor for implementing the steps of the data storage method according to any one of claims 1 to 11 when executing the computer program.

14. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program implements the steps of the data storage method according to any one of claims 1 to 11 when executed by a processor.

15. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data storage method according to any one of claims 1 to 11.

Images