Data access method and storage device

Abstract

A data access method and a storage device, the method comprising: a storage device receiving a write access request, the write access request comprising a first key and first data to be written; writing the first key and the first data in the memory of the storage device based on the first index structure; based on a second index structure, recording a corresponding relationship between a first key and first metadata in an index file, the first metadata being used for indicating a storage address of first data in a memory or a disk of the storage device, and the index file being stored in the memory of the storage device, the index file is used for reading data stored in the memory or the disk. According to the method, an index scheme with excellent read-write performance can be provided, so that the overhead and time delay of data read access can be reduced.

Description

[0001] This application claims priority to the Chinese patent application filed with the Intellectual Property Office of the People's Republic of China on July 7, 2023, with application number 202310835543.4 and invention name "A Data Access Method", the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a data access method and a storage device. Background Art

[0003] With the rapid development of computer technology, the requirements for storage performance are getting higher and higher. Among them, index information is the key path of the input / output process in the storage field, and is crucial to the read and write performance of storage products. Log-Structured Merge-Tree (LSM tree) is a basic algorithm of indexing technology, which is currently widely used in major companies and open source products.

[0004] However, the core of LSM tree is to give up some reading ability in exchange for maximizing writing ability. As more data is written to the storage system, performing read operations based on LSM tree will require a large number of disk read operations, which is time-consuming.

[0005] Therefore, how to construct an indexing solution with excellent read and write performance is still an important issue that needs to be solved urgently. Summary of the invention

[0006] Embodiments of the present application provide a data access method and a storage device for providing an indexing solution with excellent read and write performance to reduce the overhead and latency of data read access.

[0007] In a first aspect, an embodiment of the present application provides a data access method, which can be executed by a storage device, and the storage device can be a storage device that stores data in an LSM-Tree or a storage node in a storage system.

[0008] In this method, a storage device receives a write access request, wherein the write access request includes a first key and first data to be written; based on a first index structure, the first key and the first data are written in the memory of the storage device; based on the second index structure, the correspondence between the first key and the first metadata is recorded in an index file, wherein the first metadata is used to indicate the storage address of the first data in the memory or disk of the storage device, the index file is stored in the memory of the storage device, and the index file is used to read the data stored in the memory or the disk.

[0009] Through the above method, the LSM-Tree-based write IO path of the storage device is not changed, but an additional fast read path different from the traditional read IO path of the LSM-Tree is constructed, so that when the storage device receives a read access request, it can quickly access the memory or disk of the storage device based on the constructed fast read path to quickly find the target data to be read. For example, the storage device receives a read access request, and the read access request includes the first key; the first metadata is queried in the index file stored in the memory according to the first key; and the first data is read from the memory or the disk according to the first metadata.

[0010] In a possible implementation, if the first data is stored in the memory, the first metadata includes the storage address of the first data in the memory. In another possible implementation, if the first data is stored in the disk, the first metadata includes the storage address of the first data in the disk. That is to say, according to the different locations where the first data is written, the index file can also be updated accordingly, so as to timely update the correspondence between the first key corresponding to the first data and the metadata in the index file, so as to ensure that the storage device can read the correct data when processing the received read access request.

[0011] In a possible implementation, recording the correspondence between the first key and the first metadata in the index file based on the second index structure may include: when writing the first key and the first data in the memory, recording the correspondence between the first key and the first metadata in the index file based on the second index structure.

[0012] In a possible implementation, based on the second index structure, recording the correspondence between the first key and the first metadata in the index file includes: when the second data stored in the memory is merged and stored on the disk, recording the correspondence between the first key and the first metadata in the index file based on the second index structure, the second data includes the first data, and the first metadata is used to indicate the storage address of the first data in the disk (for example, layer 0).

[0013] Through the above method, in a storage device based on LSM-Tree, when the data in the memory is merged and stored in the disk, since the storage address of the corresponding data changes, such as changing from the storage address in the memory to the storage address in the disk, it is necessary to update the new metadata of these data in the index file stored in the memory. The new metadata is used to indicate the new storage address of each of these data in the disk.

[0014] In a possible implementation, the disk uses layered storage, and the method further includes: when merging the third data stored in the i-th layer of the disk and storing it in the i+1-th layer of the disk, recording the correspondence between the first key and the first metadata in the index file based on the second index structure, the third data includes the first data, and the first metadata is used to indicate the storage address of the first data in the i+1-th layer, where i is an integer greater than or equal to 0.

[0015] Through the above method, since the disk of the storage device adopts a hierarchical storage structure, when the data in the previous layer meets the conditions and needs to be merged and stored in the next layer, the storage address of these data in the disk will also change, for example, from the storage address in layer 0 to the storage address in layer 1. Accordingly, the metadata of these data needs to be updated in the index file stored in the memory. The metadata is used to indicate the new storage address of each of these data in the disk.

[0016] In a possible implementation, the first index structure includes a log structure merged with an LSM tree structure. In a possible implementation, the second index structure includes any one of the following: a hash structure, a B+ tree structure, or a learning index structure. It should be understood that the introduction of the first index structure and the second index structure here is only an example and not a limitation. The data access method of the embodiment of the present application can also be applied to other indexing schemes that need to achieve excellent read and write performance, which will not be repeated here.

[0017] In a possible implementation, the storage device may be a storage device in any of the following storage architectures: an all-flash storage architecture, a cloud storage architecture, or a distributed storage architecture. That is, the embodiments of the present application do not limit the architecture scenarios to which the solution is applicable.

[0018] In a second aspect, an embodiment of the present application provides a storage device, which may include: a transceiver unit, used to receive a write access request, the write access request including a first key and a first data to be written; a processing unit, used to write the first key and the first data in the memory of the storage device based on a first index structure; based on the second index structure, recording the correspondence between the first key and the first metadata in an index file, the index file being stored in the memory of the storage device, and the index file being used to read data stored in the memory or the disk.

[0019] In a possible implementation, the transceiver unit is also used to receive a read access request, which includes the first key; the processing unit is also used to query first metadata in the index file based on the first key; and read the first data from the memory or the disk based on the first metadata.

[0020] In a possible implementation manner, if the first data is stored in the memory, the first metadata includes a storage address of the first data in the memory.

[0021] In a possible implementation, the processing unit is configured to: when writing the first key and the first data in the memory, record a correspondence between the first key and the first metadata in the index file based on the second index structure.

[0022] In a possible implementation manner, if the first data is stored in the disk, the first metadata includes a storage address of the first data in the disk.

[0023] In a possible implementation, the processing unit is used to: when merging the second data stored in the memory to the disk, based on the second index structure, record the correspondence between the first key and the first metadata in the index file, the second data including the first data, and the first metadata being used to indicate the storage address of the first data in the disk.

[0024] In a possible implementation, the disk uses layered storage, and the processing unit is further used to: when merging the third data stored in the i-th layer of the disk and storing it in the i+1-th layer of the disk, based on the second index structure, record the correspondence between the first key and the first metadata in the index file, the third data includes the first data, and the first metadata is used to indicate the storage address of the first data in the i+1-th layer, where i is an integer greater than or equal to 0.

[0025] In a possible implementation, the first index structure includes a log structure merged with an LSM tree structure.

[0026] In a possible implementation, the second index structure includes any one of the following: a hash structure, a B+ tree structure, or a learning index structure.

[0027] In a possible implementation, the storage device may be a storage device in any of the following storage architectures: an all-flash storage architecture, a cloud storage architecture, or a distributed storage architecture.

[0028] In a third aspect, the present application further provides a storage device, the storage device comprising a processor and a memory, and may further comprise a communication interface, the processor executing program instructions in the memory to execute the method provided in the above-mentioned first aspect or any possible implementation of the first aspect. The storage device may be a storage device in different types of storage systems, such as a centralized storage device, a distributed storage device, a cloud storage device, an all-flash storage device, etc. The memory is coupled to the processor, and stores program instructions and data necessary for determining data processing (such as storing a Bloom filter). The communication interface is used to communicate with other devices.

[0029] In a fourth aspect, the present application provides a storage system, which includes at least one storage device. Each storage device includes a memory and a processor. The processor of at least one storage device is used to access the code in the memory to execute the method provided by the first aspect or any possible implementation of the first aspect.

[0030] In a fifth aspect, the present application provides a computer-readable storage medium, and when the computer-readable storage medium is executed by a computing device, the computing device executes the method provided in the aforementioned first aspect or any possible implementation of the first aspect. The storage medium stores a program. The storage medium includes but is not limited to a volatile memory, such as a random access memory, a non-volatile memory, such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD).

[0031] In a sixth aspect, the present application provides a computing device program product, the computing device program product comprising computer instructions, when executed by a computing device, the computing device performs the method provided in the aforementioned first aspect or any possible implementation of the first aspect. The computer program product may be a software installation package, and when it is necessary to use the method provided in the aforementioned first aspect or any possible implementation of the first aspect, the computer program product may be downloaded and executed on a computing device.

[0032] In a seventh aspect, the present application further provides a computer chip, wherein the chip is connected to a memory, and the chip is used to read and execute a software program stored in the memory, and to execute the method described in the above-mentioned first aspect and various possible implementation methods of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the structure of an LSM-Tree provided in an embodiment of the present application;

[0034] Figure 2A schematic diagram of the structure of another LSM-Tree provided in an embodiment of the present application;

[0035] Figure 3 A flowchart of a data access method based on a Bloom filter provided in an embodiment of the present application;

[0036] Figure 4A to Figure 4B A schematic diagram of the architecture of a system provided in an embodiment of the present application;

[0037] Figure 5 A flowchart of a data access method provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of the structure of an LSM-Tree provided in an embodiment of the present application;

[0039] Figure 7 A flowchart of a data access method provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of the structure of an LSM-Tree provided in an embodiment of the present application;

[0041] Fig. 9 A flowchart of a data access method provided in an embodiment of the present application;

[0042] Fig.10 A schematic diagram of the structure of a storage device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] Before describing the data access method provided in the embodiment of the present application, the concepts involved in the embodiment of the present application are described first:

[0044] 1. Sorted string table (SSTable).

[0045] SSTable is a persistent, ordered and immutable key value (KV) storage structure. In the embodiment of the present application, the data of SSTable is the data stored in the SSTable. SSTable includes a series of blocks of configurable size, each block is used to store key and value, and the block is a logical space of the memory, such as LUN. The index of each block is stored at the end of the SSTable. When the processor in the storage device opens an SSTable, the index stored in the SSTable will be loaded into the memory of the storage device. After that, the index of the specified key loaded into the memory is searched to determine the index of the block where the key is located, and then the position of the block where the key is located in the disk is determined according to the index of the block where the key is located, and then the data (such as value) in the block where the key is located is read from the disk.

[0046] 2. Log-structured merge-tree (LSM-Tree).

[0047] LSM-Tree is a hierarchical, ordered, disk-oriented data storage structure. LSM-Tree provides a way to write data sequentially, which can effectively improve data writing efficiency.

[0048] LSM-Tree is a data storage structure that spans the memory and disk of a storage device.

[0049] Among them, Figure 1 As shown in the figure, the memory table (MemTable) is the data structure of LSM-Tree in memory, which is only used to save the latest data and organize the data in order according to the key (Key). Since the data in the memory will be lost due to power failure, the reliability of data operations is usually ensured by maintaining the Write Ahead Logging (WAL) in the disk.

[0050] In the disk, LSM-Tree is a multi-layer structure, for example, it may include level 0, level 1, level 2, level 3, ..., level n, where n is an integer greater than or equal to 1. The storage space of different levels can be different, for example, the storage space of level 0 is 10MB, the storage space of level 1 is 100MB, and the storage space of level n is 1000MB. Each level can include multiple SSTables. Figure 1 Only some layers in the LSM-Tree are drawn as examples. When the LSM-Tree is applied to different storage systems, the number of layers of the LSM-Tree can be set according to actual needs.

[0051] When the processor of the storage device performs a write operation (such as insert (or add)), LSM-Tree will first write MemTable and WAL at the same time. When the amount of data in MemTable reaches the preset threshold, the MemTable will be automatically converted into an immutable memory table (Immutable MemTable) and flushed to the disk to form a level 0 SSTable.

[0052] Among them, the insertion of LSM-Tree is relatively simple. It only needs to be inserted into the memory table, and it does not matter whether the data already exists in the memory or on the disk. Immutable MemTable is an intermediate state that converts MemTable into SSTable on the disk. During the conversion process, the write operation is handled by the new MemTable, and the data update operation is not blocked during the process. Only the data in the memory is allowed to be updated in place. Changes to the data on the disk are only allowed to be appended, and no in-place updates are made. When the Immutable MemTable is flushed to the disk, it will also be merged with the previous data in sequence (compaction). During the merging process, the original data will not be modified at the location, but the data will be directly inserted to avoid random writes. After the data in the memory table is flushed to the disk, the backup of this data is also deleted from the WAL accordingly.

[0053] Similarly, when the data volume of multiple SSTables in layer 0 reaches the preset threshold, they will also be merged, and the merged results will be written to the SSTable of the next layer (i.e. layer 1). When the data volume of multiple SSTables in layer 1 reaches the preset threshold, they will also be merged, and the merged results will be written to the SSTable of the next layer (i.e. layer 2), and so on, until the data is written to layer n.

[0054] Since different layers of the LSM-Tree have different SSTables, and the data is newer as you go up the layer, the storage device needs to search layer by layer when performing a read operation. Figure 1 As shown in the figure, the data in memory will be searched first, including in MemTable and Immutable MemTable. If the data cannot be found in memory, it will sink according to the order of the LSM-Tree layers, and query the SSTable of each layer of the LSM-Tree in turn until the data to be read is found.

[0055] As SSTables are continuously written, the processor will need to open more and more SSTables when performing read operations, and there will be more data changes (updates, deletions) accumulated for the same key. Since SSTables are immutable, in order to reduce the number of files and clean up invalid data in a timely manner, compaction operations must be performed to merge SSTables with overlapping key intervals. Since most of the data is in the bottom-level table, the search operation will search multiple tables at multiple levels, requiring a large number of disk read operations, resulting in large resource overhead and being very time-consuming.

[0056] It should be noted that the embodiment of the present application does not limit the type of data in SSTable, which can be the data itself or the metadata of the data. In other words, SSTable stores data in the form of key-value pairs, where key is the key assigned to the data and is used to uniquely identify the data. Value can be the data itself or the metadata of the data.

[0057] 3. Bloom filter (BF):

[0058] A Bloom filter is a binary vector and a series of random mapping functions that can be used to retrieve whether an element is in a set. Currently, in some designs, such as Figure 2 As shown, a Bloom filter can be configured for the SSTable of each layer of the LSM-Tree to determine whether the data to be read is stored in the various SSTables of this layer. Each SSTable can correspond to a Bloom filter. Of course, in some scenarios, the Bloom filters in each layer of the LSM-Tree can also be flexibly set, such as multiple SSTables can correspond to one Bloom filter. The embodiment of the present application does not limit the number of SSTables corresponding to the Bloom filter.

[0059] Based on the constructed Bloom filter, when searching for KV, each Bloom filter is searched in order. The Bloom filter will determine whether the key to be queried is on the SSTable managed by the current Bloom filter. If so, the target data will be read directly from the SSTable. If not, the Bloom filter of the next SSTable will be searched until the target data is read.

[0060] like Figure 3In the flowchart shown, taking a three-level LSM-Tree structure (including layer 0, layer 1 and layer 2, or represented as L0, L1, L2 respectively) and a Bloom filter corresponding to multiple SSTables in each layer as an example, when the processor of the storage device receives a read access request from a client device, it will first search the Bloom filter of the memory table based on the Key carried in the read access request. If the Bloom filter determines that the Key exists in the memory table, the processor will read the target data from the corresponding memory table. If the Bloom filter determines that the Key does not exist in the memory table, it will search the Bloom filter of the SSTable of the next layer L0. If the Bloom filter determines that the Key to be queried exists in a certain SSTable of L0, the processor will read the target data from the corresponding SSTable. If the Bloom filter determines that the Key to be queried does not exist in L0, it will search the Bloom filter of the SSTable of the next layer L1. If the Bloom filter determines that the Key to be queried exists in L1, the processor will read the target data from the corresponding SSTable. If the Bloom filter determines that the key to be queried does not exist in L1, it will search the Bloom filter of the SSTable in the next layer L2.

[0061] In the above process, although the Bloom filter can filter out SSTables that do not contain the target data, it only searches for SSTables that may contain the target data instead of reading each SSTable, which helps to reduce a lot of read access overhead.

[0062] However, Bloom filters have a certain misjudgment rate. In the case of misjudgment, if the Bloom filter determines that the target data exists in the table, but it does not actually exist, this will result in a disk read operation. If there is no misjudgment, only one disk read is required to directly read the SSTable where the target data is located to find the corresponding data. However, due to the existence of misjudgments and the large number of SSTables in the LSM-Tree in actual applications, even if Bloom filters are used for filtering, multiple tables still need to be searched, and the performance is still poor. For example, if the misjudgment rate of the Bloom filter is 5%, for an LSM-Tree including 100 SSTables, 5 disk reads are required. If the misjudgment rate is higher or the number of SSTables is larger, the number of disk reads will be greater.

[0063] It can be seen that even if Bloom filters are introduced into LSM-Tree, it is still impossible to optimize both read and write performance.

[0064] In view of this, the embodiments of the present application propose a data access method and storage device for providing an indexing scheme with excellent read and write performance to reduce the overhead and latency of data read access. Among them, the method and the device are based on the same technical concept. Since the principles of solving problems by the method and the device are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated. In addition, in each embodiment of the present application, if there is no special explanation and logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form a new embodiment according to their inherent logical relationship.

[0065] It should be noted that in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0066] Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the priority or importance of multiple objects. For example, the first index file and the second index file are only used to distinguish different index files, rather than indicating the difference in priority or importance of the two index files.

[0067] The data access scheme of the embodiment of the present application is described in detail below in conjunction with the drawings and embodiments.

[0068] like Figure 4A As shown, it is a schematic diagram of a system architecture provided in an embodiment of the present application, and the system architecture may include an application server 100, a switch 110, and a storage system 120.

[0069] Users access data through applications. Computers running these applications are called "application servers". Application server 100 can be a physical machine or a virtual machine. Physical application servers include but are not limited to desktop computers, servers, laptops, and mobile devices. Application servers access storage systems through fiber optic switches 110 to access data. However, switch 110 is only an optional device, and application server 100 can also communicate with storage system 120 directly through the network.

[0070] Figure 4A The storage system 120 shown is a centralized storage system. The centralized storage system is characterized by having a unified entrance through which all data from external devices must pass, and this entrance is the engine 121 of the centralized storage system. The engine 121 is the core component of the centralized storage system, and many advanced functions of the storage system are implemented in it.

[0071] like Figure 4A As shown, the engine 121 has one or more controllers. Figure 4A Take the example of an engine including two controllers, where a mirror channel is provided between controller 0 and controller 1, so that the two controllers can back up each other. Engine 121 also includes a front-end interface 125 and a back-end interface 126, wherein the front-end interface 125 is used to communicate with the application server 100 and provide storage services for the application server 100. The back-end interface 126 is used to communicate with the hard disk 134 (an example of a disk) to expand the capacity of the storage system. Through the back-end interface 126, the engine 121 is connected to more hard disks 134, thereby forming a very large storage resource pool.

[0072] In hardware, such as Figure 4AAs shown, the controller 0 includes at least a processor 123 and a memory 124. The processor 123 is a central processing unit (CPU) for processing data access requests from outside the storage system (server or other storage system), and is also used to process requests generated inside the storage system. Exemplarily, when the processor 123 receives a data write request sent by the application server 100 through the front-end interface 125, the processor 123 may preferentially store the data in the memory 124, such as in the SSTable in the memory 124. When the amount of data of some or all of the SSTables in the memory 124 reaches a certain threshold, the processor 123 sends the data stored in the memory 124 to the hard disk 134 through the back-end port for persistent storage. In an embodiment of the present application, the storage space composed of the multiple hard disks 134 can be constructed in the form of an LSM-Tree. The processor 123 can execute the data access method provided by the implementation of the present application, and can provide an indexing scheme with excellent read and write performance to reduce the overhead and latency of data read access.

[0073] Memory 124 refers to an internal memory that directly exchanges data with the processor. Memory 124 includes at least two types of memories, for example, memory 124 can be either a random access memory or a read-only memory (ROM). For example, the random access memory is a dynamic random access memory (DRAM) or a storage class memory (SCM). Memory can also include other random access memories, such as static random access memory (SRAM), etc. The present application does not limit the specific type of memory 124, and any memory that can interact with the processor as memory 124 is suitable for the embodiments of the present application. Software programs are stored in memory 124, and the processor 123 can manage the hard disk by running the software programs in memory 124.

[0074] Controller 1 (and others Figure 4A The hardware components and software structure of the controller (not shown in the figure) are similar to those of the controller 0 and will not be repeated here.

[0075] Figure 4AWhat is shown is a centralized storage system with a disk-control separation. In this system, the engine 121 may not have a hard disk slot, and the hard disk 134 needs to be placed in the hard disk frame 130, and the back-end interface 126 communicates with the hard disk frame 130. The back-end interface 126 exists in the engine 121 in the form of an adapter card, and two or more back-end interfaces 126 can be used simultaneously on one engine 121 to connect multiple hard disk frames 130. Alternatively, the adapter card can also be integrated on the motherboard, in which case the adapter card can communicate with the processor 123 via the PCIE bus. Optionally, the storage system 120 can also be a storage system with a disk-control integration, the engine 121 has a hard disk slot, and the hard disk 134 can be directly deployed in the engine 121, that is, the hard disk 134 and the engine 121 are deployed on the same device.

[0076] The hard disk frame 130 includes a control unit 131 , a network card 104 and a plurality of hard disks 134 . The control unit 131 can perform address conversion and data read and write operations. The network card 104 is used to communicate with the engine 121 .

[0077] The data access method provided in the embodiment of the present application is not only applicable to centralized storage systems, but also to distributed storage systems, such as Figure 4B As shown, it is a schematic diagram of the system architecture of a distributed storage system provided in an embodiment of the present application, and the distributed storage system includes a server cluster. The server cluster includes one or more servers 140 ( Figure 4B Three servers 140 are shown, but not limited to two servers 140), and each server 140 can communicate with each other. A virtual machine 107 can be created on the server 140, and the required computing resources come from the local processor 123 and memory 124 of the server 140. Various applications can be run in the virtual machine 107.

[0078] The server 110 can execute the data access method provided in the embodiment of the present application, in terms of hardware, such as Figure 4B As shown, the server 110 includes at least a processor 123, a memory 124, a network card 104, and a hard disk 134. The processor 123, the memory 124, the network card 104, and the hard disk 134 are connected via a bus. For the functions and specific types of the processor 123, the memory 124, the network card 104, and the hard disk 134, please refer to Figure 4A The relevant instructions in will not be repeated here.

[0079] The above Figure 4A to Figure 4B The system architecture shown is only an example, and the present application does not limit the structure of the system. Any storage system built based on LSM-Tree is applicable to the embodiments of the present application. For example, the storage system can also be an all-flash storage system or a cloud storage system.

[0080] Below Figure 4AThe storage system 120 or Figure 4B The server shown is used as an example of a storage device to introduce the data access method provided by the embodiment of the present application. It should be understood that the storage device can also be any LSM-Tree-based storage device in other types of storage systems, such as an all-flash storage device or a cloud storage device, and the embodiment of the present application does not limit this.

[0081] The storage device may include a memory, a processor, and a disk. Optionally, the storage device may also include other devices, which can be found in the above combined Figure 4A-4B The relevant introduction will not be repeated here.

[0082] An index file is stored in the memory of the storage device, and the index file can be obtained using a second index structure and is used to read data stored in the memory or disk of the storage device. Exemplarily, the first index structure may include the LSM-Tree introduced above, and the second index structure may include but is not limited to any of the following: a hash structure, a B+ tree structure, or a learning index structure. Optionally, the first index structure can also be implemented as other index structures with better write performance optimization, and the second index structure can also be implemented as other index structures with better read performance optimization, which is not limited in the embodiments of the present application.

[0083] like Figure 5 As shown, the data access method may include the following steps:

[0084] S510: The storage device receives a write access request.

[0085] Exemplarily, if the storage device is the storage system shown in 4A, the write access request may come from Figure 4A Alternatively, if the storage device is a server as shown in 4B, the write access request may come from Figure 4B The client device associated with the server shown. The embodiment of the present application does not limit the architecture type to which the storage device belongs and the source of the write access request.

[0086] In a possible implementation, the write access request includes the first data to be written and the first key associated with the first data. For easy distinction, the first key may be represented as Key1 and the first data may be represented as Value1 in the following description. Key1 is a key assigned to the first data and is used to uniquely identify the first data.

[0087] S520: The storage device writes the first key and the first data into the memory based on the first index structure.

[0088] In the embodiment of the present application, taking the first index structure implemented as the LSM-Tree introduced above as an example, when implementing S520, the storage device can write the first key and the first data in the MemTable in the memory based on the first index structure.

[0089] The storage device will implement the above S520 in the same way for each received write access request, so as to ensure the optimization of the write performance of the storage device based on the LSM-Tree.

[0090] Optionally, since the data stored in the memory may be lost due to power failure, before implementing S520, the storage device may also write the first key and the first data in the WAL. The WAL may be used to perform failure (eg, power failure) recovery processing on the data lost in the memory.

[0091] In order to achieve an indexing scheme that optimizes both read and write performance, one implementation method is to not change the storage device's LSM-Tree-based write IO path, but to additionally construct a fast read path that is different from the traditional read IO path of the LSM-Tree, so that when the storage device receives a read access request, it can quickly access the storage device's memory or disk based on the constructed fast read path to quickly find the target data that needs to be read. In the embodiment of the present application, storing an index file constructed based on the second index structure in the storage device's memory is used to implement the fast read path.

[0092] S530: The storage device records the correspondence between the first key and the first metadata in the index file based on the second index structure.

[0093] In the embodiment of the present application, the index file is stored in the memory of the storage device, and the index file is used to read the data stored in the memory or the disk. The first metadata is metadata of the first data, which is used to indicate the storage address of the first data in the memory or disk of the storage device. For the convenience of distinction, the first metadata can be represented as M1 in the following description.

[0094] In the embodiment of the present application, the index file is constructed by using a second index structure different from the first index structure. For example, the first index structure may include the LSM-Tree described above, and the second index structure may include but is not limited to any of the following: a hash structure, a B+ tree structure, or a learning index structure. For ease of distinction, the following description may take the hash structure as an example of the second index structure, and the corresponding metadata generated is a hash value.

[0095] like Figure 6As shown, the index file may record the corresponding relationship between the key corresponding to the data stored in the memory and its metadata, and may also record the corresponding relationship between the key of the data stored in the disk and its metadata, such as the relationship between Key1 and M1. If the first data is stored in the memory, the first metadata M1 recorded in the index file is used to indicate the storage address of the first data in the memory; if the first data is stored in the disk, the first metadata M1 recorded in the index file is used to indicate the storage address of the first data in the disk.

[0096] Figure 6 The dotted arrow pointing from the memory table to the index file indicates that the corresponding relationship between the keys and metadata of the data stored in the memory is recorded in the index file, and the dotted arrow pointing from the disk to the index file indicates that the corresponding relationship between the keys and metadata of the data stored on the disk is also recorded in the index file. The implementation details of S530 will be introduced in conjunction with different examples below, which will not be elaborated here.

[0097] The storage device may also receive a read access request and process the read access request based on the index file. Figure 7 As shown, the method may include the following steps:

[0098] S710: The storage device receives a read access request.

[0099] Similar to the source of the write access request, if the storage device is the storage system shown in 4A, the read access request may come from Figure 4A Alternatively, if the storage device is a server as shown in 4B, the read access request may come from Figure 4B The client device associated with the server shown. The embodiment of the present application does not limit the architecture type to which the storage device belongs, and the source of the read access request.

[0100] In a possible implementation manner, the read access request includes a key corresponding to the data to be read. For ease of understanding, the read access request including the first key is taken as an example for subsequent description.

[0101] See also Figure 7 As shown, the storage device no longer processes the read access request according to the left branch process, but implements the following steps according to the right branch process:

[0102] S720: The storage device searches for first metadata in an index file stored in the memory according to the first key.

[0103] S730: The storage device reads the first data, ie, the target data, from the memory or the disk according to the first metadata.

[0104] That is, if the first metadata indicates the storage address of the first data in the memory, the storage device can read the first data from the memory. If the first metadata indicates the storage address of the first data in the disk, the storage device can read the first data from the disk.

[0105] Taking the case where the storage device receives a read access request including the first key Key1 introduced above, the storage device searches the index file according to the Key1. If the M1 corresponding to Key1 is found in the index file, and the storage address indicated by the M1 is a storage address in the memory, the first data corresponding to Key1 is read from the memory table in the memory or the immutable memory table according to the storage address. If the M1 corresponding to Key1 is found in the index file, and the storage address indicated by the M1 is a storage address in the disk, the first data corresponding to Key1 is read from the SSTable of the corresponding level of the disk according to the storage address.

[0106] Based on the above S710-S730, it can be known that when the storage device processes the read access request according to the index file, compared with the traditional read IO path based on LSM-Tree, there is no need to query multiple SSTables of LSM-Tree layer by layer, which can greatly reduce the number of disk reads and improve the overall read performance of the storage device, thereby obtaining an index file with excellent read and write performance. At the same time, compared with the data access solution based on Bloom filter, this solution does not have the problem of misjudgment, nor does it have IO overhead caused by misjudgment. This solution will not affect the write performance of the LSM-Tree-based architecture, and can also optimize the read performance for the architecture, thereby providing an index solution with excellent read and write performance.

[0107] For ease of understanding, the following takes the example of recording the correspondence between the keys and metadata of all data stored in the memory and the disk in the index file as an example, and introduces the implementation details of S530 in combination with the example.

[0108] Method 1: The storage device adopts a synchronous write method. When writing the first key and the first data in the memory, the correspondence between the first key and the first metadata is recorded in the index file based on the second index structure. The first metadata indicates the storage address of the first data in the memory.

[0109] In this manner, when the storage device performs a write operation using an algorithm based on LSM-Tree, the storage device writes the first key and the first data in the memory according to the first index structure based on LSM-Tree. Simultaneously, the storage device can also generate first metadata for the first key and the first data stored in the memory according to a preset second index structure and a corresponding algorithm, and the first metadata is used to indicate the storage address of the first data in the memory. The storage device records the correspondence between the first index and the first metadata in the index file.

[0110] Taking the second index structure as a hash structure as an example, during implementation, the storage device can use a hash function to calculate the hash value corresponding to the first key when writing the first key and the first data in the memory. The hash value is the first metadata, which is used to indicate the storage address of the first data in the memory.

[0111] Method 2: The storage device adopts an asynchronous write method, and when a merge operation occurs, the corresponding relationship between the first key and the first metadata is recorded in the index file.

[0112] Based on the previous article about Figure 1 As can be seen from the introduction, the merge operation involved in the LSM-Tree-based architecture may include a merge operation from memory to disk layer 0, and may also include a merge operation from the i-th layer to the i+1-th layer on the disk, where i is an integer greater than or equal to 0. Therefore, in the embodiment of the present application, whether merging and storing data from memory to disk layer 0, or merging and storing data from the i-th layer of the disk to the i+1-th layer, as long as it involves a change in the data storage address and a change in the metadata corresponding to the data, the corresponding relationship between the key of the data and the corresponding metadata needs to be recorded in the index file.

[0113] Taking the change of the storage address of the first data due to the merge operation as an example, the change situation may include the following examples:

[0114] Example 1: First data is changed from being stored in memory to being stored in a disk.

[0115] In this example, it may be that because the amount of data stored in the memory reaches a preset threshold, the storage device merges the second data stored in the memory and stores it in layer 0 of the disk, and the storage address of the second data is changed from the storage address in the memory to the storage address in the disk. The second data includes the first data, so the storage address of the first data is also changed from the storage address in the memory to the storage address in the disk.

[0116] When performing the merge operation in Example 1, the processor of the storage device can merge and store the second data into the corresponding SSTable of layer 0 based on the first index structure of the LSM-Tree, and record the index of the second data at the end of the SSTable.

[0117] At the same time, the processor of the storage device can generate second metadata of the second data based on the preset second index structure, and the second metadata indicates the storage address of the second data in layer 0 of the disk, and the storage device can record the corresponding relationship between the key of the second data and the binary metadata in the index file stored in the memory. Because the second data includes the first data, the second metadata includes the first metadata, and the first metadata is used to indicate the storage address of the first data in layer 0 of the disk.

[0118] Example 2: The first data is changed from being stored in the i-th layer of the disk to being stored in the i+1-th layer of the disk, where i is an integer greater than or equal to 0.

[0119] In this example, it may be that because the amount of data stored in the i-th layer reaches a preset threshold, the processor of the storage device controls the disk to merge and store the third data stored in the i-th layer into the i+1-th layer of the disk, and the storage address of the third data is changed from the storage address of the i-th layer in the disk to the storage address of the i+1-th layer in the disk. The third data may include the first data, and therefore, the storage address of the first data is also changed from the storage address of the i-th layer in the disk to the storage address of the i+1-th layer in the disk.

[0120] When performing the merge operation in Example 2, the processor of the storage device can merge and store the third data into the corresponding SSTable of the i+1th layer based on the first index structure of the LSM-Tree, and record the index of the second data at the end of the SSTable.

[0121] At the same time, the processor of the storage device can generate third metadata of the third data based on the preset second index structure, and the third metadata indicates the storage address of the third data on the i+1th layer of the disk, and the storage device can record the corresponding relationship between the key of the third data and the third metadata in the index file stored in the memory. Because the third data includes the first data, the third metadata includes the first metadata, and the first metadata is used to indicate the storage address of the first data on the i+1th layer of the disk.

[0122] Therefore, based on the above different methods and examples, the process of maintaining the information recorded in the index file when the storage device writes the first data in the memory and the first data is stored in the memory and disk step by step is introduced.

[0123] In specific implementation, due to the limitation of the content capacity of the storage device, there is a possibility that the memory is insufficient to store the correspondence between the keys of all the data stored on the disk and the metadata generated by the second index structure. In view of this situation, in one possible implementation, the correspondence between the keys of some data on the disk and the metadata is recorded in the index file stored in the memory. For example. The SSTables of different levels in the disk can be prioritized so that the metadata based on the second index structure generated for the data stored in the SSTable with a higher priority is preferentially recorded in the index file stored in the memory, while the metadata based on the second index structure generated for the data stored in the SSTable with a lower priority is stored in the index file maintained separately at the corresponding level, and the index file is also obtained by using the second index structure.

[0124] In one example, if Figure 8 As shown, the priority of SSTables at different levels can be configured according to the granularity of storage space at different levels of LSM-Tree. For example, since a large amount of data in the disk is stored in layer n, and a relatively small amount of data is stored in the memory table, layer 0 to layer n-1, the priority of the memory table, layer 0 to layer n-1 can be configured to be higher than the priority of layer n.

[0125] When writing data in the memory table, the content introduced in the first method in the previous text is used to record the corresponding relationship between the key of the data stored in the memory and the metadata generated by the second index structure in the index file stored in the memory, which will not be repeated here.

[0126] When writing data to the SSTable of layer 0 to layer n-1, the content introduced in the second method in the previous article is adopted. When the data in the memory is merged and stored in layer 0, or when the data of the i-th layer is merged and stored in the i+1-th layer (i is an integer greater than or equal to 0), the correspondence between the key of the merged data and its metadata is recorded in the index file saved in the memory. The metadata is constructed using the second index structure introduced in the previous article and will not be repeated here.

[0127] When the data of layer n-1 is merged and stored in layer n, the correspondence between the key of the merged data and its metadata is no longer recorded in the index file stored in the memory, but the correspondence between the key of the merged data and its metadata is recorded in the index file of layer n.

[0128] Further, after receiving the read access request, the storage device Fig. 9 As shown, the read access request can be processed according to the following steps:

[0129] S910: The storage device receives a read access request.

[0130] Similar to the source of the write access request, if the storage device is the storage system shown in 4A, the read access request may come from Figure 4A Alternatively, if the storage device is a server as shown in 4B, the read access request may come from Figure 4B The client device associated with the server shown. The embodiment of the present application does not limit the architecture type to which the storage device belongs, and the source of the read access request.

[0131] In a possible implementation manner, the read access request includes a key corresponding to the data to be read. For ease of understanding, the read access request including the first key is taken as an example for subsequent description.

[0132] See also Fig. 9As shown, the storage device no longer processes the read access request according to the left branch process, but implements the following steps according to the right branch process:

[0133] S920: The storage device searches for the first metadata in the index file stored in the memory according to the first key. If a match is found, the process proceeds to S930. If a match is not found, the process proceeds to S940.

[0134] S930: The storage device reads the first data, ie, the target data, from the corresponding SSTable of layer 0 to layer n-1 of the memory table or disk according to the first metadata.

[0135] S940: The storage device searches for first metadata in the index file of layer n according to the first key.

[0136] S950: The storage device reads the first data, ie, the target data, from the corresponding SSTable of layer n of the disk according to the first metadata.

[0137] Based on the above S910-S930, it can be known that even if the corresponding relationship between the keys and metadata of all data on the disk is not maintained in the index file in the memory, when the storage device processes the read access request, it queries the corresponding metadata and reads the target data based on the index file stored in the memory and the index file of layer n. Compared with the traditional read IO path based on LSM-Tree, there is no need to query multiple SSTables of LSM-Tree layer by layer, which can greatly reduce the number of disk reads and improve the overall read performance of the storage device, thereby obtaining an index document with excellent read and write performance. At the same time, compared with the data access solution based on Bloom filter, this solution does not have the problem of misjudgment, nor does it have IO overhead caused by misjudgment. This solution will not affect the write performance of the LSM-Tree-based architecture, and can also optimize the read performance for the architecture, thereby providing an index solution with excellent read and write performance.

[0138] It should be understood that the above combination Figure 8-Figure 9 The introduction is only an example and not a limitation. In other embodiments, the content of the correspondence between the index and the metadata to be stored in the second index file in the memory can also be configured according to business needs or architectural requirements. The embodiments of the present application do not limit this.

[0139] Based on the same inventive concept as the method embodiment, the embodiment of the present application further provides a storage device, which is used to execute the method executed by the storage device in the above method embodiment, such as Fig.10As shown, the storage device 1000 may include: a transceiver unit 1001, used to receive a write access request, wherein the write access request includes a first key and a first data to be written; a processing unit 1002, used to write the first key and the first data in the memory of the storage device based on a first index structure; based on the second index structure, recording the correspondence between the first key and the first metadata in an index file, wherein the index file is stored in the memory of the storage device, and the index file is used to read data stored in the memory or the disk.

[0140] In a possible implementation, the transceiver unit 1001 may also be used to receive a read access request, wherein the read access request includes the first key; the processing unit 1002 may also be used to query the first metadata in the index file according to the first key; and read the first data from the memory or the disk according to the first metadata. For detailed implementation details, see Figure 7 and Fig. 9 method steps.

[0141] In a possible implementation, if the first data is stored in the memory, the first metadata includes a storage address of the first data in the memory.

[0142] In a possible implementation manner, the processing unit 1002 may be configured to: when writing the first key and the first data into the memory, record the correspondence between the first key and the first metadata in the index file based on the second index structure. Figure 7 and Fig. 9 method steps.

[0143] In a possible implementation manner, if the first data is stored in the disk, the first metadata includes a storage address of the first data in the disk.

[0144] In a possible implementation, the processing unit 1002 may be used to: when merging the second data stored in the memory and storing them on the disk, based on the second index structure, record the correspondence between the first key and the first metadata in the index file, the second data including the first data, and the first metadata being used to indicate the storage address of the first data on the disk.

[0145] In one possible implementation, the disk adopts layered storage, and the processing unit 1002 can also be used to: when merging the third data stored in the i-th layer of the disk and storing it in the i+1-th layer of the disk, based on the second index structure, record the correspondence between the first key and the first metadata in the index file, the third data includes the first data, and the first metadata is used to indicate the storage address of the first data in the i+1-th layer.

[0146] In a possible implementation manner, the first index structure includes a log structure merged with an LSM tree structure.

[0147] In a possible implementation manner, the second index structure includes any one of the following: a hash structure, a B+ tree structure, or a learning index structure.

[0148] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0149] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a terminal device (which can be a personal computer, mobile phone, or network device, etc.) or a processor (processor) to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0150] The descriptions of the processes corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0151] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes computer program instructions, and when the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in FIG. 4 of the embodiment of the present invention are generated.

[0152] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line, or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., an SSD).

[0153] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data access method, characterized in that: Applied to a storage device, the method comprises: receiving a write access request, wherein the write access request includes a first key and first data to be written; Based on a first index structure, writing the first key and the first data into the memory of the storage device; Based on the second index structure, the correspondence between the first key and the first metadata is recorded in an index file, the first metadata is used to indicate the storage address of the first data in the memory or disk of the storage device, the index file is stored in the memory of the storage device, and the index file is used to read the data stored in the memory or the disk.

2. The method according to claim 1, characterized in that The method further comprises: receiving a read access request, wherein the read access request includes the first key; searching the first metadata in the index file according to the first key; The first data is read from the memory or the disk according to the first metadata.

3. The method according to claim 1 or 2, characterized in that: If the first data is stored in the memory, the first metadata includes a storage address of the first data in the memory.

4. The method according to claim 3, characterized in that: The recording, in the index file, the correspondence between the first key and the first metadata based on the second index structure includes: When writing the first key and the first data in the memory, the corresponding relationship between the first key and the first metadata is recorded in the index file based on the second index structure.

5. The method according to claim 1 or 2, characterized in that: If the first data is stored in the disk, the first metadata includes a storage address of the first data in the disk.

6. The method according to claim 5, characterized in that The recording, in the index file, the correspondence between the first key and the first metadata based on the second index structure includes: When the second data stored in the memory is merged and stored in the disk, the correspondence between the first key and the first metadata is recorded in the index file based on the second index structure, the second data includes the first data, and the first metadata is used to indicate the storage address of the first data in the disk.

7. The method according to claim 5, characterized in that The disk adopts hierarchical storage, and the method further includes: When the third data stored in the i-th layer of the disk is merged and stored in the i+1-th layer of the disk, the correspondence between the first key and the first metadata is recorded in the index file based on the second index structure, the third data includes the first data, and the first metadata is used to indicate the storage address of the first data in the i+1-th layer.

8. The method according to any one of claims 1 to 7, characterized in that The first index structure includes a log structure merged with an LSM tree structure.

9. The method according to any one of claims 1 to 8, characterized in that The second index structure includes any one of the following: Hash structure, B+ tree structure or learned index structure.

10. A storage device, characterized in that: The storage device comprises: A transceiver unit, configured to receive a write access request, wherein the write access request includes a first key and first data to be written; A processing unit is used to write the first key and the first data in the memory of the storage device based on a first index structure; based on the second index structure, record the correspondence between the first key and the first metadata in an index file, the first metadata is used to indicate the storage address of the first data in the memory or disk of the storage device, the index file is stored in the memory of the storage device, and the index file is used to read the data stored in the memory or the disk.

11. The storage device according to claim 10, characterized in that: The transceiver unit is further configured to receive a read access request, wherein the read access request includes the first key; The processing unit is further configured to query the first metadata in the index file according to the first key; and read the first data from the memory or the disk according to the first metadata.

12. The storage device according to claim 10 or 11, characterized in that: If the first data is stored in the memory, the first metadata includes a storage address of the first data in the memory.

13. The storage device according to claim 12, characterized in that: The processing unit is used for: When writing the first key and the first data in the memory, the corresponding relationship between the first key and the first metadata is recorded in the index file based on the second index structure.

14. The storage device according to claim 10 or 11, characterized in that: If the first data is stored in the disk, the first metadata includes a storage address of the first data in the disk.

15. The storage device according to claim 14, characterized in that: The processing unit is used for: When the second data stored in the memory is merged and stored in the disk, based on the second index structure, the correspondence between the first key and the first metadata is recorded in the index file, the second data includes the first data, and the first metadata is used to indicate the storage address of the first data in the disk.

16. The storage device according to claim 14, characterized in that: The disk adopts hierarchical storage, and the processing unit is further used for: When the third data stored in the i-th layer of the disk is merged and stored in the i+1-th layer of the disk, based on the second index structure, the correspondence between the first key and the first metadata is recorded in the index file, the third data includes the first data, and the first metadata is used to indicate the storage address of the first data in the i+1-th layer.

17. The storage device according to any one of claims 10 to 16, characterized in that: The first index structure includes a log structure merged with an LSM tree structure.

18. The storage device according to any one of claims 10 to 17, characterized in that: The second index structure includes any one of the following: Hash structure, B+ tree structure or learned index structure.

19. A storage device, characterized in that: The storage device includes a processor and a memory; The memory is used to store computer program instructions; The processor executes and calls the computer program instructions in the memory to perform the method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: When the computer-readable storage medium is executed by a computing device, the computing device executes the method according to any one of claims 1 to 9.

Images