A data processing method, storage system and storage medium

Abstract

The embodiment of the present application discloses a data processing method, a storage system and a storage medium, wherein the storage system includes a cache, the cache uses a continuous log Plog to carry data, and the method is used to cache eliminated data. When receiving a data read request including metadata of data to be read, it is determined whether the data corresponding to the Plog pointed to by the metadata is stored in the cache. If not, the data corresponding to the Plog is not read, and the data to be read is read from other storage media of the storage system, so that even if the Plog is deleted, it will not cause data reading failure, and at the same time avoid the problem of increasing write amplification or blocking the deletion process of the Plog.

Description

Technical Field

[0001] The present application relates to the field of storage, and in particular to a data processing method, a storage system, and a storage medium. Background Art

[0002] The Log-Structured Merge (LSM) tree is an efficient data index structure that implements hierarchical data storage based on key-value pairs. The LSM tree includes multiple levels, each level includes multiple subtrees, each subtree is a metadata, as the key in the key-value pair, the metadata is used to point to the corresponding persistent log (Plog), that is, Plog is the value in the key-value pair. In the process of writing data in the LSM tree, when the subtree of a certain level reaches or exceeds the upper limit of the level, the metadata in the subtree within a certain range in the level and the data pointed to by these metadata will be merged into the next level. When the storage system stores data, the LSM tree can be used as a data index structure, and the first level of the LSM tree corresponds to the data in the cache memory (hereinafter referred to as cache), and the other levels correspond to the data in other storage media in the storage system.

[0003] Cache is an important memory in the storage system, with extremely high read and write speeds, which helps to significantly improve the overall performance of the storage system. However, the capacity of the cache is very limited. When more and more data is written to the cache, the cache needs to continuously eliminate the data and write it to other storage media in the storage system to free up storage space for hot data. The cache can eliminate data by deleting the Plog corresponding to the data, but if the Plog is deleted directly, then when the index points to the metadata corresponding to the deleted Plog, it will cause data reading failure.

[0004] Currently, there is a solution to eliminate cached data: before eliminating cached data, each subtree of the LSM tree, that is, metadata, can be traversed first to determine which metadata points to the Plog that needs to be deleted, and then delete markers are inserted into these metadata so that these metadata cannot be indexed, so that data reading failure will not occur, and the Plog pointed to by these metadata can be safely deleted. However, since it takes a long time to traverse the LSM tree and a large number of delete markers are inserted, write amplification increases.

[0005] Currently, there is another solution to eliminate cached data: during the LSM tree merge process of the data in the storage system, first traverse the subtree participating in the merge, that is, the metadata, to determine which metadata points to the Plog that needs to be deleted, and then directly delete these metadata that point to the Plog that needs to be deleted without writing the merge result. Since these metadata have been deleted and cannot be indexed, data reading failure will not occur, so the corresponding Plog can be safely deleted. However, due to the long merge process, the Plog deletion process may be blocked. Summary of the invention

[0006] An embodiment of the present application provides a data processing method, a storage system, and a storage medium, wherein the method is applied to the storage system, the storage system includes a cache, the cache uses Plog to carry data, and the method is used to cache eliminated data.

[0007] To this end, the embodiments of the present application provide the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides a data processing method, comprising: when receiving a data read request, obtaining metadata of the data to be read included in the data read request, the metadata being used to point to the Plog corresponding to the data to be read, so that it can be determined whether the data corresponding to the Plog is stored in the cache; if the result of the determination is that the data corresponding to the Plog is not in the cache, the data corresponding to the Plog may not be read, and the data to be read may be read from other storage media of the storage system, so that even if the Plog is deleted, data reading failure will not be caused, while avoiding the problem of increased write amplification or blocking the Plog deletion process due to a long merging process.

[0009] In combination with the first aspect of the embodiment of the present application, in a first implementation of the first aspect of the embodiment of the present application, by storing a valid Plog list in a cache, since the valid Plog list includes multiple Plogs, the data corresponding to each Plog in the multiple Plogs are stored in the cache, and then when a data read request is received, it can be determined whether the Plog is in the valid Plog list. If it is determined that the Plog is not in the valid Plog list, it can be determined that the data corresponding to the Plog is not in the cache, so that the data corresponding to the Plog can be read without reading the data, and the data to be read can be read from other storage media of the storage system. Therefore, after creating the valid Plog list, the Plog outside the valid Plog list in the cache can be safely deleted, avoiding the problem of increased write amplification or blocking the deletion process of the Plog due to a long merging process.

[0010] In addition, if it is determined that Plog is in the valid Plog list, it can be determined that the data corresponding to Plog is stored in the cache, so the data corresponding to Plog can be read from the cache.

[0011] In combination with the first implementation method of the first aspect of the embodiment of the present application, in the second implementation method of the first aspect of the embodiment of the present application, before receiving a data read request, a valid Plog list can be created by determining each valid Plog in the cache, wherein the valid Plog is the Plog corresponding to the hot data in the cache. Since hot data is data in the cache whose access frequency is greater than or equal to the preset frequency or whose life cycle is less than or equal to the preset duration, and the valid Plog list includes each valid Plog in the cache, the creation of the valid Plog list is realized, and since the Plogs in the valid Plog list all correspond to hot data, the available space in the cache to store hot data is guaranteed.

[0012] In combination with the second implementation method of the first aspect of the embodiment of the present application, in the third implementation method of the first aspect of the embodiment of the present application, after creating the valid Plog list, each Plog in the cache except the valid Plog list can be deleted, which will not cause data reading failure and can free up storage space for hot data.

[0013] In combination with the first aspect of the embodiment of the present application, and the first to third implementation methods of the first aspect, in the fourth implementation method of the first aspect of the embodiment of the present application, since the data index structure of the valid Plog list is a hash map or a multi-way balanced search tree, fast reading of the Plog in the valid Plog list is achieved, reducing I / O consumption.

[0014] In combination with the first aspect of the embodiment of the present application, in a fifth implementation of the first aspect of the embodiment of the present application, an invalid Plog list is stored in the cache, and the invalid Plog list includes multiple Plogs, and the data corresponding to each Plog in the multiple Plogs is not in the cache, so it can be determined whether the Plog is in the invalid Plog list. When it is determined that the Plog is in the invalid Plog list, it can be determined that the data corresponding to the Plog is not in the cache, so that the data corresponding to the Plog can be read without reading the data corresponding to the Plog, and the data to be read can be read from other storage media of the storage system. Therefore, after creating the invalid Plog list, the Plog in the invalid Plog list in the cache can be safely deleted, avoiding the problem of increased write amplification or blocking the deletion process of the Plog due to a long merging process.

[0015] In addition, if it is determined that Plog is in the invalid Plog list, it can be determined that the data corresponding to Plog is stored in the cache, so the data corresponding to Plog can be read from the cache.

[0016] In combination with the fifth implementation method of the first aspect of the embodiment of the present application, in the sixth implementation method of the first aspect of the embodiment of the present application, before receiving a data read request, an invalid Plog list can be created by determining each invalid Plog in the cache, wherein the invalid Plog corresponds to the Plog of cold data in the cache. Since cold data is data in the cache whose access frequency is less than a preset frequency and whose life cycle is greater than a preset duration, and the invalid Plog list includes each invalid Plog in the cache, the creation of the invalid Plog list is realized, and since the Plogs in the invalid Plog list all correspond to cold data, the available space for storing hot data in the cache is guaranteed.

[0017] In combination with the sixth implementation method of the first aspect of the embodiment of the present application, in the seventh implementation method of the first aspect of the embodiment of the present application, after creating a valid Plog list, each Plog in the cache except for the invalid Plog list can be deleted, which will not cause data reading failure and can free up storage space for hot data.

[0018] In combination with the first aspect of the embodiment of the present application, and the fifth to seventh implementation methods of the first aspect, in the eighth implementation method of the first aspect of the embodiment of the present application, since the data index structure of the invalid Plog list is a hash map or a multi-way balanced search tree, fast reading of the Plog in the invalid Plog list is achieved, reducing I / O consumption.

[0019] In combination with the first aspect of the embodiment of the present application, and the first to eighth implementation methods of the first aspect, in the ninth implementation method of the first aspect of the embodiment of the present application, the data in the storage system uses a merged LSM tree with a log structure as a data index structure, wherein the data in the cache corresponds to the first level of the LSM tree, which fits the internal structure of the storage system, thereby achieving fast reading of data in the storage system and reducing I / O consumption.

[0020] In combination with the ninth implementation method of the first aspect of the embodiment of the present application, in the tenth implementation method of the first aspect of the embodiment of the present application, in the process of merging at least one metadata in the target level into the next level of the target level in the LSM tree, all invalid metadata in the target level can be determined and discarded. Since the invalid metadata is the metadata pointing to a Plog outside the valid Plog list (or pointing to an invalid Plog list), the storage pressure in the memory is reduced. At the same time, since there is no need to continue deleting the Plog pointed to after deleting the invalid metadata, the problem of blocking the Plog deletion process due to a long merging process is avoided.

[0021] In a second aspect, an embodiment of the present application provides a storage system, including: a cache and a processor.

[0022] The cache is used to use Plog to carry data; the processor is used to receive a data read request, the data read request includes metadata of the data to be read, and the metadata is used to point to the Plog corresponding to the data to be read; the processor is also used to determine whether the data corresponding to the Plog is stored in the cache. When the result of the determination is that the data corresponding to the Plog is not in the cache, the data corresponding to the Plog is not read, and the data to be read is read from other storage media of the storage system, so that even if the Plog is deleted, it will not cause data reading failure, and at the same time avoid the problem of increased write amplification or blocking the deletion process of the Plog due to a long merging process.

[0023] In combination with the second aspect of the embodiment of the present application, in a first implementation method of the second aspect of the embodiment of the present application, the cache is also used to store a valid Plog list, the valid Plog list includes multiple Plogs, and the data corresponding to each Plog in the multiple Plogs are stored in the cache. Then the processor specifically determines whether the Plog is in the valid Plog list. When the result of the judgment is that the Plog is not in the valid Plog list, it is determined that the data corresponding to the Plog is not in the cache, so that the data corresponding to the Plog can be read without reading the data, and the data to be read is read from other storage media in the storage system. Therefore, after creating the valid Plog list, the Plog outside the valid Plog list in the cache can be safely deleted, avoiding the problem of increased write amplification or blocking the Plog deletion process due to a long merging process.

[0024] In addition, if it is determined that Plog is in the valid Plog list, it can be determined that the data corresponding to Plog is stored in the cache, so the data corresponding to Plog can be read from the cache.

[0025] In combination with the first implementation method of the second aspect of the embodiment of the present application, in the second implementation method of the second aspect of the embodiment of the present application, before receiving a data read request, the processor may also determine each valid Plog in the cache to create a valid Plog list, wherein the valid Plog is the Plog corresponding to the hot data in the cache. Since hot data is data in the cache whose access frequency is greater than or equal to the preset frequency or whose life cycle is less than or equal to the preset duration, and the valid Plog list includes each valid Plog in the cache, the creation of the valid Plog list is realized, and since the Plogs in the valid Plog list all correspond to hot data, the available space in the cache to store hot data is guaranteed.

[0026] In combination with the second implementation method of the second aspect of the embodiment of the present application, in the third implementation method of the second aspect of the embodiment of the present application, the processor can also delete each Plog in the cache except the valid Plog list, which will not cause data reading failure and can free up storage space for hot data.

[0027] In combination with the second aspect of the embodiment of the present application, in a fourth implementation method of the second aspect of the embodiment of the present application, the cache is also used to store an invalid Plog list. Since the invalid Plog list includes multiple Plogs, the data corresponding to each Plog in the multiple Plogs is not in the cache. Therefore, the processor can determine whether the Plog is in the invalid Plog list. When the result of the judgment is that the Plog is in the invalid Plog list, it is determined that the data corresponding to the Plog is not in the cache, so that the data corresponding to the Plog can be read without reading the data, and the data to be read is read from other storage media in the storage system. Therefore, after creating the invalid Plog list, the Plog in the invalid Plog list in the cache can be safely deleted, avoiding the problem of increased write amplification or blocking the Plog deletion process due to a long merging process.

[0028] In addition, if it is determined that Plog is in the invalid Plog list, it can be determined that the data corresponding to Plog is stored in the cache, so the data corresponding to Plog can be read from the cache.

[0029] In combination with the fourth implementation method of the second aspect of the embodiment of the present application, in the fifth implementation method of the second aspect of the embodiment of the present application, before receiving a data read request, the processor may also determine each invalid Plog in the cache, wherein the invalid Plog corresponds to the Plog of cold data in the cache, and the cold data is data in the cache whose access frequency is less than the preset frequency and whose life cycle is greater than the preset duration, so that the processor can create an invalid Plog list, and the invalid Plog list includes each invalid Plog in the cache, thereby ensuring the available space for the cache to store hot data.

[0030] In combination with the fifth implementation method of the second aspect of the embodiment of the present application, in the sixth implementation method of the second aspect of the embodiment of the present application, after creating the valid Plog list, the processor can delete each Plog in the cache except for the invalid Plog list, which will not cause data reading failure and can free up storage space for hot data.

[0031] In combination with the second aspect of the embodiment of the present application and the first to sixth implementation methods of the second aspect, in the seventh implementation method of the second aspect of the embodiment of the present application, the data in the storage system uses a merged LSM tree with a log structure as a data index structure, wherein the data in the cache corresponds to the first level of the LSM tree, which fits the internal structure of the storage system, thereby achieving fast reading of data in the storage system and reducing I / O consumption.

[0032] In combination with the seventh implementation method of the second aspect of the embodiment of the present application, in the eighth implementation method of the second aspect of the embodiment of the present application, in the process of merging at least one metadata in the target level into the next level of the target level in the LSM tree, all invalid metadata in the target level can be determined and discarded. Since the invalid metadata is the metadata pointing to a Plog outside the valid Plog list (or pointing to an invalid Plog list), the storage pressure in the memory is reduced. At the same time, since there is no need to continue deleting the Plog pointed to after deleting the invalid metadata, the problem of blocking the Plog deletion process due to a long merging process is avoided.

[0033] In a third aspect, an embodiment of the present application provides a storage system, including a processor and a cache, wherein the cache stores a computer-readable program and uses Plog to carry data; by running the computer-readable program in the cache, the methods described in the above aspects are completed.

[0034] In a fourth aspect, an embodiment of the present application provides a computer program product, which, when executed, is used to execute the method described in the first aspect above.

[0035] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions are used to execute the method described in the first aspect above.

[0036] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0037] In an embodiment of the present application, by receiving a data read request, since the data read request includes metadata of the data to be read, and the metadata is used to point to the Plog corresponding to the data to be read, it is possible to determine whether the data corresponding to the Plog is stored in the cache. When the result of the judgment is that the data corresponding to the Plog is not in the cache, the data corresponding to the Plog is not read, and the data to be read is read from other storage media of the storage system, so that even if the Plog is deleted, it will not cause data reading failure, and at the same time avoid the problem of increased write amplification or blocking the Plog deletion process due to a long merging process. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1-1 It is a structural diagram of the storage system;

[0039] Figure 1-2 A schematic diagram of multiple levels in an LSM tree;

[0040] Figure 1-3 Schematic diagram of the merging process in the LSM tree;

[0041] Figure 1-4 It is a schematic diagram of traversing each subtree of the LSM tree;

[0042] Figure 1-5 A schematic diagram for inserting a deletion mark on metadata;

[0043] Figure 1-6 Schematic diagram for safe deletion of Plog;

[0044] Figure 1-7 A schematic diagram of traversing metadata in a subtree participating in a merge;

[0045] Figure 1-8 A schematic diagram of directly deleting invalid metadata in a subtree participating in a merge;

[0046] Figure 1-9 Schematic diagram for safe deletion of Plog;

[0047] Figure 2-1 A schematic diagram of an embodiment of a data processing method;

[0048] Figure 2-2 A schematic diagram for determining whether the data corresponding to the Plog is stored in the cache;

[0049] Figure 2-3 Schematic diagram for safe deletion of Plog;

[0050] Figure 3-1 A schematic diagram of an embodiment of a data processing method;

[0051] Figure 3-2 A schematic diagram of a user requesting data from a storage system;

[0052] Figure 3-3 Schematic diagram of the CPU querying Plog in the cache;

[0053] Figure 3-4 Schematic diagram of querying the valid Plog list for the CPU;

[0054] Figure 3-5 A schematic diagram of the CPU reading the data corresponding to Plog from the cache;

[0055] Figure 4-1 A schematic diagram of an embodiment of a data processing method;

[0056] Figure 4-2 Diagram of querying the invalid Plog list for the CPU. DETAILED DESCRIPTION

[0057] An embodiment of the present application provides a data processing method, which is applied to a storage system. The storage system includes a cache, and the cache uses Plog to carry data. The method is used to cache eliminated data.

[0058] It should be understood that the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0059] The storage system includes various memories, control components, and equipment (hardware) and algorithms (software) for managing information scheduling, among which the memory is the most important component of the storage system. The memory is the actual physical carrier for storing data. The more common memories on the market include hard disk drives (HDD) and solid state drives (SSD). A single memory cannot simultaneously meet the requirements of fast access speed, large storage capacity, and low cost. Therefore, the storage system needs to have multiple levels of memory with speeds ranging from slow to fast and capacities ranging from large to small, with the optimal control scheduling algorithm and reasonable cost, to form a storage system with acceptable performance.

[0060] Please refer to Figure 1-1 , is a schematic diagram of the structure of a storage system 100 , which includes a central processing unit (CPU) 110 , a random access memory (RAM) 120 , a cache 130 and a HDD 140 .

[0061] RAM 120 is an internal memory that directly exchanges data with CPU 110. It can be read and written at any time (except when refreshed) and is very fast. It is usually used as a temporary data storage medium for operating systems or other running programs. When working, RAM 120 can write (store) or read (retrieve) information from any specified address at any time.

[0062] Although RAM 120 improves the read and write speed of storage system 100, RAM 120 is expensive and has a small capacity. In addition, once the power is off, the data stored in RAM 120 will be lost. The capacity of SSD is much larger than RAM 120, and the data reading speed is much higher than HDD 140. It is also a non-volatile memory. Therefore, in the embodiment of the present application, storage system 100 uses SSD as cache 130 to improve the read and write speed of storage system 100 and the access efficiency to hot data.

[0063] In the embodiment of the present application, when the storage system 100 performs a write process, it first receives a data write request through the CPU 110, and then the CPU 110 writes the data in the RAM 120, and then writes the data to other storage media, such as the HDD 140. If the data is hot data, a copy of the data will also be written to the cache 130. When the storage system 100 performs a read process, it first receives a data read request through the CPU 110, and then searches for the data from the RAM 120. If the data is not found, it searches from the cache 130. If it is still not found, it searches from the HDD 140.

[0064] In order to quickly access and reduce I / O consumption, a data index structure may be used for the data in the storage system 100. The data index structure is a storage structure for sorting data, and its function is similar to the catalog of a book, and the required content can be quickly found according to the page numbers in the catalog.

[0065] In some feasible embodiments, the storage system 100 uses an LSM tree as a data index structure. The LSM tree is an efficient data index structure that implements hierarchical data storage based on a key-value pair. The LSM tree includes multiple levels, each level includes multiple subtrees, and each subtree is a metadata, which is a key in a key-value pair. The metadata is used to point to the corresponding Plog, which is a value in the key-value pair. It should be noted that in order to facilitate reading and writing, in the LSM tree, the metadata can be distributed in each subtree according to time.

[0066] like Figure 1-2As shown, C0, C1, C2...Ck are used as examples of multiple levels in the LSM tree. In the process of writing data in the cache 130, when the subtree of a certain level reaches or exceeds the upper limit of the level, the metadata in the subtree within a certain range in the level and the data pointed to by these metadata will be merged into the next level. In the process of reading data in the cache 130, when the storage system 100 receives a read request including metadata, it can start indexing from the first level of the LSM tree. If the metadata cannot be indexed from the first level, it will be indexed from the next level until the metadata is hit or all levels are indexed. Therefore, when a read request for data is received and the corresponding metadata is hit, the Plog pointed to by the metadata can be determined, and the storage location can be determined from the Plog, and finally the data can be obtained from the storage location. Rather than saying that the LSM tree is a data index structure, it is better to say that the LSM tree is a data index idea. Each level of the LSM tree can use a data index structure independently. For example, the C0 level can use a multi-way balanced search tree (b-tree), and the C1 level can use a hash map (hashmap).

[0067] It should be noted that Plog indicates a storage location in the storage system 100 (such as a location in a disk). When the storage system 100 stores data, it first determines a free storage location, and then generates a Plog for this storage location. This Plog is equivalent to the "name" of the storage location. After a Plog is deleted, the storage location corresponding to the Plog becomes vacant and can be used to write data. At this time, it is necessary to generate another corresponding Plog for this storage location. It should be noted that the data carried by Plog has the characteristic of append-only, that is, after a Plog is full, no new data can be written, nor can it be modified, and it can only be deleted.

[0068] Based on the read-write characteristics of the LSM tree, the LSM tree can be used in the storage system 100, and the first level of the LSM tree (such as the C0 level in the above example) can correspond to the data in the cache 130, and the other levels can correspond to the data in the HDD 140 of the storage system 100. The working principle of the cache 130 is to first provide a data copy of hot data. When the CPU 110 wants to read data, it first searches from the cache 130. If it is found, it is immediately read and processed. If it is not found, it is read from the HDD 140, and the read data is transferred to the cache 130 as hot data. The data can be read from the cache 130 later, which improves the efficiency of data reading.

[0069] Compared with HDD 140, the capacity of cache 130 is limited. When the available space of cache 130 is insufficient, low-value data needs to be eliminated to free up storage space for hot data. Cache 130 can eliminate data by deleting the Plog corresponding to the data. However, if the Plog is deleted directly, when the metadata is indexed, if the Plog pointed to by the metadata has been deleted, data reading will fail.

[0070] like Figure 1-2 In the example shown, the data pointed to by the metadata in the subtree of the CO level can be stored in the cache 130, while the data of the C1-Ck levels can be stored in the HDD 140. When the available space in the cache 130 is insufficient, that is, the amount of data written in the C0 level exceeds the upper limit, some low-value data will be eliminated, such as Figure 1-3 As shown, part of the subtree (i.e., metadata) of the C0 level and the data pointed to will be merged into the C1 level. Similarly, if the amount of data written to the C1 level exceeds its upper limit, it will also be merged into C2, and so on until the Ck level.

[0071] Currently, before cache 130 eliminates data, it can first traverse each subtree of the LSM tree to determine the metadata corresponding to each subtree (such as Figure 1-4 Whether the Plog pointed to by metadata 1 and metadata 2 in the , needs to be deleted, and then insert a deletion marker on these metadata (such as Figure 1-5 As shown in ). Metadata with deletion marks inserted cannot be indexed, so even if the Plog pointed to by these metadata is deleted, data reading failure will no longer occur. Therefore, the Plog pointed to by these metadata can be safely deleted (as shown in Figure 1-6 However, since it takes a long time to traverse the subtree of the LSM tree and there are many deletion markers, the write amplification increases.

[0072] In another solution, during the LSM tree merge operation, the metadata in the subtrees participating in the merge can be traversed to determine which metadata points to the Plog that needs to be deleted (such as Figure 1-7 , and then directly delete these metadata (such as Figure 1-8 The metadata 2 shown above is deleted without writing the merged result. Since these metadata have been deleted, they cannot be indexed. Then the Plog pointed to by these metadata can be safely deleted (such as Figure 1-9 Delete Plog2 as shown). However, since the merging process is long, it may block the deletion process of Plog.

[0073] To this end, a data processing method provided by the present application is applied to the cache 130 of the storage system 100. The cache 130 uses Plog to carry data. Please refer to Figure 2-1 , the method is executed by the processor 110, including:

[0074] 201. Receive a data read request, where the data read request includes metadata of data to be read, where the metadata is used to point to a Plog corresponding to the data to be read.

[0075] 202. Determine whether the data corresponding to the Plog is stored in the cache.

[0076] 203. When the result of the determination is that the data corresponding to the Plog is not in the cache, the data corresponding to the Plog is not read, but the to-be-read data is read from other storage media of the storage system.

[0077] In the embodiment of the present application, when the storage system receives a data read request including metadata of the data to be read, since the metadata is used to point to the Plog corresponding to the data to be read (such as Figure 1-4 Plog2 shown in the figure can determine whether the data corresponding to the Plog is stored in the cache (such as Figure 2-2 As shown), when the result of the judgment is that the data corresponding to the Plog is not in the cache, the data corresponding to the Plog is not read in the cache, but the data to be read is read from other storage media of the storage system, so that data reading failure will not occur. Therefore, the Plog pointed to by the metadata can be safely deleted (as shown Figure 2-3 As shown in the figure), it also avoids the problem of increased write amplification or blocking the Plog deletion process due to a long merging process, thereby achieving strong read and write consistency of the storage system.

[0078] For details, please refer to Figure 3-1 , is a data processing method provided by the present application, which is applied to a storage system 100, the storage system 100 includes a cache 130, and the cache 130 uses a continuous log Plog to carry data. The method includes:

[0079] 301. Determine each valid Plog in the cache, where the valid Plog is the Plog corresponding to the hot data in the cache, and the hot data is the data in the cache whose access frequency is greater than or equal to a preset frequency or whose life cycle is less than or equal to a preset duration.

[0080] The storage space of the cache is limited. In the process of continuously writing data in the cache, it is often necessary to eliminate low-value data and free up storage space to accommodate hot data. In an embodiment of the present application, hot data refers to data in the cache whose access frequency is greater than or equal to the preset frequency or whose life cycle is less than or equal to the preset duration, such as the instant geographic location, or the data on the homepage of a popular website. In contrast, data other than hot data, that is, data in the cache whose access frequency is less than the preset frequency and whose life cycle is greater than the preset duration, can be called cold data.

[0081] It can be seen that the distinction between hot data and cold data is not absolute, but relative. In some feasible embodiments, a preset duration and a preset frequency can be set to distinguish hot data from cold data. It should be noted that to distinguish whether a piece of data is hot data or cold data, both the access frequency and the life cycle can be considered, or one of the two can be selected, which is not limited here. In some feasible embodiments, the distinction between hot data and cold data can also be based on criteria other than the life cycle and the access frequency, which is not limited here.

[0082] For example, assuming the preset duration is 10 seconds and the access frequency is 10 times / minute, then in the cache, data with a life cycle less than or equal to 10 seconds or an access frequency greater than or equal to 10 times / minute is hot data, conversely, data with a life cycle greater than 10 seconds and an access frequency less than 10 times / minute is cold data.

[0083] The life cycle of data can be regarded as the life cycle of the corresponding Plog, and the access frequency of data can be regarded as the access frequency of the corresponding Plog. In the embodiment of the present application, the Plog corresponding to the hot data is a valid Plog that needs to be retained, and conversely, the Plog corresponding to the cold data is an invalid Plog that needs to be deleted. Therefore, we can determine through the above description that the valid Plog that needs to be retained is: the Plog whose life cycle in the cache is less than or equal to the preset duration or the access frequency is greater than or equal to the preset frequency is a valid Plog that needs to be retained; the Plog whose life cycle in the cache is greater than the preset duration and the access frequency is less than the preset frequency is an invalid Plog that needs to be deleted. It should be noted that to determine whether a Plog is a valid Plog or an invalid Plog, both the access frequency and the life cycle can be considered at the same time, or one of the two can be considered, which is not limited here.

[0084] Taking Table 1 as an example, assuming that the data shown in Table 1 is stored in the storage system, the name of the employee can be metadata (key), and the corresponding contact information can be the data corresponding to the pointed Plog (value). If the user reads the contact information of some employees, the employee name of the employee, such as Cindy, can be pointed to Plog3, and the data corresponding to Plog3 can be obtained, that is, the contact information = 13532345678.

[0085] Table 1

[0086] Staff Name Plog pointing to Contact Details Amy Plog1 13512345678 Betty Plog2 13522345678 Cindy Plog3 13532345678 Danny Plog4 13542345678 Ella Plog5 13552345678 Fable Plog6 13562345678

[0087] In this example, we can use both the life cycle and access frequency as the basis to determine whether the corresponding data is hot data or cold data, and then whether the Plog is valid Plog or invalid Plog. In this example, we set: the life cycle is less than or equal to 30 seconds or the access frequency is greater than or equal to 10 times / minute. As long as one of the two is met, it is determined to be hot data, otherwise it is cold data.

[0088] Through statistics, we can obtain the life cycle and access frequency of each piece of data as shown in Table 2:

[0089] Table 2

[0090] Staff Name Plog pointing to Contact Details life cycle Frequency of visit Hot / Cold Data Amy Plog1 13512345678 15 seconds 15 times Hot Data Betty Plog2 13522345678 2 minutes 5 times Cold Data Cindy Plog3 13532345678 1 year 30 times Hot Data Danny Plog4 13542345678 5 years 1 time Cold Data Ella Plog5 13552345678 30 minutes 0 times Cold Data Fable Plog6 13562345678 10 seconds 0 times Cold Data

[0091] As shown in Table 2, the hot data are the entries with metadata Amy, Cindy and Fabel, and the rest of the entries are cold data. The list of hot data is shown in Table 3:

[0092] Table 3

[0093] Staff Name Plog pointing to Contact Details Amy Plog1 13512345678 Cindy Plog3 13532345678 Fable Plog6 13562345678

[0094] As can be seen from Table 3, each valid Plog that needs to be retained in the cache is Plog1, Plog3 and Plog6.

[0095] 302. Create a valid Plog list, where the valid Plog list includes each valid Plog in the cache.

[0096] In an embodiment of the present application, after each valid Plog in the cache is determined, a valid Plog list may be created, the valid Plog list including each valid Plog in the cache.

[0097] As in the example described in step 301, after obtaining the valid Plogs that need to be retained in the cache (as shown in Table 3), the valid Plog list shown in Table 4 can be obtained:

[0098] Table 4

[0099] Valid Plog List Plog1 Plog3 Plog6

[0100] It should be noted that the valid Plog list is stored in a cache, and a data index structure such as a hash map (haspmap) or a multi-way balanced search tree (b-tree) can be added to the valid Plog list, which is not limited here.

[0101] 303. Delete every Plog in the cache except the valid Plog list.

[0102] In the embodiment of the present application, after the valid Plog list is created, the valid Plog list can be used as a sieve to filter the read request pointing to the invalid Plog, so that the data corresponding to the invalid Plog in the cache cannot be read, so the Plog outside the valid Plog list can be safely deleted without causing data reading failure. It should be noted that when a Plog is deleted, the data stored in the Plog can be stored in the HDD or deleted, which is not limited here.

[0103] 304. Receive a data read request, where the data read request includes metadata of the data to be read, and the metadata is used to point to the Plog corresponding to the data to be read.

[0104] In an embodiment of the present application, the storage system may receive a data read request, the data read request including metadata of the data to be read, the metadata being used to point to the Plog corresponding to the data to be read. Figure 3-2 , the user operates in the terminal device or the interactive system of the storage system to query the data to be read (3021), the terminal device or the interactive system of the storage system sends a data read request to the storage system according to the user's operation (3022), and then the storage system performs business processing according to the data read request (3023), and finally returns the data read result (3024). It should be noted that the data read result can be the data to be read or a data read failure message.

[0105] To avoid data read failure after the storage system receives a data read request with metadata, refer to Figure 3-3 In an embodiment of the present application, the storage system receives a data read request carrying metadata through the CPU, and then the CPU obtains the metadata from the data read request and searches the cache for the metadata. Specifically, the CPU sends a data read request carrying metadata to the cache (3031), and then indexes the metadata in the cache data index structure (3032). If the metadata is not in the cache data index structure, the cache will return a cache miss to the CPU (3033), and the CPU needs to search for the data to be read from the cached HDD; if the cache hits the metadata in the data index structure, it will hit the Plog pointed to by the metadata, and finally return the Plog pointed to by the metadata to the CPU (3034).

[0106] In the embodiment of the present application, after determining Plog, since it is unclear whether the Plog has been deleted in the cache, the CPU does not directly read data from the storage location indicated by Plog, but executes the following step 305 to avoid data reading failure.

[0107] 305. Determine whether Plog is in the valid Plog list.

[0108] Please refer to Figure 3-4 , the CPU of the storage system sends a query request for Plog to the cache (3041), and the cache confirms whether Plog is in the valid Plog list (3042). If Plog is in the valid Plog list, the cache returns a confirmation message to the CPU (3043), which means that the Plog is a valid Plog and should be stored in the cache without being deleted, so the data should be read from the storage location indicated by the Plog. If Plog is not in the valid Plog list, the cache returns a cache miss to the CPU (3044), indicating that the Plog is an invalid Plog and should have been deleted in step 303, so the CPU will not attempt to read data from the storage location indicated by the Plog.

[0109] For example, through Table 4, it can be queried that Plog1 is in the valid Plog list, it can be considered that Plog1 has not been deleted, and data can be read from the storage location indicated by Plog1 in the cache; but Plog2 is not in the valid Plog list, it can be considered that Plog2 should be deleted, so data cannot be read from the storage location indicated by Plog2 in the cache.

[0110] Since the read of data in the cache may not hit, that is, the data corresponding to the Plog may not be read after the Plog is determined, even if the Plog is deleted, it will not cause data reading failure. Therefore, after creating the valid Plog list, the Plog outside the valid Plog list can be safely deleted, while avoiding the increase of write amplification and the blocking of the Plog deletion process due to the long merging process.

[0111] 306. When the result of the determination is that Plog is not in the valid Plog list, it is determined that the data corresponding to Plog is not in the cache, the data corresponding to Plog is not read, and the to-be-read data is read from the HDD of the storage system.

[0112] In the embodiment of the present application, when the CPU receives a cache miss returned by the cache, according to the data index structure of the LSM tree of the storage system, the CPU will search for the data indicated by the metadata in the next level corresponding to the C0 level corresponding to the cache, that is, the C1 level. Since the data of the C1 level is stored in the HDD, the CPU can read the to-be-read data corresponding to the metadata from the HDD. If it is not found in C1, it will continue to search in the next level, which will not be repeated here.

[0113] 307. When the result of the determination is that Plog is in the valid Plog list, it is determined that the data corresponding to Plog is stored in the cache, and the data corresponding to Plog is read.

[0114] After the CPU determines that Plog is stored in the cache, when the result of the judgment is that Plog is in the valid Plog list, that is, it is determined that the data corresponding to Plog is stored in the cache, the data corresponding to Plog can be read from the cache. For details, please refer to Figure 3-5 , the CPU sends a data read request for the data corresponding to Plog to the cache (3051), and the cache determines the storage location indicated by Plog (3052), then obtains the data to be read from the storage location and returns it to the CPU (3053).

[0115] 308. Discard metadata pointing to invalid Plog.

[0116] According to the characteristics of the LSM tree, when the amount of data at a certain level exceeds a threshold, the data within a certain key range in the level is merged with the data within the same key range in the next level. In an embodiment of the present application, if the metadata points to an invalid Plog, such metadata can be considered invalid metadata, and the invalid metadata should be discarded to reduce the storage pressure in the cache. In an embodiment of the present application, the metadata to be merged can be traversed during the LSM tree merge to determine and delete the invalid metadata.

[0117] For example, in the example of Table 1 above, if the metadata "Betty" (pointing to Plog2, which is an invalid Plog) or "Danny" (pointing to Plog4, which is an invalid Plog) participates in the merging process of the LSM tree, these metadata can be discarded.

[0118] In the above embodiments, the data read request is filtered by creating a valid Plog list. In some feasible embodiments, the data read request can also be filtered by creating an invalid Plog list. Please refer to Figure 4-1, which is a data processing method also provided in the present application, and is applied to the cache 130 of the storage system 100, where the cache 130 uses Plog to carry data, and includes:

[0119] 401. Determine each invalid Plog in the cache. No valid Plog corresponds to the Plog of cold data in the cache. Cold data is data in the cache whose access frequency is less than a preset frequency or whose life cycle is greater than a preset duration.

[0120] In the embodiment of the present application, for the definition and determination of cold data and hot data, invalid Plog and valid Plog in the cache, reference may be made to step 301 and will not be repeated here.

[0121] Taking Table 1 in step 301 as an example, according to the determination method in step 301, invalid Plog can be determined, thereby obtaining a list of thermal data shown in Table 5:

[0122] Table 5

[0123] Staff Name Plog pointing to Contact Details Betty Plog2 13522345678 Danny Plog4 13542345678 Ella Plog5 13552345678

[0124] As can be seen from Table 5, each invalid Plog that needs to be deleted in the cache is Plog2, Plog4 and Plog5.

[0125] 402. Create an invalid Plog list, which includes each invalid Plog in the cache.

[0126] In the embodiment of the present application, after each invalid Plog in the cache is determined, an invalid Plog list can be created, and the invalid Plog list includes each valid Plog in the cache. As in the example described in step 401, after the invalid Plog to be deleted in the cache is obtained (as shown in Table 5), the invalid Plog list shown in Table 6 can be obtained:

[0127] Table 6

[0128] Valid Plog List Plog2 Plog4 Plog5

[0129] It should be noted that the invalid Plog list is stored in a cache, and a data index structure such as a hash map or a multi-way balanced search tree may be added to the invalid Plog list, which is not limited here.

[0130] 403. Delete every Plog in the cache except the valid Plog list.

[0131] In the embodiment of the present application, after the invalid Plog list is created, the invalid Plog list can be used as a sieve to filter the read request pointing to the invalid Plog, so that the data corresponding to the invalid Plog in the cache cannot be read, so the Plog in the invalid Plog list can be safely deleted without causing data reading failure. It should be noted that when a Plog is deleted, the data stored in the Plog can be stored in the HDD or deleted, which is not limited here.

[0132] 404. Receive a data read request, where the data read request includes metadata of the data to be read, and the metadata is used to point to the Plog corresponding to the data to be read.

[0133] Step 404 is the same as step 304 and will not be described in detail here.

[0134] 405. Determine whether Plog is in the invalid Plog list.

[0135] Please refer to Figure 4-2 , the CPU of the storage system sends a query request for Plog to the cache (4021), and the cache confirms whether Plog is in the invalid Plog list (4022). If Plog is not in the invalid Plog list, the cache returns a confirmation message to the CPU (4023), which means that the Plog is a valid Plog and should be stored in the cache without being deleted, so the data should be read from the storage location indicated by the Plog. If Plog is in the invalid Plog list, the cache returns a cache miss to the CPU (4024), indicating that the Plog is an invalid Plog and should have been deleted in step 403, so the CPU will not attempt to read data from the storage location indicated by the Plog.

[0136] For example, it can be found from Table 4 that Plog1 is not in the invalid Plog list, so it can be considered that Plog1 has not been deleted, and data can be read from the storage location indicated by Plog1 in the cache; while Plog2 is in the invalid Plog list, it can be considered that Plog2 should be deleted, so data cannot be read from the storage location indicated by Plog2 in the cache.

[0137] Since the read of data in the cache may not hit, that is, the data corresponding to the Plog may not be read after the Plog is determined, even if the Plog is deleted, it will not cause data reading failure. Therefore, after creating the valid Plog list, the Plog outside the valid Plog list can be safely deleted, while avoiding the increase of write amplification and the blocking of the Plog deletion process due to the long merging process.

[0138] 406. When the result of the determination is that Plog is in the invalid Plog list, it is determined that the data corresponding to Plog is not in the cache, the data corresponding to Plog is not read, and the to-be-read data is read from the HDD of the storage system.

[0139] Please refer to step 306, which will not be described in detail here.

[0140] 407. When the result of the determination is that Plog is not in the invalid Plog list, it is determined that the data corresponding to Plog is stored in the cache, and the data corresponding to Plog is read.

[0141] After the CPU determines that Plog is stored in the cache through the above steps, the data corresponding to Plog can be read from the cache. For details, please refer to step 307, which will not be described here.

[0142] 408. Discard metadata pointing to invalid Plog.

[0143] Step 408 is the same as step 308 and will not be described in detail here.

[0144] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a program, and the program executes some or all of the steps recorded in the above method embodiment.

[0145] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0146] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center to another website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0148] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0151] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), RAM, disk or CD-ROM and other media that can store program codes.

Claims

1. A data processing method, characterized in that: Applied to a storage system, the storage system includes a cache, the cache uses a continuous log Plog to carry data, and the method includes: receiving a data read request, the data read request including metadata of data to be read, the metadata being used to point to a Plog corresponding to the data to be read; Determine whether the data corresponding to the Plog is stored in the cache; When the result of the judgment is that the data corresponding to the Plog is not in the cache, the data corresponding to the Plog is not read, and the data to be read is read from other storage media of the storage system; wherein, The cache stores a valid Plog list, the valid Plog list includes multiple Plogs, and data corresponding to each Plog in the multiple Plogs is stored in the cache; The determining whether the data corresponding to the Plog is stored in the cache specifically includes: Determine whether the Plog is in the valid Plog list; When the result of the judgment is that the Plog is not in the valid Plog list, it is determined that the data corresponding to the Plog is not in the cache; or, The cache stores an invalid Plog list, the invalid Plog list includes multiple Plogs, and data corresponding to each Plog in the multiple Plogs is not in the cache; The determining whether the data corresponding to the Plog is stored in the cache specifically includes: Determine whether the Plog is in the invalid Plog list; When the result of the determination is that the Plog is in the invalid Plog list, it is determined that the data corresponding to the Plog is not in the cache.

2. The method according to claim 1, characterized in that: Before receiving the data reading request, the method further includes: Determine each valid Plog in the cache, where the valid Plog is the Plog corresponding to hot data in the cache, where the hot data is data in the cache whose access frequency is greater than or equal to a preset frequency or whose life cycle is less than or equal to a preset duration; The valid Plog list is created, and the valid Plog list includes each valid Plog in the cache.

3. The method according to claim 2, characterized in that: After creating the valid Plog list, it also includes: Each Plog in the cache except the valid Plog list is deleted.

4. The method according to any one of claims 1 to 3, characterized in that: The data index structure of the effective Plog list is a hash map or a multi-way balanced search tree.

5. The method according to claim 1, characterized in that: Before receiving the data reading request, the method further includes: Determine each invalid Plog in the cache, where the invalid Plog corresponds to the Plog of cold data in the cache, where the cold data is data in the cache with an access frequency less than a preset frequency and a life cycle greater than a preset duration; The invalid Plog list is created, and the invalid Plog list includes each invalid Plog in the cache.

6. The method according to claim 5, characterized in that: After creating the valid Plog list, it also includes: Delete each Plog in the cache except for the invalid Plog list.

7. The method according to claim 1, 5 or 6, characterized in that: The data index structure of the invalid Plog list is a hash map or a multi-way balanced search tree.

8. The method according to claim 1 or 2 or 3 or 5 or 6, characterized in that: The data in the storage system uses a merged LSM tree of a log structure as a data index structure, wherein the data in the cache corresponds to the first level of the LSM tree.

9. A storage system, characterized in that: include: Cache, used to carry data using Plog; A processor, configured to receive a data read request, wherein the data read request includes metadata of data to be read, wherein the metadata is used to point to a Plog corresponding to the data to be read; The processor is further configured to determine whether the data corresponding to the Plog is stored in the cache. When the result of the determination is that the data corresponding to the Plog is not in the cache, the data corresponding to the Plog is not read, but the data to be read is read from other storage media of the storage system; wherein, The cache is further used to store a valid Plog list, the valid Plog list includes multiple Plogs, and data corresponding to each Plog in the multiple Plogs is stored in the cache; The processor is specifically configured to determine whether the Plog is in the valid Plog list, and when the result of the determination is that the Plog is not in the valid Plog list, determine that the data corresponding to the Plog is not in the cache; or, The cache is further used to store an invalid Plog list, wherein the invalid Plog list includes multiple Plogs, and data corresponding to each Plog in the multiple Plogs is not in the cache; The processor is specifically configured to determine whether the Plog is in the invalid Plog list, and when the result of the determination is that the Plog is in the invalid Plog list, determine that the data corresponding to the Plog is not in the cache.

10. The storage system according to claim 9, characterized in that: The processor is further configured to determine each valid Plog in the cache, wherein the valid Plog is a Plog corresponding to hot data in the cache, and the hot data is data in the cache whose access frequency is greater than or equal to a preset frequency or whose life cycle is less than or equal to a preset duration; The processor is further configured to create the valid Plog list, wherein the valid Plog list includes each valid Plog in the cache.

11. The storage system according to claim 10, characterized in that: The processor is further configured to delete each Plog in the cache except the valid Plog list.

12. The storage system according to claim 9, characterized in that: The processor is further configured to determine each invalid Plog in the cache, wherein the invalid Plog corresponds to the Plog of cold data in the cache, wherein the cold data is data in the cache whose access frequency is less than a preset frequency and whose life cycle is greater than a preset duration; The processor is further configured to create the invalid Plog list, wherein the invalid Plog list includes each invalid Plog in the cache.

13. The storage system according to claim 12, characterized in that: The processor is further configured to delete each Plog in the cache except for the invalid Plog list.

14. A storage system, characterized in that: Includes processor and cache, where The cache stores a computer-readable program and uses Plog to carry data; The method according to any one of claims 1 to 8 is implemented by running the computer-readable program in the cache.

15. A computer program product, characterized in that When the computer program product is executed, it is used to perform the method according to any one of claims 1 to 8.

16. A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, wherein the instructions are used to execute the method according to any one of claims 1 to 8.

Images