Data management methods, apparatus, computer equipment, readable storage media and program products

Abstract

This application relates to a data management method, apparatus, computer device, computer-readable storage medium, and computer program product. The method is applied to a storage system comprising multi-level caches, each level of which contains a dataset stored based on a hierarchical caching mechanism. The method includes: responding to a data read instruction, sequentially querying for a first target data in the datasets of each level of the storage system; and when the first target data is found in the target cache level, providing feedback on the first target data. This method can improve data read and write efficiency.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data management method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Current storage systems provide data services in clusters. To improve user read and write performance, storage systems typically allocate a certain proportion of memory at the data entry point to cache newly written data or store frequently read data, thereby improving data read performance.

[0003] In traditional methods, in order to improve the read hit rate, in addition to allocating memory in the storage system, a pre-read strategy is also adopted. Based on the pre-read strategy, the dataset that the user may read is estimated and stored in memory. When the user initiates a read data command, the data is first queried from memory to improve the data reading efficiency.

[0004] However, traditional methods are limited by memory capacity and can only store a small amount of data. As a result, the data read hit rate is still low, which leads to low data reading efficiency. Summary of the Invention

[0005] Therefore, it is necessary to provide a data management method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0006] In a first aspect, this application provides a data management method applied to a storage system, the storage system including a multi-level cache, each level of the cache including a dataset stored based on a hierarchical caching mechanism, the method comprising:

[0007] In response to a data read command, the first target data is queried sequentially in the datasets of each cache level of the storage system;

[0008] When the first target data is found in the target cache level, feedback is provided on the first target data.

[0009] In this embodiment, multi-level caching is used to store data in multiple levels. Each level of caching contains a dataset stored based on the hierarchical caching mechanism. Therefore, when reading data, the data can be queried level by level in each level of caching based on the hierarchical storage mechanism, thereby improving the data read hit rate.

[0010] In one embodiment, the multi-level cache includes memory cache, solid-state drive cache and disk cache in order of hierarchy from first to last;

[0011] The memory cache includes a first-level cache, the solid-state drive cache includes a second-level cache and a third-level cache, and the disk cache includes a fourth-level cache.

[0012] In this embodiment, a multi-level caching mechanism is set up in the storage system. The multi-level cache includes memory cache, solid-state drive cache and disk cache. The problem of limited memory cache capacity is solved by adding solid-state drive cache. Furthermore, the multi-level cache stores datasets with different reading frequencies. The higher the reading frequency of the data, the greater the probability of it being hit first, thereby improving data reading efficiency.

[0013] In one embodiment, the first-level cache contains a first dataset, the second-level cache contains a second dataset, and the third-level cache contains a third dataset; the second dataset contains the first dataset.

[0014] In this embodiment, the multi-level cache in the storage system is intelligently divided and used, allowing data to reside in different caches for different durations as needed. After adding a warm data cache layer, the amount of data that can be cached at the front end is greatly increased, and the proportion of reads hitting the front end cache is greatly increased. This can shorten the data path depth of user read requests and provide higher read performance.

[0015] In one embodiment, the step of sequentially performing a first target data query in the datasets of each level of the storage system's cache in response to a data read instruction includes:

[0016] In response to a data read command, based on a preset multi-level cache priority strategy, the first target data is queried level by level in the multi-level cache in descending order of priority.

[0017] In this embodiment, data is managed by caching it into different levels of cache in the storage system according to hot data, warm data, low-temperature data and cold data. When reading data, the multi-level cache residency and eviction strategy ensures that the same data is preferentially stored in the hot data cache and warm data cache. This can effectively make up for the low capacity of simply using memory, and can significantly shorten the user's read path and improve the user's read performance.

[0018] In one embodiment, the step of responding to a data read instruction and querying the first target data level by level in the multi-level cache according to a preset multi-level cache priority strategy, in descending order of priority, includes:

[0019] If the first target data is not found in the first dataset of the first-level cache, then the first target data is queried in the second-level cache.

[0020] If the first target data does not exist in the second dataset of the second-level cache, query the third-level cache to see if the first target data exists.

[0021] If the first target data is not present in the third dataset of the third-level cache, data is retrieved from the disk cache and written into the third-level cache, and the first target data is read from the updated third dataset.

[0022] In this embodiment, data is managed by caching it into different levels of cache in the storage system according to hot data, warm data, low-temperature data, and cold data. This ensures that when data is read, the same data is preferentially stored in the hot data cache and the warm data cache through the residence and eviction strategy of the multi-level cache. This can significantly shorten the path for users to read data and improve the performance of data reading.

[0023] In one embodiment, the method further includes:

[0024] If the first target data exists in the second dataset of the second-level cache, then the first target data is loaded into the second-level cache.

[0025] In this embodiment, data retrieval is performed through hierarchical caching to shorten the user's reading path as much as possible and improve the efficiency of user data reading.

[0026] In one embodiment, the step of loading the first target data into the second-level cache if the second dataset in the second-level cache contains the first target data includes:

[0027] If a portion of the first target data exists in the second dataset of the second-level cache, the portion of the first target data that exists is loaded from the second-level cache.

[0028] Load the first target data that does not exist in the second dataset of the second level cache from the third level cache, write the first target data that does not exist into the second level cache, and perform the step of loading the first target data that exists from the second level cache.

[0029] In this embodiment, the lifecycle (i.e., update cycle) of the data (warm data and low temperature data) in the second and third datasets and the data (cold data) in the fourth dataset is configured. The cache space is reclaimed by sorting by popularity. The second-level cache, the third-level cache and the fourth-level cache all support the function of writing back missing data, which can solidify the missing data in the cache media at each level in a timely manner, improve the read hit rate of hot data, and set different lifecycles for the data cache in the second, third and fourth datasets to ensure that the low temperature data cache can cache more data.

[0030] In one embodiment, the method further includes:

[0031] In response to a data write instruction, the second target data corresponding to the data write instruction is written to the first-level cache;

[0032] Check the cache status of the second-level cache. If the second-level cache is in a normal state and meets the preset data caching period, write the second target data in the first-level cache into the second-level cache.

[0033] The second target data is written to the third level cache through the second level cache, and the second target data is synchronously written to the disk cache through the second level cache.

[0034] In this embodiment, the computer device creates write requests for the local flash drive and backend storage through the second-level cache and the third-level cache, and completes the operation of the two requests simultaneously without interfering with each other. That is, the second target data is written to the third-level cache through the second-level cache (i.e., creating a local flash drive write request), and then the second target data is synchronously written to the disk cache directly through the second-level cache.

[0035] Secondly, this application also provides a data management device, comprising:

[0036] The query module is used to query the first target data in the dataset of each level of cache in the storage system in response to the data read command.

[0037] The sending module is used to send feedback on the first target data when the first target data is found in the target cache level.

[0038] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0039] In response to a data read command, the first target data is queried sequentially in the datasets of each cache level of the storage system;

[0040] When the first target data is found in the target cache level, feedback is provided on the first target data.

[0041] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0042] In response to a data read command, the first target data is queried sequentially in the datasets of each cache level of the storage system;

[0043] When the first target data is found in the target cache level, feedback is provided on the first target data.

[0044] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0045] In response to a data read command, the first target data is queried sequentially in the datasets of each cache level of the storage system;

[0046] When the first target data is found in the target cache level, feedback is provided on the first target data.

[0047] The aforementioned data management method, apparatus, computer equipment, computer-readable storage medium, and computer program product are applied to a storage system containing multi-level caches. Each level of the cache contains a dataset stored based on a hierarchical caching mechanism. The method includes: responding to a data read instruction, sequentially performing a first target data query in each level of the storage system's caches; and when the first target data is found in the target cache level, providing feedback on the first target data. This method, by setting up multi-level caches for multi-level data storage, with each level containing a dataset stored based on a hierarchical caching mechanism, allows for step-by-step data queries in each cache level during data reads, improving the data read hit rate. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a schematic diagram of the storage system in one embodiment;

[0050] Figure 2 This is a flowchart illustrating a data management method in one embodiment;

[0051] Figure 3 This is a schematic diagram of the structure of a multi-level cache in a storage system in one embodiment;

[0052] Figure 4 This is a flowchart illustrating a method for querying first target data in one embodiment;

[0053] Figure 5 This is a flowchart illustrating the step of querying the first target data level by level in one embodiment;

[0054] Figure 6 This is a flowchart illustrating the step of hitting the first target data in the second-level cache in one embodiment;

[0055] Figure 7 This is a flowchart illustrating the step of partially hitting the first target data in the second-level cache in one embodiment.

[0056] Figure 8 This is a flowchart illustrating the user data writing step in one embodiment;

[0057] Figure 9 This is a schematic diagram illustrating a specific process of a user reading data in one embodiment.

[0058] Figure 10 This is a schematic diagram illustrating a specific process of a user writing data in one embodiment.

[0059] Figure 11 This is a structural block diagram of a data management method apparatus in one embodiment;

[0060] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] Before introducing the specific content of this application, let's first introduce the technical terms involved in this application:

[0063] DPC: Data Page Cache (Front-end Memory Cache);

[0064] UNC: A data caching acceleration module in a distributed storage system. It improves the data read and write performance of the storage system by using a cache disk to create a data cache area, manages the lifecycle of data in the cache, judges the data popularity, and dynamically adjusts the cache location of the data.

[0065] Hot data: Data recently written by users, and data that is frequently read (most frequently read) within a short period of time;

[0066] Warm data: After being written by a user, it is accessed frequently, but less frequently than hot data;

[0067] Low-temperature data: After being written by the user, there is an access frequency, but the frequency is lower than that of high-temperature data;

[0068] Cold data: Data that is written by a user but not accessed for a long period of time.

[0069] The data management method provided in this application embodiment can be applied to, for example... Figure 1 The storage system shown contains multi-level caches, each containing a dataset stored based on a hierarchical caching mechanism. Specifically, in response to a data read command, the computer device sequentially queries the datasets of each level of the storage system for the first target data; when the first target data is found in the target cache level, it provides feedback on the first target data. Therefore, this storage system, for its multi-level caches, implements hierarchical data storage management based on data popularity, enabling data querying and data storage at each level of the cache during data read and write operations, thus improving data management efficiency.

[0070] In one exemplary embodiment, such as Figure 2 As shown, a data management method is provided, which can be applied to... Figure 1 The storage system described herein is integrated into a computer device, and the data management method is implemented through the computer device. This storage system includes multi-level caches, with each level containing a dataset stored based on a hierarchical caching mechanism. The specific process includes steps 202 to 206. Wherein:

[0071] Step 202: In response to the data read command, perform a first target data query in the datasets of each level of cache in the storage system.

[0072] In practice, current storage systems provide services on a cluster or per user basis. To improve read performance, storage systems typically allocate a certain proportion of computational memory to cache recently written or frequently read data. However, this design is limited by the upper limit of computer memory capacity, allowing only a very small amount of data to be cached. Furthermore, as users continuously update and clean up their data, the probability of a user hitting data residing in memory during random read / write operations is significantly reduced.

[0073] Storage systems also utilize read-ahead mechanisms to anticipate the location of data users expect to read and then complete the data read in advance. However, this is still limited by memory capacity, allowing only a small amount of data to reside and being quickly overwritten by new data. This often leads to scenarios with limited cache space, resulting in frequent data refreshes and exchanges. This not only increases software complexity but also poses a risk of data inconsistency. Due to the unpredictability of user behavior, the data read in advance may not yield significant read benefits. The result of this approach is an increase in read / write operations and disk concurrent I / O for the backend storage system.

[0074] To address this issue, this application utilizes SSDs (Solid State Disk / Drive) or NVMe (Non-Volatile Memory Express) for data caching, resolving the memory space constraint problem. Therefore, the storage system design incorporates multi-level caching, including both disk caching and memory caching. Disk caching stores a portion of the data, minimizing the length of the I / O read / write path, thereby improving read hit rate and performance. Optionally, terabyte-level solid-state drives can be used for data storage, effectively improving the read performance of the storage system.

[0075] Specifically, the storage system utilizes multi-level caching, with each level containing datasets stored using a tiered caching mechanism. Each level of cache stores datasets with varying read frequencies. When a user triggers a data read command for the first target data in the storage system via a client on the computer device, the computer device responds by sequentially querying the datasets in each level of the storage system's cache for that first target data.

[0076] Specifically, the caching mechanism for each preset tier in the storage system stores data layer by layer according to the sequential order of the caching tiers (i.e., first-tier cache → second-tier cache → third-tier cache → fourth-tier cache) and the corresponding data update cycle of each tier. For example, data is first written to the first-tier cache. When the storage duration of data in the first-tier cache reaches its data update cycle, and no read instructions have been received during the data storage period, the data in the first-tier cache is written to the next tier, i.e., the second-tier cache. Simultaneously, the data in the first-tier cache is released and cleared to receive newly written data. For data in the second-tier cache, the decision is based on the second-tier cache's data update cycle. When the storage duration of data in the second-tier cache reaches its data update cycle, and no read instructions have been received during the data storage period, the data in the second-tier cache is written to the next tier, i.e., the third-tier cache. Simultaneously, the data in the second-level cache is released and cleared to make room for new data written from the first-level cache. Data in the third-level cache remains in the third-level cache using the same storage method. When the data update cycle of the third-level cache arrives and no data read instruction is received, the data in the third-level cache will be further written to the fourth-level cache. If the fourth-level cache is the last cache level in the storage system, the data will remain in the fourth-level cache and will not be passed on further.

[0077] Step 204: When the first target data is found in the target cache level, feedback is given to the first target data.

[0078] In implementation, the computer device performs a layer-by-layer cache query based on the query order of the multi-level cache in the storage system. If the first target data is not found in the current level cache, the system queries whether the dataset stored in the next level cache contains the first target data. This process continues until the first target data is found in the target cache level. At this point, the storage system in the computer device sends the first target data back to the client where the computer device is located, thus providing the user with the first target data.

[0079] The aforementioned data management method is applied to a storage system containing multi-level caches. Each level of the cache contains a dataset stored based on a hierarchical caching mechanism. The method includes: responding to a data read instruction, sequentially querying for a first target data in each level of the storage system's caches; and when the first target data is found in the target cache level, feeding the first target data back to the client. By employing this method, multi-level caching is used for multi-level data storage, with each level containing a dataset stored based on a hierarchical caching mechanism. Therefore, during data reads, data can be queried level by level in each cache based on the hierarchical storage mechanism, improving the data read hit rate.

[0080] In one exemplary embodiment, the multi-level cache includes a memory cache, a solid-state drive (SSD) cache, and a disk cache in descending order of level. Specifically, the memory cache includes a first-level cache, the SSD cache includes second-level and third-level caches, and the disk cache includes a fourth-level cache.

[0081] In implementation, the multi-level caching system of a storage system includes memory cache, solid-state drive cache, and disk cache in a sequential order. Specifically, the order is: Level 1 cache, Level 2 cache, Level 3 cache, and Level 4 cache. This sequential order also reflects the order of read and / or write priorities from highest to lowest. The memory cache includes Level 1, Level 2, and Level 3 caches. The sequential order of the multi-level caching is defined based on the frequency of data access in each cache from highest to lowest. Therefore, Level 1 cache can be called hot data cache, Level 2 cache, Level 3 cache, and Level 4 cache. Figure 3 As shown, Figure 3 This is a schematic diagram of the multi-level cache structure in a storage system. The hot data cache uses physical memory as its storage medium, the warm data cache uses a cache disk, the low-temperature data cache uses a cache disk, and the cold data cache uses a traditional mechanical hard disk.

[0082] In this embodiment, a multi-level caching mechanism is set up in the storage system. The multi-level cache includes memory cache, solid-state drive cache and disk cache. The problem of limited memory cache capacity is solved by adding solid-state drive cache. Furthermore, the multi-level cache stores datasets with different reading frequencies. The higher the reading frequency of the data, the greater the probability of it being hit first, thereby improving data reading efficiency.

[0083] In one exemplary embodiment, a first-level cache contains a first dataset, a second-level cache contains a second dataset, and a third-level cache contains a third dataset. The second dataset contains the first dataset.

[0084] In implementation, the storage system includes multi-level caches that store data based on a preset caching mechanism. Each level of cache is continuously updated and cleaned up as users write data. Therefore, within the data update cycle corresponding to each level of cache, the first level cache contains the first dataset, the second level cache contains the second dataset, the third level cache contains the third dataset, and the fourth level cache (disk cache) contains the fourth dataset. Since the read frequency (i.e., the access frequency) of the data stored in each level of cache differs, the first dataset, based on its read frequency, can also be called the "hot dataset," the second dataset can be called the "warm dataset," and the third dataset can be called the "low-temperature dataset." Furthermore, because the preset caching mechanism involves writing data layer by layer, after a user performs a data write operation, the data is first written to the first level cache (i.e., the hot data cache). When the data update cycle of the first level cache ends, the first dataset in the first level cache is written to the second level cache, and the first level cache then clears the first dataset. Subsequently, the data in the initially written first dataset is passed to subsequent caches in the storage system. Because different levels of cache correspond to data with varying read frequencies, the data update cycle for each cache level increases as the read frequency decreases. Therefore, the data update cycle for the first-level cache is shorter than that for the second-level cache. The second dataset stored in the second-level cache contains all the data from the first-level cache; in other words, the second dataset includes the first dataset.

[0085] In this embodiment, the multi-level cache in the storage system is intelligently divided and used, allowing data to reside in different caches for different durations as needed. After adding a warm data cache layer, the amount of data that can be cached at the front end is greatly increased, and the proportion of reads hitting the front end cache is greatly increased. This can shorten the data path depth of user read requests and provide higher read performance.

[0086] In one exemplary embodiment, such as Figure 4 As shown, step 202 includes the following steps:

[0087] Step 402: In response to the data read instruction, based on the preset multi-level cache priority strategy, the first target data is queried level by level in the multi-level cache in order of priority from high to low.

[0088] In implementation, a dedicated software module is set up in the storage system to manage the residence and eviction of hot data, warm data, low-temperature data, and cold data caches on different media within the storage system. Hot data, warm data, low-temperature data, and cold data are defined based on data read frequency (also known as data read intensity), and will not be explained again in the following embodiments. Therefore, when a user initiates a data read command, the computer device runs the software module in the storage system to respond to the command. Based on a preset multi-level cache priority strategy, it queries the first target data level by level in the multi-level cache according to the order of priority from high to low. Specifically, the second-level cache (warm data cache) supplements the first-level cache (hot data cache), compensating for the limited amount of hot data stored in the first-level cache; the third-level cache acts as a bridge between the second and fourth-level caches, also belonging to the high-speed cache area, solving the problem of insufficient cache levels affecting data read performance. Therefore, during data querying, the data is queried level by level from the first-level cache to the fourth-level cache until the first target data is found.

[0089] In this embodiment, data is managed by caching it into different levels of cache in the storage system according to hot data, warm data, low-temperature data and cold data. When reading data, the multi-level cache residency and eviction strategy ensures that the same data is preferentially stored in the hot data cache and warm data cache. This can effectively make up for the low capacity of simply using memory, and can significantly shorten the user's read path and improve the user's read performance.

[0090] In an exemplary embodiment, when the data read by the user is not in the DPC or the second-level cache, the data will be loaded from the third-level cache first. After the data is returned to the user, a copy will be synchronously stored in the third-level cache to improve the hit rate of subsequent reads on the front-end cache. If the data is not in the third-level cache, it will be loaded from the fourth-level cache. This data will be synchronously stored in both the third-level and second-level caches. Since a copy has already been stored in the second-level cache, the data in the third-level cache will enter the recycling process earlier. Figure 5 As shown, the specific processing steps of step 202 or step 402 include:

[0091] Step 502: If the first target data is not found in the first dataset of the first-level cache, then query the first target data in the second-level cache.

[0092] In implementation, when querying data level by level, if the first target data exists in the first dataset stored in the first-level cache, the first target data can be directly returned to the user. If the first target data does not exist in the first dataset stored in the first-level cache, the computer will query the first target data in the second-level cache.

[0093] Step 504: If the first target data is not present in the second dataset of the second-level cache, query the third-level cache to see if the first target data exists.

[0094] In implementation, if the first target data is still not found in the second dataset of the second-level cache, the computer device will continue to query the next level of the multi-level cache, that is, check whether the first target data exists in the third-level cache. If the first target data exists in the second dataset of the second-level cache, the computer device can load the first target data from the second-level cache into the first-level cache to provide feedback to the user.

[0095] Step 506: If the first target data is not present in the third dataset of the third-level cache, retrieve the data from the disk cache and write it into the third-level cache, and then read the first target data from the updated third dataset.

[0096] In implementation, if the first target data is still not present in the third dataset stored in the third-level cache, the computer device will retrieve data from the disk cache and write it into the third-level cache, and then read the first target data from the updated third dataset. If the first target data exists in the third dataset in the third-level cache, the computer device can load the first target data from the third-level cache into the second-level cache and the first-level cache in sequence, and then provide feedback to the user.

[0097] In this embodiment, data is managed by caching it into different levels of cache in the storage system according to hot data, warm data, low-temperature data, and cold data. This ensures that when data is read, the same data is preferentially stored in the hot data cache and the warm data cache through the residence and eviction strategy of the multi-level cache. This can significantly shorten the path for users to read data and improve the performance of data reading.

[0098] In one exemplary embodiment, such as Figure 6 As shown, the cache disk (disk) supports read miss data write-back functionality. This method also includes:

[0099] Step 602: If the first target data exists in the second dataset of the second-level cache, then load the first target data into the second-level cache.

[0100] In implementation, if the first target data exists in the second dataset of the second-level cache, the computer device loads the first target data into the second-level cache, and writes the first target data back into the first-level cache, thereby feeding the first target data back to the user.

[0101] In this embodiment, data retrieval is performed through hierarchical caching to shorten the user's reading path as much as possible and improve the efficiency of user data reading.

[0102] In one exemplary embodiment, such as Figure 7 As shown, in response to a special case during data reading, namely, if only part of the first target data exists in the current level cache, the computer device will further query the data in the next level cache until the complete first target data is retrieved. Specifically, the processing steps in step 602 include:

[0103] Step 702: If a portion of the first target data exists in the second dataset of the second-level cache, load the portion of the first target data from the second-level cache.

[0104] In implementation, if a portion of the first target data exists in the second dataset of the second-level cache, the computer device loads the portion of the first target data into the second-level cache and loads the existing portion of the first target data into the first-level cache, thereby writing back the data so that the user can read that portion of the first target data.

[0105] Step 704: Load the first target data that does not exist in the second dataset of the second-level cache from the third-level cache, write the first target data that does not exist into the second-level cache, and execute the step of loading the first target data that exists from the second-level cache.

[0106] In implementation, the query for the first target data continues in the third-level cache. Specifically, the first target data that is not present in the second dataset of the second-level cache is loaded from the third-level cache. This non-existent portion of the first target data is written to the second-level cache, and step 702 above is executed again. This step-by-step write-back process provides feedback to the user regarding their query data.

[0107] In this embodiment, the lifecycle (i.e., update cycle) of the data (warm data and low temperature data) in the second and third datasets and the data (cold data) in the fourth dataset is configured. The cache space is reclaimed by sorting by popularity. The second-level cache, the third-level cache and the fourth-level cache all support the function of writing back missing data, which can solidify the missing data in the cache media at each level in a timely manner, improve the read hit rate of hot data, and set different lifecycles for the data cache in the second, third and fourth datasets to ensure that the low temperature data cache can cache more data.

[0108] Optionally, the front-end warm data cache module (second-level cache and third-level cache) is designed in read-only storage mode. User data will be written to both the front-end warm data cache module (second-level cache and third-level cache) and the back-end storage module (fourth-level cache) at the same time. Feedback to the user end must be sent only after the back-end storage medium has been completed and returned. Data will be written to the back-end low-temperature data cache first. This solution can improve the performance of writing data.

[0109] In one exemplary embodiment, such as Figure 8 As shown, the UNC-managed front-end memory cache, with its small cached data volume and high data eviction frequency, is a core module of the storage system. The front-end warm data cache (including warm and cold data) can store more user data, improving upon the limitations of the front-end memory cache space. It reclaims space based on read / write activity and is also a core module. The back-end cold data cache module complements the front-end warm data cache. Both the warm data cache (second-level cache, third-level cache) and the cold data cache (fourth-level cache) support data popularity sorting and space reclamation based on data popularity. Therefore, this application provides a method for users to write data, which includes:

[0110] Step 802: In response to the data write instruction, write the second target data corresponding to the data write instruction into the first-level cache.

[0111] In practice, when a user performs a write operation on data, the computer device responds to the data write instruction by preferentially writing the second target data corresponding to the data write instruction into the first-level cache (hot data cache).

[0112] Step 804: Check the cache status of the second-level cache. If the second-level cache is in a normal state and meets the preset data caching period, write the second target data in the first-level cache into the second-level cache.

[0113] In implementation, the computer device generates a user write IO and submits it to the front-end input module (DPC) of the storage system. At this time, the DPC will first check the cache status of the second-level cache and the third-level cache in the current storage environment to see if they can operate normally. If the cache status of the second-level cache and the third-level cache is normal, the computer device will submit the second target data to be written to the second-level cache and the third-level cache.

[0114] Step 806: Write the second target data to the third level cache through the second level cache, and synchronously write the second target data to the disk cache through the second level cache.

[0115] In this embodiment, the computer device creates write requests for the local flash drive and backend storage through the second-level cache and the third-level cache, and completes the operation of the two requests simultaneously without interfering with each other. That is, the second target data is written to the third-level cache through the second-level cache (i.e., creating a local flash drive write request), and then the second target data is synchronously written to the disk cache directly through the second-level cache.

[0116] Optionally, UNC will retain one copy of the data in the hot data cache (first-level cache) and one copy in the warm data cache (second-level cache), and write it to the cold data cache (fourth-level cache). Since this data has already been cached at the front end, UNC will inform the back end cold data cache through an agreed protocol whether the data has been retained in the warm data cache. If it has been retained in the warm data cache, the data in the cold data cache can enter the reclamation process earlier to release storage space. UNC performs global management of data distribution to ensure that each data cache in the storage system can achieve maximum caching efficiency.

[0117] Optionally, the warm data caching module (including warm data cache and low temperature data cache, i.e., second-level cache and third-level cache), as a data front-end caching module, may encounter disk anomalies, resulting in scenarios where writing to the cache disk fails. This application stipulates that if the write to the backend storage system is successful, the user write IO request returns normally; conversely, if the write to the backend storage system fails, regardless of whether the warm data cache write to disk is successful, the user write IO request returns an exception.

[0118] Optionally, the warm data cache module (including warm data cache and low temperature data cache, i.e., second-level cache and third-level cache) supports dynamic expansion and contraction. The cache disk managed by the warm data cache can be added or removed according to user needs and hardware conditions. This operation will not affect the user's read and write performance and will not cause data consistency issues.

[0119] Optionally, if a write failure occurs on the cache disk read / written by the warm data cache module (including warm data cache and low temperature data cache, i.e., second-level cache and third-level cache), the warm data cache module will mark the faulty cache disk as unwritable but readable, thus maximizing the performance of the front-end cache and improving the read hit rate.

[0120] Optionally, the cache disk for warm data caching cannot reside all user data. Therefore, the warm data cache will also sort the data by popularity and clean up low-popularity data periodically to ensure that newly written data or hot data with a high number of read hits is prioritized.

[0121] In a specific embodiment, such as Figure 9 As shown, this application provides a data management method involving the user's data reading process. To better reflect the hierarchical storage of data based on data read frequency in the storage system, the method uses hot data cache, warm data cache, low-temperature data cache, and cold data cache to describe each cache layer in the multi-level cache. The process specifically includes the following steps:

[0122] Step 901: The user sends a data read request for the first target data.

[0123] In step 902, the computer device responds to the data read request by checking the hot data cache for a match. If the first target data is not matched, step 903 is executed. If the first target data is matched, the next data read request can be executed.

[0124] Step 903: The computer device checks the warm data cache for a match. If the first target data is not matched, proceed to step 904; if the first target data is partially or fully matched, proceed to step 907.

[0125] Step 904: The computer device checks the low-temperature data cache for a match. If some of the first target data is not matched, proceed to step 905. If the first target data is matched, proceed to step 906.

[0126] Step 905: Obtain the first target data from the HDD (Hard Disk Drive).

[0127] Step 906: Read the first target data from the low-temperature data cache and load it into the front-end cache so that feedback can be sent to the user.

[0128] Step 907: Load data from the warm data cache; if all the first target data is hit in the warm data cache, proceed to step 908; if some of the first target data is hit in the warm data cache, read the cached data of the hit portion and proceed to step 909.

[0129] Step 908: Load the first target data from the warm data cache.

[0130] Step 909: Load the missing cached data from the low-temperature data cache.

[0131] In a specific embodiment, such as Figure 10 As shown, this application provides a data management method involving the user's data writing process. To better reflect the hierarchical storage of data based on data read frequency in the storage system, the method uses hot data cache, warm data cache, low-temperature data cache, and cold data cache to describe each cache layer in the multi-level cache. The process specifically includes the following steps:

[0132] Step 1001: The user sends a data write request for the second target data;

[0133] In step 1002, the computer device responds to the data write request by writing the second target data into the hot data cache.

[0134] Step 1003: After the hot data cache update cycle arrives, the computer device's warm data cache writes the second target data into the warm data cache.

[0135] Step 1004: The computer device checks the data cache status in the warm data cache. If the data cache status is normal, proceed to step 1005; if the data cache status is abnormal, proceed to step 1007.

[0136] Step 1005: Write the second target data into the warm data cache.

[0137] Step 1006: After passing through the warm data cache, the second target data is cached to the disk cache and synchronized to the backend storage system.

[0138] Step 1007: The computer device determines that the warm data cache is in an unserviceable state;

[0139] Step 1008: Check the data cache status in the low-temperature data cache through the UNC backend service module. If the data cache status is normal, proceed to step 1009. If the data cache status is abnormal, proceed to step 1010.

[0140] Step 1009: Write the second target data into the low-temperature data cache.

[0141] Step 1010: Write the second target data to the disk cache (cold data storage pool, mechanical hard drive).

[0142] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0143] Based on the same inventive concept, this application also provides a data management device for implementing the data management method described above. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data management device embodiments provided below can be found in the limitations of the data management method described above, and will not be repeated here.

[0144] In one exemplary embodiment, such as Figure 11 As shown, a data management device is provided, including: a query module 1101 and a sending module 1102, wherein:

[0145] The query module 1101 is used to query the first target data in the dataset of each level of cache in the storage system in response to the data read command.

[0146] The sending module 1102 is used to send feedback on the first target data when the first target data is found in the target cache level.

[0147] In one embodiment, the multi-level cache includes memory cache, solid-state drive cache and disk cache in order of hierarchy from first to last;

[0148] Among them, memory cache includes first-level cache, solid-state drive cache includes second-level cache and third-level cache, and disk cache includes fourth-level cache.

[0149] In one embodiment, the first-level cache contains a first dataset, the second-level cache contains a second dataset, and the third-level cache contains a third dataset; the second dataset contains the first dataset.

[0150] In one embodiment, the query module 1101 is specifically used to respond to a data read instruction and, based on a preset multi-level cache priority strategy, query the first target data level by level in the multi-level cache in order of priority from high to low.

[0151] In one embodiment, the query module 1101 is specifically used to query the first target data in the second-level cache if the first target data does not exist in the first dataset of the first-level cache;

[0152] If the first target data is not found in the second dataset of the second-level cache, then query the third-level cache to see if the first target data exists.

[0153] If the first target data is not present in the third dataset of the third-level cache, retrieve the data from the disk cache and write it into the third-level cache, and then read the first target data from the updated third dataset.

[0154] In one embodiment, the device further includes:

[0155] The loading module is used to load the first target data from the second dataset in the second-level cache if the first target data already exists in the second dataset.

[0156] In one embodiment, the loading module is specifically used to load the partially existing first target data from the second-level cache if a portion of the first target data exists in the second dataset of the second-level cache.

[0157] Load the first target data that does not exist in the second dataset of the second dataset in the second-level cache from the third-level cache, write the first target data that does not exist into the second-level cache, and perform the step of loading the first target data that exists from the second-level cache.

[0158] In one embodiment, the device further includes:

[0159] The response module is used to respond to a data write command and write the second target data corresponding to the data write command into the first-level cache.

[0160] The inspection module is used to check the cache status of the second-level cache. If the second-level cache is in a normal state and meets the preset data caching period, the second target data in the first-level cache is written into the second-level cache.

[0161] The write module is used to write the second target data to the third level cache through the second level cache, and to synchronously write the second target data to the disk cache through the second level cache.

[0162] Each module in the aforementioned data management device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0163] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a data management method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0164] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0165] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0166] In response to a data read command, the first target data is queried sequentially in the datasets of the caches at each level of the storage system;

[0167] When the first target data is found in the target cache level, feedback is provided for the first target data.

[0168] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0169] In response to a data read command, based on a preset multi-level cache priority strategy, the system queries the first target data level by level in the multi-level cache in descending order of priority.

[0170] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0171] If the first target data is not found in the first dataset of the first-level cache, then the first target data is queried in the second-level cache.

[0172] If the first target data is not found in the second dataset of the second-level cache, then query the third-level cache to see if the first target data exists.

[0173] If the first target data is not present in the third dataset of the third-level cache, retrieve the data from the disk cache and write it into the third-level cache, and then read the first target data from the updated third dataset.

[0174] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0175] If the first target data exists in the second dataset of the second-level cache, then the first target data is loaded into the second-level cache.

[0176] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0177] If the second dataset in the second-level cache contains a portion of the first target data, load the portion of the first target data from the second-level cache.

[0178] Load the first target data that does not exist in the second dataset of the second dataset in the second-level cache from the third-level cache, write the first target data that does not exist into the second-level cache, and perform the step of loading the first target data that exists from the second-level cache.

[0179] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0180] In response to a data write command, the second target data corresponding to the data write command is written to the first-level cache;

[0181] Check the cache status of the second-level cache. If the second-level cache is in a normal state and meets the preset data caching period, write the second target data in the first-level cache into the second-level cache.

[0182] The second target data is written to the third level cache through the second level cache, and then synchronously written to the disk cache through the second level cache.

[0183] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0184] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0185] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0186] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0187] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data management method, characterized in that, The method is applied to a storage system that includes multi-level caches, each level of which contains a dataset stored based on a hierarchical caching mechanism. The method includes: In response to a data read command, the first target data is queried sequentially in the datasets of each cache level of the storage system; When the first target data is found in the target cache level, the first target data is fed back; if only part of the first target data exists in the current level cache, the data is further queried in the next level cache until the complete first target data is found. The multi-level cache, in descending order of level, includes memory cache, solid-state drive cache, and disk cache; The memory cache includes a first-level cache, the solid-state drive cache includes a second-level cache and a third-level cache, and the disk cache includes a fourth-level cache; the second dataset stored in the second-level cache contains all the data from the first-level cache.

2. The method according to claim 1, characterized in that, The first-level cache contains a first dataset, the second-level cache contains a second dataset, and the third-level cache contains a third dataset; the second dataset contains the first dataset.

3. The method according to claim 1, characterized in that, The step of responding to a data read instruction by sequentially querying the datasets of each level of the storage system cache for the first target data includes: In response to a data read command, based on a preset multi-level cache priority strategy, the first target data is queried level by level in the multi-level cache in descending order of priority.

4. The method according to claim 3, characterized in that, The step of responding to a data read command by querying the first target data level by level in the multi-level cache according to a preset multi-level cache priority strategy, in descending order of priority, includes: If the first target data is not found in the first dataset of the first-level cache, then the first target data is queried in the second-level cache. If the first target data does not exist in the second dataset of the second-level cache, query the third-level cache to see if the first target data exists. If the first target data is not present in the third dataset of the third-level cache, data is retrieved from the disk cache and written into the third-level cache, and the first target data is read from the updated third dataset.

5. The method according to claim 4, characterized in that, The method further includes: If the first target data exists in the second dataset of the second-level cache, then the first target data is loaded into the second-level cache.

6. The method according to claim 5, characterized in that, If the first target data exists in the second dataset of the second-level cache, then loading the first target data into the second-level cache includes: If a portion of the first target data exists in the second dataset of the second-level cache, the portion of the first target data that exists is loaded from the second-level cache. Load the first target data that does not exist in the second dataset of the second level cache from the third level cache, write the first target data that does not exist into the second level cache, and perform the step of loading the first target data that exists from the second level cache.

7. The method according to claim 1, characterized in that, The method further includes: In response to a data write instruction, the second target data corresponding to the data write instruction is written to the first-level cache; Check the cache status of the second-level cache. If the second-level cache is in a normal state and meets the preset data caching period, write the second target data in the first-level cache into the second-level cache. The second target data is written to the third level cache through the second level cache, and the second target data is synchronously written to the disk cache through the second level cache.

8. A data management device, characterized in that, The apparatus is applied to a storage system, the storage system including multi-level caches, each level of cache containing a dataset stored based on a hierarchical caching mechanism, the apparatus comprising: The query module is used to query the first target data in the dataset of each level of cache in the storage system in response to the data read command. The sending module is used to send feedback on the first target data when the first target data is found in the target cache level; if only part of the first target data exists in the current level cache, the data is further queried in the next level cache until the complete first target data is found. The multi-level cache, in descending order of level, includes memory cache, solid-state drive cache, and disk cache; The memory cache includes a first-level cache, the solid-state drive cache includes a second-level cache and a third-level cache, and the disk cache includes a fourth-level cache; the second dataset stored in the second-level cache contains all the data from the first-level cache.

9. The apparatus according to claim 8, characterized in that, The first-level cache contains a first dataset, the second-level cache contains a second dataset, and the third-level cache contains a third dataset; the second dataset contains the first dataset.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images