Layered storage read cache management method and system, storage medium and product

Abstract

The invention provides a hierarchical storage read cache management method and system, a storage medium and a product, and relates to the field of data storage, the method comprises the following steps: dividing all data blocks into three data hierarchies of hot data, warm data and cold data for respective storage according to access popularity, the data storage positions, the storage data proportions and the information recording precision of the three data levels are different; when the cold data is accessed, the accessed target cold data is upgraded to the temperature data, and the temperature data is updated; when the hot data is accessed, the complete metadata information of the accessed target hot data is cached in the memory, and the hot data is sorted and updated again; and when the temperature data is accessed, determining whether to replace the target hot data with the target temperature data or not according to the size relationship between the first historical access time of the accessed target temperature data and the second historical access time of the target hot data with the lowest access heat. By implementing the method, the occupation of memory resources can be reduced when the popularity of mass data is tracked.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data storage, and in particular to a read cache management method and system for hierarchical storage, a storage medium and a product. BACKGROUND

[0002] With the rapid development of technologies such as big data, cloud computing and artificial intelligence, the total amount of global data is growing at an unprecedented rate, and data storage systems have entered the PB (Petabyte) or even EB (Exabyte) level. When building such a large-scale storage system, the balance between cost and performance becomes a core challenge in design. High-performance storage media such as solid-state drives (SSDs) and memories (DRAMs) have fast read and write speeds, but the cost per unit capacity is high; while traditional mechanical hard drives (HDDs) are low in cost, but their random read and write performance is far inferior to that of solid-state media. Therefore, the hierarchical storage architecture using SSDs as a cache layer and HDDs as a large-capacity data storage layer has become the mainstream solution in the industry to balance performance and cost. Under this architecture, how to design an efficient read cache management algorithm to accurately place the most frequently accessed data (i.e. hot data) in the cache layer is the key to determining the performance of the entire system.

[0003] In related technologies, in order to improve cache hit rate, an advanced adaptive cache replacement algorithm is widely used, such as the classic ARC (Adaptive Replacement Cache) algorithm. This algorithm dynamically adjusts the proportion of cache space used to record "recent access history" and "frequent access history" to adapt to changing workloads. Its implementation is to maintain two core LRU (Least Recently Used) lists in memory: the T1 list is used to store new data blocks that have been accessed only once, and the T2 list is used to store frequent data blocks that have been accessed twice or more. In addition, in order to make more accurate replacement decisions, the ARC algorithm also maintains two "ghost lists" (Ghost List): B1 and B2, which record the metadata information of data blocks evicted from T1 and T2. When a data block is accessed, the algorithm checks all four lists to determine its access history and decides its flow or replacement between T1 and T2.

[0004] However, as the data scale managed by the storage system jumps from TB level to PB level, the total amount of data blocks can reach billions or even hundreds of millions. In order to maintain its adaptability and high hit rate, the ARC algorithm requires tracking the access history of a large number of data blocks, which means that even if a data block is not currently cached in the memory, its metadata (such as the complete ID of the data block) may need to be retained in the shadow list B1 or B2. Therefore, when the number of "non-hot" but potentially hot data blocks that the system needs to track reaches the order of hundreds of millions, the memory overhead for storing these metadata will expand dramatically. Each metadata entry needs to consume tens of bytes, and the cumulative effect will result in tens of GB or even more valuable system memory being used purely for cache management, which constitutes a huge operating burden and cost pressure in resource-sensitive storage servers. SUMMARY

[0005] The application provides a hierarchical storage read cache management method, system, storage medium and product, which is used to solve the problem of excessive memory overhead caused by tracking all data hotness of traditional cache algorithms under PB-level massive data.

[0006] In a first aspect, the application provides a hierarchical storage read cache management method, which is applied to a hierarchical storage system including a memory, a solid state disk and a mechanical hard disk, and the method comprises: all data blocks are divided into three data levels of hot data, warm data and cold data according to access hotness for storage, the hot data table corresponding to the hot data and the warm data table corresponding to the warm data have different recording accuracies of hotness information, and the access hotness is determined according to the last access time of data; in response to a read request for the cold data, the target cold data to be accessed is promoted to warm data, and the warm data is sorted and eliminated at the end according to the least recently used strategy, and the cold data and the warm data are stored in the mechanical hard disk; in response to a read request for the hot data, the complete metadata information of the target hot data to be accessed is cached in the memory, and the hot data is re-sorted and updated according to the least recently used strategy, and the target hot data is stored in the cache area of the solid state disk and the mechanical hard disk; in response to a read request for the warm data, it is judged whether the first historical access time of the target warm data to be accessed is greater than the second historical access time of the target hot data with the lowest access hotness, the warm data table corresponding to the warm data stores hotness information in the memory using a multi-path group associative way of information compression, and the complete metadata information of the warm data is stored in the solid state disk; if yes, the target warm data is used to replace the target hot data; and if no, the warm data is re-sorted and updated according to the least recently used strategy.

[0007] By adopting the technical scheme, the system can divide data into hot, warm and cold three levels, and adopt different accuracy heat recording strategies for different levels. Specifically, the system only retains complete metadata for the hottest hot data in the memory, and for the large number of warm data, the heat information (not complete metadata) is efficiently stored in the memory in the form of information compression, and the complete metadata is stored in the solid state disk. This differentiated management mode greatly reduces the occupation of valuable memory resources when the system tracks the heat of massive data. Finally, while maintaining high-precision hot data identification and high cache hit rate, the core problem of excessive memory overhead faced by traditional algorithms under PB-level data scale is effectively solved, and the optimization balance of cost and performance is realized.

[0008] In some embodiments, the step of storing the heat information of the warm data table corresponding to the warm data in the memory using the information compression multi-way set associative manner specifically includes: calculating a hash value of a 64-bit logical identifier of each data block, the hash value being used to realize uniform distribution of the data block among groups; determining a target group number according to the hash value, the target group number being used to indicate a storage group of each data block; extracting a low 16-bit of the hash value as a tag value, the tag value being used to uniquely identify the data block within the storage group, so that the collision probability of a plurality of different logical identifiers of the data block having the same tag value within the same storage group is lower than a preset collision threshold; recording a current access time as a 16-bit timestamp, the timestamp recording a recent access time in minutes; combining the tag value and the timestamp to form a 32-bit compressed record; storing the compressed record in a storage group corresponding to the target group number, each storage group having a capacity of 32 records and being sorted and managed in the least recently used strategy within the group.

[0009] By adopting the technical scheme, the system can efficiently compress and store the heat information of the warm data into the memory. By performing hash calculation on the logical identifier of the data block, uniform distribution of the data among different storage groups is realized, and hot spot concentration is avoided. Then, the low 16-bit of the hash value is extracted as a tag to identify the data block within the group with extremely low collision probability, and a 16-bit timestamp is combined to form a compressed record of only 32 bits. This way, the metadata information originally requiring dozens of bytes is compressed to 4 bytes, greatly reducing the memory space occupied by the heat tracking of a single data block, so that the system can track the access history of hundreds of millions of warm data at extremely low memory cost.

[0010] In some embodiments, the step of judging whether the first historical access time of the accessed target warm data is less than the second historical access time of the target hot data with the lowest access frequency, specifically comprises: extracting the last access time of the warm data from the compressed record of the warm data table; obtaining the last access time of the hot data from the tail of the double-linked list of the hot data table; if the last access time of the warm data is greater than the last access time of the hot data, determining that the first historical access time of the accessed target warm data is greater than the second historical access time of the target hot data with the lowest access frequency; if the last access time of the warm data is less than or equal to the last access time of the hot data, determining that the first historical access time of the accessed target warm data is less than or equal to the second historical access time of the target hot data with the lowest access frequency.

[0011] By using the above technical solution, the system directly extracts the last access time from the compressed record of the warm data in the memory, and obtains the last access time of the current least active (with the lowest access frequency) hot data from the tail of the double-linked list of the hot data table. This process does not need to access the complete metadata in the low-speed medium (solid state disk or mechanical hard disk), and is completely completed in the memory. By directly comparing the size of the two time stamps, the system can quickly and lowly judge whether a accessed warm data has a higher access frequency than the existing hot data.

[0012] In some embodiments, the step of replacing the target hot data with the target warm data, specifically comprises: reading the data content of the target warm data from the mechanical hard disk, the data content including the actual stored user data; allocating new cache space in the solid state disk, the cache space being used to store the target warm data to be promoted; writing the data content into the cache space to complete the physical storage of the target warm data in the high-speed layer; constructing complete metadata information including logical identification, solid state disk address and mechanical hard disk address, the complete metadata information providing accurate data positioning; inserting the complete metadata information into the head of the hot data table, and marking the target warm data as the current hottest state; deleting the replaced item at the tail of the hot data table, and releasing the solid state disk space and the complete metadata in the memory occupied by the replaced item, and updating the level state of the replaced item to cold data.

[0013] By adopting the technical scheme, the system reads target temperature data from a mechanical hard disk and writes the target temperature data into a newly allocated cache space of a solid state disk, so that the data is accelerated on a physical storage layer. Then, complete metadata containing accurate address information is constructed in the memory and inserted into a head of a hot data table, while the coldest data in a tail of the original hot data table and all resources (memory metadata and solid state disk space) occupied by the coldest data are eliminated and cleaned. The method ensures that the migration process of the data from the warm layer to the hot layer not only updates the logical state of the data, but also synchronously completes physical caching of the data on a high-speed medium and metadata updating of the data in the memory, so that the data access performance is improved.

[0014] In some embodiments, the method further comprises: In the periodic time reference adjustment process, a new time origin is set as a current time minus a preset number of days, and the time origin is used to determine data validity; All existing compressed records in the groups are traversed to extract an access time field of each record; A relative time difference of each record from the time origin is calculated; Expired records with a negative relative time difference are deleted, and time fields of remaining records are updated to the relative time difference.

[0015] By adopting the technical scheme, the system can periodically maintain and clean expired hotness information in the warm data table. By setting a new time origin and traversing all compressed records, the system can calculate the relative timeliness of the access time of each record. For those records with too long access time (negative relative time difference), the system will directly delete the records, so as to automatically "forget" the cold data information that has not been accessed for a long time and has no potential to become hot. At the same time, the time stamp of the remaining records is updated to a relative value. This process effectively recovers the limited space of the warm data table, prevents unlimited accumulation of invalid historical data, and ensures the validity and timeliness of the hotness records.

[0016] In some embodiments, the step of storing the compressed records into the storage group corresponding to the target group number specifically comprises: If the storage group is full, an existing compressed record with the earliest last access time in the storage group is identified as a to-be-eliminated item according to a least recently used strategy in the group; The to-be-eliminated item is replaced by a new compressed record corresponding to the to-be-accessed data block, and an elimination error rate caused by the storage group capacity limitation is controlled to be below a preset error rate threshold.

[0017] By adopting the technical scheme, when a new access occurs and the corresponding storage group is full, the system accurately identifies and eliminates the oldest record in the group according to a least recently used (LRU) strategy in the group, and replaces the record with a new compressed record. This design allows the latest access history to be retained in limited storage space. Through reasonable design of the hash and group capacity, the elimination error rate (i.e. the error of eliminating data that may be accessed in the future) caused by space limitation can be controlled below an acceptable threshold, thereby ensuring the overall accuracy of the heat tracking while greatly saving memory.

[0018] In some embodiments, the step of responding to the read request of the temperature data specifically comprises: allocating a preset number of bits as an access count field in the 32-bit compressed record of the temperature data table corresponding to the temperature data; When the target temperature data is accessed, if the access count field corresponding to the target temperature data does not reach a preset promotion count value, only the count value and the access time corresponding to the target temperature data are updated; When the access count field of the target temperature data has reached the promotion count value, the step of judging whether the first historical access time of the accessed target temperature data is greater than the second historical access time of the target hot data with the lowest heat is performed.

[0019] By adopting the technical scheme, the system adds a "frequency" threshold for the promotion of temperature data to hot data to prevent frequent fluctuations in data levels caused by accidental single access. By adding an access count field in the 32-bit compressed record, the system not only tracks the "recency" (access time) of the data, but also tracks the "frequency" (access count). Only when the access frequency of a temperature data reaches a preset promotion count value in a short period of time, the system will start the heat comparison program with the coldest hot data. This mechanism can more accurately identify "true hot" data that is continuously and frequently accessed in a short period of time, effectively filtering the interference caused by "glitch" access, and improving the stability and efficiency of the cache.

[0020] In a second aspect, the present application provides a hierarchical storage system, comprising: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is configured to store computer program code, the computer program code comprising computer instructions, and the one or more processors invoke the computer instructions so that the system can implement the hierarchical storage read cache management method provided by the above-mentioned embodiments, which will not be described here.

[0021] In a third aspect, the present application provides a computer readable storage medium comprising instructions which, when executed on a tiered storage system, cause the system to implement a tiered storage read cache management method as described above in the embodiments, which will not be repeated here.

[0022] In a fourth aspect, the present application provides a computer program product which, when executed on a tiered storage system, cause the system to implement a tiered storage read cache management method as described above in the embodiments, which will not be repeated here.

[0023] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. A metadata precision tiered management system is constructed which maps to the data physical tiers (memory, SSD, HDD). It abandons the traditional algorithm which uses the same precision to track all non-cache data, and designs a high compression ratio heat recording mechanism for the largest number of warm data, storing only its 32-bit compressed heat information in memory, and placing the complete metadata on the SSD, so that the system can track the access heat of massive data at a very low memory cost (significantly reduced compared to traditional solutions), fundamentally solving the problem of high memory overhead caused by tracking too much historical information in a PB-level storage system, and achieving excellent scalability.

[0024] 2. A dual-determination promotion mechanism based on access frequency and recent access time is used to ensure that cache resources are allocated to high-value data. It introduces an access counter as the first layer of determination condition, effectively filtering data that may be incorrectly promoted due to accidental or low-frequency access, and preventing invalid occupation of hot cache resources. Only after the access frequency meets the requirements, the system will start the second layer of determination, i.e., quickly comparing the access timestamp of the data in memory with the timestamp of the coldest hot data. This dual-filtering mechanism improves the accuracy of hot data identification and the stability of the cache, effectively suppressing the cache replacement overhead caused by inaccurate promotion decisions.

[0025] 3. An extensible and self-maintaining memory data structure is designed for the heat information of the large number of warm data. Through hash grouping, in-group LRU eviction, and low-conflict probability label design, this structure can accommodate massive compressed records in limited memory space and efficiently handle access updates. In addition, it introduces a periodic time reference adjustment and expired record cleaning mechanism. This mechanism can actively identify and remove data records that have not been accessed for a long time, automatically recycle storage space and maintain the effectiveness and timeliness of the heat information library, ensuring the performance stability of the cache management system in long-term operation. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1is a flowchart of a read cache management method of a tiered storage system according to an embodiment of the present application; Figure 2 is another flowchart of a read cache management method of a tiered storage system according to an embodiment of the present application; Figure 3 is a schematic diagram of an example scenario of data reading by a tiered storage system according to an embodiment of the present application; Figure 4 is a schematic diagram of an example scenario of building a temperature data table by a tiered storage system according to an embodiment of the present application; Figure 5 is a schematic diagram of an example structure of an entity device of a tiered storage system according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The terminology used in the following embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the description of the embodiments of the present application and the appended claims, the singular forms "a," "an" and "the" are intended to include both singular and plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used in the description of the embodiments of the present application, refers to any one or combination of any

[0028] Hereinafter, the terms "first" and "second" are used only for the purpose of description and should not be understood as implying or suggesting relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0029] The method provided by the present embodiment is described in the following flow. Please refer to Figure 1 is a flowchart of a read cache management method of a tiered storage system according to an embodiment of the present application;

[0030] In order to more intuitively understand the tiered storage architecture proposed by the present application, please refer to Figure 3FIG. 1 is a schematic diagram of an exemplary scenario of data reading by a tiered storage system in an embodiment of the present application, showing the data reading architecture of the tiered storage system designed by the present application. The diagram clearly reveals how the three logical data tiers (hot, warm, and cold) are mapped to the three physical storage media (memory, solid state disk, and mechanical hard disk). It depicts the access paths of data of different hotness through arrows, and intuitively shows how the system realizes performance optimization through differential management: when accessing the “hot data” with the highest frequency, the system reads the metadata from the memory at high speed and reads the data content from the solid state disk, with the shortest path and the fastest speed; when accessing the “cold data”, it directly obtains it from the mechanical hard disk with the lowest cost, without occupying valuable cache resources.

[0031] Specifically, reading hot data utilizes the “hot data table” and “metadata cache” in the memory, and the “data read cache” in the solid state disk, wherein the data in the “data read cache” is called from the “data capacity layer” of the mechanical hard disk, and is adjusted according to the change of the hot data tier; reading warm data involves the “warm data table” in the memory, the “metadata” in the solid state disk, and the “data capacity layer” in the mechanical hard disk, and the final data source is the mechanical hard disk; reading cold data directly accesses the “data capacity layer” of the mechanical hard disk. This flowchart is a macroscopic manifestation of the core idea of the present application, “metadata precision tiered management system”, and shows how the system balances cost and performance to provide the most efficient access service for data of different values.

[0032] S101, divide all data blocks into three data tiers of hot data, warm data, and cold data according to the access hotness for storage respectively.

[0033] Among them, the access hotness refers to the frequency of reading data blocks, which is used to represent the activity state of data; the data block is the basic unit of data management in the storage system, which is divided by 256KiB in the present system, and a 64-bit integer unique number is used as the metadata id identifier; the hot data refers to a type of data with the highest access frequency, accounting for 5%~10% of the total data, consistent with the read cache capacity; the warm data refers to a type of data with an access frequency between the hot data and the cold data, accounting for 25% of the total data; the cold data refers to a type of data with the lowest access frequency, accounting for 60%~65% of the total data, which is all data other than the hot data and the warm data; the data tier is a logical tier for classifying and managing data according to the access hotness, and different tiers correspond to different storage locations, storage proportions, and information recording precision.

[0034] Specifically, the core basis for the system to divide the data hierarchy is the last access time of the data, which quantifies the access heat of the data. In terms of storage resource allocation, the data content of hot data is stored in the cache area of the solid state disk and the mechanical hard disk, and the complete metadata is stored in the memory; the data content of warm data is only stored in the mechanical hard disk, and the complete metadata is stored in the solid state disk, and the compressed heat information is stored in the memory; the data content of cold data is only stored in the mechanical hard disk. In terms of information recording accuracy, hot data needs to record complete metadata information and accurately mark the heat, warm data records heat information in a compressed manner, and cold data does not record any heat information. The system completes the initial data hierarchy division by combining the preset data total amount proportion threshold and the historical access time statistics of the data, and dynamically updates the hierarchy of the data according to the heat change in the subsequent data access process, to ensure that the data hierarchy matches the actual access heat.

[0035] Optionally, the system can first count the access times and the last access time of all data blocks in a preset period, set the hot data access time threshold and the time threshold (such as the access times in the last 7 days ≥ 50 times and the last access time within 24 hours), the warm data access time threshold and the time threshold (such as the access times in the last 7 days 10-49 times and the last access time within 7 days), and the rest of the data is determined as cold data; then allocate solid state disk cache space and mechanical hard disk storage space for hot data, and only allocate mechanical hard disk storage space for warm data and cold data; finally, a data hierarchy mapping table is established to record the metadata id, the hierarchy to which each data block belongs, the storage location and other information, which is stored in the memory and the solid state disk. Optionally, the system can first preliminarily group the data blocks according to their logical address ranges, sample and analyze the access heat of each group of data; then determine the proportion of hot, warm and cold data in each group of data according to the sampling results, to ensure that the overall proportion meets the preset proportion; then configure the corresponding storage medium and information recording rule for different levels of data, cache complete metadata to memory for hot data, generate 32-bit compressed heat records for warm data, and do not generate heat records for cold data; finally, write the grouped data of different levels to the corresponding storage medium, and update the hierarchy management index synchronously.

[0036] S102, in response to the read request for cold data, the target cold data to be accessed is promoted to warm data, and the warm data is sorted and removed from the end according to the least recently used strategy.

[0037] Among them, the read request refers to the instruction of the application or business to the storage system to obtain data; the target cold data refers to the cold data block specified by the read request to be accessed; the least recently used strategy refers to a management strategy that sorts data according to access time and preferentially removes data that has not been accessed for the longest time; the end removal refers to removing the warm data item at the end of the sorting and the earliest access time when the warm data storage capacity reaches the limit.

[0038] Specifically, after receiving a data read request, the system verifies the current level of the target data through the data popularity query module. Once confirmed as cold data, it initiates the level promotion process. First, the status of the target cold data is updated, promoting it from the cold data level to the warm data level, while simultaneously recording its access time as initial popularity information. Since the total amount of warm data has a fixed percentage limit, promoting new warm data might cause the total amount of warm data to exceed the threshold. In this case, the system uses a least recently used strategy to sort all warm data, eliminating the warm data item at the bottom of the sort that has not been accessed for the longest time. This eliminated item is then downgraded back to cold data, ensuring that the total amount of warm data remains within the preset percentage range, while also ensuring that the warm data consists of relatively active data.

[0039] Optionally, the system can first parse the logical data ID in the read request and query the metadata mapping table to determine that the current level of the data is cold data. Then, it can generate a 32-bit compressed heat record for the data, which includes a hash label calculated based on the logical ID and the current access timestamp. Next, it can insert the compressed record into the storage group corresponding to the warm data table. If the group has reached 32 records, it can sort all records in the group according to the least recently used strategy, delete the record with the earliest access time at the end, and downgrade the corresponding warm data to cold data. Finally, it can update the warm data management index and the data level mapping table to mark the target cold data as warm data. Optionally, the system can first receive a read request, then query the target data status through the hierarchical cache in memory to confirm that it is cold data; then read the content of the target cold data from the mechanical hard drive and temporarily cache it to the solid-state drive to improve the read performance; next, the access information of the data is recorded in the warm data temporary record table, triggering the overall sorting of warm data, arranged from newest to oldest according to the most recent access time; if the total amount of warm data exceeds 25% of the total data, the excess quantity is calculated, and the corresponding number of warm data are eliminated from the end of the sort, updating their level to cold data; finally, the target cold data is officially included in the warm data management, generating the corresponding heat record and synchronizing it to the warm data table in memory.

[0040] S103. In response to the request to read temperature data, determine whether the first historical access time of the target temperature data being accessed is greater than the second historical access time of the target hot data with the lowest access popularity.

[0041] Among them, the target temperature data refers to the temperature data block that the read request specifies to be accessed; the first historical access time refers to the most recent access time of the target temperature data before this access; the target hot data with the lowest access popularity refers to the hot data block at the end of the sorted list in the hot data table that has not been accessed for the longest time; and the second historical access time refers to the most recent access time of the target hot data with the lowest access popularity.

[0042] Specifically, after receiving a request to read temperature data, the system first extracts the 32-bit compressed heat record of the target temperature data through the temperature data recording module, and parses out the first historical access time from it. Simultaneously, it accesses the doubly linked list maintained by the heat data recording module, retrieves the target hot data with the lowest access heat at the tail of the list, and extracts its second historical access time. The system then compares these two times, with the core judgment logic being whether the activity level of the target temperature data exceeds that of the currently least active hot data. If the first historical access time is greater than the second historical access time, it indicates that the target temperature data is more active than the coldest or hottest data and is eligible for promotion; if this condition is not met, the target temperature data remains at the temperature data level, and only its own heat information is updated.

[0043] Optionally, the system can first parse the logical data ID in the read request, query the compressed record of the target temperature data through hash grouping of the temperature data table, and extract a 16-bit timestamp as the first historical access time; then access the doubly linked list of hot data, locate the tail node of the list, and read the last access time of the target hot data stored in that node as the second historical access time; then convert the first and second historical access times into a unified time format and compare their numerical values; finally, record the comparison result. If the first time is greater than the second time, mark the target temperature data as eligible for promotion; otherwise, mark it as maintaining the original level. Optionally, the system can also first receive the temperature data read request, trigger the temperature data heat recording module to unlock the compressed record of the target data, decode it to obtain the first historical access time; simultaneously, send a query request to the hot data management module to obtain the access times of the last 10% of hot data in the hot data table, take the minimum value as the second historical access time; then start the time comparison thread, considering the difference in time precision (temperature data is in minutes, hot data is in seconds), convert the first historical access time to seconds before comparison; finally, generate a comparison report to clarify whether the target temperature data meets the promotion conditions and provide feedback to the heat promotion and elimination module.

[0044] S104. In response to the request to read hot data, cache the complete metadata information of the accessed target hot data in memory, and reorder and update the hot data according to the least recently used strategy.

[0045] Among them, target hot data refers to the hot data block that is specified for access by the read request; complete metadata information refers to data containing all location and attribute information such as logical data ID, solid-state drive physical block number, hard disk physical block number, and last access time; re-sorting and updating refers to adjusting the position of hot data in the sorting queue according to the current access time to ensure that the queue order matches the popularity.

[0046] Specifically, after receiving the hot data read request, the system quickly locates the complete metadata information of the target hot data through the hot data metadata cache in the memory, without reading the metadata from the solid state disk or the mechanical hard disk, and directly acquires the data storage location in the solid state disk based on the metadata, thereby realizing high-speed data reading. At the same time, since the current access improves the heat of the target hot data, the system removes the target hot data from the current position in the double-linked list and reinserts it into the head of the linked list according to the least recently used strategy, and marks it as the current hottest data. Through this sorting update, it is ensured that the hot data linked list is always sorted from new to old according to the access time, and the head of the linked list is the hottest data and the tail is the coldest data.

[0047] Optionally, the system can first parse the logical data id in the read request, accurately match in the hot data metadata cache in the memory, find the complete metadata of the target hot data, then read the data from the solid state disk cache area based on the solid state disk physical block number in the metadata and return it to the requester, then update the last access time in the metadata to the current time, and lock the hot data double-linked list, finally delete the linked list node corresponding to the target hot data from the original position, reinsert it into the head of the linked list, and unlock the linked list to complete the sorting update. Optionally, the system can also first receive the hot data read request, start the metadata preloading mechanism, copy the complete metadata of the target hot data to the cache area in the memory while reading the data to improve the subsequent access speed, then update the access timestamp of the metadata to generate a sorting adjustment instruction, then traverse the hot data double-linked list to find the target data node and disconnect it from the front and rear nodes, and finally insert the node into the head of the linked list, update the head and tail pointers and node index of the linked list, and ensure that the linked list sorting accurately reflects the data heat, while recording the sorting update log.

[0048] S105, if yes, the target warm data is used to replace the target hot data; if no, the warm data is re-sorted and updated according to the least recently used strategy.

[0049] Wherein, the replacement operation refers to the promotion of the target warm data to hot data, and the demotion of the original hot data with the lowest heat to cold data; the re-sorting update refers to adjusting the sorting position of the warm data in the warm data table group according to the current access time of the warm data, to ensure that the sorting in the group conforms to the least recently used rule.

[0050] Specifically, if the result of step S103 is "Yes" (the first historical access time is greater than the second historical access time), the hot data replacement process is started. The system first reads the content of the target warm data from the mechanical hard disk, allocates new cache space in the solid state disk and writes the data, constructs the complete metadata of the target warm data and inserts it into the head of the hot data table, and marks it as the hottest data. At the same time, the original target hot data at the tail of the hot data table is deleted, the solid state disk space and memory metadata occupied by it are released, and it is downgraded to cold data. If the result is "No", the target warm data maintains the warm data level, and the system reorders the target warm data in the warm data table group to which it belongs according to the least recently used strategy, moves it to the head of the group, updates its access time, and ensures that the data in the group is always arranged from new to old according to the access time.

[0051] In the above embodiment, the system can divide data into hot, warm, and cold three levels, and adopt different precision heat recording strategies for different levels. Specifically, the system only retains complete metadata in memory for the hottest hot data, and for the large number of warm data, it uses information compression to efficiently store the heat information (not complete metadata) in memory, and the complete metadata is stored in the solid state disk. This differentiated management method greatly reduces the occupation of valuable memory resources when tracking massive data heat. Finally, while maintaining high-precision hot data identification and high cache hit rate, the core problem of excessive memory overhead faced by traditional algorithms in PB-level data is effectively solved, and the optimization balance between cost and performance is achieved.

[0052] The method provided by the present embodiment will be further described in more detail. Please refer to Figure 2 , which is another flowchart of the layered storage read cache management method in the embodiment of the present application.

[0053] Before understanding the following specific steps, please refer to Figure 4 , which is an exemplary scenario diagram of constructing a warm data table in the layered storage system in the embodiment of the present application, which details the internal organization of the memory data structure used for efficiently tracking the heat of warm data, i.e. the internal organization of the "warm data table". The structure adopts a "multi-way set-associative" design, which is composed of N independent "sets". All warm data blocks are uniformly mapped to a certain one of the N sets through hash calculation, thereby avoiding hot spot conflicts and ensuring query efficiency. This design aims to manage the access history of massive warm data with minimal memory overhead.

[0054] Within each "group", 32 record entries are fixedly stored (of course, it can also be adjusted according to actual needs). Each entry is a highly compressed 32-bit record, which is composed of a 16-bit "tag" and a 16-bit "time". The "tag" is a short identifier that uniquely identifies the data block within the group, and the "time" records the latest access time. When a group is full, if there are new access records to be stored, the system will use the "time" field to adopt the least recently used (LRU) strategy to eliminate the oldest entry in the group. The figure directly shows how the application realizes the low-cost and high-efficiency tracking of the hotness of billions of warm data through hash grouping, short tags, and LRU elimination mechanism.

[0055] S201, determining a target group number according to a hash value of a 64-bit logical identifier of each data block.

[0056] Among them, the 64-bit logical identifier refers to the unique 64-bit integer number assigned by the system to each 256KiB granularity logical data block, which is used to identify the data block; the hash value refers to the 64-bit numerical value obtained by calculating the 64-bit logical identifier through a hash algorithm, which is used to realize the uniform distribution of data blocks among groups; the target group number refers to the storage group serial number calculated according to the hash value, which is used to indicate the corresponding warm data table storage group of the data block.

[0057] This step is triggered when cold data is accessed to be promoted to warm data, or warm data is accessed to update the hotness record, and is suitable for the storage and management of warm data hotness information. It is the basic step of the multi-path group associative management of the warm data table, which ensures the uniform distribution of data blocks in the warm data table.

[0058] Specifically, the system obtains the 64-bit logical identifier of the data block to be processed (which may be warm data just promoted or existing warm data accessed). Then a preset hash algorithm is called to calculate the logical identifier to generate a 64-bit hash value. The hash value has the characteristic of uniform distribution, which can avoid the excessive concentration of data blocks in a certain storage group. Then according to the preset calculation rule, the target group number corresponding to the data block is determined based on the hash value, so that different data blocks can be reasonably allocated to different storage groups of the warm data table.

[0059] Optionally, the system can first obtain the 64-bit logical identifier of the data block to be processed; then call the hash algorithm to calculate the logical identifier, and take the low 64 bits of the result as the target hash value; then calculate the target group number according to the formula (hash value / 65536)%N (N is the total number of groups of the warm data table), to ensure that the number falls within the effective range of 0 to N-1; finally, the calculated group number is used as the target group number to locate the storage group of the data block.

[0060] S202, extract the low 16 bits of the hash value as a tag value, and record the current access time as a 16-bit timestamp.

[0061] Wherein, the tag value refers to the value extracted from the low 16 bits of the hash value, used to uniquely identify the data block in the storage group, and the collision probability is lower than the preset threshold; the 16-bit timestamp refers to the 16-bit value recorded in minutes for the current access time of the data block, used to represent the access time of the data block, and the maximum value corresponds to about 45 days.

[0062] Specifically, after obtaining the 64-bit hash value of the data block, the system directly extracts the low 16 bits of the hash value as the tag value, which can uniquely identify the data block in the storage group, and the collision probability of the same tag value appearing in the same group for different logical data blocks is only about 0.05%. At the same time, the system obtains the current system time, converts it into a time value in minutes, and then extracts the low 16 bits of the value as a 16-bit timestamp, which is used to record the access time of the data block this time.

[0063] Optionally, the system can first obtain the 64-bit hash value of the data block, obtain the low 16 bits of the hash value by bit operation, obtain the tag value and store it; then obtain the total number of milliseconds of the current system time, convert it into the number of minutes (total number of milliseconds divided by 60000); then perform 16-bit modulo operation on the number of minutes to obtain a 16-bit timestamp; finally, the tag value and the 16-bit timestamp are temporarily stored in a temporary cache for subsequent compression record combination. Optionally, the system can also first perform a shift operation on the 64-bit hash value, retain the low 16 bits as the tag value, and verify the uniqueness of the tag value; then obtain the current system date and time, calculate the total number of minutes from the preset starting time to the current time; then convert the total number of minutes into a 16-bit binary number to obtain a 16-bit timestamp; finally, encode the tag value and the timestamp respectively to ensure that the data format meets the compression record requirements.

[0064] S203, combine the tag value and the timestamp to form a 32-bit compressed record.

[0065] Wherein, the 32-bit compressed record refers to a 32-bit data record combined by a 16-bit tag value and a 16-bit timestamp, used to store the core heat information of the data; the combination refers to the operation of splicing the tag value and the timestamp into a 32-bit data according to the preset format.

[0066] Specifically, the system splices the 16-bit tag value as the high 16 bits of the 32-bit compressed record and the 16-bit timestamp as the low 16 bits according to the preset combination rule, or splices them in other preset order to form a complete 32-bit compressed record. The compressed record only occupies 4 bytes, which is reduced by 55.6% compared with the 72-bit record of ordinary multi-channel group associative management.

[0067] In addition, in the embodiments of the present application, there is a risk of overflow of the access time field (the maximum value of the 16-bit minute-level timestamp corresponds to about 45 days) for the 32-bit compressed records of the warm data table, and the effectiveness of the hotness records can be ensured through a periodic time reference adjustment mechanism. Specifically, the system can automatically perform time reference adjustment at a preset period (such as during the low-traffic period in the early morning of each day): first, a new time origin is calculated, which is the current system time minus a preset number of days (usually 30 days, taking into account the hotness tracking requirements and the risk of overflow), and the origin is used as the time reference for judging whether the warm data still has the potential to become hot; then, the 16-bit access time field of each record is extracted from the compressed records in all storage groups of the warm data table; then, the relative time difference between the access time of each record and the new origin is calculated, and if the difference is negative, it means that the warm data corresponding to the record has not been accessed for more than 30 days and has lost the potential to become hot, and the record is determined to be expired and is directly deleted; if the difference is positive, the access time field of the record is updated to the relative time difference, ensuring that the timestamp is always within the valid value range and avoiding errors in hotness judgment due to value overflow in the future.

[0068] This mechanism not only actively recovers the memory space occupied by invalid records in the warm data table, preventing storage resources from being wasted by data that has not been accessed for a long time, but also maintains the effectiveness of the access time field by updating the relative time difference, ensuring the accuracy of the hotness judgment of the warm data. For example, if the original access time of a warm data is 5 days before the new origin (difference -5), it is determined to be expired and deleted; if the original access time is 10 days after the new origin (difference 10), the access time field of the record is updated to 10, and the relative activity of the data can still be accurately reflected in subsequent judgments, providing a reliable basis for the hotness comparison between warm data and hot data.

[0069] S204, if the storage group is full, the existing compressed record with the earliest last access time in the storage group is identified as the to-be-evicted item according to the least recently used strategy in the group.

[0070] wherein the storage group is full means that the number of compressed records in the storage group reaches the preset upper limit of 32; the least recently used strategy in the group means that the compressed records in the storage group are sorted according to the access time (timestamp), and the record with the earliest access time is preferentially evicted; the existing compressed record with the earliest last access time means the compressed record with the smallest timestamp value in the storage group; and the to-be-evicted item means the compressed record identified as needing to be replaced, and the warm data corresponding to the to-be-evicted item will be downgraded to cold data.

[0071] Specifically, the system checks the current number of compressed records in the target storage group before writing a new 32-bit compressed record. If the number is less than 32, the new record is directly written; if the number is 32, the storage group is full. At this time, the system starts the least recently used strategy in the group, traverses all 32 compressed records in the group, extracts the 16-bit timestamp of each record, and compares the timestamp values. The compressed record with the smallest timestamp value (i.e., the earliest last access time) is identified as the item to be evicted, which creates space for the new record. Through reasonable hash algorithm and group capacity design, the evict error rate caused by capacity limitation can be controlled at a pre-set low level, thereby ensuring the overall accuracy of the evict decision.

[0072] S205, using the new compressed record corresponding to the data block to be accessed to replace the item to be evicted, and controlling the evict error rate caused by the capacity limitation of the storage group to be below a pre-set error rate threshold.

[0073] The new compressed record refers to a 32-bit compressed record generated based on the tag value and the current access time of the data block to be accessed; the replacement refers to the operation of deleting the item to be evicted in the storage group and writing the new compressed record; the evict error rate refers to the probability that the item to be evicted is evicted due to the capacity limitation of the storage group and the corresponding warm data is subsequently accessed, i.e., the proportion of erroneous evictions; the pre-set error rate threshold refers to the acceptable upper limit of the evict error rate set by the system, which corresponds to 0.08% in the embodiment of the system.

[0074] Specifically, the system deletes the item to be evicted from the storage group according to the position and related information of the item to be evicted, and releases the corresponding storage space. Then, the newly generated 32-bit compressed record is written to the original position of the item to be evicted, completing the replacement operation. During the entire replacement process, due to the capacity design of 32 records and LRU management of the storage group, combined with the uniform distribution characteristics of the hash value, the evict error rate caused by the capacity limitation can be controlled below the pre-set threshold (about 0.08%), ensuring that the evicted is the truly low-heat warm data, reducing the impact on subsequent data access. At the same time, the warm data corresponding to the evicted item to be evicted will be downgraded to cold data, and the heat record will not be retained.

[0075] In some embodiments, the system can also add an access count field in the compressed record to assist in determining whether to promote the target temperature data to hot data, so as to more accurately identify those "true hot" data that are continuously and frequently accessed in a short period of time, effectively filter out the interference caused by "spike" access, and improve the stability and efficiency of the cache. Of course, if the access count field is directly spliced to the tail of the 32-bit compressed record, it may sacrifice part of the memory. The specific selection can be made according to the actual needs, and it should not limit other embodiments of the present application. The process of the system adding an access count field in the compressed record is specifically shown in the following steps S206 to S209: S206, allocating a preset number of bits as an access count field in the 32-bit compressed record of the temperature data table corresponding to the temperature data.

[0076] Among them, the preset number of bits refers to the number of binary bits that the system pre-allocates from the 32-bit compressed record for storing the access count, which is usually 4-8 bits (ensure that the remaining number of bits can still meet the storage requirements of the tag value and the timestamp); the access count field refers to the newly added field in the 32-bit compressed record for counting the number of times the temperature data block is accessed, which is used to assist in determining whether the data hotness meets the frequency requirement for promoting hot data.

[0077] Specifically, the system determines the existing field allocation of the 32-bit compressed record (16-bit tag value + 16-bit timestamp), determines the preset number of bits that can be allocated (for example, 4 bits are allocated from the 16-bit timestamp, or space is released after compressing the tag value bits) without affecting the uniqueness of the tag value and the effectiveness of the timestamp. Then adjust the field structure of the compressed record, add the access count field and specify its storage rule (such as the initial value is 0, and is incremented by 1 each time it is accessed, and is reset after reaching the threshold). Finally, update the storage logic and parsing rules of the temperature data table synchronously to ensure that the newly generated compressed record can be correctly identified and processed, and at the same time, batch upgrade the historical compressed record to supplement the initial value of the access count field, and realize the compatibility of new and old record formats.

[0078] S207, the access count field is less than a preset promotion count value.

[0079] Among them, the promotion count value refers to the access frequency threshold that the system pre-sets to determine whether the temperature data has the qualification to promote to hot data, which is usually set according to the business access characteristics (such as 5 times, 10 times); the access count field is less than the promotion count value, which means that the cumulative access number of the temperature data has not yet reached the condition to trigger the hotness comparison (compared with the coldest hot data), and the access number needs to be accumulated.

[0080] Specifically, after receiving a read request of the warm data, the system first extracts the current value of the access count field from the corresponding 32-bit compressed record, and then calls the system configuration module to obtain the preset promotion count value. The two values are compared. If the access count value is less than the promotion count value, it means that the access frequency of the warm data has not reached the standard of "hot" data. Therefore, the hotness comparison process with hot data does not need to be started, and only the access count value and the access timestamp need to be updated, and the warm data needs to be continued to be retained in the warm data level for observation. This process can effectively reduce unnecessary level adjustment overhead and improve system running efficiency.

[0081] S208, updating the count value and the access time corresponding to the target warm data.

[0082] Specifically, the system locks the 32-bit compressed record corresponding to the target warm data to prevent data inconsistency caused by concurrent access. Then the current access count value is read and increased by 1 (if the maximum value of the count field is reached, it is reset to the maximum value to avoid overflow). The count value update is completed. Then the current system time is obtained, and the timestamp is converted into the corresponding numerical value according to the timestamp format requirement (such as minute level), and the original timestamp in the compressed record is replaced, and the access time update is completed. Finally, the compressed record is unlocked, and the sorting position of the record in the warm data table (such as moving to the head in the LRU chain table in the group) is updated synchronously to ensure that the group sorting always reflects the latest access situation of the data. The whole process is completed in memory, without accessing solid state disk or mechanical hard disk, and the operation delay is extremely low.

[0083] S209, extracting the last access time of the warm data from the compressed record of the warm data table, and obtaining the last access time of the hot data from the tail of the bidirectional linked list of the hot data table.

[0084] Among them, the last access time of the warm data refers to the actual access time corresponding to the timestamp field extracted from the 32-bit compressed record of the target warm data, which is used to represent the time when the warm data was last accessed; the bidirectional linked list of the hot data table refers to a bidirectional linked list structure for managing hot data, which is sorted by access time, the head is the hottest data, and the tail is the coldest data; the last access time of the hot data refers to the last access time of the hot data block corresponding to the tail node of the bidirectional linked list of the hot data table, which is used to represent the activity level of the current hottest data.

[0085] Specifically, the system locates the 32-bit compressed record of the target warm data in the warm data table, extracts the timestamp field through a preset field parsing rule (such as a field structure of “16-bit tag value + 4-bit access count field + 12-bit timestamp”), and converts the timestamp into an actual last access time (such as May 20, 2024, 14:25) according to the encoding rule of the timestamp (such as minute level, origin time). At the same time, the system directly locates the tail node of the doubly linked list maintained by the hot data record module (the node corresponds to the hottest hot data), and reads the last access time (such as May 20, 2024, 14:10) from the complete metadata stored in the node. The extraction of the two times is completed in memory, without the need to access external storage media, ensuring low latency of the comparison operation.

[0086] S210, the last access time of the warm data is greater than the last access time of the hot data.

[0087] Specifically, the system converts the extracted last access time of the warm data and the last access time of the hot data into a unified time format (such as Unix timestamp, YYYY-MM-DDHH:MM:SS), and then performs numerical comparison. If the time value of the warm data is greater than the time value of the hot data, it means that the warm data was accessed more recently and is more active, and has the qualification to replace the hottest hot data. At this time, the system records the judgment result as “yes” and triggers the subsequent warm data promotion and hot data elimination process; if the time value of the warm data is less than or equal to the time value of the hot data, the judgment result is “no”, and only the sorting position of the warm data needs to be updated, without performing the promotion operation. The judgment process is completely completed in memory, with extremely short time consumption, and will not affect the response speed of the foreground read request.

[0088] S211, determine that the first historical access time is less than or equal to the second historical access time.

[0089] Specifically, after extracting the first historical access time of the target warm data and the second historical access time of the coldest hot data, the system first converts them into a unified time format (such as minute-level timestamp) to ensure consistent comparison accuracy. Then, the time values are compared, and if the value corresponding to the first historical access time is less than or equal to the value corresponding to the second historical access time, it means that the target warm data was accessed more recently and is less active than the coldest hot data. At this time, the system determines that the target warm data does not meet the promotion condition, and does not need to start the hot data replacement process, but only needs to update the sorting position of the target warm data in the warm data table according to the least recently used strategy, to ensure that the relatively more active data is always retained in the warm data table, while avoiding the occupation of hot cache resources by low-value data due to false promotion.

[0090] S212, allocate a new cache space in the solid state disk, read the data content of the target temperature data from the mechanical hard disk, and write the data content into the cache space.

[0091] Specifically, the system calls the solid state disk management module, applies for a continuous cache space according to the data block size (256KiB) of the target temperature data, ensures that the space is not occupied by other data and has stable read-write performance. Then through the mechanical hard disk IO interface, according to the mechanical hard disk physical address recorded in the target temperature data metadata, the corresponding data content is read, and data integrity check (such as CRC check) is performed during the reading process to prevent data damage. After the check passes, the system writes the data content into the new cache space allocated in the solid state disk, and checks the data consistency again after the writing is completed to ensure that the written content is completely consistent with the original data in the mechanical hard disk. At the same time, the system records the physical address of the solid state disk cache space.

[0092] S213, build complete metadata information containing logical identification, solid state disk address and mechanical hard disk address, and insert the complete metadata information into the head of the hot data table to mark the target temperature data as the current hottest state.

[0093] Specifically, the system collects the core positioning information of the target temperature data: logical identification (64-bit metadata ID), solid state disk address (physical block number of allocated cache space), mechanical hard disk address (original storage address), and supplements the current access time (second level precision) as the attribute field of the metadata, to build complete metadata information. Then access the double-linked list maintained by the hot data record module, lock the list to prevent concurrent modification, encapsulate the built complete metadata information as a new linked list node, and insert it into the head position of the double-linked list. Since the hot data table adopts the least recently used strategy for management, the head node represents the current hottest data, and inserting into the head completes the operation of "marking as the current hottest state". Finally, update the index of the hot data table (such as the mapping relationship between the metadata ID and the linked list node), to ensure that subsequent access can quickly locate the metadata node through logical identification, and release the linked list lock to allow other hot data operations to proceed normally.

[0094] S214, delete the replaced item at the tail of the hot data table, release the solid state disk space and the complete metadata in the memory occupied by the replaced item, and update the level state of the replaced item to cold data.

[0095] Specifically, the system locates the tail node of the hot data bidirectional linked list, and the hot data corresponding to the node is the replaced item. Then the node is locked to prevent concurrent access, the complete metadata stored in the node is read, and the cache space address of the replaced item in the solid state disk and the metadata index information in the memory are obtained. Then the resource release operation is performed: first, send an instruction to the solid state disk management module to mark the cache space corresponding to the replaced item as idle, and release the storage resources of the area; second, delete the complete metadata structure of the replaced item in the memory and the mapping relationship in the hot data index table, and recycle the memory resources. Finally, the level state of the replaced item is updated, the data is removed from the hot data management system, and the level of the data is marked as cold data, and the hot data related hotness recording and sorting management is no longer performed on the data, and the level mapping table of the system is updated to ensure that subsequent access can accurately identify the cold data property of the data.

[0096] The layered storage system of the embodiment of the application is applied to an electronic device, Figure 5 An architecture schematic diagram of an electronic device suitable for implementing the embodiment of the application is shown.

[0097] It should be noted that, Figure 5 The electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiment of the application.

[0098] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions (computer programs) or controlled by instructions (computer programs) related hardware, and the instructions can be stored in a computer readable storage medium and loaded and executed by a processor. The electronic device of the embodiment includes a storage medium and a processor, wherein the storage medium stores a plurality of instructions, and the instructions can be loaded by the processor to execute any step of the method provided by the embodiment of the application.

[0099] Specifically, the storage medium and the processor are directly or indirectly electrically connected to realize the transmission or interaction of data. For example, the elements can be electrically connected to each other through one or more signal lines. The storage medium stores computer execution instructions for realizing the data access control method, including at least one software function module stored in the storage medium in the form of software or firmware. The processor executes various function applications and data processing by running the software program and the module stored in the storage medium. The storage medium can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. Among them, the storage medium is used to store programs, and the processor executes the programs after receiving execution instructions.

[0100] Further, the software program and the module in the storage medium can also include an operating system, which can include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and can communicate with various hardware or software components to provide a running environment for other software components. The processor can be an integrated circuit chip with signal processing capability. The processor described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., which can realize or execute the methods, steps and logic flow diagrams disclosed in the embodiments. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0101] Due to the instructions stored in the storage medium, the steps in any method provided by the embodiments of the present application can be executed, and thus the beneficial effects of any method provided by the embodiments of the present application can be achieved. Details are described in the foregoing embodiments, which will not be repeated here.

[0102] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, any changes or replacements within the technical scope disclosed by the present application, which can be easily thought by those skilled in the art, should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A layered storage read cache management method applied to a layered storage system comprising a memory, a solid state disk and a mechanical hard disk, characterized in that, The method comprises: all data blocks are divided into three data levels of hot data, warm data and cold data according to access heat for storage, the hot data table corresponding to the hot data and the warm data table corresponding to the warm data have different recording accuracies of heat information, the access heat is determined according to the last access time of data; in response to a read request for the cold data, target cold data to be accessed is promoted to warm data, and the warm data is sorted and eliminated at the end according to the least recently used strategy, the data content of the cold data and the data content of the warm data are stored in a mechanical hard disk, and complete metadata information of the warm data is stored in a solid state disk; in response to a read request for the hot data, complete metadata information of target hot data to be accessed is cached in the memory, and the hot data is re-sorted and updated according to the least recently used strategy, the target hot data is stored in a cache area of a solid state disk and a mechanical hard disk; in response to a read request for the warm data, it is judged whether the first historical access time of target warm data to be accessed is greater than the second historical access time of target hot data with the lowest access heat, the warm data table corresponding to the warm data stores heat information in the memory using an information compression multi-path group associative way, and complete metadata information of the warm data is stored in a solid state disk; if yes, the target warm data is used to replace the target hot data; if no, the warm data is re-sorted and updated according to the least recently used strategy.

2. The method of claim 1, wherein, The step of storing the heat information in the memory using the information compression multi-path group associative way by the warm data table corresponding to the warm data comprises: calculating the hash value of the 64-bit logical identifier of each data block, the hash value is used to realize the uniform distribution of the data block among groups; determining the target group number according to the hash value, the target group number is used to indicate the storage grouping of each data block; extracting the low 16 bits of the hash value as a tag value, the tag value is used to uniquely identify the data block within the storage grouping, so that the conflict probability of multiple different logical identifiers of the data block with the same tag value within the same storage grouping is lower than a preset conflict threshold; record the current access time as a 16-bit timestamp, the timestamp records the latest access time in minutes; combine the tag value and the timestamp to form a 32-bit compressed record; store the compressed record in the storage grouping corresponding to the target group number, the capacity of each storage grouping is 32 records, and the least recently used strategy is used for sorting and management within the group.

3. The method of claim 2, wherein, The step of judging whether the first historical access time of target warm data to be accessed is greater than the second historical access time of target hot data with the lowest access heat comprises: extracting the last access time of the warm data from the compressed record of the warm data table; obtaining the last access time of the hot data from the tail of the bidirectional linked list of the hot data table; if the last access time of the warm data is greater than the last access time of the hot data, it is determined that the first historical access time of the target warm data to be accessed is greater than the second historical access time of the target hot data with the lowest access heat. If the last access time of the warm data is less than or equal to the last access time of the hot data, it is determined that the first historical access time of the accessed target warm data is less than or equal to the second historical access time of the target hot data with the lowest access frequency.

4. The method of claim 1, wherein, The step of replacing the target hot data with the target warm data specifically comprises: reading data content of the target warm data from the mechanical hard disk, the data content including actual stored user data; allocating a new cache space in the solid-state disk, the cache space being used to store the target warm data to be promoted; writing the data content into the cache space to complete physical storage of the target warm data in the high-speed layer; constructing complete metadata information including a logical identifier, a solid-state disk address and a mechanical hard disk address, the complete metadata information providing accurate data positioning; inserting the complete metadata information into a head of a hot data table and marking the target warm data as being in a current hottest state; deleting a replaced item at a tail of the hot data table and releasing a solid-state disk space and complete metadata in memory occupied by the replaced item, and updating a level state of the replaced item to cold data.

5. The method of claim 2, further comprising: in a periodic time reference adjustment process, setting a new time origin as a current time minus a preset number of days, the time origin being used to determine data validity; traversing all existing compressed records in the group to extract an access time field of each record; calculating a relative time difference between each record and the time origin; deleting expired records with a negative relative time difference and updating a time field of remaining records to the relative time difference.

6. The method of claim 2, wherein, The step of storing the compressed record into the storage group corresponding to the target group number specifically comprises: if the storage group is full, identifying an existing compressed record with the earliest last access time in the storage group as a to-be-evicted item according to a least recently used strategy in the group; replacing the to-be-evicted item with a new compressed record corresponding to the to-be-accessed data block, and controlling an evicted error rate caused by a capacity limit of the storage group to be below a preset error rate threshold.

7. The method of claim 1, wherein, The step of responding to the read request of the warm data specifically comprises: allocating a preset number of bits as an access count field in a 32-bit compressed record of a warm data table corresponding to the warm data; when the target warm data is accessed, if an access count field corresponding to the target warm data has not reached a preset promotion count value, only updating a count value and an access time corresponding to the target warm data; when the access count field of the target warm data has reached the promotion count value, performing the step of determining whether the first historical access time of the accessed target warm data is greater than the second historical access time of the target hot data with the lowest access frequency.

8. A tiered storage system, characterized by, The system comprises one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is configured to store computer program code including computer instructions, and the one or more processors invoke the computer instructions to cause the system to perform the method of any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are run on a hierarchical storage system, they cause the system to perform the method of any of claims 1-7.

10. A computer program product, characterised in that, When the computer program product is run on a hierarchical storage system, it causes the system to perform the method of any of claims 1-7.

Images