A hierarchical storage management method for mass data

Abstract

This invention relates to the field of data storage and management technology, specifically disclosing a hierarchical storage management method for massive data. This application constructs time-anchor groupings based on data generation timestamps, maintains a dedicated metadata linked list and state mapping table for each group, triggers atomic updates of the corresponding group's heat status during access, periodically compares the group's heat status with the actual storage level, identifies groups to be cooled and preheated, and performs batch cooling migration and proactive preheating migration for the entire group as units of time-anchor group. This invention solves the technical problems of existing technologies where the overhead of per-block heat statistics increases linearly with the amount of data, heat determination is lagging, and it cannot adapt to periodic batch access. It decouples the hierarchical storage decision overhead from the total amount of data blocks, completes predictive batch preheating from point to surface, reduces storage system IO and computing power consumption, and significantly improves the performance of batch access to massive data and the efficiency of hierarchical storage management.

Description

Technical Field

[0001] This invention relates to the field of data storage and management technology, and in particular to a hierarchical storage management method for massive amounts of data. Background Technology

[0002] In the hierarchical storage management scenario of distributed storage systems, the common technical solution in the industry is usually that the system maintains access time records independently for each data block. By periodically scanning the access time records of all data blocks, the recorded values ​​are compared with preset thresholds to determine the hot or cold status of the data blocks, and finally a storage layer migration task for a single data block is generated.

[0003] In real-world applications involving hundreds of billions of data blocks, such as daily financial logs and IoT time-series data, the data exhibits significant temporal locality. This means that data blocks generated at the same time exhibit highly consistent access patterns, and there are periodic batch access requirements for compliance audits and other purposes. In this scenario, existing technologies face several technical challenges. Firstly, the scanning overhead for popularity determination is linearly tied to the total number of data blocks. Full scanning of hundreds of billions of access records is extremely time-consuming. Extending the scanning interval to reduce overhead would directly lead to severe delays in popularity determination. Furthermore, the system struggles to leverage the temporal correlation of data to predict batch access, only able to perform cold-layer recall on already accessed individual data blocks. When periodic batch access occurs, a large amount of unaccessed data remains in the cold layer, resulting in extremely high initial latency and a significant decrease in overall execution efficiency.

[0004] Therefore, there is an urgent need for a hierarchical storage management method for massive amounts of data to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide a hierarchical storage management method for massive amounts of data, comprising the following steps: Obtain the generation timestamp of the data block, truncate the generation timestamp according to the preset time anchor granularity to obtain the time anchor group identifier, write the entity of the data block into the hot storage layer, and append the global identifier of the data block to the group metadata chain list corresponding to the group identifier. In the group status mapping table, initialize the current actual storage level of the group to the hot layer. When a read access request for any data block is received, the group identifier to which the data block belongs is parsed, the last access time of the group is updated to the current system time, and the actual storage location is located according to the global identifier of the data block to read the data. The current actual storage level of each group in the group status mapping table is periodically traversed. The last access time of each group is compared with a preset threshold to identify groups whose actual storage level is inconsistent with their logical heat status. A list of groups to be migrated, including groups to be cooled down and groups to be warmed up, is generated. For a group to be cooled, obtain the corresponding group metadata linked list, traverse the data block identifiers recorded on the linked list, migrate the corresponding data entities from the hot storage layer to the cold storage layer in batches, and update the current actual storage level of the group in the group status mapping table to the cold layer after completion. For a group to be preheated, obtain the corresponding group metadata linked list, migrate the data entities corresponding to the data block identifiers recorded on the linked list from the cold storage layer back to the hot storage layer in batches, and update the current actual storage level of the group in the group status mapping table to the hot layer.

[0006] Furthermore, the present invention also discloses a hierarchical storage management system for massive data, comprising: The acquisition module is used to acquire the generation timestamp of the data block, truncate the generation timestamp according to the preset time anchor granularity to obtain the time anchor group identifier, write the entity of the data block into the hot storage layer, append the global identifier of the data block to the group metadata chain list corresponding to the group identifier, and initialize the current actual storage level of the group to the hot layer in the group status mapping table. The parsing module is used to parse the group identifier to which the data block belongs when a read access request for any data block is received, update the last access time of the group to the current system time, and locate the actual storage location to read the data based on the global identifier of the data block. The generation module is used to periodically traverse the current actual storage level of each group in the group status mapping table, compare the last access time of each group with a preset threshold, identify groups whose actual storage level is inconsistent with the logical heat status, and generate a list of groups to be migrated that includes groups to be cooled down and groups to be warmed up. The first processing module is used to obtain the corresponding group metadata linked list for the group to be cooled, traverse the data block identifiers recorded on the linked list, migrate the corresponding data entities from the hot storage layer to the cold storage layer in batches, and update the current actual storage level of the group in the group status mapping table to the cold layer after completion. The second processing module is used to obtain the corresponding group metadata linked list for the group to be preheated, migrate the data entities corresponding to the data block identifiers recorded on the linked list from the cold storage layer back to the hot storage layer in batches, and update the current actual storage level of the group in the group status mapping table to the hot layer after completion.

[0007] Furthermore, the first processing module includes: The acquisition unit is used to acquire the identifier of the group to be cooled, locate and load the group metadata linked list corresponding to the group identifier from the metadata service; The extraction unit is used to traverse the nodes on the grouped metadata linked list, extract the global identifiers of data blocks stored in each node, and aggregate them into a full list of data block identifiers. The writing unit is used to read the corresponding data entities in batches from the hot storage layer according to the full data block identifier list, and write the read data entities sequentially into the cold storage layer. The update unit is used to batch update the storage location field in the metadata of the corresponding data block after confirming that the data entity has been successfully written to the cold storage layer, and update the current actual storage level of the group in the group status mapping table to the cold layer.

[0008] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described hierarchical storage management method for massive data.

[0009] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described hierarchical storage management method for massive data.

[0010] The beneficial effects of this application are as follows: Firstly, this invention reconstructs the smallest unit of hierarchical storage management from a single data block to a time-anchored group. The heat determination and migration decision only require scanning the group-level records, and the number of scans only increases with the time span. This can decouple the decision overhead from the total amount of data blocks, keep the decision overhead constant in the case of massive data, and avoid the problem of lagging heat determination.

[0011] Secondly, this invention implements an active preheating mechanism based on grouping, which can trigger the batch migration of the entire group of data back to the hot layer when any data block in the group is accessed. It can automatically adapt to periodic batch access without business configuration, eliminate the initial high latency of batch access, and greatly improve the performance of batch reading of historical data.

[0012] Third, the present invention adopts a batch migration execution mode to transform discrete random IO into continuous sequential IO, reducing the IO overhead of the storage system and the pressure on metadata services, while avoiding fragmented storage of data in the same group in cold and hot layers, and simplifying the management of the entire data lifecycle. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of a method flow proposed in an embodiment of this application.

[0014] Figure 2 This is a schematic diagram of the system structure proposed in an embodiment of the present invention.

[0015] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0016] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0017] like Figure 1 As shown, this application provides a hierarchical storage management method for massive amounts of data, including the following steps: S1: Obtain the generation timestamp of the data block, truncate the generation timestamp according to the preset time anchor granularity to obtain the time anchor group identifier, write the entity of the data block into the highest performance hot storage layer, and append the global identifier of the data block to the group metadata linked list corresponding to the group identifier. S2: When a read access request for any data block is received, the group identifier to which the data block belongs is parsed, the "last access time of the group" of the group is updated to the current system time, and the actual storage location is located according to the global identifier of the data block to read the data. S3: Periodically compare the "last access time of the group" of each group with the preset threshold, as well as the current actual storage level of each group recorded in the "group status mapping table", to identify the groups to be cooled down and the groups to be warmed up that are inconsistent with the actual storage level and logical heat status. S4: For the group to be cooled, obtain the corresponding group metadata linked list, traverse all data block identifiers recorded on the linked list, migrate the corresponding data entities from the hot storage layer to the cold storage layer in batches, and update the current actual storage level of the group in the "group status mapping table" to the cold layer after completion. S5: For the group to be preheated, obtain the corresponding group metadata linked list, migrate the data entities corresponding to all data block identifiers recorded on the linked list from the cold storage layer back to the hot storage layer in batches, and update the current actual storage level of the group in the "group status mapping table" to the hot layer.

[0018] As described in steps S1-S5 above, in distributed storage systems that handle massive data scenarios such as financial transaction log archiving, industrial IoT time-series data acquisition, and video surveillance data retention, the total number of data blocks within the system can reach hundreds of billions or even trillions. Furthermore, the data exhibits significant temporal locality, meaning that data blocks generated at the same time tend to have highly similar future access patterns. Routine business access is concentrated on recently generated "hot" data, while low-frequency compliance audits and historical backtracking operations initiate batch accesses to the entire dataset within a specific time frame. Existing technologies use individual data blocks as the basic unit for hierarchical storage management, failing to leverage the temporal correlation of data for aggregation management. This leads to several technical problems in massive data scenarios. Full-scale hotness determination requires traversing and scanning the access records of all individual data blocks. Scanning time and computational overhead increase linearly with the total number of data blocks. Increasing the scanning interval to reduce overhead results in severe lag in hotness determination. Periodic batch accesses can only be responded to passively, performing only cold-layer recall on already accessed individual data blocks, unable to predict access demands for other data in the same group in advance. This results in extremely high initial latency for batch access tasks and a significant decrease in overall execution efficiency.

[0019] In existing technologies, the standard process of hierarchical storage management has technical defects throughout the entire chain. In the data writing stage, only the storage of a single data block and independent metadata records are completed, without time-based aggregation and grouping. Metadata is completely scattered at the single data block level. In the read access processing stage, only the independent access records of the accessed single data block are updated, which cannot map a single access behavior to the overall heat status change of batch data. In the heat determination and migration decision stage, all access records of single data blocks must be fully scanned. The decision overhead is strongly tied to the total number of data blocks, making it impossible to achieve low-overhead periodic determination. In the migration execution stage, the single data block is used as the smallest execution unit, which cannot achieve batch operation, resulting in low IO efficiency and huge pressure on metadata service writing. In the cold data access stage, only the cold layer recall of single data blocks can be passively executed, without an active preheating mechanism, which cannot adapt to the business needs of periodic batch access.

[0020] This invention reconstructs the smallest decision-making and execution unit of a distributed hierarchical storage system, establishing a closed-loop hierarchical storage management process centered on time-anchored grouping. It achieves fully automated management of the entire process, from time-dimensional grouping aggregation during data writing, group-level heat status updates triggered by read access, periodic lightweight group status comparison and migration decisions, and batch cooling migration and proactive warm-up migration of entire groups. This completely decouples the computational and I / O overhead of hierarchical storage management from the total number of data blocks in the system. Simultaneously, it enables proactive and predictive data warm-up for periodic batch accesses with time correlations. This solves the core technical problems of existing technologies in scenarios with hundreds of billions of data blocks, such as excessive computational overhead for full-scale scanning of hierarchical storage management, delayed heat determination, and severely insufficient batch access response performance.

[0021] This invention is based on the temporal locality of data access, constructing a fully closed-loop hierarchical storage management mechanism that triggers single-point access, links group statuses, and enables batch migration of entire groups. During the data writing phase, the system aggregates all data blocks whose generation times fall within the same preset time interval into a single time-anchor group. Each group maintains a dedicated metadata linked list, access status record, and storage hierarchy record, integrating the previously scattered metadata of individual data blocks into an aggregated metadata system based on groups. This elevates the management granularity of hierarchical storage from single data blocks to time-anchor groups. During the read access processing phase, a read request from the front-end business to any data block within a group triggers a real-time update of the overall access status of that group, transforming a single-point access behavior into a predictive signal for the access needs of the entire group of data. During the migration decision-making phase, the system only needs to periodically scan the access status records and storage tier records at the group level. Through standardized threshold comparison and status consistency verification, it can complete the cold / hot status determination of all groups, identifying target groups to be cooled or warmed up. The entire decision-making process does not require scanning the access records of any single data block, and the decision-making overhead is only related to the time span and is completely independent of the total number of data blocks. During the migration execution phase, the system uses groups as units, and collects the index information of all data blocks in the same group at once through the group metadata linked list, performing batch cooling or warming migration of the entire group. After the migration is completed, the storage tier status of the group is updated synchronously, forming a complete management closed loop. When any data block in a group that has been archived to the cold layer is accessed, the access status of that group will be updated in real time. In subsequent decision cycles, the system will identify inconsistencies in the status of the group and automatically trigger the warming migration of the entire group of data, ensuring that subsequent batch accesses can directly read data from the high-performance hot layer, achieving proactive and predictive data readiness for periodic batch accesses.

[0022] In one embodiment, step S1 specifically includes: S11: Extract the generation timestamp from the data block metadata, and read the system's preset time anchor granularity parameter. The time anchor granularity parameter is a fixed time length set according to the temporal locality characteristics of data access, and its value is accurate to the hour or day. S12: Convert the generated timestamp into a numerical representation, then divide the value of the generated timestamp by the value corresponding to the time anchor granularity parameter, round down the quotient, and then multiply the rounded quotient by the value corresponding to the time anchor granularity parameter. The resulting product is the time anchor group identifier. S13: In the metadata service, query whether there is a group metadata linked list indexed by the group identifier. If not, create an empty linked list containing only the head node, and create a record for the group identifier in the "Last access time of the group" table, and set the initial value of the record to null. S14: Generate a globally unique identifier for the data block, append the identifier as a new node to the tail of the group metadata linked list, and create a record in the "group status mapping table" with the group identifier as the primary key, and initialize the value of its current actual storage level field to the hot layer.

[0023] As described in steps S11-S14 above, by standardizing the truncation processing of data block generation timestamps, initializing and maintaining the group metadata linked list, and synchronously creating group-related metadata records, a metadata system centered on time-anchored groups is constructed. The decision-making unit for hierarchical storage management is reconstructed from a single data block to a time-anchored group, providing a unified metadata foundation for subsequent heat statistics, migration decisions, and batch data operations based on groups. This fundamentally decouples the hierarchical storage decision-making overhead from the total number of data blocks.

[0024] In distributed hierarchical storage scenarios dealing with massive data blocks, such as daily log archiving in financial transaction systems and time-series data storage for the Internet of Things, the total number of data blocks can reach tens of billions. Data blocks generated at the same time have highly similar access patterns. Existing technologies use individual data blocks as decision-making units, requiring each data block to maintain access records and metadata independently. This leads to a linear increase in the computational overhead of heat statistics with the total number of data blocks. At the same time, it is impossible to use the temporal locality of data access to achieve predictive management of batch data. Therefore, it is necessary to establish a standardized grouped metadata construction mechanism with time anchors as the core, which aggregates data blocks with the same time attributes and access patterns into a unified management unit. This solves the technical problems of excessive computing power overhead and inability to adapt to periodic batch access caused by overly fine-grained decision-making units in existing technologies.

[0025] Step S11 first extracts the generation timestamp from the data block metadata and reads the system's preset time anchor granularity parameter. The time anchor granularity parameter is a fixed time length set according to the temporal locality characteristics of data access, and its value is accurate to the hour or day. The generation timestamp is a metadata field that is generated synchronously when the data block is created and records the time when the data block is generated. It is the core carrier of the time attribute of the data block. The time anchor granularity parameter is a parameter pre-configured by the system according to the access rules of the business scenario. For example, in the financial daily log cutting scenario, business access is executed in batches on a daily basis, and the time anchor granularity parameter can be set to 1 day. In the IoT high-frequency data acquisition scenario, business access is executed on an hourly basis, and the time anchor granularity parameter can be set to 1 hour. This can clearly define the time dimension standard for grouping and aggregation, ensure that all data blocks are grouped according to a unified time granularity, and ensure that data blocks in the same group have consistent time attributes and highly similar access rules.

[0026] Step S12 converts the generated timestamp into a numerical representation, then divides the generated timestamp value by the value corresponding to the time anchor granularity parameter, rounds the quotient down, and then multiplies the rounded quotient by the value corresponding to the time anchor granularity parameter. The resulting product is the time anchor group identifier. The calculation formula for the time anchor group identifier is as follows: ; Among them, the This represents the time anchor group identifier. The final generated group timestamp value serves as a unified group identifier for all data blocks within the same time granularity. This represents the original timestamp value of the data block, extracted from the data block metadata, and converted from a time format to a Unix timestamp value in seconds. The system defaults to a time anchor granularity parameter, set according to the locality of business time (accurate to the hour / day), and converts it to a second-level value (e.g., 1 day = 86400 seconds). This converts the generated timestamp into a numerical representation, specifically converting the time-formatted timestamp to a Unix timestamp value in seconds. This standardizes the numerical calculation of time attributes. The time anchor granularity parameter converts a fixed time length into a value with the same unit; for example, when the time anchor granularity parameter is 1 day, the corresponding value is 86400 seconds. Through standardized numerical calculations, the system achieves uniform truncation of the generated timestamps, ensuring that all data blocks within the same time anchor granularity generate identical group identifiers. Reproducible and standardized numerical calculations enable the aggregation and classification of data block time attributes, avoiding errors in the time format conversion process and ensuring the accuracy and consistency of group identifier generation. Technically, this achieves the core goal of aggregating scattered single data blocks into a unified management unit according to the time dimension.

[0027] Step S13: Query the metadata service to see if a group metadata linked list indexed by the group identifier exists. If not, create an empty linked list containing only the head node, and create a record for the group identifier in the group's last access time table, setting the initial value of the record to null. The metadata service is the core service in the distributed storage system responsible for managing the metadata of all data blocks. The group metadata linked list is an ordered linked list maintained for each time anchor group. The linked list nodes are used to record the global identifiers and storage locations of all data blocks within the group. The group's last access time table is a database table maintained by the system, using the group identifier as the primary key, used to record... Recording the most recent read access time for each group is significant because it synchronously initializes the core metadata structure corresponding to that group when the first data block is written. This ensures that each group has a corresponding metadata linked list to record data block information within the group, and also initializes the access time records required for group popularity statistics. This provides a data carrier for subsequent popularity status updates on a group-by-group basis, avoiding statistical anomalies caused by missing metadata structures during subsequent popularity statistics. Furthermore, it ensures that the metadata of all data blocks within the same group is aggregated into the same linked list, providing a complete data block index for subsequent batch migration of the entire group.

[0028] Step S14 generates a globally unique identifier for the data block, appends this identifier as a new node to the tail of the group metadata linked list, and creates a record in the group status mapping table, using the group identifier as the primary key and initializing the value of its current actual storage level field to the hot layer. The globally unique identifier of the data block is a unique identity generated by the system for each data block within the entire distributed storage system, used to locate and index the corresponding data entity. The group status mapping table is a data structure maintained by the system to record the current actual storage level of each time-anchor group. In this step, appending the globally unique identifier of the data block to the tail of the group metadata linked list uses an append-only write method, ensuring high performance of linked list writing while ensuring that all data block identifiers within the same group are completely recorded in their corresponding linked lists. The creation and initialization of records as hot layers is because data block entities are stored in the highest-performance hot storage layer by default when written. This initialization operation ensures that the actual storage level record of the group is completely consistent with the actual storage location of the data entity, avoiding data inconsistency problems during subsequent state comparison. The above steps can complete the metadata binding of a single data block with its group, and at the same time complete the initialization of the group's storage level state. This allows for the direct retrieval of all data block information within the group through the group metadata linked list and the acquisition of the actual storage state of the group through the group state mapping table during subsequent migration decisions. This provides a complete index and state basis for the subsequent migration of the entire group, realizing a complete closed loop from single data block management to group management. Ultimately, this serves the core invention objective of reducing the overhead of hierarchical storage management under massive data and realizing proactive preheating for periodic batch access.

[0029] In one embodiment, step S2 specifically includes: S21: In response to the received read access request, parse out the global identifier of the target data block carried in the request, and query and obtain the time anchor group identifier to which the data block belongs from the metadata; S22: Obtain the current system time, and perform an update operation in the "Last Accessed Time of Group" table using the group identifier as the key, setting the value of the corresponding record to the current system time. The update operation uses an atomic replacement method to support concurrent access. S23: Read the current storage level and storage path information recorded in the metadata of the data block. If the storage level is a cold layer, concatenate the cold storage root path and the relative path to obtain the complete access address. If it is a hot layer, directly use the hot layer access address and initiate data reading to the corresponding storage engine. S24: Encapsulate the data entity content returned by the storage engine and return it to the requester, completing the full response for this read access.

[0030] As described in steps S21-S24 above, by embedding a group-level atomic update mechanism for the heat status in the full response process of the read access request, the read response of the target data block and the logical heat status update of the group to which it belongs are completed synchronously. This achieves lightweight heat status maintenance with time anchor group as the smallest statistical unit, reducing the heat statistics operation overhead corresponding to a single data block access from a linear complexity that increases with the number of data blocks to a fixed time complexity. At the same time, it provides accurate logical heat status basis and triggering conditions for subsequent migration decisions and preheating of the entire group of data, solving the core technical problems of excessive computing power overhead, delayed heat determination, and inability to predict periodic batch access caused by per-data-block heat statistics in the existing technology.

[0031] In distributed storage systems that handle hundreds of billions of data blocks, such as daily log storage in core banking systems and time-series data acquisition and storage in industrial IoT scenarios, front-end business read access to data blocks exhibits significant temporal locality. This means that accessing any one data block within a group at the same time anchor often indicates that other data blocks within that group will soon be accessed in batches. Current technologies only update access records for the individual accessed data block, requiring the system to maintain a separate access timestamp for each block. Each read access only triggers a change in the heat status of a single data block. This not only leads to the continuous accumulation of metadata update overhead due to frequent read operations as the total number of data blocks grows, but also fails to leverage the temporal correlation of data access to predict group-level access demands. Ultimately, this results in persistently high computational overhead for full-scale heat determination and severe delays in responding to periodic batch accesses in scenarios such as compliance auditing.

[0032] Step S21 responds to the received read access request by parsing the global identifier of the target data block carried in the request and querying and retrieving the time anchor group identifier to which the data block belongs from the metadata. The read access request is a standardized data read request initiated by the front-end business system to the distributed storage system. The request message always carries a globally unique identifier for the target data block. This identifier is a system-wide unique identity generated for each data block in step S14. The time anchor group identifier to which the data block belongs is the group affiliation information generated in step S12 through standardized numerical calculations and persistently stored synchronously with the data block's metadata. The technical implementation of this step is as follows: after the system receives the read access request, it first performs standardized parsing of the request message, extracts the globally unique identifier of the target data block carried in the request, uses this identifier as an index key to initiate a targeted query to the metadata service, retrieves the full metadata record corresponding to the data block, and extracts the pre-stored time anchor group identifier from the metadata record. The significance of this step lies in quickly binding the target data block to its respective group management unit at the beginning of the read access process, accurately mapping the access behavior of a single data block to the corresponding time anchor group, providing an accurate index basis for subsequent group-level heat status updates, ensuring that every business read access can trigger the heat status change of the corresponding group, directly serving the core inventive concept of this invention with the time anchor group as the smallest management unit, and avoiding the problem of access behavior being disconnected from the batch data management unit in the prior art.

[0033] Step S22 retrieves the current system time and, using the group identifier as the key, performs an update operation on the group's last accessed time table, setting the value of the corresponding record to the current system time. The update operation employs an atomic replacement method to support concurrent access. The current system time is a unified and accurate timestamp maintained by the time synchronization service within the storage system cluster. The group's last accessed time table is a structured data table created in step S13 for each time anchor group, using the group identifier as the primary key. The atomic replacement method refers to using the database's native atomic update operation to ensure that multiple concurrent read accesses to the same group do not cause update conflicts or data inconsistencies. Even if multiple read requests for data blocks within the same group occur simultaneously, only the latest system time will be retained as the group's last accessed time. The technical implementation of this step is as follows: after obtaining the unified current system time of the cluster, the system uses the time anchor group identifier obtained in step S21 as the query key to perform an atomic update operation on the group's last accessed time table, directly replacing the field value of the corresponding record with the current system time. There is no need to read the original value and modify it. After the operation is completed, it directly enters the subsequent data reading stage without waiting for additional confirmation procedures. This method updates the group-level popularity status with a fixed time complexity. The operation time does not increase with the number of data blocks in the group. Regardless of how many data blocks the group contains at a given time point, a single read access triggers only one atomic update of a single record. Compared to existing technologies that update the independent access record of a single data block with each read access, this significantly reduces the write pressure on the metadata service. At the same time, the atomic update method ensures the accuracy of the group popularity status in high-concurrency business scenarios and avoids data anomalies caused by concurrent access. This avoids the excessive computational overhead of updating access records block by block in existing technologies. Furthermore, the real-time update of the last access time of the group provides accurate logical popularity status basis for subsequent migration decisions and provides triggering conditions for the proactive preheating of the entire group of data.

[0034] Step S23 reads the current storage level and storage path information recorded in the metadata of the data block. If the storage level is a cold layer, the cold storage root path and the relative path are concatenated to obtain the complete access address. If it is a hot layer, the hot layer access address is used directly, and data reading is initiated to the corresponding storage engine. The current storage level and storage path information of the data block are metadata fields initialized in step S14 and synchronously updated after the migration operations in steps S4 and S5 are completed. The cold storage root path and hot layer access address are fixed access entries pre-configured by the system for the corresponding cold and hot storage layers. The storage engines include the all-flash storage engine for the corresponding hot storage layer and the object storage engine for the corresponding cold storage layer. The specific implementation of this step is as follows: the system reads the current storage level field and the relative storage path field from the metadata record of the target data block, and generates a corresponding complete access address based on the different storage levels. For data blocks in hot storage, the absolute hot access address recorded in the metadata is used directly. For data blocks in cold storage, the system's preset cold storage root path is concatenated with the relative path in the metadata to generate a complete object storage access address. Subsequently, the system initiates a standardized data read request to the storage engine of the corresponding storage layer, passing in the generated complete access address. The above steps are the core functional implementation of the read access request and belong to a mature standardized data access process in the storage system. While completing the group heat status update, it accurately locates the actual storage location of the target data block, ensuring the accuracy of data reading. Moreover, this step runs parallel to the group heat update operation without conflict, and will not add extra core path latency to business read access, ensuring that the heat statistics mechanism of this invention will not affect the access performance of normal business.

[0035] Step S24 encapsulates the data entity content returned by the storage engine and returns it to the requester, completing the full response for this read access. After receiving the binary data entity of the target data block returned by the storage engine, the system encapsulates the data entity according to the standardized message format agreed upon in the read access request, adds necessary fields such as response headers and checksums, and then sends the encapsulated response message to the front-end business system that initiated the request. This fully satisfies the read access needs of the business system, ensuring that the grouping popularity update mechanism of this invention is completely embedded in the existing standardized read access process, completely unaware of the front-end business system, and achieving seamless integration of popularity statistics and business access.

[0036] In one embodiment, step S3 specifically includes: S31: Obtain the hot layer dwell threshold and the cold layer preheating activation threshold from the system configuration, wherein the hot layer dwell threshold and the cold layer preheating activation threshold are both preset time length constants, and the value of the hot layer dwell threshold is greater than the value of the cold layer preheating activation threshold. S32: Traverse all group identifiers in the "Group Status Mapping Table" whose current actual storage level is the hot layer. For each group traversed, read the last access time recorded in the "Group Last Access Time" table, calculate the difference between the current system time and the last access time, and use the difference as the silent duration of the group. S33: Compare the calculated silence duration of each hot layer group with the hot layer dwell threshold. If the silence duration of one or more hot layer groups is greater than the hot layer dwell threshold, determine that the logical heat state of the one or more hot layer groups is cold, and add the group identifier of the one or more hot layer groups to the list of groups to be cooled. S34: Traverse all group identifiers in the "Group Status Mapping Table" whose current actual storage level is cold layer. For each traversed cold layer group, read the last access time recorded in the "Group Last Access Time" table, calculate the difference between the current system time and the last access time, and use the difference as the silent duration of the cold layer group. S35: Compare the calculated silent duration of each cold layer group with the cold layer preheating activation threshold. If the silent duration of one or more cold layer groups is less than or equal to the cold layer preheating activation threshold, then determine that the logical heat state of the one or more cold layer groups is hot, and add the group identifier of the one or more cold layer groups to the list of groups to be preheated. The list of groups to be cooled and the list of groups to be preheated together constitute the output of this step.

[0037] As described in steps S31-S35 above, by using a standardized comparison logic with preset dual thresholds, group traversal and silent duration calculation by storage level, and consistency verification between logical heat status and actual storage level, lightweight migration decision-making based on time-anchored groups is achieved. This reduces the scanning scope of full heat determination from hundreds of billions of single data block access records to group-level records that only increase with time span, completely decoupling the overhead of hierarchical storage decision-making from the total amount of data blocks. At the same time, it accurately identifies inactive groups that need to be cooled down and potentially active groups that need to be warmed up, providing accurate decision-making basis for subsequent batch migration of the entire group. This solves the core technical problems of excessive full-scan overhead, delayed heat determination, and inability to predict batch access needs in existing technologies.

[0038] In distributed storage systems handling long-term archiving scenarios with hundreds of billions of data blocks, such as bank transaction logs requiring retention for over 5 years or data collected by IoT devices requiring long-term archiving, the number of time-anchor groupings within the system continuously accumulates over time. Simultaneously, the boundary between hot and cold data dynamically changes with business access. Frequent daily accesses concentrate on recently generated hot data, while infrequent batch accesses such as compliance audits and historical backtracking trigger concentrated readings of long-archived cold data. Current technologies require a full scan of all single data block access records for heat determination and migration decisions. When the data volume reaches hundreds of billions, a single full scan can take hours or even tens of hours, consuming significant system computing power and causing severe delays in heat determination. This prevents timely responses to changes in the hot and cold data boundary caused by business accesses and makes it impossible to trigger preheating decisions for the entire group of data through a single access.

[0039] Step S31 retrieves the hot layer residency threshold and cold layer preheating activation threshold from the system configuration. Both the hot layer residency threshold and the cold layer preheating activation threshold are preset time length constants, and the value of the hot layer residency threshold is greater than the value of the cold layer preheating activation threshold. The hot layer residency threshold is a time length pre-configured by the system based on the available capacity of the hot storage layer and the time range of high-frequency business access. For example, in a financial transaction log scenario, 99% of daily access is concentrated in data within 3 days, so the hot layer residency threshold can be set to 3 days, corresponding to a value of 259200 seconds. The cold layer preheating activation threshold is a time length pre-configured by the system based on the business characteristics of batch access, used to determine whether there is a batch access requirement for the cold layer group; for example, it can be set to 3 days, corresponding to a value of 259200 seconds. Specifically, when the system starts a periodic migration decision task, it first reads the two parameters—the pre-configured and persistently stored hot layer residency threshold and cold layer preheating activation threshold—from the system configuration management module to initialize the judgment benchmark for this decision task. By using differentiated dual-threshold settings, the system adapts to two different business scenarios: cooling data in the hot layer and warming data in the cold layer. This provides a unified and clear quantitative benchmark for determining the logical heat status of all groups. At the same time, by using parameter constraints such as the hot layer residence threshold being greater than the cold layer warming activation threshold, the system avoids frequent invalid migrations of groups between the hot and cold storage layers, ensuring the stability of system operation. This step is the benchmark prerequisite for the entire migration decision-making process, ensuring that the determination of the hot and cold status of all groups follows a unified standard.

[0040] Step S32 iterates through all group identifiers in the group status mapping table whose current actual storage level is the hot layer. For each group iterates through, it reads the last access time recorded in the group's last access time table, calculates the difference between the current system time and the last access time, and uses this difference as the quiet duration of the group. The group status mapping table is the core data structure created in step S14 and updated synchronously in subsequent migration operations. It records the current actual storage level of each time anchor group. The group's last access time table is the data structure created in step S13 and updated in real-time during the read access process. It records the timestamp of the most recent access for each group. The current system time is a unified and accurate timestamp maintained within the storage system cluster through a time synchronization service. The system uses a group status mapping table as its data source. It filters all group identifiers currently in the hot storage layer, creating a hot layer group traversal list. Then, it sequentially traverses each group identifier in the list, using that identifier as the key to retrieve the last access time from the group's last access time table. Subtracting this last access time from the current system time calculates the group's quiet duration. If the group's last access time is initially null, its creation time is used as the calculation benchmark. By traversing hot layer groups in a targeted manner, the scanning scope is narrowed from all groups to hot layer groups, further reducing scanning overhead. Simultaneously, the standardized calculation of quiet duration accurately quantifies the inactivity level of each hot layer group, providing a quantitative basis for subsequent cooling-down decisions. This simplifies the access interval calculation, which originally required statistical analysis per data block, to a single difference calculation at the group level, directly reducing the computational overhead for hotness determination and serving the technical goal of decoupling decision-making overhead from the total number of data blocks in this invention.

[0041] Step S33 compares the calculated silent duration of each hot layer group with the hot layer residency threshold. If the silent duration of one or more hot layer groups exceeds the hot layer residency threshold, the logical heat status of the one or more hot layer groups is determined to be cold, and the group identifier of the one or more hot layer groups is added to the list of groups to be cooled. The system compares the silent duration of each hot layer group calculated in step S32 with the hot layer residency threshold obtained in step S31. When the silent duration of a group exceeds the hot layer residency threshold, it means that the group has not been accessed for a preset time, and the business access demand is extremely low. The system determines its logical heat status to be cold, and at the same time verifies that its actual storage level is hot. After confirming the inconsistency, the group identifier is added to the list of groups to be cooled. By standardizing numerical comparisons, the system automates and accurately determines the cooling needs of hot zone groups, identifying groups whose actual storage level does not match their logical heat status. This ensures that inactive data that has not been accessed for a long time is promptly migrated out of the high-performance hot zone, freeing up valuable storage resources for frequently accessed active data. Furthermore, this determination process only requires a single comparison of group-level records, without scanning any data block information within the group, resulting in extremely low computational overhead. The determination results directly form a list of groups to be cooled, providing a direct basis for subsequent batch cooling of the entire group.

[0042] Step S34 iterates through all group identifiers in the group status mapping table that are currently in the cold layer. For each cold layer group, it reads the last access time recorded in the group's last access time table, calculates the difference between the current system time and the last access time, and uses this difference as the quiet duration of the cold layer group. The system uses the group status mapping table as the data source, filters out all group identifiers that are currently in the cold layer through filtering conditions, and forms a cold layer group traversal list. Then, it traverses each group identifier in the list in sequence, uses the group identifier as the key, queries the corresponding last access time from the group's last access time table, and subtracts the last access time from the current system's cluster unified time to calculate the quiet duration of the cold layer group. By traversing cold layer groups in a targeted manner, all cold data groups in archive storage are accurately covered. Through standardized calculation of the silent duration, the latest activity level of cold layer groups is quantified, providing a quantitative basis for subsequent warm-up decisions. The core value of this step lies in capturing the state changes triggered by a single access to a cold layer group through group-level state scanning, providing a trigger signal for subsequent whole-group warm-up decisions, and realizing proactive prediction of upcoming batch accesses, thus solving the core problem that existing technologies cannot predict periodic batch accesses.

[0043] Step S35 compares the calculated silent duration of each cold layer group with the cold layer preheating activation threshold. If the silent duration of one or more cold layer groups is less than or equal to the cold layer preheating activation threshold, the logical heat state of the one or more cold layer groups is determined to be hot. The group identifier of the one or more cold layer groups is added to the list of groups to be preheated. The list of groups to be cooled down and the list of groups to be preheated together constitute the output of this step. The system compares the silence duration of each cold layer group calculated in step S34 with the cold layer preheating activation threshold obtained in step S31. When the silence duration of a group is less than or equal to the cold layer preheating activation threshold, it indicates that the group has been accessed recently. Based on the temporal locality of data access, other data blocks in the group have a very high probability of being accessed in batches. The system determines its logical hotness status as hot and verifies that its actual storage level is cold. After confirming the inconsistency, the system adds the group identifier to the list of groups to be preheated. Finally, the list of groups to be de-cooled and the list of groups to be preheated are integrated into a complete list of groups to be migrated, which is passed as the output of this step to the subsequent migration execution stage. By standardizing numerical comparisons, the system accurately determines the preheating requirements of cold data groups. Utilizing the group status changes triggered by a single data block access, it identifies cold data groups that are about to be accessed in batches, enabling proactive preheating decisions from specific points to the entire system. This completely solves the problem that existing technologies can only passively respond to batch access and cannot predict it in advance. At the same time, this determination process only requires a single comparison of group-level records, resulting in extremely low computational overhead. The final output list of groups to be preheated provides a direct basis for subsequent batch preheating of the entire group, ensuring that all group data is ready in the high-performance hot layer in advance when batch access arrives, thereby improving the processing performance of batch access scenarios.

[0044] In one embodiment, step S4 specifically includes: S41: Obtain the list of cooling groups to be cooled output in step S3, extract one or more cooling group identifiers from it, locate and load the group metadata linked list corresponding to each cooling group identifier from the metadata service into memory; S42: For each loaded group metadata linked list, sequentially traverse each node on the linked list, extract the global identifier of the data block stored in each node, aggregate all global identifiers of the data blocks belonging to the same group, and form a full list of data block identifiers corresponding to the group. S43: Using the full list of data block identifiers as input, initiate a batch read request to the hot storage layer to obtain the ordered data entity stream corresponding to the list, and simultaneously write the data entity stream into the pre-allocated continuous storage space in the cold storage layer through the network in a sequential writing manner. S44: After confirming that all data entities corresponding to the full list of data block identifiers have been successfully persisted to the cold storage layer, update the storage location field in the metadata of this batch of data blocks in batches, so that it points to the new address of the cold storage layer, and update the value of the current actual storage level field of the group in the "group status mapping table" to the cold layer.

[0045] As described in steps S41-S44 above, the entire group cooling migration is achieved by batch loading of the group metadata linked list, aggregating and collecting full data block identifiers, streaming batch read and write between hot and cold storage layers, and atomic batch updating of metadata and group status. This optimizes the original discrete migration operation per data block into a streaming batch operation of the entire group, significantly reducing the system resource consumption of the migration operation, while ensuring the consistency and integrity of data during the migration process. Ultimately, this achieves efficient and reliable migration of inactive data from the hot layer to the cold layer, releasing high-performance storage resources in the hot layer, and solving the core technical problems of low migration efficiency, high system overhead, and frequent metadata updates caused by per-data-block migration in the existing technology.

[0046] In long-term operation scenarios where distributed storage systems handle hundreds of billions of data blocks, such as a bank's core transaction system generating billions of log data entries daily or an industrial IoT platform collecting massive amounts of time-series data daily, the high-performance storage resources of the hot storage layer have limited capacity. Every day, a large amount of inactive data exceeding the hot layer's residency threshold needs to be migrated from the hot layer to a low-cost cold storage layer. Current technologies perform data cooling migration on a per-block basis. For billions of data entries generated daily, billions of single-block read, write, and metadata update operations are required. This not only results in extremely long migration times but also continuously consumes significant storage system I / O resources and metadata service computing power, severely impacting normal access to online services. Furthermore, discrete single-block migration cannot guarantee the complete and synchronous cooling of data within the same anchor group at the same time, easily leading to fragmentation where some data in the same group resides in the hot layer and some in the cold layer, adding additional overhead to subsequent data access and management.

[0047] Step S41 obtains the list of groups to be cooled output in step S3, extracts one or more group identifiers to be cooled, and locates and loads the group metadata linked list corresponding to each group identifier into memory from the metadata service. The list of groups to be cooled is generated in step S33 and contains all time-anchor group identifiers that meet the cooling conditions. The group metadata linked list is an ordered linked list created in step S13 and continuously updated during data block writing. The linked list completely records the globally unique identifiers of all data blocks within the corresponding group. The metadata service is the core service module in the distributed storage system responsible for managing all metadata. After the system migration execution module receives the list of groups to be cooled output from the upstream decision-making stage, it first parses the list, extracts all time-anchor group identifiers to be cooled, and then uses each group identifier as an index key to initiate a query request to the metadata service, locates the corresponding group's group metadata linked list, and loads the complete linked list structure into the migration execution module's memory, providing direct data support for subsequent data block identifier collection. By loading the full metadata list of the corresponding group at once using the group identifier, the operation that originally required querying metadata block by block is simplified to a single group-level linked list loading operation. This significantly reduces the number of interactions with the metadata service and lowers the access pressure on the metadata service. At the same time, the complete linked list loaded in memory provides a complete and accurate index for the subsequent batch aggregation of full data blocks in the same group, ensuring that no data block in the same group is missed during the cold migration, and directly serves the core execution logic of the whole group batch migration of this invention.

[0048] Step S42 involves sequentially traversing each node of each loaded group metadata linked list, extracting the global identifiers of data blocks stored in each node, and aggregating all global identifiers of data blocks belonging to the same group to form a full list of data block identifiers corresponding to that group. The technical implementation of this step is as follows: for each loaded group metadata linked list in memory, the system performs a full traversal starting from the head node, following the append order of the linked list. It extracts the globally unique identifier of the corresponding data block stored in each linked list node. After the traversal is complete, all extracted identifiers are aggregated in an ordered manner to form a full list of data block identifiers corresponding one-to-one with that group. The order of the list is consistent with the order of the linked list nodes. By traversing the linked list once, all data block identifiers within the same group are collected in one go, forming a complete execution list for batch migration. This ensures that all data blocks within the same anchor group at the same time are included in the scope of this cooling migration, avoiding data omissions or fragmentation of data within the same group. At the same time, the aggregated ordered identifier list provides a prerequisite for subsequent batch IO optimization of the storage system, enabling the system to perform continuous batch read operations according to the ordered list, transforming random reads into sequential reads, significantly improving read efficiency, and reducing IO pressure on the hot storage layer.

[0049] Step S43 takes the full list of data block identifiers as input and initiates a batch read request to the hot storage layer to obtain an ordered data entity stream corresponding to the list. Simultaneously, this data entity stream is written sequentially over the network to a pre-allocated contiguous storage space in the cold storage layer. The hot storage layer is the highest-performance all-flash storage layer in the system, while the cold storage layer is a low-cost archive object storage layer. The full list of data block identifiers is an ordered list of identifiers for all data blocks within a group, generated in step S42. The contiguous storage space is a storage range with contiguous logical addresses pre-allocated by the cold storage layer for this entire group migration. The system uses the full list of data block identifiers generated in step S42 as input and initiates a standardized batch read request to the storage engine of the hot storage layer. The hot storage engine reads the corresponding data block entities sequentially according to the list order, outputting an ordered, continuous data entity stream. Upon receiving the data entity stream, the system writes it directly to the pre-allocated contiguous storage space in the cold storage layer without going through the local disk cache, achieving synchronous streaming execution of reads and writes. By combining batch reading with streaming sequential writing, the original discrete random I / O operation per data block is transformed into batch sequential reading in the hot layer and streaming sequential writing in the cold layer. This fully utilizes the storage system's performance optimization capabilities for batch sequential I / O, significantly improving the execution efficiency of cross-layer data migration. At the same time, the streaming synchronous execution method avoids local data caching, reducing disk I / O overhead and memory usage, and minimizing the resource contention of online services caused by migration operations.

[0050] Step S44, after confirming that all data entities corresponding to the full list of data block identifiers have been successfully persisted to the cold storage layer, updates the storage location field in the metadata of these data blocks in batches, making it point to the new address in the cold storage layer, and updates the value of the current actual storage level field of the group in the group status mapping table to the cold layer. The confirmation of successful persistence of all data entities corresponding to the full list of data block identifiers is accomplished jointly by the write success check code returned by the cold storage layer and the data integrity check result. The storage location field in the data block metadata is the core metadata item recording the actual storage address of the data block, and the group status mapping table is the core data structure recording the current actual storage level of each group, serving as the core basis for subsequent migration decisions. After receiving the successful full data write response from the cold storage layer, the system first performs integrity verification on the written data entities to ensure that all data blocks have been written completely and accurately to the cold storage layer. After the verification passes, it sends a batch update request to the metadata service to update the storage location field in the metadata of all data blocks being migrated at once, making it point to the corresponding new address in the cold storage layer. After the batch metadata update is completed, it sends an update request to the metadata service to update the value of the current actual storage layer field of the corresponding group in the group status mapping table from the hot layer to the cold layer, thus completing the closed loop of the entire group's cooling migration process. By executing the steps of first verifying, then batch updating metadata, and finally updating group status, the consistency and integrity of data during the migration process are ensured. This avoids data access anomalies caused by data being migrated but metadata not being updated, or metadata being updated but data migration failing. At the same time, the batch metadata update method simplifies the original block-by-block metadata update operation into a single batch update, significantly reducing the write pressure on the metadata service. Finally, the group status mapping table is updated synchronously to ensure that the actual storage level record of the group is completely consistent with the actual storage location of the data. This provides an accurate status basis for subsequent periodic migration decisions and avoids problems such as repeated migration or incorrect status determination. This step completes the final closed loop of the entire group cooling migration and realizes the full-process hierarchical storage management based on time-anchored groups in this invention.

[0051] In one embodiment, step S5 specifically includes: S51: Receive the list of groups to be preheated generated in step S3, extract one or more group identifiers to be preheated from it, and verify the integrity and accessibility of the data entities corresponding to each group identifier in the cold storage layer. S52: For each group identifier to be preheated, obtain the group metadata linked list corresponding to the group identifier, traverse each node on the linked list, extract the global identifier of the data block recorded in each node, and aggregate all global identifiers of the data blocks belonging to the same group into a preheating identifier set. S53: Based on the preheating concurrency parameter preset by the system, control the number of tasks that read data entities from the cold storage layer at the same time, and based on each preheating identifier set, read data entities from the cold storage layer in batches in a manner that does not exceed the concurrency limit specified by the preheating concurrency parameter. S54: Asynchronously write the read data entity to the target area of ​​the hot storage layer. During the writing process, monitor the input and output processing pressure index of the hot storage layer in real time, and dynamically adjust the writing rate according to the fluctuation of the input and output processing pressure index to ensure that the writing operation does not have a significant impact on online services. S55: After all data entities have been successfully written to the hot storage layer, the storage location field in the metadata of this batch of data blocks is updated in batches to point to the new address of the hot storage layer. The current actual storage level field corresponding to the group in the "group status mapping table" is updated to the hot layer. A cleanup command is sent to the cold storage layer to delete the copies of data entities that have been successfully migrated to the hot storage layer in order to release the cold layer storage space.

[0052] As described in steps S51-S55 above, through pre-verification of the integrity of the groups to be preheated, aggregation of full data block identifiers, concurrent and controllable cold layer batch reading, adaptive rate limiting of hot layer write rate, atomic batch update of metadata and group status, and synchronous cleanup of cold layer data, the whole group proactive preheating is achieved with time anchor group as the unit. The passive single data block cold layer recall in the existing technology is optimized into proactive whole group batch preheating. Without affecting the normal operation of online business, the full data of the cold layer that will be accessed in batches is migrated back to the high-performance hot storage layer in advance, which completely solves the core technical problems of high initial access latency, low efficiency of data block recall, and uncontrollable system resource consumption in the batch access scenario of the existing technology.

[0053] In batch access scenarios of distributed storage systems, such as compliance auditing, historical data backtracking, and periodic report generation—for example, when bank auditors need to backtrack the full transaction logs of a specific day a year ago, or industrial enterprises need to backtrack the time-series data collected by equipment for an entire month—the target data is archived in a low-cost cold storage layer, and access is characterized by batch reading of entire groups. In existing technologies, cold data retrieval can only be passively triggered when a single data block is accessed, and each access can only retrieve the currently requested single data block. When facing batch access of entire groups, two core problems arise. First, all data in the initial access needs to be read from the cold layer in real time, resulting in extremely high access latency and a significant decrease in the overall execution efficiency of batch tasks. Second, the high frequency of single-data-block cold layer read requests puts enormous pressure on the access gateway of the cold storage layer and the system's IO resources. Simultaneously, the metadata update operation per data block continuously consumes the computing power of the metadata service, affecting the operation of normal online business, and it is impossible to pre-warm up other data in the same group that will soon be accessed.

[0054] Step S51 receives the list of groups to be preheated generated in step S3, extracts one or more group identifiers from it, and verifies the integrity and accessibility of the data entities corresponding to each group identifier in the cold storage layer. The list of groups to be preheated is the list of all time-anchor group identifiers that meet the preheating conditions, generated in step S35. The corresponding data entities in the cold storage layer are the full data block entities within the corresponding group that were written to the cold storage layer during the cooling migration in step S4. The technical implementation of this step is as follows: after the system preheating execution module receives the list of groups to be preheated from the upstream decision-making process, it first parses the list to extract all time-anchor group identifiers to be preheated. Then, for each group identifier, it initiates a pre-check request to the cold storage layer to verify the storage integrity, metadata consistency, and accessibility of the full data entities corresponding to that group, confirming that the data is undamaged, unlost, and can be read normally. The significance of this step is to perform pre-verification at the beginning of the preheating process, identify abnormalities in the cold layer data in advance, avoid failure of the preheating task due to damage or missing cold layer data, ensure the complete execution of the entire preheating process, and at the same time, the verification results can be used as the basis for adjusting the execution priority of subsequent preheating tasks to ensure that only complete and accessible groups will enter the subsequent preheating execution stage, avoiding unnecessary occupation of system resources.

[0055] Step S52, for each group identifier to be preheated, obtains the group metadata linked list corresponding to that group identifier, traverses each node in the linked list, extracts the global identifiers of the data blocks recorded in each node, and aggregates all global identifiers of data blocks belonging to the same group into a preheating identifier set. The group metadata linked list is an ordered linked list created during initialization in step S13 and continuously updated during data block writing. It completely records the globally unique identifiers of all data blocks within the corresponding group and serves as the core index for the entire group's preheating. For each preheated group identifier that passes the pre-verification, the system initiates a query request to the metadata service using that identifier as the index key to obtain the corresponding complete group metadata linked list. Then, starting from the head node of the linked list, it performs a full sequential traversal, extracting the globally unique identifiers of the data blocks stored in each node. After the traversal is complete, all identifiers are aggregated into a preheating identifier set corresponding one-to-one with that group. This set contains all data block identifiers within that group that need to be migrated back for preheating. By performing a single linked list traversal to collect all data block identifiers in the same group, a complete execution list for the entire group's preheating is formed. This ensures that all data blocks within the same anchor group at the same time are included in the scope of this preheating migration, completely solving the problem that existing technologies can only recall accessed data blocks one by one. This achieves the core goal of triggering the preheating of the entire group of data with a single access. At the same time, the aggregated identifier set provides the prerequisite for subsequent batch reading optimization of the cold storage layer, enabling the system to perform batch sequential reading operations and significantly improving the reading efficiency of cold layer data.

[0056] Step S53 controls the number of tasks simultaneously reading data entities from the cold storage layer based on the system's preset preheating concurrency parameters. Based on each preheating identifier set, data entities are read in batches from the cold storage layer without exceeding the concurrency limit set by the preheating concurrency parameters. The preheating concurrency parameters are constant parameters pre-configured by the system based on the gateway processing capacity of the cold storage layer, cluster egress bandwidth, and bandwidth usage of online services. These parameters limit the maximum number of concurrent read tasks for preheating tasks, preventing excessive consumption of system bandwidth resources during preheating. First, the preset preheating concurrency parameters are read from the system configuration management module to determine the upper limit of the number of concurrent cold layer read tasks. Then, using the preheating identifier set generated in step S52 as input, a corresponding number of batch read tasks are created according to the preset concurrency limit. Each read task is responsible for reading a portion of the data block identifiers within its set. The total concurrency of all read tasks always does not exceed the preset concurrency parameter limit, allowing for the orderly batch reading of corresponding data entities from the cold storage layer. By setting up concurrency parameters, precise traffic control is achieved for cold layer read operations, avoiding unlimited concurrent read requests from overwhelming the gateway bandwidth and processing capacity of the cold storage layer. This prevents the warm-up operation from affecting other normal business accesses in the cold storage layer. At the same time, the batch read method fully utilizes the performance optimization capabilities of the cold storage layer for large files and batch sequential reads, significantly improving the reading efficiency of cold layer data, shortening the total warm-up time, and ensuring that the migration of all data is completed before the batch access arrives.

[0057] Step S54 asynchronously writes the read data entities to the target area of ​​the hot storage layer. During the writing process, the input / output processing pressure indicators of the hot storage layer are monitored in real time, and the writing rate is dynamically adjusted according to the fluctuations of the input / output processing pressure indicators to ensure that the writing operation does not significantly affect online services. The input / output processing pressure indicators of the hot storage layer include core performance indicators such as IOPS utilization, bandwidth utilization, and average read / write latency, which are real-time collected data on the operating status of the hot storage layer. The system continuously writes the data entities read from the cold storage layer to the pre-allocated target storage area of ​​the hot storage layer asynchronously. During the writing process, the system collects the input / output processing pressure indicators of the hot storage layer in real time through the performance monitoring module. When the pressure indicators exceed a preset safety threshold, the writing rate of the preheated data is automatically reduced; when the pressure indicators fall back below the safety threshold, the writing rate is gradually restored, achieving adaptive dynamic adjustment of the writing rate. By combining asynchronous writing with adaptive rate limiting, fine-grained control over the resource usage of the hot storage layer is achieved during the pre-warming write operation. This avoids resource contention between the pre-warming write operation and the hot layer read / write operation of online services, ensuring that the access performance of online services is not affected by the pre-warming operation. This solves the core problem in existing technologies where uncontrolled data migration operations can easily lead to performance fluctuations in online services. At the same time, the asynchronous writing method enables the parallel execution of cold layer reads and hot layer writes, further improving the overall pre-warming execution efficiency.

[0058] Step S55: After all data entities have been successfully written to the hot storage layer, the storage location field in the metadata of these data blocks is updated in batches to point to the new address in the hot storage layer. The current actual storage level field corresponding to the group in the group status mapping table is updated to the hot layer, and a cleanup command is sent to the cold storage layer to delete the copies of the data entities that have been successfully migrated to the hot storage layer, thereby freeing up cold storage space. The confirmation that all data entities have been successfully written to the hot storage layer is achieved through a write success response returned by the hot storage layer and the data integrity verification result. The storage location field in the data block metadata is the core metadata item recording the actual storage address of the data block, and the group status mapping table is the core data structure recording the current actual storage level of each group, serving as the core basis for subsequent migration decisions. After receiving the full data write success response from the hot storage layer, the system first performs integrity and consistency verification on all data entities written to the hot layer to ensure that all data blocks have been completely and accurately written to the hot storage layer. After the verification passes, a batch update request is initiated to the metadata service to update the storage location field in the metadata of all data blocks in this pre-warming process, making it point to the corresponding new address in the hot storage layer. After the batch metadata update is completed, an update request is sent to the metadata service to update the value of the current actual storage level field of the corresponding group in the group status mapping table from the cold layer to the hot layer. Finally, a data cleanup command is sent to the cold storage layer to delete the full data entity copy corresponding to the group in the cold storage layer, releasing the archive storage space of the cold layer, thus completing the closed loop of the entire group preheating process. By executing the order of first verifying, then batch updating metadata, then updating group status, and finally cleaning up cold layer copies, the consistency and integrity of data during the preheating process are strictly guaranteed, avoiding access anomalies caused by data dual copies or inconsistencies between metadata and actual storage locations. The batch metadata update method simplifies the original block-by-block metadata update operation into a single batch update, significantly reducing the write pressure on the metadata service. Synchronously updating the group status mapping table ensures that the actual storage level record of the group is completely consistent with the actual storage location of the data, providing accurate status basis for subsequent periodic migration decisions. Finally, the cold layer data copy is cleaned up, avoiding the storage space waste caused by dual copies of cold and hot layers. This enables efficient utilization of storage resources, completing the final closed loop of the entire group's proactive preheating and fully realizing the core inventive goal of this invention: proactive and predictive data readiness for periodic batch access.

[0059] like Figure 2 As shown, the present invention also discloses a hierarchical storage management system for massive data, comprising: The acquisition module is used to acquire the generation timestamp of the data block, truncate the generation timestamp according to the preset time anchor granularity to obtain the time anchor group identifier, write the entity of the data block into the hot storage layer, append the global identifier of the data block to the group metadata chain list corresponding to the group identifier, and initialize the current actual storage level of the group to the hot layer in the group status mapping table. The parsing module is used to parse the group identifier to which the data block belongs when a read access request for any data block is received, update the last access time of the group to the current system time, and locate the actual storage location to read the data based on the global identifier of the data block. The generation module is used to periodically traverse the current actual storage level of each group in the group status mapping table, compare the last access time of each group with a preset threshold, identify groups whose actual storage level is inconsistent with the logical heat status, and generate a list of groups to be migrated that includes groups to be cooled down and groups to be warmed up. The first processing module is used to obtain the corresponding group metadata linked list for the group to be cooled, traverse the data block identifiers recorded on the linked list, migrate the corresponding data entities from the hot storage layer to the cold storage layer in batches, and update the current actual storage level of the group in the group status mapping table to the cold layer after completion. The second processing module is used to obtain the corresponding group metadata linked list for the group to be preheated, migrate the data entities corresponding to the data block identifiers recorded on the linked list from the cold storage layer back to the hot storage layer in batches, and update the current actual storage level of the group in the group status mapping table to the hot layer after completion.

[0060] In one embodiment, the first processing module includes: The acquisition unit is used to acquire the identifier of the group to be cooled, locate and load the group metadata linked list corresponding to the group identifier from the metadata service; The extraction unit is used to traverse the nodes on the grouped metadata linked list, extract the global identifiers of data blocks stored in each node, and aggregate them into a full list of data block identifiers. The writing unit is used to read the corresponding data entities in batches from the hot storage layer according to the full data block identifier list, and write the read data entities sequentially into the cold storage layer. The update unit is used to batch update the storage location field in the metadata of the corresponding data block after confirming that the data entity has been successfully written to the cold storage layer, and update the current actual storage level of the group in the group status mapping table to the cold layer.

[0061] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described hierarchical storage management method for massive data.

[0062] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described hierarchical storage management method for massive data.

[0063] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in this application and in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0064] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0065] The above description is merely a preferred embodiment of the present invention and does not limit the scope of this application. Any equivalent results or equivalent process transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection of this application.

Claims

1. A hierarchical storage management method for mass data, characterized by, Includes the following steps: Obtain the generation timestamp of the data block, truncate the generation timestamp according to the preset time anchor granularity to obtain the time anchor group identifier, write the entity of the data block into the hot storage layer, and append the global identifier of the data block to the group metadata chain list corresponding to the group identifier. In the group status mapping table, initialize the current actual storage level of the group to the hot layer. When a read access request for any data block is received, the group identifier to which the data block belongs is parsed, the last access time of the group is updated to the current system time, and the actual storage location is located according to the global identifier of the data block to read the data. The current actual storage level of each group in the group status mapping table is periodically traversed. The last access time of each group is compared with a preset threshold to identify groups whose actual storage level is inconsistent with their logical heat status. A list of groups to be migrated, including groups to be cooled down and groups to be warmed up, is generated. For a group to be cooled, obtain the corresponding group metadata linked list, traverse the data block identifiers recorded on the linked list, migrate the corresponding data entities from the hot storage layer to the cold storage layer in batches, and update the current actual storage level of the group in the group status mapping table to the cold layer after completion. For a group to be preheated, obtain the corresponding group metadata linked list, migrate the data entities corresponding to the data block identifiers recorded on the linked list from the cold storage layer back to the hot storage layer in batches, and update the current actual storage level of the group in the group status mapping table to the hot layer.

2. The hierarchical storage management method for massive data according to claim 1, characterized in that, The step of initializing the current actual storage level of the group as a hot layer in the group state mapping table includes: Extract the generation timestamp from the metadata of the data block, and read the system's preset time anchor granularity parameter, which is a fixed time length set according to the temporal locality characteristics of data access; The generated timestamp is converted into a numerical representation, and the converted value is truncated according to the time anchor granularity parameter to obtain the time anchor group identifier, so that data blocks whose generation time falls into the same time interval have the same group identifier. In the metadata service, query whether a group metadata list indexed by the group identifier exists. If it does not exist, create the list and create a record for the group identifier at the time when the group was last accessed and set it to the initial value. Generate a globally unique identifier for the data block, append the identifier as a new node to the end of the group metadata linked list, and create a record in the group status mapping table with the group identifier as the primary key, initializing the value of its current actual storage level field to a hot layer.

3. The hierarchical storage management method for massive data according to claim 1, characterized in that, The step of locating the actual storage location based on the global identifier of the data block to read the data includes: In response to a received read access request, the global identifier of the target data block carried in the request is parsed out, and the time anchor group identifier to which the data block belongs is retrieved from the metadata. Obtain the current system time, and using the group identifier as the key, atomically update the value of the corresponding record in the last accessed time of the group to the current system time; Read the current storage level and storage path information recorded in the metadata of the data block, generate the corresponding complete access address according to the storage level, and initiate data reading to the corresponding storage engine; The data entity content returned by the storage engine is encapsulated and returned to the requester.

4. The hierarchical storage management method for massive data according to claim 1, characterized in that, The step of generating a list of groups to be migrated, including groups to be cooled and groups to be preheated, includes: Obtain the preset hot layer residence threshold and cold layer preheating activation threshold; Traverse the groups in the group status mapping table whose current actual storage level is hot layer, read the last access time of each group, calculate the difference between the current system time and the last access time of the group, and mark the hot layer groups whose difference is greater than the hot layer dwell threshold as groups to be cooled. Traverse the groups in the group status mapping table whose current actual storage level is cold layer, read the last access time of each group, calculate the difference between the current system time and the last access time of the group, and mark the cold layer groups whose difference meets the cold layer preheating activation threshold condition as groups to be preheated. The groups to be cooled and preheated are summarized and output as a list of groups to be migrated.

5. The hierarchical storage management method for massive data according to claim 1, characterized in that, The step of updating the current actual storage level of the group in the group status mapping table to the cold layer includes: Obtain the identifier of the group to be cooled, locate and load the linked list of group metadata corresponding to the group identifier from the metadata service; Traverse the nodes on the grouped metadata linked list, extract the global identifiers of data blocks stored in each node, and aggregate them into a full list of data block identifiers; According to the full list of data block identifiers, the corresponding data entities are read in batches from the hot storage layer and written sequentially to the cold storage layer. After confirming that the data entity has been successfully written to the cold storage layer, the storage location field in the corresponding data block metadata is updated in batches, and the current actual storage level of the group in the group status mapping table is updated to the cold layer.

6. The hierarchical storage management method for massive data according to claim 1, characterized in that, The step of updating the current actual storage level of the group in the group status mapping table to the hot layer includes: Obtain the identifier of the group to be preheated, and verify the integrity and accessibility of the data entity corresponding to the group identifier in the cold storage layer; Get the group metadata linked list corresponding to the group identifier, traverse the nodes on the linked list, extract the global identifier of the data block recorded in each node, and aggregate them into a set of warm-up identifiers. Based on the preheating identifier set, data entities are read in batches from the cold storage layer and asynchronously written to the target area of ​​the hot storage layer; After all data entities are successfully written to the hot storage layer, the storage location field in the corresponding data block metadata is updated in batches, the current actual storage level corresponding to the group in the group status mapping table is updated to the hot layer, and the migrated data entity copies in the cold storage layer are cleaned up.

7. A hierarchical storage management system for massive data, characterized in that, include: The acquisition module is used to acquire the generation timestamp of the data block, truncate the generation timestamp according to the preset time anchor granularity to obtain the time anchor group identifier, write the entity of the data block into the hot storage layer, append the global identifier of the data block to the group metadata chain list corresponding to the group identifier, and initialize the current actual storage level of the group to the hot layer in the group status mapping table. The parsing module is used to parse the group identifier to which the data block belongs when a read access request for any data block is received, update the last access time of the group to the current system time, and locate the actual storage location to read the data based on the global identifier of the data block. The generation module is used to periodically traverse the current actual storage level of each group in the group status mapping table, compare the last access time of each group with a preset threshold, identify groups whose actual storage level is inconsistent with the logical heat status, and generate a list of groups to be migrated that includes groups to be cooled down and groups to be warmed up. The first processing module is used to obtain the corresponding group metadata linked list for the group to be cooled, traverse the data block identifiers recorded on the linked list, migrate the corresponding data entities from the hot storage layer to the cold storage layer in batches, and update the current actual storage level of the group in the group status mapping table to the cold layer after completion. The second processing module is used to obtain the corresponding group metadata linked list for the group to be preheated, migrate the data entities corresponding to the data block identifiers recorded on the linked list from the cold storage layer back to the hot storage layer in batches, and update the current actual storage level of the group in the group status mapping table to the hot layer after completion.

8. The hierarchical storage management system for massive data according to claim 7, characterized in that, The first processing module includes: The acquisition unit is used to acquire the identifier of the group to be cooled, locate and load the group metadata linked list corresponding to the group identifier from the metadata service; The extraction unit is used to traverse the nodes on the grouped metadata linked list, extract the global identifiers of data blocks stored in each node, and aggregate them into a full list of data block identifiers. The writing unit is used to read the corresponding data entities in batches from the hot storage layer according to the full data block identifier list, and write the read data entities sequentially into the cold storage layer. The update unit is used to batch update the storage location field in the metadata of the corresponding data block after confirming that the data entity has been successfully written to the cold storage layer, and update the current actual storage level of the group in the group status mapping table to the cold layer.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images