Archive storage method and system based on hot and cold data layering

Abstract

The invention relates to the technical field of data processing, and discloses an archive storage method and system based on hot and cold data stratification, and the method comprises the steps: carrying out the access monitoring of archive data in a storage device, and obtaining the access log information of the archive data; performing cold and hot state evaluation on the archive data to obtain a data popularity value; performing threshold comparison judgment on the data popularity value to obtain a hierarchical judgment result; performing smooth migration on cold data in the archive data; after the whole process of smooth migration is responded, the global data index of the storage device is updated, and the storage position information of the cold data in the storage device and the hierarchy judgment result are associated and integrated to obtain index associated information; when an access request of the archive data is responded, the updated global data metadata index is inquired, and the access request is routed to different types of storage layers of the storage device for data response processing; according to the invention, the efficiency of archive storage based on hot and cold data stratification can be improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for archive storage based on hot and cold data stratification. Background Technology

[0002] In the field of archival data storage, traditional storage methods fail to effectively differentiate between data access frequency and timeliness, storing all archival data uniformly in the same type of storage layer. This indiscriminate storage model results in access to frequently accessed data being limited by uneven allocation of storage resources, failing to obtain efficient response support, while infrequent access to cold data occupies high-speed storage resources for extended periods, leading to high storage costs and low resource utilization.

[0003] Existing technologies lack a dynamic and accurate mechanism for determining hot and cold data, making it difficult to adjust data storage strategies in real time based on changes in data access behavior. They also cannot achieve smooth migration of cold data or dynamic optimization of storage resources. This results in data storage systems facing not only ever-increasing storage pressure when dealing with massive amounts of archival data, but also problems such as low data access routing efficiency and significant response latency, severely impacting the overall performance of archival data storage and access. Summary of the Invention

[0004] This invention provides a file storage method and system based on hot and cold data stratification to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a file storage method based on hot and cold data stratification, comprising:

[0006] S1. Monitor access to the archive data in the storage device to obtain access log information of the archive data;

[0007] S2. Based on the preset time decay heat benchmark and the access log information, evaluate the hot and cold status of the archive data to obtain the data heat value of the archive data.

[0008] S3. Perform threshold comparison on the data heat value to obtain the hierarchical judgment result of the archive data;

[0009] S4. Based on the hierarchical judgment result, perform smooth migration of cold data in the archive data;

[0010] S5. After the entire smooth migration process is completed, update the global data index of the storage device, and associate and integrate the storage location information of the cold data in the low-speed storage layer of the storage device with the hierarchical determination result to obtain the index association information of the storage device.

[0011] S6. When responding to the access request of the archive data, query the updated global data metadata index, and route the access request to different types of storage layers of the storage device for data response processing according to the index association information.

[0012] In a preferred embodiment, the step of monitoring access to the archive data in the storage device to obtain access log information of the archive data includes:

[0013] By deploying a monitoring agent along the data access path of the storage device, access operations to archive data can be intercepted in real time.

[0014] The access operation is parsed, and the parsed operation subject identifier, target metadata identifier, operation timestamp identifier and operation type are integrated into the access record of the archive data.

[0015] According to a preset time window, the operation subject identifier, the target metadata identifier, and the operation timestamp identifier are described by text adaptation. The operation records of the target metadata within the time window are regularized and reorganized with the corresponding text descriptions to obtain the aggregated access summary of the archive data.

[0016] The aggregated access summary is written in a structured format to the log storage area of ​​the storage device to construct and update the access log information of the archive data.

[0017] In a preferred embodiment, the step of evaluating the popularity of the archive data based on a preset time decay popularity benchmark and the access log information to obtain the data popularity value of the archive data includes:

[0018] Obtain a preset time decay heat benchmark, which includes at least a time decay coefficient for defining the timeliness of access behavior and a set of operation type weights for distinguishing different access operations.

[0019] Based on the archive data, read the historical associated access records from the access log information;

[0020] Based on the time decay heat benchmark, the associated access records are initially weighted to obtain the weighted access record set of the archive data;

[0021] Based on the time difference between the operation timestamp in the associated access record and the current evaluation time, and in conjunction with the time decay coefficient, the initial weight value of the access record in the weighted access record set is adjusted for time-related decay to generate the dynamic contribution weight of the archive data.

[0022] Based on the dynamic contribution weight, the weighted access record set is weighted and statistically analyzed, and the statistical results are normalized to generate the data popularity value of the archive data.

[0023] In a preferred embodiment, the formula for calculating the dynamic contribution weight is as follows:

[0024] ;

[0025] In the formula, In order to be at the current assessment moment The data popularity value, The weight set for the operation type and the operation type The corresponding basic operation weight value, The total number of associated access records corresponding to the aforementioned archive data. The time decay coefficient, The time difference between the operation timestamp in the associated access record and the current evaluation time. It is the hyperbolic tangent function. It is a function with maximum value. It is a logarithmic function with base 10.

[0026] In a preferred embodiment, the step of performing a threshold comparison judgment on the data popularity value to obtain the hierarchical judgment result of the archive data includes:

[0027] The distribution of popularity values ​​of archive data in the storage device is periodically analyzed, and the analysis results are quantified in conjunction with the current storage resource load to determine the dynamic hot and cold threshold of the archive data.

[0028] The deviation between the data heat value and the dynamic hot / cold determination threshold is compared.

[0029] When the data heat value is greater than or equal to the dynamic hot / cold determination threshold, the archive data is determined to be hot data, so as to obtain the hot data identifier of the archive data;

[0030] When the data heat value is less than the dynamic hot / cold determination threshold, the archive data is determined to be cold data, so as to obtain the cold data identifier of the archive data;

[0031] The hot data identifier and the cold data identifier are used together as the result of the hierarchical determination of the archive data.

[0032] In a preferred embodiment, the step of smoothly migrating cold data in the archive data based on the hierarchical judgment result includes:

[0033] The archive metadata identified as cold data from the hierarchical judgment results is used as a list of cold data to be migrated.

[0034] In the low-speed storage layer of the storage device, storage space is allocated for independent data in the cold data list to be migrated, and the complete data content of the independent data is copied from the high-speed storage layer to the allocated storage space to obtain a data copy of the independent data.

[0035] Perform a data consistency check on the data copy to ensure that the data copy is consistent with the original data;

[0036] In response to the data consistency check passing, the logical access path in the storage device that points to the original data is atomically updated to point to the data copy in the low-speed storage layer, thus completing the atomic path switching operation of the storage device;

[0037] After the atomic path switching operation is successfully completed, the asynchronous space reclamation task of the storage device is executed. The physical space occupied by the original data stored in the high-speed storage layer is marked as releasable. The data release operation is performed during the low load period of the storage device to complete the smooth migration of cold data in the archive data.

[0038] In a preferred embodiment, updating the global data index of the storage device after the entire smooth migration process is completed includes:

[0039] Acquire the migration completion confirmation signal, storage information identifier, and new storage location descriptor in response to the smooth migration process;

[0040] Based on the storage information identifier, the corresponding index entry is queried and locked in the original data index of the storage device;

[0041] Write the new storage location descriptor into the storage location field of the index entry, and write the cold data status identifier in the hierarchy determination result into the data hierarchy field of the same index entry to complete the association update of the original data index;

[0042] The updated original data index is used as the global data index of the storage device.

[0043] In a preferred embodiment, the step of associating and integrating the storage location information of the cold data in the low-speed storage layer of the storage device with the hierarchical determination result to obtain the index association information of the storage device includes:

[0044] Batch commit and synchronization operations are performed on the updated index entries in the global data index to ensure that the index copy data held by each access node in the storage device is consistent;

[0045] After the batch submission and synchronization operation is successfully confirmed, based on the updated global data index, the integrated storage location field and data level field information are extracted to construct the index association information of the storage device.

[0046] In a preferred embodiment, the step of querying the updated global data metadata index when responding to an access request for the archive data, and routing the access request to different types of storage layers of the storage device for data response processing based on the index association information, includes:

[0047] Receive the access request for the archive data;

[0048] Query the global data metadata index, and extract the current storage level identifier of the archive data from the index association information based on the unique identifier of the archive data;

[0049] Based on the current storage tier identifier, the target storage tier of the storage device is determined, and the access request is forwarded to the target storage tier to achieve tiered storage of the archive data.

[0050] To address the aforementioned problems, the present invention also provides an archive storage system based on hot and cold data stratification, the system comprising:

[0051] The log collection module is used to monitor access to archive data in the storage device and obtain access log information of the archive data;

[0052] The hot / cold status assessment module is used to assess the hot / cold status of the archive data based on a preset time decay heat benchmark and the access log information, and obtain the data heat value of the archive data.

[0053] The hierarchy determination module is used to perform threshold comparison and determination on the data popularity value to obtain the hierarchy determination result of the archive data.

[0054] The data smooth migration module is used to smoothly migrate cold data in the archive data according to the hierarchical judgment result;

[0055] The global index update association module is used to update the global data index of the storage device after the smooth migration process is completed, and to associate and integrate the storage location information of the cold data in the low-speed storage layer of the storage device with the hierarchical determination result to obtain the index association information of the storage device.

[0056] The intelligent access routing module is used to query the updated global data metadata index when responding to an access request for the archive data, and to route the access request to different types of storage layers of the storage device for data response processing based on the index association information.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. This invention quantifies data popularity by combining a time-decaying heat benchmark with structured access log analysis and a dynamic contribution weighting formula, along with a dynamic hot / cold data determination threshold, to accurately distinguish between hot and cold data. Cold data is smoothly migrated to a low-speed storage layer, while hot data is retained in a high-speed storage layer, avoiding resource waste, reducing storage costs, and improving the overall utilization of storage resources.

[0059] 2. This invention ensures data integrity and smooth migration by performing consistency checks, atomic path switching, and space reclamation during low-load periods for cold data migration. The global data index is updated in real time and associated with storage location and hierarchical information. Access requests can be accurately routed to the target storage layer through the index, significantly reducing response latency and improving the overall performance of archive data storage and access. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating an embodiment of the present invention of an archive storage method based on hot and cold data stratification;

[0061] Figure 2 A functional block diagram of an archive storage system based on hot and cold data stratification, provided as an embodiment of the present invention;

[0062] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0063] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0064] This application provides a file storage method based on hot and cold data tiering. The executing entity of this file storage method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the file storage method based on hot and cold data tiering can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0065] Reference Figure 1 The diagram shown is a flowchart illustrating a file storage method based on hot and cold data stratification according to an embodiment of the present invention. In this embodiment, the file storage method based on hot and cold data stratification includes:

[0066] S1. Monitor access to the archive data in the storage device to obtain access log information of the archive data;

[0067] In this embodiment of the invention, the step of monitoring access to archive data in the storage device to obtain access log information of the archive data includes:

[0068] By deploying a monitoring agent along the data access path of the storage device, access operations to archive data can be intercepted in real time.

[0069] The access operation is parsed, and the parsed operation subject identifier, target metadata identifier, operation timestamp identifier and operation type are integrated into the access record of the archive data.

[0070] According to a preset time window, the operation subject identifier, the target metadata identifier, and the operation timestamp identifier are described by text adaptation. The operation records of the target metadata within the time window are regularized and reorganized with the corresponding text descriptions to obtain the aggregated access summary of the archive data.

[0071] The aggregated access summary is written in a structured format to the log storage area of ​​the storage device to construct and update the access log information of the archive data.

[0072] The monitoring agent deployed on the data access path in the storage device runs continuously. This monitoring agent is directly embedded in the transmission link of data reading, writing, modification and other access actions inside the storage device. When an access action for archive data is initiated, the monitoring agent immediately identifies and captures all process information of the access action, realizing real-time interception of archive data access operations.

[0073] For intercepted access operations to archive data, request parsing is performed. During the parsing process, the operation subject identifier corresponding to the object initiating the access operation, the target metadata identifier corresponding to the accessed archive data, the operation timestamp identifier corresponding to the moment the access operation occurs, and the specific operation type corresponding to the access operation are extracted one by one. Then, the above-extracted operation subject identifier, target metadata identifier, operation timestamp identifier, and operation type are systematically integrated to form the access record for the archive data.

[0074] According to a pre-set time window, text adaptation descriptions are performed on the operation subject identifier, target metadata identifier, and operation timestamp identifier in the access records. The specific process of text adaptation description is as follows: generate corresponding subject identity description text based on the inherent attributes of the operation subject identifier; generate corresponding metadata pointing description text based on the exclusive characteristics of the target metadata identifier; and generate corresponding time interval description text based on the specific time period corresponding to the operation timestamp identifier. Then, all operation records of the same target metadata within the time window are organized and reorganized with the corresponding subject identity description text, metadata pointing description text, and time interval description text. During the organization and reorganization, the records are first classified and collected according to the target metadata identifier, then all operation records of the same target metadata are arranged in the order of the operation timestamp identifier, and finally, the corresponding description texts are matched next to each operation record to obtain the aggregated access summary of the archive data.

[0075] The generated archive data aggregated access summary is written to the log storage area of ​​the storage device in a structured format. The specific execution method of the structured format is to arrange the contents of the aggregated access summary in the following order: target metadata identifier description text, operation subject identifier description text, operation timestamp identifier time interval description text, operation type, and fixed fields of operation record. During the writing process, it directly connects to the dedicated writing interface of the log storage area of ​​the storage device to store the aggregated access summary completely in this area. At the same time, the original archive data access log information in the storage device is updated based on the newly written aggregated access summary. During the update, the new aggregated access summary is added to the access log entry of the corresponding target metadata to build a complete and continuous archive data access log information.

[0076] The beneficial effects are that by deploying a monitoring agent on the data access path of the storage device, the real-time and accurate interception of archival data access operations can be achieved. Based on a clear request parsing process, standardized and unified access records are integrated. Combined with a preset time window, the access records and text adaptation descriptions are reorganized to generate an aggregated access summary. Then, through structured format writing and log information updates, the complete construction and dynamic maintenance of archival data access logs can be achieved, ensuring that archival data access behavior is traceable and access information is sortable, thereby improving the standardization and effectiveness of archival data management.

[0077] S2. Based on the preset time decay heat benchmark and the access log information, evaluate the hot and cold status of the archive data to obtain the data heat value of the archive data.

[0078] In this embodiment of the invention, the step of evaluating the hot / cold status of the archive data based on a preset time decay heat benchmark and the access log information to obtain the data heat value of the archive data includes:

[0079] Obtain a preset time decay heat benchmark, which includes at least a time decay coefficient for defining the timeliness of access behavior and a set of operation type weights for distinguishing different access operations.

[0080] Based on the archive data, read the historical associated access records from the access log information;

[0081] Based on the time decay heat benchmark, the associated access records are initially weighted to obtain the weighted access record set of the archive data;

[0082] Based on the time difference between the operation timestamp in the associated access record and the current evaluation time, and in conjunction with the time decay coefficient, the initial weight value of the access record in the weighted access record set is adjusted for time-related decay to generate the dynamic contribution weight of the archive data.

[0083] Based on the dynamic contribution weight, the weighted access record set is weighted and statistically analyzed, and the statistical results are normalized to generate the data popularity value of the archive data.

[0084] The formula for calculating the dynamic contribution weight is as follows:

[0085] ;

[0086] In the formula, In order to be at the current assessment moment The data popularity value, The weight set for the operation type and the operation type The corresponding basic operation weight value, The total number of associated access records corresponding to the aforementioned archive data. The time decay coefficient, The time difference between the operation timestamp in the associated access record and the current evaluation time. It is the hyperbolic tangent function. It is a function with maximum value. It is a logarithmic function with base 10.

[0087] In the formula, the current evaluation time corresponding to the data heat value is the time point when the evaluation operation is carried out. The operation type weight set and the basic operation weight value corresponding to the operation type are derived from the preset time decay heat benchmark. The total number of associated access records is the number of historical associated access records corresponding to the read archive data. The time decay coefficient is derived from the preset time decay heat benchmark. The time difference is the interval between the operation timestamp in the associated access record and the current evaluation time. The hyperbolic tangent function is used to handle the relationship between the time difference and the time decay coefficient. The maximum value function is used to select the larger value. The logarithmic function with base 10 is used to handle the total number of associated access records.

[0088] The formula combines the relationship between the basic operation weight value corresponding to the operation type, the time decay coefficient and the time difference, and then accumulates the results after processing with the hyperbolic tangent function. Finally, it compares the result with the result processed by the maximum value function to obtain the data popularity value of the archive data at the current evaluation time, thereby realizing the quantitative calculation of the popularity of the archive data.

[0089] The smaller the time difference between the associated access records, the closer the result obtained after processing with the hyperbolic tangent function is to the associated value corresponding to the time decay coefficient, resulting in a larger cumulative result and a correspondingly higher data popularity value. Conversely, the larger the time difference between the associated access records, the closer the result obtained after processing with the hyperbolic tangent function is to zero, resulting in a smaller cumulative result and a correspondingly lower data popularity value. At the same time, the more associated access records there are, the larger the result after processing with the logarithmic function and the maximum value function will be, which will reduce the data popularity value accordingly.

[0090] The time decay heat benchmark is retrieved from the preset configuration storage area. This time decay heat benchmark includes a time decay coefficient used to define the timeliness of access behavior and a set of operation type weights used to distinguish different access operations. The retrieval process directly connects to the dedicated read interface of the configuration storage area. The time decay heat benchmark file is retrieved according to the preset read command. The contents of the file are verified line by line to confirm that the time decay coefficient and the set of operation type weights have been completely included. This ensures that the obtained time decay heat benchmark is complete and accurate and can be directly used for subsequent weight calculation.

[0091] Based on the unique identifier of the archival data, the historical access records corresponding to the archival data are retrieved and read in the log storage area of ​​the storage device. The retrieval process performs precise matching according to the association mapping relationship between the archival data identifier and the access records. Using the unique identifier of the archival data as the search keyword, all access log entries in the log storage area are traversed, and access records in the entry whose target metadata identifier is completely consistent with the unique identifier of the archival data are selected. The selected access records are arranged in chronological order according to the operation timestamp identifier to ensure that the obtained historical access records completely correspond to the target archival data and that access records of other irrelevant archival data are not mixed in.

[0092] Based on the retrieved time decay heat benchmark, initial weight allocation is performed on the retrieved historical associated access records. During the allocation process, the operation type corresponding to each associated access record is identified one by one. The identified operation type is compared with the operation type weight set in the time decay heat benchmark, and the weight value corresponding to the operation type is matched. The weight value is directly assigned to the corresponding associated access record. At the same time, a weight mark field is added to the access record, and the matched weight value is entered into this field. The weight assignment of all historical associated access records is completed, and finally a weighted access record set of the archive data is formed.

[0093] For each access record in the weighted access record set, the time difference between its operation timestamp and the current evaluation time is determined. Specifically, the specific time corresponding to the operation timestamp identifier in the access record is read, and this time is compared with the specific time of the current evaluation time. The interval between the two times is calculated, and then the initial weight value of the access record is adjusted for time-related decay based on the time decay heat benchmark. During the adjustment process, the change in weight value is adjusted according to the length of the time difference. The longer the time difference, the greater the decay of the weight value, and the shorter the time difference, the smaller the decay of the weight value. The adjusted weight value is the dynamic contribution value of the access record to the heat of the archival data. The adjusted weight values ​​of all access records are summarized to generate the dynamic contribution weight of the archival data.

[0094] Based on the generated dynamic contribution weights, a weighted statistical operation is performed on the weighted access record set. During the statistical process, each access record in the weighted access record set is traversed one by one, and the dynamic contribution weight value corresponding to each record is extracted. All extracted dynamic contribution weight values ​​are accumulated to obtain the statistical total value. Then, the statistical total value is normalized. The specific method of normalization is to map the statistical total value to a preset fixed value range, divide the upper limit of the preset value range by the statistical total value to obtain the normalization ratio coefficient, and then multiply the statistical total value by the ratio coefficient to make the final result fall within the range, and finally generate the data popularity value of the archive data.

[0095] The beneficial effects are as follows: by retrieving a preset time decay heat benchmark, a unified and clear assessment basis is provided for the assessment of the popularity of archival data; relying on the unique identifier of archival data, the corresponding historical access records are accurately read, ensuring the accuracy and relevance of the assessment data source; based on the time decay heat benchmark, an initial weight allocation is completed to form a weighted access record set; by combining the time difference between the operation timestamp and the current assessment time and the time decay coefficient, the weights are adjusted to achieve timeliness decay, generating dynamic contribution weights that fit the actual access timeliness; and then through weighted statistics and normalization processing, the data popularity value of archival data is generated, realizing a scientific quantitative assessment of the popularity of archival data and improving the rationality and accuracy of the data popularity assessment results.

[0096] S3. Perform threshold comparison on the data heat value to obtain the hierarchical judgment result of the archive data;

[0097] In this embodiment of the invention, the step of performing a threshold comparison judgment on the data popularity value to obtain the hierarchical judgment result of the archive data includes:

[0098] The distribution of popularity values ​​of archive data in the storage device is periodically analyzed, and the analysis results are quantified in conjunction with the current storage resource load to determine the dynamic hot and cold threshold of the archive data.

[0099] The deviation between the data heat value and the dynamic hot / cold determination threshold is compared.

[0100] When the data heat value is greater than or equal to the dynamic hot / cold determination threshold, the archive data is determined to be hot data, so as to obtain the hot data identifier of the archive data;

[0101] When the data heat value is less than the dynamic hot / cold determination threshold, the archive data is determined to be cold data, so as to obtain the cold data identifier of the archive data;

[0102] The hot data identifier and the cold data identifier are used together as the result of the hierarchical determination of the archive data.

[0103] The data popularity values ​​of all archive data in the storage device are statistically analyzed at fixed intervals. During the statistical process, the generated data popularity value of each archive data entry is extracted, and different value intervals are defined according to the popularity value. The number of archive data entries contained in each interval is counted, and the proportion of data in each interval to the total number of archive data entries in the storage device is calculated, thus forming the data popularity value distribution analysis results. Simultaneously, the storage capacity utilization ratio of the storage device is statistically analyzed, i.e., the ratio of used storage capacity to total storage capacity is calculated. Data read / write response times are sampled multiple times and averaged to statistically analyze storage resources. The available capacity is the remaining value after subtracting the used storage capacity from the total storage capacity. This method is used to obtain the current storage resource load. Then, the results of the heat value distribution analysis are combined with the current storage resource load for quantitative processing. Specifically, the proportion of data in each interval of the heat value distribution is matched and correlated with various quantitative indicators of storage resource load. If the proportion of data in a certain interval is high and the storage resource load indicator shows that resources are tight, the judgment threshold for that interval is increased accordingly. If the proportion of data in a certain interval is low and the storage resource load is relaxed, the judgment threshold is decreased accordingly. Finally, the dynamic hot and cold judgment threshold of the archive data that is suitable for the current storage status is determined.

[0104] The deviation of the data popularity value corresponding to each piece of archival data is compared with the determined dynamic hot and cold judgment threshold. Specifically, for each piece of archival data, the generated data popularity value is retrieved first, and then the determined dynamic hot and cold judgment threshold is retrieved. The two values ​​are directly compared to determine whether the data popularity value of the archival data is higher than, equal to or lower than the dynamic hot and cold judgment threshold. In this way, the deviation of the data popularity value of a single piece of archival data from the dynamic hot and cold judgment threshold is obtained.

[0105] When the data heat value of the archive data is greater than or equal to the dynamic hot / cold determination threshold, the archive data is directly determined to be hot data. At the same time, a corresponding identification field is added to the metadata information of the archive data, and the exclusive information representing the hot data is entered in the field. The identification field containing the exclusive information is the hot data identifier of the archive data.

[0106] When the data popularity value of the archive data is less than the dynamic hot / cold data determination threshold, the archive data is directly determined to be cold data. At the same time, a corresponding identification field is added to the metadata information of the archive data, and the exclusive information representing cold data is entered in the field. The identification field containing the exclusive information is the cold data identifier of the archive data.

[0107] The hot data identifier and cold data identifier added to the archive data will be integrated. Specifically, in the metadata area of ​​the archive data, the fields corresponding to these two identifiers will be associated and bound to ensure that each piece of archive data corresponds to only one identifier. The bound identifier combination will be directly used as the hierarchical judgment result of the archive data, and will be used for subsequent operations such as adjusting the storage hierarchy of the archive data.

[0108] The beneficial effects are as follows: by periodically statistically analyzing the distribution of archive data popularity values ​​and combining it with the storage resource load, a dynamic hot / cold threshold suitable for the current storage state can be accurately determined, ensuring the adaptability of the threshold to the actual storage scenario; by comparing the deviation between the data popularity value and the dynamic threshold, the hot / cold attributes of the archive data can be clearly identified and corresponding labels can be added, achieving accurate differentiation of the hot / cold status of archive data; by integrating the hot / cold labels to form a hierarchical judgment result, a clear basis can be provided for subsequent operations such as adjusting the storage hierarchy of archive data, improving the utilization efficiency of storage resources and the rationality of data management.

[0109] S4. Based on the hierarchical judgment result, perform smooth migration of cold data in the archive data;

[0110] In this embodiment of the invention, the step of smoothly migrating cold data in the archive data based on the hierarchical judgment result includes:

[0111] The archive metadata identified as cold data from the hierarchical judgment results is used as a list of cold data to be migrated.

[0112] In the low-speed storage layer of the storage device, storage space is allocated for independent data in the cold data list to be migrated, and the complete data content of the independent data is copied from the high-speed storage layer to the allocated storage space to obtain a data copy of the independent data.

[0113] Perform a data consistency check on the data copy to ensure that the data copy is consistent with the original data;

[0114] In response to the data consistency check passing, the logical access path in the storage device that points to the original data is atomically updated to point to the data copy in the low-speed storage layer, thus completing the atomic path switching operation of the storage device;

[0115] After the atomic path switching operation is successfully completed, the asynchronous space reclamation task of the storage device is executed. The physical space occupied by the original data stored in the high-speed storage layer is marked as releasable. The data release operation is performed during the low load period of the storage device to complete the smooth migration of cold data in the archive data.

[0116] From the hierarchical judgment results of the archive data, the archive metadata marked as cold data is filtered out. During the filtering process, the hierarchical judgment results corresponding to each piece of archive data in the storage device are examined one by one to accurately identify the identification information representing cold data. All archive metadata marked as cold data are collected and organized in chronological order of data generation time. At the same time, a migration mark is added to each piece of cold data metadata, and finally a well-structured list of cold data to be migrated is formed.

[0117] In the low-speed storage layer of the storage device, unoccupied storage blocks are scanned. Based on the actual data capacity of each independent data in the cold data list to be migrated, the required storage space size is calculated, and a dedicated contiguous storage block of the exact same size is allocated. After allocation, a unique association mapping relationship between the dedicated storage space and the corresponding independent data is established in the mapping table of the storage device. Then, the data transfer process is started, and the complete data content of the independent data is read from the high-speed storage layer. During the reading process, the data blocks are retrieved one by one in order. Then, the complete data content is accurately copied to the dedicated storage space allocated in the low-speed storage layer according to the established data transfer protocol. After the copy is completed, data identification information is written to the header of the storage space, and finally, a data copy corresponding to the independent data is generated.

[0118] A comprehensive data consistency check is performed on the data copy generated in the low-speed storage layer and the original data in the high-speed storage layer. During the check, a byte-by-byte comparison method is used to compare the content of each byte of the data copy and the original data in turn. At the same time, the data attribute information, data check value and data integrity mark of the two are verified. If all the comparison items are completely consistent, the data consistency check is determined to be passed. If any of the comparison items are inconsistent, the subsequent operation is immediately terminated and the data retransmission mechanism is triggered to ensure the consistency between the data copy and the original data.

[0119] In response to the successful data consistency check, the logical access path update procedure for the storage device is immediately initiated. This procedure locks the access path configuration table of the storage device and atomically updates the logical access path in the storage device that originally pointed to the original data in the high-speed storage layer. The atomic update operation adopts a transaction-level execution mode to ensure that the path update operation either succeeds completely or fails completely, without any abnormal situations such as path confusion or interruption. After the update is completed, the access path configuration table is saved and unlocked, and finally the logical access path is stably modified to point to the data copy in the low-speed storage layer, completing the atomic path switching operation of the storage device.

[0120] After the atomic path switching operation is successfully completed, the system automatically triggers the asynchronous space reclamation task built into the storage device. This task first retrieves the specific location and block range of the physical space occupied by the original data in the high-speed storage layer, updates the status of the physical space from occupied to releasable in the space management table of the storage device, and then enters the background running mode to continuously monitor the CPU utilization, data read and write throughput and storage I / O load of the storage device. When the storage device is detected to be in a preset low load state period for a continuous period of time, the data release operation of the physical space is immediately executed, clearing all the contents of the original data in the high-speed storage layer in block order, and marking the block as free in the space management table, thereby completing the smooth migration of cold data in the archive data.

[0121] The beneficial effects are as follows: First, by accurately filtering cold data metadata from the hierarchical judgment results to form a list of cold data to be migrated, the accuracy of the cold data migration objects is ensured. Second, by allocating matching low-speed storage layer space to the cold data to be migrated and generating data copies, combined with byte-by-byte consistency checks, the data after migration is ensured to be indistinguishable from the original data. Third, atomic path switching operations avoid abnormal access paths, ensuring the continuity and stability of data access. Fourth, asynchronous space reclamation tasks release high-speed storage layer space during periods of low device load, achieving smooth migration of cold data, effectively improving the resource utilization of storage devices, and optimizing the configuration effect of data storage hierarchy.

[0122] S5. After the entire smooth migration process is completed, update the global data index of the storage device, and associate and integrate the storage location information of the cold data in the low-speed storage layer of the storage device with the hierarchical determination result to obtain the index association information of the storage device.

[0123] In this embodiment of the invention, updating the global data index of the storage device after the entire smooth migration process is completed includes:

[0124] Acquire the migration completion confirmation signal, storage information identifier, and new storage location descriptor in response to the smooth migration process;

[0125] Based on the storage information identifier, the corresponding index entry is queried and locked in the original data index of the storage device;

[0126] Write the new storage location descriptor into the storage location field of the index entry, and write the cold data status identifier in the hierarchy determination result into the data hierarchy field of the same index entry to complete the association update of the original data index;

[0127] The updated original data index is used as the global data index of the storage device.

[0128] The step of associating and integrating the storage location information of the cold data in the low-speed storage layer of the storage device with the hierarchical determination result to obtain the index association information of the storage device includes:

[0129] Batch commit and synchronization operations are performed on the updated index entries in the global data index to ensure that the index copy data held by each access node in the storage device is consistent;

[0130] After the batch submission and synchronization operation is successfully confirmed, based on the updated global data index, the integrated storage location field and data level field information are extracted to construct the index association information of the storage device.

[0131] Receive and acquire the migration completion confirmation signal fed back from the end of the cold data smooth migration process. At the same time, extract the storage information identifier generated synchronously in the process to uniquely identify the migrated cold data, as well as the new storage location descriptor containing the specific storage block address, access path, and storage block range of the cold data in the low-speed storage layer. Verify the validity of the migration completion confirmation signal to ensure that the three types of information—the migration completion confirmation signal, the storage information identifier, and the new storage location descriptor—are complete and correspond one-to-one with the migrated cold data, without any missing or mismatched information.

[0132] Using the obtained storage information identifier as the unique search keyword, a precise query operation is performed in the original data index of the storage device. During the search process, all index entries in the original data index are traversed one by one, and the identifier field in the index entry is matched with the storage information identifier. After locating the corresponding target index entry, the index entry locking mechanism is immediately activated to perform an exclusive locking operation on the target index entry. During the locking period, any other operation is prohibited from reading or modifying the entry, so as to avoid other operations from tampering with or interfering with the entry during subsequent update processes.

[0133] The new storage location descriptor is completely written into the storage location field of the locked target index entry, completely overwriting the original high-speed storage layer location information in that field. At the same time, the corresponding cold data status identifier is retrieved from the hierarchy judgment result and accurately written into the data hierarchy field of the same index entry, replacing the original data hierarchy information in that field. After the content of the two fields is updated, the accuracy of the written content is checked again. Once it is confirmed to be correct, the lock on the index entry is released, thereby realizing the correlation update of the original data index.

[0134] The original data index after the association update operation is completed is directly set as the global data index of the storage device. This global data index will serve as the unified and unique basis for all access nodes in the storage device to query the data location and hierarchical information. At the same time, the index is marked as globally effective in the index configuration table of the storage device to ensure that each access node can obtain the latest storage status information of the migrated cold data based on this global data index when it initiates a data query request.

[0135] A batch commit operation is performed on the updated index entries in the global data index. All updated index entry information is submitted to the index management module of the storage device in a pre-defined order. A commit log is generated and stored during the commit process. Then, the index synchronization mechanism is started, and the updated index entry information is pushed synchronously to each access node of the storage device through the internal communication link of the storage device. During the synchronization process, each access node automatically overwrites its local original index copy after receiving the information. After the synchronization is completed, each access node sends a synchronization success signal. The index management module verifies the consistency between the synchronized index copy of each node and the global data index to ensure that the index copy data held by each access node in the storage device is consistent.

[0136] After receiving confirmation signals from the batch submission and synchronization operations, the updated global data index is used as the core data source. The storage location field and data level field information of all index entries are extracted. The two types of field information are integrated and classified. During the integration process, the data level field is grouped according to its content. Index entries of the same data level are grouped into the same group. Within each group, the data level field is sorted in order according to the block address information of the storage location field. After sorting, an association mapping table containing the data level and the corresponding storage location is generated. Finally, the index association information of the storage device is constructed.

[0137] The beneficial effects are as follows: by obtaining relevant confirmation signals and storage information after the smooth migration of cold data, accurate and complete basis is provided for updating the original data index; the target index entry is accurately located and locked based on the storage information identifier, ensuring the targeting of the index update operation; the index association update is completed by writing the new storage location descriptor and the cold data status identifier, ensuring that the index information is consistent with the actual storage status of the data; the updated index is set as the global data index and batch commit and synchronization operations are performed to ensure the consistency of the index replicas of each access node and avoid data access deviation; finally, index association information is constructed based on the global data index, improving the management efficiency of the storage device data index and the accuracy of data access.

[0138] S6. When responding to the access request of the archive data, query the updated global data metadata index, and route the access request to different types of storage layers of the storage device for data response processing according to the index association information.

[0139] In this embodiment of the invention, when responding to an access request for the archive data, querying the updated global data metadata index, and routing the access request to different types of storage layers of the storage device for data response processing based on the index association information, includes:

[0140] Receive the access request for the archive data;

[0141] Query the global data metadata index, and extract the current storage level identifier of the archive data from the index association information based on the unique identifier of the archive data;

[0142] Based on the current storage tier identifier, the target storage tier of the storage device is determined, and the access request is forwarded to the target storage tier to achieve tiered storage of the archive data.

[0143] The system receives access requests for archive data initiated by the client. During the receiving process, a dedicated communication link is established between the access request and the client. The access permission information carried in the access request is verified. After confirming that the client has the qualification to access the corresponding archive data, the unique identifier of the archive data contained in the access request is extracted. At the same time, the initiation time of the access request is recorded to ensure that the received access request is complete and valid.

[0144] The global data metadata index within the storage device is retrieved. Using the unique identifier of the archive data extracted from the access request as the search keyword, a search operation is performed in the global data metadata index. After locating the index entry that matches the unique identifier, the corresponding current storage level identifier is extracted from the index association information associated with the index entry. This current storage level identifier directly represents the storage level information where the archive data is located. After extraction, the validity of the identifier is verified to ensure that the extracted current storage level identifier is consistent with the actual storage state of the archive data.

[0145] Based on the extracted current storage level identifier, the system matches the preset storage level and level identifier within the storage device to determine the target storage level where the archive data is located. Then, the access request forwarding program is started, and the verified access request is pushed to the determined target storage level according to the preset forwarding rules. After receiving the access request, the target storage level retrieves the corresponding archive data and sends it back to the accessing end, thereby realizing the hierarchical storage of archive data.

[0146] The beneficial effects are as follows: by verifying access permissions, the system receives archive data access requests, ensuring the legality and validity of access operations; by querying the global data metadata index based on the unique identifier of the archive data and extracting the current storage level identifier, the system can accurately locate the data storage location; by determining the target storage layer based on the storage level identifier and forwarding the access request, the system can ensure that the access request efficiently reaches the corresponding storage layer, and ultimately realize the hierarchical storage of archive data, improving the data access efficiency and resource allocation rationality of storage devices.

[0147] like Figure 2 The diagram shown is a functional block diagram of an archive storage system based on hot and cold data stratification provided in an embodiment of the present invention.

[0148] The archive storage system 100 based on hot and cold data stratification described in this invention can be installed in an electronic device. Depending on the functions implemented, the archive storage system 100 may include a log acquisition module 101, a hot / cold status assessment module 102, a hierarchy determination module 103, a data smooth migration module 104, a global index update association module 105, and an intelligent access routing module 106. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0149] In this embodiment, the functions of each module / unit are as follows:

[0150] The log collection module 101 is used to monitor access to archive data in the storage device and obtain access log information of the archive data;

[0151] The hot and cold status assessment module 102 is used to assess the hot and cold status of the archive data based on a preset time decay heat benchmark and the access log information, and obtain the data heat value of the archive data.

[0152] The hierarchy determination module 103 is used to perform threshold comparison determination on the data heat value to obtain the hierarchy determination result of the archive data.

[0153] The data smooth migration module 104 is used to perform smooth migration of cold data in the archive data according to the hierarchical judgment result;

[0154] The global index update association module 105 is used to update the global data index of the storage device after the smooth migration process is completed, and to associate and integrate the storage location information of the cold data in the low-speed storage layer of the storage device with the hierarchical determination result to obtain the index association information of the storage device.

[0155] The intelligent access routing module 106 is used to query the updated global data metadata index when responding to the access request of the archive data, and to route the access request to different types of storage layers of the storage device for data response processing according to the index association information.

[0156] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0157] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0159] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0160] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for archival storage based on hot and cold data stratification, characterized in that, The method includes: S1. Monitor access to the archive data in the storage device to obtain access log information of the archive data; S2. Based on the preset time decay heat benchmark and the access log information, evaluate the hot and cold status of the archive data to obtain the data heat value of the archive data. S3. Perform threshold comparison on the data heat value to obtain the hierarchical judgment result of the archive data; S4. Based on the hierarchical judgment result, perform smooth migration of cold data in the archive data; S5. After the entire smooth migration process is completed, update the global data index of the storage device, and associate and integrate the storage location information of the cold data in the low-speed storage layer of the storage device with the hierarchical determination result to obtain the index association information of the storage device. S6. When responding to the access request of the archive data, query the updated global data metadata index, and route the access request to different types of storage layers of the storage device for data response processing according to the index association information.

2. The archive storage method based on hot and cold data stratification as described in claim 1, characterized in that, The process of monitoring access to archive data in the storage device and obtaining access log information for the archive data includes: By deploying a monitoring agent along the data access path of the storage device, access operations to archive data can be intercepted in real time. The access operation is parsed, and the parsed operation subject identifier, target metadata identifier, operation timestamp identifier and operation type are integrated into the access record of the archive data. According to a preset time window, the operation subject identifier, the target metadata identifier, and the operation timestamp identifier are described by text adaptation. The operation records of the target metadata within the time window are regularized and reorganized with the corresponding text descriptions to obtain the aggregated access summary of the archive data. The aggregated access summary is written in a structured format to the log storage area of ​​the storage device to construct and update the access log information of the archive data.

3. The archive storage method based on hot and cold data stratification as described in claim 1, characterized in that, The process of evaluating the popularity of the archive data based on a preset time decay popularity benchmark and the access log information to obtain the data popularity value of the archive data includes: Obtain a preset time decay heat benchmark, which includes at least a time decay coefficient for defining the timeliness of access behavior and a set of operation type weights for distinguishing different access operations. Based on the archive data, read the historical associated access records from the access log information; Based on the time decay heat benchmark, the associated access records are initially weighted to obtain the weighted access record set of the archive data; Based on the time difference between the operation timestamp in the associated access record and the current evaluation time, and in conjunction with the time decay coefficient, the initial weight value of the access record in the weighted access record set is adjusted for time-related decay to generate the dynamic contribution weight of the archive data. Based on the dynamic contribution weight, the weighted access record set is weighted and statistically analyzed, and the statistical results are normalized to generate the data popularity value of the archive data.

4. The archive storage method based on hot and cold data stratification as described in claim 3, characterized in that, The formula for calculating the dynamic contribution weight is as follows: ； In the formula, In order to be at the current assessment moment The data popularity value, The weight set for the operation type and the operation type The corresponding basic operation weight value, The total number of associated access records corresponding to the aforementioned archive data. The time decay coefficient, The time difference between the operation timestamp in the associated access record and the current evaluation time. It is the hyperbolic tangent function. It is a function with maximum value. It is a logarithmic function with base 10.

5. The archive storage method based on hot and cold data stratification as described in claim 1, characterized in that, The step of performing a threshold comparison on the data popularity value to obtain the hierarchical judgment result of the archive data includes: The distribution of popularity values ​​of archive data in the storage device is periodically analyzed, and the analysis results are quantified in conjunction with the current storage resource load to determine the dynamic hot and cold threshold of the archive data. The deviation between the data heat value and the dynamic hot / cold determination threshold is compared. When the data heat value is greater than or equal to the dynamic hot / cold determination threshold, the archive data is determined to be hot data, so as to obtain the hot data identifier of the archive data; When the data heat value is less than the dynamic hot / cold determination threshold, the archive data is determined to be cold data, so as to obtain the cold data identifier of the archive data; The hot data identifier and the cold data identifier are used together as the result of the hierarchical determination of the archive data.

6. The archive storage method based on hot and cold data stratification as described in claim 1, characterized in that, The step of smoothly migrating cold data in the archive data based on the hierarchical judgment result includes: The archive metadata identified as cold data from the hierarchical judgment results is used as a list of cold data to be migrated. In the low-speed storage layer of the storage device, storage space is allocated for independent data in the cold data list to be migrated, and the complete data content of the independent data is copied from the high-speed storage layer to the allocated storage space to obtain a data copy of the independent data. Perform a data consistency check on the data copy to ensure that the data copy is consistent with the original data; In response to the data consistency check passing, the logical access path in the storage device that points to the original data is atomically updated to point to the data copy in the low-speed storage layer, thus completing the atomic path switching operation of the storage device; After the atomic path switching operation is successfully completed, the asynchronous space reclamation task of the storage device is executed. The physical space occupied by the original data stored in the high-speed storage layer is marked as releasable. The data release operation is performed during the low load period of the storage device to complete the smooth migration of cold data in the archive data.

7. The archive storage method based on hot and cold data stratification as described in claim 1, characterized in that, After the entire smooth migration process is completed, updating the global data index of the storage device includes: Acquire the migration completion confirmation signal, storage information identifier, and new storage location descriptor in response to the smooth migration process; Based on the storage information identifier, the corresponding index entry is queried and locked in the original data index of the storage device; Write the new storage location descriptor into the storage location field of the index entry, and write the cold data status identifier in the hierarchy determination result into the data hierarchy field of the same index entry to complete the association update of the original data index; The updated original data index is used as the global data index of the storage device.

8. The archive storage method based on hot and cold data stratification as described in claim 7, characterized in that, The step of associating and integrating the storage location information of the cold data in the low-speed storage layer of the storage device with the hierarchical determination result to obtain the index association information of the storage device includes: Batch commit and synchronization operations are performed on the updated index entries in the global data index to ensure that the index copy data held by each access node in the storage device is consistent; After the batch submission and synchronization operation is successfully confirmed, based on the updated global data index, the integrated storage location field and data level field information are extracted to construct the index association information of the storage device.

9. The archive storage method based on hot and cold data stratification as described in claim 1, characterized in that, When responding to an access request for the archive data, the process of querying the updated global data metadata index and, based on the index association information, routing the access request to different storage layers of the storage device for data response processing includes: Receive the access request for the archive data; Query the global data metadata index, and extract the current storage level identifier of the archive data from the index association information based on the unique identifier of the archive data; Based on the current storage tier identifier, the target storage tier of the storage device is determined, and the access request is forwarded to the target storage tier to achieve tiered storage of the archive data.

10. An archive storage system based on hot and cold data stratification, characterized in that, The system for implementing the archive storage method based on hot and cold data stratification as described in claim 1 includes: The log collection module is used to monitor access to archive data in the storage device and obtain access log information of the archive data; The hot / cold status assessment module is used to assess the hot / cold status of the archive data based on a preset time decay heat benchmark and the access log information, and obtain the data heat value of the archive data. The hierarchy determination module is used to perform threshold comparison and determination on the data popularity value to obtain the hierarchy determination result of the archive data. The data smooth migration module is used to smoothly migrate cold data in the archive data according to the hierarchical judgment result; The global index update association module is used to update the global data index of the storage device after the smooth migration process is completed, and to associate and integrate the storage location information of the cold data in the low-speed storage layer of the storage device with the hierarchical determination result to obtain the index association information of the storage device. The intelligent access routing module is used to query the updated global data metadata index when responding to an access request for the archive data, and to route the access request to different types of storage layers of the storage device for data response processing based on the index association information.

Images