A cold and hot data migration prediction method based on dynamic load analysis

Abstract

The application discloses a cold and hot data migration prediction method based on dynamic load analysis and relates to the technical field of data storage management, which comprises the following steps: collecting data access events, storage layer performance records and migration scheduling records, obtaining data object load records, storage layer load records and migration quota records; calculating current access heat according to the data object load records and calculating a dynamic load coefficient according to the storage layer load records; obtaining next period prediction heat in combination with the current access heat, heat history cache and the dynamic load coefficient, and determining a candidate target storage layer; calculating a migration benefit value and forming a migration prediction list; performing fragment replication, incremental write record playback, data digest verification and access pointing switching according to the migration prediction list, and obtaining migration records. The application realizes the coupling between cold and hot data migration prediction and a dynamic load coefficient, and forms a prediction result which is restricted by migration benefits and admission conditions.

Description

Technical Field

[0001] This invention relates to the field of data storage management technology, and in particular to a method for predicting the migration of hot and cold data based on dynamic load analysis. Background Technology

[0002] With the continuous operation of digital service supply chain collaboration, business processes such as orders, inventory, warehousing, logistics, settlement, and after-sales service generate a large number of data objects with significant differences in access frequency, access time, and data size. To improve storage resource utilization, tiered storage systems typically configure high-frequency access data in high-performance storage tiers and low-frequency access or archived data in capacity-type storage tiers based on data hotness or coldness. By periodically collecting information such as access frequency, access time, and data volume, and combining this with preset hot / cold thresholds, data migration tasks are generated to achieve data scheduling and storage performance management between different storage tiers.

[0003] In conventional methods for predicting hot and cold data migration, on the one hand, the determination of hot and cold data often relies on the access popularity of the current period or historical periods. The coupling between dynamic load changes such as storage layer queue depth, capacity occupancy, and tail latency and the popularity prediction of the next period is insufficient, which can easily cause the migration judgment to lag behind the fluctuation of business load. On the other hand, the joint constraints on migration benefits, replica consistency status, remaining capacity of the target layer, and migration quota are not sufficient when generating migration tasks, which can easily lead to migration prediction results with insufficient executability or unclear benefits. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method for predicting the migration of hot and cold data based on dynamic load analysis to solve the problems of insufficient coupling between dynamic load and heat prediction and insufficient constraints on the executability of migration prediction results in the prior art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides a method for predicting hot and cold data migration based on dynamic load analysis. The method includes: collecting data access events, storage access counters, link tracing records, data object metadata directories, storage layer performance records, and migration scheduling records generated during a monitoring period in a digital service supply chain collaborative business; obtaining data object load records including read frequency, write frequency, access bytes, access recentity, business timeliness level, current storage layer, object data volume, and replica consistency status; and obtaining storage layer load records including queue depth, capacity usage, tail latency, and remaining available capacity, as well as migration quota records; calculating the current access heat using a weighted fusion method based on access frequency, access bytes, access recentity, and business timeliness level, and further calculating the current access heat based on the storage layer load records using a weighted fusion method based on queue depth, capacity usage, and tail latency. The system uses a delay-weighted fusion approach to calculate the dynamic load factor. Based on the current access popularity, historical access history cache, and dynamic load factor, it employs a combination of historical baseline popularity and load trend correction to obtain the predicted popularity for the next period. Based on the predicted popularity for the next period and the cold / hot migration threshold, it determines candidate storage layers for promotion, demotion, and target storage. Based on these candidate storage layers, the object data volume, replica consistency status, remaining available capacity, migration quota records, and the dynamic load factor, it calculates the migration benefit value and forms a migration prediction list. According to the migration prediction list, it performs sharded replication, incremental write record replay, data digest verification, and access direction switching on data objects, obtaining migration records including migration direction, metadata version number, data digest verification results, and access direction switching results.

[0008] As a preferred embodiment of the cold and hot data migration prediction method based on dynamic load analysis described in this invention, within the monitoring period, data access events, storage access counters, and link tracing records are associated with data object identifiers and transaction identifiers to obtain the number of read operations, the number of write operations, the cumulative number of accessed bytes, the most recent access timestamp, and the business service identifier for each data object. Based on the number of read operations, the number of write operations, and the cumulative number of accessed bytes for each data object, normalization is performed according to the historical statistical benchmark of similar data objects to obtain the read frequency, write frequency, and number of accessed bytes. The access recentity is calculated based on the time interval between the most recent access timestamp and the end time of the current monitoring period, and the business timeliness level is matched from the supply chain business catalog and service level agreement based on the business service identifier. The current storage layer and object data volume of each data object are read from the data object metadata catalog, and the replica consistency status is determined based on the metadata version number, primary replica log sequence number, and readable replica log sequence number in the data object metadata catalog. The read frequency, write frequency, number of accessed bytes, access recentity, business timeliness level, current storage layer, object data volume, and replica consistency status are written according to the data object identifier and monitoring period number to form a data object load record.

[0009] As a preferred embodiment of the cold and hot data migration prediction method based on dynamic load analysis described in this invention, the following steps are taken: Based on storage layer performance records, the average pending input / output queue depth, rated available capacity, current occupied capacity, 95th percentile access latency, and remaining available capacity after deducting reserved capacity are read for each storage layer within the current monitoring period. The queue depth is calculated based on the average pending input / output queue depth and the queue depth normalization benchmark, and the tail latency is calculated based on the 95th percentile access latency and the tail latency normalization benchmark. Capacity occupancy is calculated based on the ratio between the current occupied capacity and the rated available capacity, and the queue depth, capacity occupancy, tail latency, and remaining available capacity are written according to the storage layer number and the monitoring period number to form a storage layer load record. Based on the migration scheduling record, the remaining migration bandwidth quota after deducting scheduled migration tasks within the current monitoring period is read, and the monitoring period duration of the current monitoring period is obtained. The remaining migration bandwidth quota and the monitoring period duration are converted into the upper limit of the amount of data allowed to migrate in the current monitoring period, and written according to the monitoring period number to form a migration quota record.

[0010] As a preferred embodiment of the cold and hot data migration prediction method based on dynamic load analysis described in this invention, the following steps are taken: First, the read frequency, write frequency, accessed bytes, access recentity, business timeliness level, and current storage layer of the data object within the current monitoring period are read from the data object load record. Second, the read frequency and write frequency are fused according to their respective read / write ratios to obtain the access frequency of the data object within the current monitoring period. Third, the access frequency, accessed bytes, access recentity, and business timeliness level are weighted and fused according to their respective heat weights to obtain the current access heat of the data object within the current monitoring period. Fourth, the current access heat is written to the heat history cache according to the data object identifier and monitoring period number, and a correspondence is established between the current access heat and the current storage layer of the data object. Fifth, an object heat record is generated based on the data object identifier, monitoring period number, current storage layer, and current access heat to calculate the predicted heat for the next period.

[0011] As a preferred embodiment of the cold and hot data migration prediction method based on dynamic load analysis described in this invention, the following steps are taken: First, the queue depth, capacity occupancy, tail latency, and storage layer number of the storage layer within the current monitoring period are read from the storage layer load record. Then, based on the load weights corresponding to the queue depth, capacity occupancy, and tail latency of the storage layer, a set of load indicators participating in the dynamic load calculation within the current monitoring period is determined. Next, the queue depth, capacity occupancy, and tail latency are weighted and fused to obtain the dynamic load coefficient of the storage layer within the current monitoring period. Finally, the dynamic load coefficient is written into the storage layer load analysis record according to the storage layer number and the monitoring period number. Based on the current storage layer of the data object, the dynamic load coefficient of the corresponding storage layer is associated with the data object to generate a storage layer load association record for calculating the predicted heat of the next period.

[0012] As a preferred embodiment of the cold and hot data migration prediction method based on dynamic load analysis described in this invention, the following steps are taken: First, the historical access popularity of a data object before the current monitoring period is obtained from the historical popularity cache according to the data object identifier and monitoring period number, forming a historical popularity sequence. Then, based on the number and value of valid historical access popularity in the historical popularity sequence, the historical baseline popularity of the data object in the current monitoring period is calculated. Next, based on the difference between the current access popularity and the historical baseline popularity, the popularity change of the data object in the current monitoring period is calculated. Then, based on the current storage layer of the data object, the dynamic load coefficient of the corresponding storage layer in the current monitoring period is obtained, and the load trend correction is calculated based on the popularity change and the dynamic load coefficient. Finally, the predicted popularity for the next period is calculated based on the current access popularity, the historical baseline popularity, and the load trend correction, and the predicted popularity for the next period is limited to between 0 and 1.

[0013] As a preferred embodiment of the hot and cold data migration prediction method based on dynamic load analysis described in this invention, the following steps are taken: Hot migration threshold and cold migration threshold are obtained from the migration strategy file; the predicted heat for the next cycle is compared with the hot migration threshold; if the predicted heat for the next cycle is not less than the hot migration threshold, the data object is identified as a candidate for promotion; the predicted heat for the next cycle is compared with the cold migration threshold; if the predicted heat for the next cycle is not greater than the cold migration threshold, the data object is identified as a candidate for demotion; if the predicted heat for the next cycle is greater than the cold migration threshold but less than the hot migration threshold, the data object is identified as a retention object, and its current storage layer is retained; the target layer heat matching value of each storage layer is read from the storage layer configuration table; for promotion candidates, candidate target storage layers are determined from storage layers whose target layer heat matching value is higher than the target layer heat matching value of the current storage layer; for demotion candidates, candidate target storage layers are determined from storage layers whose target layer heat matching value is lower than the target layer heat matching value of the current storage layer.

[0014] As a preferred embodiment of the cold and hot data migration prediction method based on dynamic load analysis described in this invention, the following steps are taken: First, candidate objects for upgrading and candidate objects for downgrading are selected from the cold and hot prediction result records. The current storage layer, candidate target storage layer, predicted heat for the next period, and business timeliness level are obtained for each candidate object. Then, the target layer heat matching value is obtained based on the candidate target storage layer, and the business timeliness level is converted into a business direction matching value based on the migration direction corresponding to the candidate objects. Next, the load release value for migrating from the current storage layer to the candidate target storage layer is calculated based on the dynamic load coefficient corresponding to the current storage layer and the dynamic load coefficient corresponding to the candidate target storage layer. Then, the migration cost value of the data object in the current monitoring period is calculated based on the object data volume and migration quota records. Finally, the migration benefit value for each candidate object for upgrading and each candidate object for downgrading is calculated based on the predicted heat for the next period, the target layer heat matching value, the business direction matching value, the load release value, the migration cost value, and the replica consistency status.

[0015] As a preferred embodiment of the cold and hot data migration prediction method based on dynamic load analysis described in this invention, the following steps are taken: First, a consistency admission check is performed on candidate objects for upgrading and candidate objects for downgrading based on replica consistency status. Data objects whose replica consistency status does not meet the migration conditions are marked as non-migrated objects. Second, a capacity admission check is performed on data objects that pass the consistency admission check based on the remaining available capacity of the candidate target storage layer and the amount of object data. Data objects whose object data amount is greater than the remaining available capacity of the candidate target storage layer are marked as non-migrated objects. Third, a migration quota admission check is performed on data objects that pass the capacity admission check based on migration quota records and the amount of object data. Data objects whose migration quota is insufficient in the current monitoring period are marked as non-migrated objects. Fourth, a benefit admission check is performed on data objects that pass the migration quota admission check based on migration benefit value. Data objects whose migration benefit value is not greater than zero are marked as non-migrated objects. Fifth, data objects that pass the benefit admission check are sorted from high to low according to their migration benefit value, and the data object identifier, current storage layer, candidate target storage layer, migration direction, predicted heat for the next period, object data amount, migration benefit value, and execution priority are written into the migration prediction list.

[0016] As a preferred embodiment of the cold and hot data migration prediction method based on dynamic load analysis described in this invention, the following steps are taken: According to the execution priority in the migration prediction list, the data object identifier, current storage layer, candidate target storage layer, migration direction, and object data volume are obtained. At the start of migration, the metadata version number and current access pointer are obtained from the data object metadata directory. The number of shards is determined based on the object data volume and migration shard size. Data objects in the current storage layer are copied to the candidate target storage layer according to their shard numbers. During the shard copying process, committed writes generated after the metadata version number are recorded, forming incremental write records. The incremental write records are replayed to the candidate target storage layer according to their write sequence numbers, ensuring that the data object content in the candidate target storage layer is consistent with the committed data in the current storage layer. Data digests are calculated for the data objects in both the current and candidate target storage layers, and the two data digests are compared to obtain a data digest verification result. When the data digest verification result is successful, the access pointer of the data object is switched from the current storage layer to the candidate target storage layer, and a migration record including the migration direction, metadata version number, data digest verification result, and access pointer switching result is generated.

[0017] The beneficial effects of this invention are as follows: By using a weighted fusion calculation of access frequency, access byte volume, access recentity, and business timeliness level to calculate the current access heat, and by using a weighted fusion calculation of queue depth, capacity occupancy, and tail latency to calculate the dynamic load coefficient, the joint analysis of the hotness and coldness of data objects and the real-time load status of the storage layer is realized, improving the adaptability of hot and cold data migration judgment to changes in business access and storage operation pressure; by combining the current access heat, historical heat cache, and dynamic load coefficient, and using a combination of historical baseline heat and load trend correction to obtain the predicted heat for the next period, the future hot and coldness trend of data objects can be predicted in advance, reducing the lag caused by relying solely on the current access heat or fixed historical statistical results for migration prediction. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a method for predicting the migration of hot and cold data based on dynamic load analysis.

[0020] Figure 2 A flowchart for the formation of basic records and load analysis.

[0021] Figure 3 A flowchart for determining the predicted heat level and forming the migration prediction list for the next cycle.

[0022] Figure 4 A flowchart for the execution of cold and hot data migration and the generation of migration records.

[0023] Figure 5 This is a comparison chart of access latency at the 95th percentile.

[0024] Figure 6 A comparison chart showing the proportion of tasks that were effectively migrated. Detailed Implementation

[0025] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0026] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0027] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0028] Reference Figures 1-6 This is one embodiment of the present invention, which provides a method for predicting the migration of hot and cold data based on dynamic load analysis, including the following steps:

[0029] S1. By collecting data access events, storage access counters, link tracing records, data object metadata catalogs, storage layer performance records, and migration scheduling records generated during the monitoring period of digital service supply chain collaborative business, obtain data object load records including read frequency, write frequency, access byte volume, access proximity, business timeliness level, current storage layer, object data volume, and replica consistency status, and obtain storage layer load records including queue depth, capacity usage, tail latency, and remaining available capacity, as well as migration quota records.

[0030] Furthermore, during the monitoring period, data access events are obtained from order services, inventory services, warehousing services, logistics services, settlement services, and after-sales services of the digital service supply chain collaboration business. Data access events include data object identifiers, access timestamps, access types, access byte counts, business service identifiers, and transaction identifiers.

[0031] Specifically, the number of read operations, the number of write operations, and the cumulative number of bytes accessed for a data object during the current monitoring period are obtained from the storage access counter.

[0032] Obtain the call association between the business service identifier and the data object identifier from the link tracing record; read the current storage layer, object data volume, and object type from the data object metadata directory.

[0033] Obtain the queue depth, capacity usage, tail latency, and remaining available capacity of each storage layer from the storage layer performance records.

[0034] Obtain the allowable migration bandwidth quota for the current monitoring period from the migration scheduling record.

[0035] Furthermore, object identification alignment is performed on data access events, storage access counters, and link tracing records.

[0036] Specifically, by using the data object identifier and transaction identifier as association fields, read operations, write operations, and access bytes belonging to the same data object within the same monitoring period are grouped into the same data object load record.

[0037] When both access logs and storage access counters exist, the number of read operations, the number of write operations, and the cumulative number of accessed bytes in the storage access counter are used as the measurement values. Access logs are used to supplement access timestamps and business service identifiers, and link tracing records are used to supplement the correspondence between business service identifiers and supply chain business links.

[0038] Furthermore, data objects are identified at the granularity of database partitions, index segments, file fragments, object storage objects, and block storage extents.

[0039] Specifically, when the same business data has both a logical table name and a physical shard name, the physical shard name in the data object metadata directory is used as the data object identifier, and the logical table name is written into the business attribution field of the data object load record.

[0040] Furthermore, based on the number of read operations, write operations, and cumulative access bytes of the data object within the current monitoring period, normalized values ​​for read frequency, write frequency, and access bytes are generated respectively. When the normalized baseline value is greater than 0, it is represented as: ; ; ;

[0041] in, Represents data objects In the The normalized value of the reading frequency for each monitoring cycle. Represents data objects In the The normalized value of the write frequency for each monitoring cycle. Represents data objects In the The normalized value of the number of bytes accessed per monitoring period. Represents data objects In the Number of read operations per monitoring cycle Represents data objects In the Number of write operations per monitoring cycle Represents data objects In the Cumulative access bytes per monitoring period Represents data objects Normalization benchmark for the read frequency of data objects of the same type Represents data objects Normalized benchmark for the write frequency of data objects of the same type Represents data objects The normalized baseline for the number of bytes accessed by data objects of the same type.

[0042] It should be noted that the normalization benchmarks for similar data objects are determined jointly based on the business domain, object type, and current storage layer. The normalization benchmarks for frequency normalization, write frequency normalization, and access byte size normalization are respectively determined by the most recent... The 95th percentile values ​​of the number of read operations, write operations, and cumulative bytes accessed for the same type of data object within a monitoring period are used to form the data. The value is set by the running statistics strategy and ranges from 10 to 2880.

[0043] The frequency normalization benchmark is formed by the 95th percentile of the number of read operations on similar data objects, with a common value range of 1 to 10 times per monitoring cycle. 6 Write frequency normalization benchmark is formed by the 95th percentile of the number of write operations on similar data objects, with a common value range of 1 to 10 times per monitoring cycle. 5 The access byte volume normalization benchmark is formed by the 95th percentile value of the cumulative access bytes of the same type of data object, with a common value range of 4KB / monitoring cycle to 1TB / monitoring cycle; when the 95th percentile value exceeds the corresponding common value range, it is limited according to the boundary value of the corresponding common value range; when the 95th percentile value is 0, the corresponding normalization result is recorded as 0.

[0044] The normalized values ​​for read frequency, write frequency, and access byte count are all limited to the range of 0 to 1. When the number of read operations on a data object in the current monitoring period exceeds the normalized read frequency baseline, the normalized read frequency value is recorded as 1. When the number of write operations on a data object in the current monitoring period exceeds the normalized write frequency baseline, the normalized write frequency value is recorded as 1. When the cumulative access byte count of a data object in the current monitoring period exceeds the normalized access byte count baseline, the normalized access byte count value is recorded as 1, in order to avoid abnormal access peaks in a single monitoring period causing subsequent current access popularity calculation results to exceed the normalization range.

[0045] It should be noted that the 95th percentile value is used as the benchmark for normalization of acquisition frequency, write frequency, access bytes, and tail latency because supply chain collaboration business will generate a small number of extreme access peaks during promotions, centralized replenishment, batch settlement, and centralized updates of logistics status. Directly using the maximum value will easily lead to an overall underestimation of the normalization results, while directly using the average value will not be able to reflect the business pressure during high-load periods. The 95th percentile value can retain the characteristics of high-load access while reducing the impact of a small number of abnormal peaks on the calculation results, thereby making the normalization of read frequency, write frequency, access bytes, and tail latency more stable.

[0046] When the read frequency normalization benchmark, write frequency normalization benchmark, or access byte volume normalization benchmark is 0, or when the historical samples of the same type of data object are empty, the corresponding normalization value is not directly recorded as 0. Instead, an alternative normalization benchmark is determined according to the priority order of the global historical statistical benchmark of the same type of data object in the same business domain, the statistical benchmark of the same type of data object in the current monitoring period, and the preset minimum normalization benchmark. Among them, the global historical statistical window is the most recent 7 to 30 days, the number of statistical objects of the same type of data object in the current monitoring period is not less than 10, the preset minimum normalization benchmark for read frequency is 1 time / monitoring period, the preset minimum normalization benchmark for write frequency is 1 time / monitoring period, and the preset minimum normalization benchmark for access byte volume is 4KB / monitoring period. The normalized values ​​of read frequency, write frequency, and access byte volume calculated using the alternative normalization benchmark are still limited to the range of 0 to 1.

[0047] Furthermore, based on the data object The access recentity is calculated using the timestamp of the most recent valid access and the end time of the current monitoring period, and is expressed as: ;

[0048] in, Represents data objects In the Recentness of visits per monitoring period Represents data objects The most recent access interval since the end of the current monitoring period. Indicates the recent decay scale, Represents the natural constant.

[0049] It should be noted that the units for the most recent access interval and the recentity decay scale are the same, both using minutes as the time unit; the recentity decay scale is formed offline based on the median of the historical access intervals of the same business domain, and its value is greater than 0.

[0050] When there are no data object access events in the current monitoring period, the most recent access interval is formed based on the interval between the most recent valid access timestamp and the end time of the current monitoring period. The number of read operations, the number of write operations, and the cumulative number of accessed bytes are all recorded as 0.

[0051] Furthermore, the business timeliness level corresponding to the data object is obtained from the supply chain business catalog and service level agreement.

[0052] Specifically, order payment, inventory deduction, warehouse inbound / outbound confirmation, and logistics anomaly handling correspond to high timeliness levels; logistics trajectory query, after-sales work order processing, and settlement reconciliation correspond to medium timeliness levels; and historical archiving, audit retention, and offline analysis correspond to low timeliness levels. The business timeliness levels are normalized to the range of 0 to 1.

[0053] It should be noted that the source of the business timeliness level is determined according to the priority order of the service level agreement, the supply chain business catalog, and the data governance catalog; when the service level agreement configures level 1, level 2, level 3, and level 4, they are mapped to 1, 0.75, 0.5, and 0.25, respectively.

[0054] When the current data object is not recorded in the service level agreement, it is mapped according to the business process in the supply chain business catalog; when the current data object is not recorded in the supply chain business catalog, it is mapped according to the data category in the data governance catalog.

[0055] Furthermore, the current storage layer and object data volume are read from the data object metadata directory.

[0056] The current storage layer is represented by the layer number in the storage layer configuration table. The hot layer, warm layer, and cold layer are set with layer numbers in order of access latency from low to high. The object data volume is represented by the actual number of storage bytes that need to be transferred when the data object is migrated.

[0057] It should be noted that the amount of object data and the remaining available capacity of the storage layer are measured in the same unit.

[0058] Specifically, when a data object has a compressed copy, the object data volume is formed according to the actual number of bytes stored after compression; when a data object has multiple copies, the object data volume is formed according to the amount of primary copy data that needs to be transferred in one migration, without repeatedly accumulating the capacity of redundant copies that did not participate in the migration.

[0059] Furthermore, the replica consistency status of the data object is read from the replica synchronization status record.

[0060] Specifically, when the metadata version number, primary replica log sequence number, and readable replica log sequence number of a data object are all consistent, the replica consistency status is recorded as 1; when the data object has replica repair, incomplete log replay, or inconsistent metadata version, the replica consistency status is recorded as 0.

[0061] It should be noted that the replica consistency status takes the value of 0 or 1, where 1 indicates that the current data object meets the basic consistency conditions for entering the migration prediction judgment, and 0 indicates that the current data object will not enter the migration prediction list in the current monitoring period. The replica consistency status is formed by reading the replica synchronization status record of the storage cluster. It is a publicly available and mature status detection information in storage management. This embodiment will not elaborate on the underlying implementation process of replica log replication.

[0062] Furthermore, storage layer load records are generated based on storage layer performance records.

[0063] The storage layer load record includes queue depth normalization value, capacity occupancy normalization value, tail latency normalization value, and remaining available capacity.

[0064] Specifically, the queue depth normalization value is formed based on the average unprocessed I / O queue depth of the storage layer in the current monitoring period, the tail latency normalization value is formed based on the 95th percentile access latency of the storage layer in the current monitoring period, and the capacity occupancy normalization value is formed based on the rated capacity and remaining available capacity of the storage layer.

[0065] When the corresponding normalized baseline value is greater than 0, the formulas for calculating the queue depth normalization value and the tail delay normalization value are expressed as follows: ; ;

[0066] The formula for calculating the normalized value of capacity occupancy is expressed as: ;

[0067] in, Indicates the storage layer number. Represents the storage layer In the The queue depth normalization value for each monitoring period. Represents the storage layer In the The tail delay normalization value for each monitoring period, Represents the storage layer In the Normalized value of capacity occupancy for each monitoring cycle Represents the storage layer In the Average unprocessed I / O queue depth over each monitoring period Represents the storage layer In the The first monitoring cycle Percentile access delay Represents the storage layer The queue depth normalization benchmark, Represents the storage layer Tail delay normalization benchmark, Represents the storage layer In the The remaining available capacity after deducting reserved capacity for each monitoring cycle Represents the storage layer The rated available capacity.

[0068] It should be noted that the queue depth normalization baseline is formed based on the recommended upper limit of queue depth in the storage controller interface specification and the running stress test records, and the value is greater than 0; the tail latency normalization baseline is formed based on the upper limit of tail latency in the storage layer service level agreement and the running statistics records, and the value is greater than 0; the rated available capacity is formed by reading the storage resource management records, and the value is greater than 0; the remaining available capacity and the object data volume are in the same unit, both expressed in bytes.

[0069] Furthermore, a migration quota record is generated based on the migration scheduling record. The migration quota record includes the maximum amount of data that can be migrated in the current monitoring period. , is represented as: ;

[0070] in, Indicates the first The maximum amount of data that can be migrated per monitoring cycle. Indicates the first The remaining migration bandwidth quota after deducting scheduled migration tasks within each monitoring period This indicates the duration of the monitoring period.

[0071] It should be noted that the monitoring period duration is set by the migration strategy file, with a value ranging from 5 seconds to 300 seconds; the remaining migration bandwidth quota is calculated from the migration scheduling record and the network bandwidth control record, and the value is not less than 0; when the remaining migration bandwidth quota is 0, the data volume limit is 0, and the migration quota record of the current monitoring period is retained as a valid record.

[0072] Furthermore, the normalized values ​​of read frequency, write frequency, access byte volume, access recentity, business timeliness level, current storage layer, object data volume, and replica consistency status are written into the data object load record according to the data object identifier and monitoring cycle number.

[0073] Write the queue depth, capacity occupancy, tail delay, and remaining available capacity into the storage layer load record according to the storage layer number and monitoring cycle number.

[0074] Write the maximum amount of data that can be migrated in the current monitoring period into the migration quota record according to the monitoring period number.

[0075] Furthermore, the data object load record, storage layer load record, and migration quota record together constitute the basic operational record for the current monitoring cycle.

[0076] S2. Based on the data object load record, calculate the current access popularity using a weighted fusion method that combines access frequency, access byte size, access recentity, and business timeliness level. Based on the storage layer load record, calculate the dynamic load coefficient using a weighted fusion method that combines queue depth, capacity usage, and tail latency.

[0077] Furthermore, based on the data object load record, the data object is retrieved. In the The reading frequency, writing frequency, number of bytes accessed, access recentity, and business timeliness level of each monitoring period are combined to form the access frequency.

[0078] Specifically, access frequency is expressed as: ;

[0079] in, Represents data objects In the Access frequency for each monitoring cycle This indicates the percentage of read frequencies.

[0080] It should be noted that the read frequency percentage is determined based on the supply chain business access statistics records, and the value ranges from 0 to 1. When the data object belongs to the read-intensive data corresponding to order query, inventory query, and logistics trajectory query, the read frequency percentage is close to 1. When the data object belongs to the write-sensitive data corresponding to order status writing, inventory deduction, and logistics node addition, the read frequency percentage is close to 0. When the data object does not form a clear read / write type in the business access statistics records, the read frequency percentage is 0.5.

[0081] Furthermore, based on access frequency, access byte volume, access recentity, and business timeliness level, a weighted fusion method is used to calculate the current access popularity, expressed as:

[0082] in, Represents data objects In the Current access popularity for each monitoring period Indicates access frequency weight. Indicates the weight of the number of bytes accessed. Indicates the weight of recent visits. This indicates the weight of the timeliness level of the business.

[0083] It should be noted that the access frequency weight, access byte volume weight, access relevance weight, and business timeliness level weight are formed offline based on historical supply chain access samples, business service level records, and storage access statistics records. The access frequency weight ranges from [0.30, 0.55], the access byte volume weight ranges from [0.15, 0.35], the access relevance weight ranges from [0.10, 0.30], and the business timeliness level weight ranges from [0.05, 0.30], and the sum of the four weights is 1.

[0084] Furthermore, the current access popularity is written to the popularity history cache and a corresponding relationship is established with the data object identifier, monitoring period number, and current storage layer.

[0085] Based on the storage layer load record, obtain the storage layer In the The queue depth, capacity usage, and tail latency for each monitoring period are calculated using a weighted fusion method to determine the dynamic load system.

[0086] Specifically, the dynamic load factor is expressed as: ;

[0087] in, Represents the storage layer In the Dynamic load factor for each monitoring cycle Indicates queue depth weight. Indicates the capacity occupancy weight. This represents the tail delay weight.

[0088] It should be noted that the queue depth weight, capacity occupancy weight, and tail latency weight are determined based on storage layer stress test records, latency constraints in the service level agreement, and operational statistics records. The common value range for queue depth weight is [0.30, 0.50], the common value range for capacity occupancy weight is [0.15, 0.35], and the common value range for tail latency weight is [0.25, 0.50], and the sum of the three weights is 1.

[0089] Furthermore, the dynamic load coefficient is written into the storage layer load analysis record according to the storage layer number and monitoring cycle number.

[0090] It should be noted that when any of the queue depth, capacity usage, or tail latency is missing from the storage layer performance record, the missing item will not be used in the calculation of the dynamic load coefficient for the current monitoring period, and the weights of the remaining valid items will be normalized according to their original weight ratios.

[0091] When queue depth, capacity utilization, and tail latency are all missing, the dynamic load coefficient of the previous monitoring period for the same storage layer is used, and the missing performance data status is recorded. At the same time, the number of consecutive missing performance data for the same storage layer is counted. When the number of consecutive missing performance data does not exceed 3, the storage layer is allowed to continue to participate in the dynamic load coefficient association and candidate target storage layer judgment. When the number of consecutive missing performance data exceeds 3, the storage layer is marked as a storage layer with unknown load status. Storage layers with unknown load status are not considered as candidate target storage layers for promotion or demotion. Data objects already located in storage layers with unknown load status are only allowed to be migrated out, not migrated in.

[0092] When there is no dynamic load factor from the previous monitoring period, the dynamic load factor is recorded as 0.5, indicating that the current storage layer load status is at an intermediate level.

[0093] Furthermore, after calculating the current access popularity and dynamic load coefficient, a hot / cold prediction input record is generated.

[0094] The cold and hot prediction input records include data object identifier, monitoring period number, current access popularity, current storage layer, dynamic load coefficient, and popularity history cache index.

[0095] S3. Based on the current access popularity, historical popularity cache, and dynamic load coefficient, the predicted popularity for the next period is obtained by combining historical baseline popularity with load trend correction. Based on the predicted popularity for the next period and the cold / hot migration threshold, the candidate storage layer for promotion, the candidate storage layer for demotion, and the candidate target storage layer are determined.

[0096] Furthermore, based on the cold and heat forecast input records, data objects are obtained. In the The system collects the current access popularity, current storage layer, and dynamic load coefficient of the current storage layer for each monitoring period, and retrieves the historical access popularity of the data object before the current monitoring period from the historical popularity cache to form a historical popularity sequence.

[0097] Specifically, historical access popularity is retrieved from the historical popularity cache according to the data object identifier and monitoring period number. The historical access popularity is arranged in chronological order, and the retrieval range is the consecutive periods before the current monitoring period. One monitoring cycle.

[0098] It should be noted that, This indicates the number of consecutive historical monitoring periods included in the historical baseline heat index calculation. The number of consecutive historical monitoring periods is determined by rounding up the ratio of the historical heat index statistical window duration to the monitoring period duration, with a value ranging from 3 to 2880. The historical heat index statistical window duration is configured in the migration strategy file according to the supply chain business timeliness level, with a value ranging from 15 minutes to 24 hours.

[0099] When the number of valid historical access popularity records in the popularity history cache is less than At any given time, a historical popularity sequence is formed based on the actual number of valid historical visits.

[0100] When the data object does not exist in the popularity history cache When determining the effective historical access popularity, the current access popularity is used as the historical baseline popularity.

[0101] Furthermore, data objects are calculated based on historical heat sequences. The historical baseline heat is expressed as: ;

[0102] in, Represents data objects In the Historical baseline heat for each monitoring period This indicates the number of valid historical periods used in the historical baseline heat calculation. Indicates the historical cycle offset number. Represents data objects In the Historical visit volume for each monitoring period.

[0103] It should be noted that the range of the number of valid historical periods is 1~ Among these, order payment, inventory deduction, warehouse inbound / outbound confirmation, and logistics anomaly handling correspond to high-efficiency business processes. The value range is 3~120; this applies to medium-time-efficiency services such as logistics tracking, after-sales work order processing, and settlement reconciliation. The value range is 121~960; historical archiving, audit retention, and offline analysis correspond to low-time-efficiency business. The value range is 961 to 2880.

[0104] When there is no valid historical access popularity of the current data object in the historical popularity cache, the calculation of the historical popularity average will no longer be performed. Instead, the current access popularity formed by the current data object in the current monitoring period will be used as the historical baseline popularity. The number of valid historical periods participating in the historical baseline popularity calculation will be determined as one monitoring period to ensure that newly created data objects or data objects entering the monitoring range for the first time can continue to participate in the prediction popularity calculation of the next period.

[0105] Furthermore, the change in popularity is calculated based on the current access popularity and the historical baseline popularity, and is expressed as follows: ;

[0106] in, Represents data objects In the Changes in heat intensity over a monitoring period.

[0107] Furthermore, based on the change in heat intensity and the dynamic load coefficient corresponding to the current storage layer, the load trend correction is calculated, expressed as: ;

[0108] in, Represents data objects In the Load trend correction amount for each monitoring period Represents data objects The current storage layer is in the first The dynamic load factor for each monitoring cycle.

[0109] When the change in heat is greater than 0, it indicates that the data object... The access popularity has increased relative to the historical baseline. The higher the current storage layer load, the greater the load trend correction.

[0110] When the change in temperature is less than When, it represents a data object. The access popularity has decreased relative to the historical baseline. The higher the current storage layer load, the smaller the load trend correction and the negative correction.

[0111] Furthermore, the predicted popularity for the next period is calculated based on the current access popularity, historical baseline popularity, and load trend correction, and is expressed as follows: ;

[0112] in, Represents data objects The predicted popularity in the next monitoring cycle, This indicates the current percentage of visits. This represents the load trend correction factor.

[0113] It should be noted that the current access popularity ratio is determined based on historical access replay tests, with a value range of [0,1]; the load trend correction coefficient is determined based on migration replay tests and storage layer stress test records, with a value range of [0,0.5].

[0114] Furthermore, the hot and cold migration thresholds are read from the migration strategy file. The hot and cold migration thresholds include a hot migration threshold and a cold migration threshold, and the hot migration threshold is greater than the cold migration threshold and both thresholds are between 0 and 1.

[0115] It should be noted that the hot migration threshold is determined based on the hot layer capacity constraint, the distribution of access popularity of high-timeliness services, and historical migration hit records, with a value range of [0.6, 0.95]; the cold migration threshold is determined based on the cold layer capacity constraint, the distribution of access popularity of low-access objects, and historical migration back records after demotion, with a value range of [0.05, 0.4]; a heat hysteresis interval is formed between the hot migration threshold and the cold migration threshold to reduce the probability of data objects repeatedly migrating near the hot and cold boundaries.

[0116] Historical migration hit records refer to migration records in the migration effect observation window that meet the migration effect judgment conditions after the data object completes the upgrade or downgrade migration according to the historical migration prediction list. The migration effect observation window is determined according to the business timeliness level, and the value range is 3 monitoring cycles to 120 monitoring cycles. High timeliness business uses 3 to 20 monitoring cycles, medium timeliness business uses 21 to 60 monitoring cycles, and low timeliness business uses 61 to 120 monitoring cycles.

[0117] For upgrade migration, when the migrated data object meets the following conditions within the migration effect observation window: the access latency is lower than the average access latency of the same access type before migration, the difference between the predicted heat of the next period and the heat matching value of the target storage layer is within the preset matching error range, no downgrade migration occurs, and the migration benefit value is positive, the corresponding migration record is recorded as an upgrade migration hit record. For downgrade migration, when the migrated data object does not experience upgrade migration back within the migration effect observation window, the access latency does not exceed the latency limit allowed by the corresponding business timeliness level, the predicted heat of the next period is not higher than the hot migration threshold, and the migration benefit value is positive, the corresponding migration record is recorded as a downgrade migration hit record.

[0118] The preset matching error range is configured by the migration strategy file, with a value range of 0.05 to 0.2. When a migrated data object experiences a short-term relocation within the migration effect observation window, the access delay exceeds the upper limit of the delay allowed by the corresponding business timeliness level, or the migration benefit value is not greater than 0, the corresponding migration record will be recorded as a missed migration record, and the migration trigger sensitivity under the same business domain and the same data object type will be reduced during subsequent threshold tuning.

[0119] Furthermore, the predicted heat for the next cycle is compared with the hot migration threshold and the cold migration threshold to determine the migration candidate status of the data object.

[0120] Specifically, when the predicted heat of the next cycle is not less than the hot migration threshold, the data object is marked as a candidate for promotion; when the predicted heat of the next cycle is not greater than the cold migration threshold, the data object is marked as a candidate for demotion; when the predicted heat of the next cycle is greater than the cold migration threshold but less than the hot migration threshold, the data object is marked as a retention object and the current storage layer is retained.

[0121] Furthermore, the target layer heat matching value for each storage layer is obtained from the storage layer configuration table.

[0122] Among them, the target layer thermal matching value of the thermal layer is higher than that of the warm layer, and the target layer thermal matching value of the warm layer is higher than that of the cold layer.

[0123] It should be noted that the target layer heat matching value is determined during the deployment phase based on storage media access latency, unit capacity cost, rated input / output performance, and supply chain business service level constraints. The heat matching value ranges from [0.7, 1] for the hot layer, from [0.3, 0.7] for the warm layer, and from [0, 0.3] for the cold layer.

[0124] Furthermore, for candidate storage layers, among those whose target layer heat matching value is higher than the current storage layer target layer heat matching value, the storage layer with the smallest difference from the predicted heat of the next cycle is selected as the candidate target storage layer. When there are two or more storage layers with the same difference from the predicted heat of the next cycle, the candidate target storage layers are determined in order of increasing dynamic load coefficient. When the dynamic load coefficient is still the same, the candidate target storage layers are determined in order of decreasing remaining available capacity. When the remaining available capacity is still the same, the candidate target storage layers are determined in order of increasing unit capacity cost. When the unit capacity cost is still the same, the candidate target storage layers are determined in order of storage layer number in the storage layer configuration table.

[0125] For candidate storage layers, the storage layer with the smallest difference from the predicted heat value of the next period is selected from among the storage layers whose target layer heat matching value is lower than that of the current storage layer target layer. When there are two or more storage layers with the same difference from the predicted heat value of the next period, the candidate target storage layers are determined in order of increasing dynamic load coefficient. When the dynamic load coefficient is still the same, the candidate target storage layers are determined in order of decreasing remaining available capacity. When the remaining available capacity is still the same, the candidate target storage layers are determined in order of increasing unit capacity cost. When the unit capacity cost is still the same, the candidate target storage layers are determined in order of storage layer number in the storage layer configuration table.

[0126] It should be noted that the dynamic load factor comes from the storage layer load analysis record, the remaining available capacity comes from the storage layer load record, the unit capacity cost comes from the storage layer configuration table, and the storage layer number comes from the storage layer configuration table. By determining the candidate target storage layer through the above priority order, it is possible to prioritize the storage layer with lower load pressure, more capacity margin, and lower migration cost when the degree of heat matching is the same, so as to avoid the candidate target storage layer being not unique.

[0127] When a candidate object for promotion is already in the storage layer with the highest heat matching value, the current data object is marked as a boundary-preserving object. Boundary-preserving objects are not included in the promotion migration prediction list, and the reason for boundary preservation is written in the cold and hot prediction result record.

[0128] Furthermore, a high thermal stress determination is performed on the boundary-preserving object. When the predicted heat of the boundary-preserving object in the next cycle is not lower than the thermal migration threshold in several consecutive monitoring cycles, and the dynamic load coefficient of the current storage layer reaches the high load determination threshold of the thermal layer, a boundary high-heat handling record is generated. The number of consecutive monitoring cycles is configured by the migration strategy file, and the value ranges from 2 to 5 monitoring cycles. The high load determination threshold of the thermal layer is formed by the access latency constraint in the service level agreement and the storage layer stress test record, and the value ranges from 0.75 to 0.90.

[0129] Furthermore, the boundary high-heat handling record includes data object identifier, current storage layer, predicted heat for the next cycle, dynamic load coefficient, number of consecutive high-heat cycles, reason for boundary maintenance, and handling suggestion identifier; the handling suggestion identifier includes read cache preheating identifier, replica expansion suggestion identifier, and hot layer load alarm identifier.

[0130] It should be noted that the read cache preheating flag is used to prompt the caching service to establish a read cache before the peak of business access corresponding to the current data object; the replica expansion suggestion flag is used to prompt the replica management module to add readable replicas or expand the read request carrying capacity of the current data object; the hot layer load alarm flag is used to prompt the operation and maintenance management module to pay attention to the load pressure of the storage layer with the highest heat matching value. The boundary high heat handling record does not change the cold and hot migration direction of the current data object, nor is it directly included in the upgrade migration prediction list, but rather serves as a supplementary handling basis when the current data object is already in the highest storage layer but still experiences continuous high heat pressure.

[0131] When a candidate object for demotion is already in the storage layer with the lowest heat matching value, the data object is marked as a boundary-preserving object.

[0132] Furthermore, the data object identifier, monitoring period number, current access popularity, historical baseline popularity, popularity change, load trend correction, next period predicted popularity, migration candidate status, current storage layer, and candidate target storage layer are written into the hot and cold prediction result record.

[0133] S4. Calculate the migration benefit value based on the candidate objects for promotion, candidate objects for demotion, candidate target storage layer, object data volume, replica consistency status, remaining available capacity, migration quota records, and dynamic load coefficient, and form a migration prediction list according to the migration benefit value.

[0134] Furthermore, based on the hot and cold prediction results, the migration candidate status, current storage layer, candidate target storage layer, predicted heat for the next period, and business timeliness level of the data object are obtained. Based on the data object load record, the object data volume and replica consistency status are obtained. Based on the storage layer load record, the remaining available capacity of the candidate target storage layer is read. Based on the migration quota record, the upper limit of the amount of data allowed to be migrated in the current monitoring period is obtained. Based on the storage layer load analysis record, the dynamic load coefficients corresponding to the current storage layer and the candidate target storage layer are obtained respectively.

[0135] Furthermore, candidate states are filtered for data objects in the cold and heat prediction results records, retaining candidates for upgrading and downgrading, and removing objects that are kept in place and those that are kept at the boundary.

[0136] Among them, the "maintained object" refers to the data object whose predicted popularity in the next period is between the cold migration threshold and the hot migration threshold, and the "boundary maintained object" refers to the candidate object for promotion that is already in the storage layer with the highest popularity matching value, or the candidate object for demotion that is already in the storage layer with the lowest popularity matching value.

[0137] Furthermore, the candidate target storage layer is read from the retained promotion and demotion candidate objects. The target layer heat matching value is used to form a business direction matching value based on the migration candidate status, as follows:

[0138] in, Represents data objects In the Business direction matching value for each monitoring period Represents data objects The business timeliness level.

[0139] It should be noted that the range of the business direction matching value is [0,1]. For candidates for upgrading, the higher the business timeliness level, the larger the business direction matching value. For candidates for downgrading, the lower the business timeliness level, the larger the business direction matching value.

[0140] Furthermore, when the maximum amount of data that can be migrated in the current monitoring period is greater than 0, the migration benefit value is calculated based on the predicted heat of the next period, the target layer heat matching value, the business direction matching value, the current dynamic load coefficient of the storage layer, the dynamic load coefficient of the candidate target storage layer, the amount of object data, the migration quota record, and the replica consistency status, and is expressed as: ;

[0141] in, Represents data objects In the Each monitoring cycle migrates to the candidate target storage layer. The migration benefit value, Represents data objects In the The replica's consistent status over each monitoring period. Indicates the matching weight based on popularity. This indicates the matching weight of business direction. Indicates the load release weight. Indicates the migration cost weight. Represents data objects Current storage layer In the Dynamic load factor for each monitoring cycle Indicates candidate target storage layer In the Dynamic load factor for each monitoring cycle Represents data objects The amount of object data, Indicates the first The maximum amount of data that can be migrated per monitoring cycle.

[0142] It should be noted that the four weights are determined based on the service level agreement, storage layer cost records, historical migration time records, and migration hit records, and their values ​​range from 0 to 1, and the sum of the four weights is 1.

[0143] The common value ranges for the heat matching weight are [0.30, 0.50], the common value ranges for the business direction matching weight are [0.10, 0.30], the common value ranges for the load release weight are [0.15, 0.35], and the common value ranges for the migration overhead weight are [0.10, 0.30]. The four weights are adjusted within their respective ranges according to the service level requirements of the current supply chain business, the storage layer load status, and the migration bandwidth usage, and the sum of the four weights is kept to be 1.

[0144] Specifically, when the system prioritizes ensuring that hot data hits the target storage layer, the example weight for hot matching is 0.45, the example weight for business direction matching is 0.20, the example weight for load release is 0.20, and the example weight for migration overhead is 0.15.

[0145] When the system prioritizes alleviating the load pressure on the current storage layer, the example weight for heat matching is 0.35, the example weight for business direction matching is 0.15, the example weight for load release is 0.35, and the example weight for migration overhead is 0.15.

[0146] When the system migration bandwidth is limited, or the migration window duration in the migration scheduling record is between 5 minutes and less than 30 minutes, the example weight for heat matching is 0.35, the example weight for business direction matching is 0.15, the example weight for load release is 0.20, and the example weight for migration overhead is 0.30. These example values ​​are used to increase the impact of migration overhead on migration benefits, and to prevent data migration tasks within the current migration window from consuming business bandwidth or failing to complete the access direction switch due to excessive object data volume.

[0147] It should also be noted that when the maximum amount of data allowed to be migrated is 0, no migration prediction list to be executed in the current monitoring period is generated, and insufficient migration quota is recorded in the migration prediction results. At the same time, for data objects that have formed upgrade or downgrade candidates, the candidate benefit ranking value is still calculated based on the predicted popularity of the next period, the candidate target storage layer, the business direction matching value, the dynamic load coefficient, and the object data volume, and data objects with a candidate benefit ranking value greater than 0 are written into the delayed migration candidate record.

[0148] Among them, the candidate records for delayed migration include data object identifier, current storage layer, candidate target storage layer, migration direction, predicted popularity in the next cycle, candidate revenue ranking value, object data volume, reason for delay, and time of delay; the reasons for delay include insufficient migration quota and unavailable migration window.

[0149] Specifically, when the migration prediction list is generated in the next monitoring period, the candidate records for delayed migration are read first, and the data objects in the candidate records for delayed migration are re-verified for the predicted popularity, replica consistency status, remaining available capacity of the candidate target storage layer, and the upper limit of the amount of data allowed to be migrated in the current monitoring period. Data objects that pass the verification participate in the migration benefit value ranking in the next monitoring period, and data objects that fail the verification are removed from the candidate records for delayed migration, and the reason for removal is written.

[0150] When the replica consistency status is 0, the current data object is not written into the migration prediction list, and the replica inconsistency is recorded in the reason for non-migration; when the replica consistency status of the current data object is restored to meet the migration conditions in a subsequent monitoring period, the migration benefit value is recalculated based on the data object load record, storage layer load record and hot / cold prediction result record of the current monitoring period.

[0151] Furthermore, the candidate data objects are sorted in descending order according to their migration benefit values.

[0152] Specifically, when two candidate data objects have the same migration benefit value, they are sorted by the size of the data objects from smallest to largest; when both the migration benefit value and the size of the data objects are the same, they are sorted by the lexicographical order of the data object identifiers to obtain the candidate migration ranking result.

[0153] Furthermore, admission checks are performed sequentially according to the candidate migration ranking results. These checks include replica consistency status checks, candidate target storage layer capacity checks, current monitoring period migration quota checks, and migration benefit checks. Specifically, the replica consistency status check requires that the replica consistency status meets the migration conditions; the candidate target storage layer capacity check requires that the amount of object data is not greater than the remaining available capacity of the candidate target storage layer during the ranking check process; the current monitoring period migration quota check requires that the sum of the total amount of object data already written into the migration prediction list and the current object data amount is not greater than the upper limit of the amount of data allowed to be migrated in the current monitoring period; and the migration benefit check requires that the migration benefit value is greater than zero.

[0154] Specifically, when the current candidate data object fails the replica consistency status check or migration benefit check, the current candidate data object is marked as an unmigrated object, and the admission check continues for the next candidate data object in the candidate migration ranking result; when the current candidate data object fails the candidate target storage layer capacity check or the current monitoring period migration quota check, the current candidate data object is written into the delayed migration candidate record, and the admission check continues for the next candidate data object in the candidate migration ranking result, so that the remaining capacity and remaining migration quota in the current monitoring period can continue to be used for subsequent data objects that meet the conditions.

[0155] The candidate records for delayed migration include the data object identifier, current storage layer, candidate target storage layer, migration direction, predicted popularity for the next cycle, migration benefit value, object data volume, failed checks, and delay time. When subsequent candidate data objects pass the replica consistency status check, candidate target storage layer capacity check, current monitoring cycle migration quota check, and migration benefit check, the subsequent candidate data objects are written into the migration prediction list, and the remaining available capacity of the candidate target storage layer during the sorting check process and the remaining migration quota for the current monitoring cycle are deducted simultaneously.

[0156] After the admission checks of all candidate data objects are completed in the current monitoring period, the migration prediction list and the postponement migration candidate record are retained. The migration prediction list is used for data migration execution in the current monitoring period, and the postponement migration candidate record is used to participate in the migration benefit ranking after re-verifying the prediction popularity, replica consistency status, remaining available capacity of candidate target storage layer and migration quota record in the next monitoring period.

[0157] It should be noted that the above-mentioned admission check adopts a deterministic strategy of prioritizing revenue ranking, skipping those with insufficient quotas, and continuing to check subsequent objects. This is to ensure that the process of forming the migration prediction list is reproducible, while avoiding the blockage of multiple small and medium-sized objects from entering the migration prediction list due to the large amount of data of a single large object. Large objects that are skipped will not be directly discarded, but will be written into the postponement migration candidate record and will continue to participate in the migration judgment after being re-verified in the next monitoring cycle.

[0158] Furthermore, for data objects that pass the admission check, the data object identifier, current storage layer, candidate target storage layer, migration direction, predicted popularity for the next cycle, migration benefit value, object data volume, and execution priority are written into the migration prediction list. After writing, the remaining available capacity of the candidate target storage layer during the sorting check process and the remaining migration quota for the current monitoring cycle are deducted.

[0159] The execution priority is generated according to the order in which candidate data objects are written into the migration prediction list.

[0160] Furthermore, data objects that fail the access checks will not be written into the migration prediction list, and reasons for non-migration will be generated based on the failed check items.

[0161] Among them, when the replica consistency check fails, the reason for not migrating is recorded as replica inconsistency; when the candidate target storage layer capacity check fails, the reason for not migrating is recorded as insufficient target storage layer capacity; when the current monitoring period migration quota check fails, the reason for not migrating is recorded as insufficient migration quota; when the migration benefit check fails, the reason for not migrating is recorded as insufficient migration benefit.

[0162] Furthermore, after completing the sorting and checking of all candidate data objects, a migration prediction list and a record of reasons for non-migration are generated for the current monitoring period.

[0163] S5. According to the migration prediction list, perform shard copying, incremental write record playback, data digest verification, and access direction switching on the data objects to obtain migration records including migration direction, metadata version number, data digest verification result, and access direction switching result.

[0164] Furthermore, based on the migration prediction list, the data object identifier, current storage layer, candidate target storage layer, migration direction, object data volume, migration benefit value, and execution priority of the data objects are obtained, and the migration order of the data objects in the current monitoring period is determined according to the execution priority from high to low.

[0165] Furthermore, for each data object in the migration prediction list, the metadata version number and current access pointer are obtained in real time from the data object metadata directory before the migration begins.

[0166] Among them, the real-time read metadata version number represents the metadata version number of the data object at the start of a monitoring cycle migration, and the real-time read current access pointer represents the access pointer of the data object to the current storage layer at the start of a monitoring cycle migration.

[0167] It should be noted that the metadata version number and the current access pointer are not directly inherited from the migration prediction list, but are read in real time from the data object metadata directory based on the data object identifier at the start of the migration, so as to avoid version deviation between the time when the migration prediction list is formed and the time when the migration is executed; when the metadata version number fails to be read, the current data object does not perform sharded replication, and a metadata read failure is written in the migration record.

[0168] Furthermore, the data objects are calculated based on the object data volume and the migration shard size. The number of fragments is expressed as: ;

[0169] in, Represents data objects The number of fragments, Represents data objects The size of the migration fragment during the migration process. Represents data objects The amount of object data.

[0170] It should be noted that the migration fragment size is determined based on the migration policy file, the storage layer read / write block size, and the network bandwidth control record, with a value range of 4MB to 1024MB.

[0171] When the object data volume is less than the migration shard size and the object data volume is greater than 0, the shard quantity is 1, and data shard replication is performed as a single shard. When the object data volume is 0, the current data object is treated as an empty object for migration. The empty object still participates in the migration benefit ranking and admission check, but the corresponding object data volume is recorded as 0 in the migration cost calculation, and the upper limit of the data volume allowed for migration in the current monitoring period is not deducted.

[0172] When performing migration processing on empty objects, data sharding and incremental write record replay are not performed. Instead, an empty object metadata record is created in the candidate target storage layer. The empty object metadata record includes the data object identifier, object type, candidate target storage layer, metadata version number, access permission identifier, and creation time. After the empty object metadata record is created, metadata version verification and access pointer switching are performed.

[0173] When the empty object metadata record is successfully created and the metadata version verification passes, the access pointer is switched from the current access pointer read at the start of the migration to the target access pointer corresponding to the candidate target storage layer, and the data digest verification result is recorded as passed in the migration record; when the empty object metadata record creation fails or the metadata version verification fails, the access pointer remains the current access pointer read at the start of the migration, and the reason for the empty object metadata creation failure or the metadata version verification failure is written in the migration record.

[0174] It should be noted that empty object migration does not consume the maximum amount of data allowed to be migrated in the current monitoring period, but it does occupy one migration execution slot. The migration execution slot comes from the migration scheduling record and is used to limit the number of metadata switching tasks that can be executed within the current monitoring period, avoiding the pressure on the metadata directory to be written due to a large number of empty objects being migrated at the same time. After an empty object enters the migration prediction list, its execution priority is still determined according to the migration benefit value, ensuring that the empty object migration record, access to the switching result, and the storage layer metadata status are consistent.

[0175] Furthermore, the data objects are sorted in ascending order of their fragment numbers. Data shards in the current storage layer are copied to the candidate target storage layer. Before copying each shard, the metadata version number of the current data object is checked to see if it has changed. If the metadata version number changes, the copying of the current shard is paused and the newly written records during the version change period are read.

[0176] Furthermore, during the sharding replication process, new writes occurring after the metadata version number are generated into incremental write records based on the write sequence number, write offset, write length, and write content. These incremental write records are then associated with data object identifiers and candidate target storage layers.

[0177] The write sequence number is formed by reading from the storage access log or transaction commit log, the write offset and write length are formed by reading from the storage access counter or object write log, and the write content is formed by reading from the committed data in the current storage layer.

[0178] It should be noted that incremental write records only record writes that have been successfully committed after the migration baseline version, and do not record cached content of uncommitted transactions; when there are multiple writes in the same offset interval, the write order is preserved in ascending order of write sequence number, and the last committed write is used as the final content of the corresponding offset interval of the candidate target storage layer.

[0179] Furthermore, after the fragmented replication is completed, the incremental write records are replayed to the candidate target storage layer in ascending order of write sequence number, and the latest write sequence number is read at the end of each round of replay. When new writes continue to occur during the replay, the new writes are appended to the incremental write records and the next round of incremental replay begins.

[0180] The incremental replay is set to have a maximum number of replay rounds, which is configured by the migration strategy file and ranges from 2 to 5 times. When the incremental replay is completed within the maximum number of replay rounds, and the content of the data object in the candidate target storage layer has covered all committed writes corresponding to the latest write sequence number, the data digest verification step is initiated.

[0181] Specifically, when there are still new writes after the incremental replay has reached the maximum number of replay rounds, the switchover preparation phase begins. During the switchover preparation phase, a write barrier is set for the current data object, newly arriving write requests are temporarily stored in the switchover buffer queue, and the last round of incremental replay is performed on the committed writes after the write barrier is established, so that the content of the data object in the candidate target storage layer covers all committed writes before the write barrier was established.

[0182] The write barrier retention time is determined by the service level agreement and migration strategy file, with a value range of 100 milliseconds to 5 seconds. When the last round of incremental replay is completed within the write barrier retention time, data digest verification and access pointer switching are performed. After the access pointer switching is completed, the write barrier is released, and the write requests in the switching buffer queue are directed to the candidate target storage layer.

[0183] Furthermore, if the last round of incremental replay fails to complete within the write fence hold time, or if the number of write requests in the switching buffer queue exceeds the buffer limit configured in the migration strategy file, the migration of the current data object is paused, the access pointer remains the current access pointer read at the start of the migration, the replica in the candidate target storage layer is marked as a replica to be cleaned up, and the reason for the migration not being completed is written in the migration record; among which, the reasons for the migration not being completed include incremental replay not converging, write fence timeout, and switching buffer queue overflow.

[0184] It should be noted that the maximum number of replay rounds is used to avoid infinite loops of incremental replay in continuous write scenarios, the write barrier is used to form a definite write cutoff boundary before the access pointer is switched, and the switching buffer queue is used to temporarily store new write requests that arrive during the switching preparation phase. By constraining the maximum number of replay rounds, the write barrier hold time, and the upper limit of the switching buffer queue, it can be ensured that the incremental write record replay process has definite completion conditions and failure exit conditions.

[0185] Furthermore, fragment digests are calculated for data objects in the current storage layer and the candidate target storage layer, and an object digest value is formed based on each fragment digest, represented as:

[0186] in, Represents data objects In the Data summary values ​​for each monitoring period, Represents data objects The Each fragment summary value, Indicates the fragment number. This represents the function for calculating the summary. This indicates that the pieces are spliced ​​together according to their segment numbers.

[0187] It should be noted that the digest calculation function uses the SHA-256 digest algorithm to perform digest calculations on the fragmented content of the data object; the current storage layer and the candidate target storage layer use the same digest algorithm, the same fragment boundaries, and the same fragment sequence number to perform digest calculations, ensuring that the data digest values ​​of the current storage layer and the candidate target storage layer are comparable.

[0188] Furthermore, the data digest value of the current storage layer is compared with the data digest value of the candidate target storage layer.

[0189] Before performing data digest verification, a consistency snapshot of the current data object is created for switching preparation, and the digest verification snapshot version number is recorded. The digest verification snapshot version number is determined by the metadata version number read at the start of the migration, the latest write sequence number when the incremental write record playback is completed, and the write barrier establishment time. It is used to limit the current storage layer and the candidate target storage layer to use the same data version when performing digest calculation.

[0190] After the digest verification snapshot version number is determined, a digest is calculated for the data object content in the current storage layer up to the digest verification snapshot version number to obtain the data digest value of the current storage layer; a digest is calculated for the data object content in the candidate target storage layer up to the same digest verification snapshot version number to obtain the data digest value of the candidate target storage layer; the current storage layer and the candidate target storage layer use the same digest algorithm, the same fragmentation boundary, and the same fragmentation sequence for digest calculation.

[0191] When the current storage layer data digest value matches the candidate target storage layer data digest value, and the incremental write record has been replayed to the digest verification snapshot version number, the data digest verification result is recorded as passed; when the current storage layer data digest value does not match the candidate target storage layer data digest value, or the incremental write record has not been replayed to the digest verification snapshot version number, the data digest verification result is recorded as failed, the access pointer of the data object remains the current access pointer read at the start of the migration, and the replica in the candidate target storage layer is marked as a verification failed replica.

[0192] When the data digest verification result fails, a digest review is performed first. The digest review uses the same digest verification snapshot version number to recalculate the current storage layer data digest value and the candidate target storage layer data digest value. When the digest review passes, the data digest verification result is corrected to pass and the access direction switching step is initiated. When the digest review still fails, the verification failure copy in the candidate target storage layer is marked as a copy to be cleaned up. It will not be used as a business access copy, nor as the basis for continued incremental verification in the next monitoring cycle.

[0193] The copy to be cleaned is retained for fault verification during the cleanup waiting time, which is configured by the migration policy file and ranges from 5 minutes to 24 hours. After the cleanup waiting time is exceeded, the copy to be cleaned is deleted from the candidate target storage layer, and the deletion result of the copy to be cleaned is written into the migration record.

[0194] When the same data object fails data digest verification three times consecutively in the same candidate target storage layer, a migration execution anomaly alarm is generated. When the number of data objects that fail data digest verification in the same candidate target storage layer within the same monitoring period reaches the anomaly number threshold configured in the migration strategy file, a target storage layer write anomaly alarm is generated, and the candidate target storage layer is suspended from being used as a new migration target layer.

[0195] It should be noted that the snapshot version number for summary verification is used to ensure that the current data summary value of the storage layer and the data summary value of the candidate target storage layer are calculated based on the same consistency moment; the failed verification replica is only used for fault review and is not open to business access; the replica to be cleaned up does not participate in the migration benefit calculation and incremental replay reuse in the next monitoring cycle. When the same data object needs to be migrated in the next monitoring cycle, the metadata version number is read again from the current storage layer and the sharding replication, incremental write record replay and data summary verification are re-executed.

[0196] Furthermore, when the data digest verification result is passed, the access pointer is switched for the data object.

[0197] Specifically, before performing the access target switching, the migration execution service obtains the switching lease of the current data object and marks the current data object as a switching ready state; the switching lease is used to ensure that only one access target switching task exists for the same data object at the same time.

[0198] Furthermore, in the switchover preparation state, business read requests still access the current storage layer according to the current access pointer read at the start of the migration, while business write requests enter the switchover buffer queue. When the data digest verification result is passed and the committed writes in the switchover buffer queue have been replayed to the candidate target storage layer, the access pointer switch is performed.

[0199] Furthermore, the access pointer switching is performed using a metadata version comparison and exchange method. When the current metadata version number has a version inheritance relationship with the metadata version number read at the start of the migration, and the current access pointer has not changed, the access pointer is atomically switched to the target access pointer pointing to the candidate target storage layer, and a new metadata version number is generated.

[0200] It should be noted that the metadata version comparison and exchange method is used to ensure that the access pointer is successfully switched only once. After the access pointer is successfully switched, subsequent business read and write requests are routed to the candidate target storage layer. When the access pointer fails to switch, business read and write requests continue to be routed to the current storage layer. The replicas in the candidate target storage layer are not open to business access, and the reason for the switching failure is written in the migration record.

[0201] Furthermore, when an unreplayed new write is detected during the access pointer switching, the access pointer switching is paused, the new write is reread and replayed to the candidate target storage layer, and the data digest verification is performed again. If the verification passes again, the access pointer switching is performed. If the verification fails again, the current access pointer read at the start of the migration is retained, the replica in the candidate target storage layer is marked as a verification failure replica, and the reason for the access pointer switching failure is written in the migration record.

[0202] Furthermore, the failed verification copy is not open to business access and is deleted after the cleanup waiting time expires; the cleanup waiting time is configured by the migration strategy file and ranges from 5 minutes to 24 hours.

[0203] Furthermore, when the number of consecutive access switching failures for the same data object reaches 3, the current data object is written into the migration suppression record, and automatic migration is suspended or the migration benefit value is reduced for the next 3 to 10 monitoring cycles.

[0204] Furthermore, after completing the migration process of the data objects, a migration record is generated. The migration record includes the data object identifier, migration direction, current storage layer, candidate target storage layer, metadata version number, object data volume, number of shards, migration start time, migration end time, data digest verification result, access pointer switching result, and reason for non-migration.

[0205] It should be noted that migration records, as the migration execution results of the current monitoring period, are no longer included in the formation of the migration prediction list for the current monitoring period. Migration records not only record successful migration results but also migration failures, summary verification failures, access pointer switching failures, delayed migrations, copies awaiting cleanup, and migration suppression status. The migration direction in the migration record represents the actual execution direction of upgrade and downgrade migrations; the metadata version number is used to trace the data version at the time of migration execution; the data summary verification result records the consistency verification status between the candidate target storage layer copy and the current storage layer copy; and the access pointer switching result represents whether the business access path has been switched. When a migration is not successfully completed, the migration record also includes the reason for failure, the reason for delay, the identifier of copies awaiting cleanup, and the identifier for review in the next monitoring period.

[0206] In this embodiment, Figure 5 This paper compares the 95th percentile access latency changes of conventional hot / cold migration methods and hot / cold data migration prediction methods based on dynamic load analysis over 24 consecutive monitoring periods. As shown in the figure, the access latency of the conventional hot / cold migration method exhibits significant peaks across multiple monitoring periods, indicating that migration judgment is prone to lag behind storage layer load changes when generating migration tasks solely based on static hot / cold thresholds or historical access statistics. This invention calculates the current access heat using data object load records, calculates the dynamic load coefficient using storage layer load records, and combines this with historical heat cache to obtain the predicted heat for the next period. This allows migration prediction to reflect queue depth, capacity occupancy, and tail latency changes in advance. After forming a migration prediction list based on these prediction results, the matching between data objects and candidate target storage layers can better reflect the current operating state, thereby reducing tail latency fluctuations during high-load periods and achieving coordinated control between hot / cold data migration prediction and dynamic load changes.

[0207] In this embodiment, Figure 6This paper compares the changes in the proportion of effective migration tasks between the conventional cold / hot migration method and the cold / hot data migration prediction method based on dynamic load analysis over 24 consecutive monitoring periods. As shown in the figure, the proportion of effective migration tasks decreased significantly in some monitoring periods using the conventional cold / hot migration method, indicating that candidate migration tasks may have issues such as insufficient target storage layer capacity, unmet replication consistency requirements, insufficient migration quota, or unclear migration benefits. This invention, after determining candidate upgrade / downgrade targets and candidate target storage layers, further calculates migration benefit values ​​by combining object data volume, replication consistency status, remaining available capacity, migration quota records, and dynamic load coefficients. A migration prediction list is then formed based on admission checks and migration benefit values. This process reduces the number of migration tasks that do not meet execution conditions from entering the list, making the migration prediction results more consistent with on-site scheduling constraints, thereby improving the executability of the migration prediction list and the rationality of migration task sequencing.

[0208] In summary, this invention achieves joint analysis of the hotness / coldness of data objects and the real-time load status of the storage layer by: using a weighted fusion calculation of access frequency, access byte volume, access recentity, and business timeliness level to calculate the current access heat; and using a weighted fusion calculation of queue depth, capacity occupancy, and tail latency to calculate the dynamic load coefficient. This improves the adaptability of hot / cold data migration judgment to changes in business access and storage operation pressure. By combining the current access heat, historical heat cache, and dynamic load coefficient, and using a combination of historical baseline heat and load trend correction to obtain the predicted heat for the next period, this invention enables advance judgment of the future hot / coldness trend of data objects, reducing the lag caused by relying solely on the current access heat or fixed historical statistical results for migration prediction.

[0209] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for predicting the migration of hot and cold data based on dynamic load analysis, characterized in that, include: By collecting data access events, storage access counters, link tracing records, data object metadata catalogs, storage layer performance records, and migration scheduling records generated during the monitoring period of digital service supply chain collaborative business, we can obtain data object load records including read frequency, write frequency, access bytes, access recentity, business timeliness level, current storage layer, object data volume, and replica consistency status. We can also obtain storage layer load records and migration quota records including queue depth, capacity usage, tail latency, and remaining available capacity. Based on the data object load records, the current access popularity is calculated by weighting and combining access frequency, access byte volume, access recentity, and business timeliness level. Based on the storage layer load records, the dynamic load coefficient is calculated by weighting and combining queue depth, capacity usage, and tail latency. Based on the current access popularity, historical popularity cache, and dynamic load coefficient, the predicted popularity for the next period is obtained by combining historical baseline popularity with load trend correction. Based on the predicted popularity for the next period and the cold / hot migration threshold, the candidate storage layer for promotion, the candidate storage layer for demotion, and the candidate target storage layer are determined. Based on the candidate objects for upgrading, candidate objects for downgrading, candidate target storage layer, object data volume, replica consistency status, remaining available capacity, migration quota records, and dynamic load coefficient, the migration benefit value is calculated, and a migration prediction list is formed according to the migration benefit value. According to the migration prediction list, the data objects are sharded and copied, incrementally written and replayed, data digests are verified and access points are switched, and migration records including migration direction, metadata version number, data digest verification results and access point switching results are obtained.

2. The method for predicting cold and hot data migration based on dynamic load analysis as described in claim 1, characterized in that, The process involves collecting data access events, storage access counters, link tracing records, data object metadata directories, storage layer performance records, and migration scheduling records generated during the monitoring period of digital service supply chain collaborative business. This data collection yields data object load records including read frequency, write frequency, access bytes, access recentity, business timeliness level, current storage layer, object data volume, and replica consistency status. During the monitoring period, data access events, storage access counters, and link tracing records are associated with data object identifiers and transaction identifiers to obtain the number of read operations, the number of write operations, the cumulative number of bytes accessed, the most recent access timestamp, and the business service identifier for each data object. Based on the number of read operations, the number of write operations, and the cumulative number of bytes accessed for each data object, normalization is performed according to the historical statistical benchmark of similar data objects to obtain the read frequency, write frequency, and number of bytes accessed. The access recentity is calculated based on the time interval between the most recent access timestamp and the end time of the current monitoring period, and the business timeliness level is matched from the supply chain business catalog and service level agreement based on the business service identifier. The current storage layer and data volume of each data object are read from the data object metadata directory, and the replica consistency status is determined based on the metadata version number, primary replica log sequence number and readable replica log sequence number in the data object metadata directory. The read frequency, write frequency, number of bytes accessed, access recentity, business timeliness level, current storage layer, object data volume, and replica consistency status are written according to the data object identifier and monitoring cycle number to form a data object load record.

3. The method for predicting cold and hot data migration based on dynamic load analysis as described in claim 1, characterized in that, The acquisition of storage layer load records and migration quota records, including queue depth, capacity usage, tail latency, and remaining available capacity, includes: Based on the storage layer performance records, read the average pending input / output queue depth, rated available capacity, current occupied capacity, 95th percentile access latency, and remaining available capacity after deducting reserved capacity for each storage layer in the current monitoring period; The queue depth is calculated based on the average unprocessed input / output queue depth and the queue depth normalization benchmark, and the tail delay is calculated based on the 95th percentile access delay and the tail delay normalization benchmark. The capacity occupancy is calculated based on the ratio between the current occupied capacity and the rated available capacity. The queue depth, capacity occupancy, tail delay, and remaining available capacity are written according to the storage layer number and monitoring cycle number to form a storage layer load record. Read the remaining migration bandwidth quota after deducting the scheduled migration tasks within the current monitoring period based on the migration scheduling record, and obtain the monitoring period duration of the current monitoring period; The remaining migration bandwidth quota and monitoring period duration are converted into the maximum amount of data that can be migrated in the current monitoring period, and written according to the monitoring period number to form a migration quota record.

4. The method for predicting cold and hot data migration based on dynamic load analysis as described in claim 2, characterized in that, The calculation of current access popularity based on data object load records, using a weighted fusion method of access frequency, access byte volume, access recentity, and business timeliness level, includes: Read the data object's read frequency, write frequency, number of bytes accessed, access recentity, business timeliness level, and current storage layer from the data object load record during the current monitoring period; Based on the read and write ratios corresponding to the read and write frequencies, the read and write frequencies are fused to obtain the access frequency of the data object within the current monitoring period. Based on the popularity weights corresponding to access frequency, access byte volume, access recentity, and business timeliness level, the access frequency, access byte volume, access recentity, and business timeliness level are weighted and integrated to obtain the current access popularity of the data object in the current monitoring period. Write the current access popularity into the popularity history cache according to the data object identifier and monitoring period number, and establish a correspondence between the current access popularity and the current storage layer of the data object; Based on the data object identifier, monitoring period number, current storage layer, and current access popularity, generate an object popularity record for calculating the predicted popularity of the next period.

5. The method for predicting cold and hot data migration based on dynamic load analysis as described in claim 3, characterized in that, The dynamic load coefficient is calculated based on the storage layer load record, using a weighted fusion method that considers queue depth, capacity occupancy, and tail latency. Read the queue depth, capacity usage, tail latency, and storage layer number of the storage layer in the current monitoring period from the storage layer load record; Based on the load weights corresponding to the queue depth, capacity usage, and tail latency of the storage layer, determine the set of load indicators that participate in dynamic load calculation within the current monitoring period. The queue depth, capacity usage, and tail latency are weighted and fused to obtain the dynamic load coefficient of the storage layer in the current monitoring period. Write the dynamic load coefficient into the storage layer load analysis record according to the storage layer number and monitoring cycle number; Based on the current storage layer of the data object, the data object is associated with the dynamic load coefficient of the corresponding storage layer to generate a storage layer load association record for calculating the predicted heat of the next period.

6. The method for predicting cold and hot data migration based on dynamic load analysis as described in claim 4 or 5, characterized in that, The process of obtaining the predicted popularity for the next period based on current access popularity, historical popularity cache, and dynamic load coefficient, using a combination of historical baseline popularity and load trend correction, includes: Based on the data object identifier and monitoring period number, retrieve the historical access popularity of the data object before the current monitoring period from the historical popularity cache to form a historical popularity sequence; Based on the number and value of valid historical access popularity in the historical popularity sequence, calculate the historical baseline popularity of the data object in the current monitoring period; Calculate the change in popularity of the data object during the current monitoring period based on the difference between the current access popularity and the historical baseline popularity; Based on the current storage layer of the data object, obtain the dynamic load coefficient of the corresponding storage layer in the current monitoring period, and calculate the load trend correction amount based on the heat change and the dynamic load coefficient. The predicted popularity for the next period is calculated based on the current access popularity, historical baseline popularity, and load trend correction, and the predicted popularity for the next period is limited to between 0 and 1.

7. The method for predicting cold and hot data migration based on dynamic load analysis as described in claim 6, characterized in that, The step of determining candidate storage layers for promotion, candidate storage layers for demotion, and candidate target storage layers based on the predicted heat and cold migration thresholds for the next cycle includes: Obtain the hot migration threshold and cold migration threshold from the migration strategy file; The predicted heat of the next cycle is compared with the heat migration threshold. If the predicted heat of the next cycle is not less than the heat migration threshold, the data object is identified as a candidate for upgrading. The predicted heat of the next period is compared with the cold migration threshold. If the predicted heat of the next period is not greater than the cold migration threshold, the data object is identified as a candidate for downgrading. When the predicted heat in the next cycle is greater than the cold migration threshold but less than the hot migration threshold, the data object is identified as a retained object and its current storage layer is retained. Read the target layer heat matching value of each storage layer from the storage layer configuration table. For upgrade candidates, determine the candidate target storage layer from the storage layers whose target layer heat matching value is higher than the target layer heat matching value of the current storage layer. For downgrade candidates, determine the candidate target storage layer from the storage layers whose target layer heat matching value is lower than the target layer heat matching value of the current storage layer.

8. The method for predicting cold and hot data migration based on dynamic load analysis as described in claim 4 or 5, characterized in that, The calculation of migration benefit value based on promotion candidate objects, demotion candidate objects, candidate target storage layer, object data volume, replica consistency status, remaining available capacity, migration quota records, and dynamic load coefficient includes: Filter the upgrade and downgrade candidates from the hot and cold prediction results records, and obtain the current storage layer, candidate target storage layer, next cycle prediction heat and business timeliness level for each upgrade and downgrade candidate; Obtain the target layer heat matching value based on the candidate target storage layer, and convert the business timeliness level into a business direction matching value based on the migration direction corresponding to the upgrade candidate object and the downgrade candidate object; Based on the dynamic load coefficient of the current storage layer and the dynamic load coefficient of the candidate target storage layer, calculate the load release value for migrating from the current storage layer to the candidate target storage layer. Calculate the migration cost of the data object within the current monitoring period based on the object data volume and migration quota records; Based on the predicted heat of the next cycle, the target layer heat matching value, the business direction matching value, the load release value, the migration overhead value, and the replica consistency status, calculate the migration benefit value for each upgrade candidate and each downgrade candidate.

9. The method for predicting cold and hot data migration based on dynamic load analysis as described in claim 8, characterized in that, The process of forming a migration forecast list based on migration benefit values ​​includes: Based on the replica consistency status, perform consistency admission checks on the promotion and demotion candidate objects, and mark data objects whose replica consistency status does not meet the migration conditions as non-migration objects. Based on the remaining available capacity of the candidate target storage layer and the amount of object data, a capacity admission check is performed on the data objects that have passed the consistency admission check, and data objects whose object data volume is greater than the remaining available capacity of the candidate target storage layer are marked as non-migrated objects. Based on the migration quota records and object data volume, migration quota access checks are performed on data objects that have passed the capacity access check, and data objects with insufficient migration quota in the current monitoring period are marked as non-migrated objects; Based on the migration benefit value, perform a benefit access check on the data objects that have passed the migration quota access check, and mark the data objects with a migration benefit value not greater than zero as non-migrated objects. Data objects that pass the benefit admission check are sorted from highest to lowest according to their migration benefit value, and the data object identifier, current storage layer, candidate target storage layer, migration direction, predicted popularity in the next cycle, object data volume, migration benefit value, and execution priority are written into the migration prediction list.

10. The method for predicting cold and hot data migration based on dynamic load analysis as described in claim 1, characterized in that, The process of performing sharded replication, incremental write record replay, data digest verification, and access direction switching on data objects according to the migration prediction list, and obtaining migration records including migration direction, metadata version number, data digest verification result, and access direction switching result, includes: According to the execution priority in the migration prediction list, obtain the data object identifier, current storage layer, candidate target storage layer, migration direction and object data volume, and obtain the metadata version number and current access point from the data object metadata directory when the migration starts; The number of shards is determined based on the amount of object data and the size of the migration shards. Data objects in the current storage layer are copied to the candidate target storage layer according to the shard number. During the shard copying process, the committed writes generated after the metadata version number are recorded to form an incremental write record. The incremental write records are replayed to the candidate target storage layer according to the write sequence number, so that the content of the data objects in the candidate target storage layer is consistent with the committed data in the current storage layer. Data digests are calculated for data objects in the current storage layer and the candidate target storage layer, and the two data digests are compared to obtain the data digest verification result. When the data digest verification result is passed, the access pointer of the data object is switched from the current storage layer to the candidate target storage layer, and a migration record is generated that includes the migration direction, metadata version number, data digest verification result, and access pointer switching result.

Images