A data hierarchical storage control method and data processing device

Abstract

This invention discloses a data tiered storage control method and a data processing device. The method includes: determining a data temperature score based on the access characteristics of data objects; dynamically adjusting the tiering threshold according to the current state parameters of the storage layer; determining the target storage layer of the data object based on the data temperature score and the adjusted tiering threshold; when the target storage layer is inconsistent with the current storage layer, determining whether the data object meets a preset hysteresis condition; when the hysteresis condition is met, calculating the migration benefit value of migrating the data object from the current storage layer to the target storage layer; and executing the migration of the data object when the migration benefit value meets the preset condition. This invention can improve the accuracy of hot and cold data identification, reduce frequent migrations caused by short-term access fluctuations, avoid low-value migrations occupying system resources, and improve storage resource utilization and data access performance.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, specifically to a data tiered storage control method and a data processing device. Background Technology

[0002] As the data scale of enterprise business systems and database systems continues to grow, the contradiction between data storage costs and data access efficiency is becoming increasingly prominent. Different types of data vary significantly in terms of access frequency, access timeliness, update frequency, and value density. Existing data processing systems typically use a single storage medium or a static tiered strategy for data storage. Single storage medium solutions struggle to balance high performance and low cost; static tiered solutions often rely on manually configured rules, such as migration based on data generation time, table name, file directory, or fixed periods, failing to accurately reflect the true access frequency of data. While some systems can count access frequency, they usually only determine hot / cold data based on access frequency, ignoring recent access time, business priority, write stability, migration costs, storage layer load, and access latency sensitivity, easily leading to problems such as erroneous migration, frequent migration, hot spot fluctuations, and decreased access performance. Furthermore, existing technologies often trigger migration only based on hot / cold data results without determining whether the migration benefits outweigh the migration costs, resulting in low-value or ineffective migrations that consume system resources. Summary of the Invention

[0003] To address the problems of inaccurate data migration decisions, migration jitter, and ineffective migration resource consumption in existing technologies, this application proposes a data tiered storage control method and data processing device. This method dynamically adjusts the tiering threshold based on the storage tier status and makes migration decisions by combining hysteresis conditions and migration benefit judgments, thereby achieving synergistic optimization of storage resource utilization and migration overhead.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: A data tiered storage control method includes: determining a data temperature score based on the access characteristics of a data object; dynamically adjusting a tiered threshold according to the current state parameters of the storage layer; determining the target storage layer of the data object based on the data temperature score, the adjusted tiered threshold, and a preset hysteresis condition; calculating the migration benefit value of the data object from the current storage layer to the target storage layer; and performing the migration when the migration benefit value meets a preset condition.

[0005] As one implementation, the step of dynamically adjusting the tiering threshold based on the current state parameters of the storage layer includes: adjusting the tiering threshold based on the storage layer's capacity utilization and I / O load; and forming a hysteresis interval between a hot data threshold and a cold data threshold, wherein the hot data threshold is higher than the cold data threshold.

[0006] As one implementation, adjusting the tiering threshold based on the storage layer's capacity utilization and I / O load includes: increasing the hot data threshold when the storage layer's capacity utilization is higher than the target utilization; and increasing the hot data threshold when the storage layer's I / O load is higher than the target load.

[0007] As one implementation method, the preset hysteresis condition includes: the data object must meet a preset duration condition before migration can be triggered.

[0008] As one implementation method, the preset duration condition includes: the data object meets the migration condition for multiple consecutive statistical periods.

[0009] In one implementation, the migration benefit value is calculated based on access benefits, storage cost benefits, and migration costs.

[0010] In one implementation, the access revenue is calculated based on the difference between the predicted access probability and the access latency; the storage cost revenue is calculated based on the difference between the size of the data object and the unit storage cost; and the migration cost is calculated based on the size of the data object and the available migration bandwidth.

[0011] As one implementation, the method further includes: determining the execution priority of the migration task based on the migration benefit value, the size of the data object, and the system load.

[0012] As one implementation, the data temperature score is calculated based on at least one of the following: access frequency, access recentity, business priority, latency sensitivity, write stability, and migration cost.

[0013] Furthermore, the present invention also provides a data processing device, comprising: The temperature rating module is used to determine the temperature rating of data based on the access characteristics of data objects. The threshold adjustment module is used to dynamically adjust the layering threshold based on the current state parameters of the storage layer. The target layer determination module is used to determine the target storage layer of the data object based on the data temperature score and the adjusted layer threshold. The hysteresis determination module is used to determine whether the data object meets the preset hysteresis condition when the target storage layer is inconsistent with the current storage layer. The revenue calculation module is used to calculate the migration revenue value of the data object from the current storage layer to the target storage layer when the data object meets the preset hysteresis condition. The migration execution module is used to perform the migration of the data object when the migration benefit value meets the preset conditions.

[0014] Beneficial effects: This invention dynamically adjusts the tiering threshold based on storage layer capacity utilization and I / O load, enabling the tiering strategy to adapt to the real-time state of storage resources and solving the problem of static thresholds being unable to cope with load fluctuations. By setting hysteresis intervals and duration conditions, requiring the migration-in threshold to be higher than the migration-out threshold and meeting the conditions for multiple consecutive cycles, migration jitter caused by short-term fluctuations in access frequency is effectively suppressed, reducing system resource waste. By calculating a comprehensive migration benefit value including access benefits, storage cost benefits, and migration costs before migration, migration is only performed when the benefit condition is met, avoiding low-value or ineffective migrations and improving the overall operating efficiency of the storage system. In addition, by calculating a data temperature score through multi-dimensional features, the true hot or cold state of data can be more accurately identified, ensuring the access performance of high-value data. Attached Figure Description

[0015] Figure 1 This is a flowchart of a data hierarchical storage control method provided in an embodiment of the present invention; Figure 2 This is a flowchart of dynamic threshold adjustment provided in an embodiment of the present invention; Figure 3 This is a flowchart of a hysteresis condition determination method provided in an embodiment of the present invention; Figure 4 This is a structural block diagram of a data processing device provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Implementation To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0019] Example 1: like Figure 1 As shown, this embodiment provides a data tiered storage control method. This method can be executed by a data processing device, which may be a database server, a storage array controller, or a cloud storage management node; this embodiment does not limit the specific device to this type.

[0020] Specifically, the method includes the following steps: Step S100: Determine the data temperature score based on the access characteristics of the data object.

[0021] In this embodiment, a data object refers to the smallest management unit in the storage system, and its granularity can be flexibly defined according to the actual application scenario. For example, in a database scenario, a data object can be a table partition, data page, or index block; in a file system scenario, a data object can be a file, file block, or directory; and in an object storage scenario, a data object can be an object entity. Access characteristics are used to describe the access status of a data object within a preset statistical period, including but not limited to access frequency, access recentity, business priority, latency sensitivity, and write stability. Data temperature score is a quantitative representation of the "popularity" of a data object. A higher score usually indicates that the data object is frequently accessed or is more important to the business, and is more suitable for storage in a higher-performance storage layer; conversely, a lower score indicates that the data object is less popular and is more suitable for storage in a lower-cost storage layer. By comprehensively calculating the temperature score through multi-dimensional access characteristics, misjudgments caused by a single dimension (such as judging only by the number of accesses) can be avoided, thereby more accurately identifying the true value of the data.

[0022] Step S200: Dynamically adjust the layering threshold based on the current state parameters of the storage layer.

[0023] In this embodiment, the storage layer typically includes multiple layers of storage media with varying performance, such as a high-performance hot data layer (e.g., NVMe SSD), a medium-performance warm data layer (e.g., SATA SSD), and a large-capacity, low-cost cold data layer (e.g., HDD or object storage). The tiering threshold is a critical value used to distinguish between hot and cold data states, such as a hot data threshold and a cold data threshold. Unlike existing technologies that use fixed thresholds, this step introduces a dynamic adjustment mechanism to adjust the tiering threshold in real time based on the current state parameters of the storage layer. These current state parameters reflect the real-time load of the storage layer, such as capacity utilization, I / O load, and bandwidth utilization. When the high-performance storage layer is under resource strain (e.g., high capacity utilization), the system can appropriately increase the hot data threshold. This means that only truly high-value data can enter the high-performance layer, thus preventing low-value data from crowding out valuable high-performance resources. This dynamic adjustment mechanism allows the tiering strategy to adapt to the real-time state of the storage system, solving the problem that static thresholds cannot cope with load fluctuations.

[0024] Step S300: Based on the data temperature score, the adjusted stratification threshold, and the preset hysteresis condition, determine the target storage layer of the data object.

[0025] In this embodiment, the target storage layer refers to the ideal location where the data object should be stored. The system compares the data temperature score obtained in step S100 with the stratification threshold adjusted in step S200 to initially determine whether the data object belongs to hot data, warm data, or cold data. However, to avoid frequent jumps between different storage layers due to short-term fluctuations in access characteristics (i.e., "migration jitter"), this step introduces a preset hysteresis condition. The hysteresis condition requires that the state change of the data object must meet certain stability requirements, such as continuously meeting the migration condition within multiple consecutive statistical periods, before triggering the final migration decision. By filtering with the hysteresis condition, the interference of short-term access fluctuations on the decision can be effectively suppressed, ensuring the stability of the migration decision.

[0026] Step S400: Calculate the migration benefit value of migrating the data object from the current storage layer to the target storage layer, and execute the migration when the migration benefit value meets the preset conditions.

[0027] In this embodiment, even if the target storage layer of the data object is inconsistent with the current storage layer, the system will not immediately perform migration, but will first conduct a benefit assessment. The migration benefit value is used to quantify the comprehensive value that the migration can bring, which comprehensively considers the improvement of access performance after migration (access benefit), the saving of storage costs (storage cost benefit), and the consumption of system resources by the migration process itself (migration cost). The preset condition can be that the migration benefit value is greater than zero, or greater than a certain preset benefit threshold. Only when the migration benefit value meets the preset condition will the system generate a migration task and execute the data migration. This mechanism can effectively filter out low-value or invalid migration tasks, avoid the waste of system resources caused by "migration for the sake of migration", and ensure that every migration brings positive benefits.

[0028] Through the above steps, the method of this embodiment forms a complete control closed loop: first, the data value is accurately evaluated through temperature scoring; second, the storage resource status is adaptively adjusted through dynamic thresholds; third, jitter is suppressed through hysteresis conditions; and finally, invalid migration is avoided through benefit determination, thereby achieving synergistic optimization of storage performance, storage cost, and system stability.

[0029] Example 2: This embodiment describes the method for determining the data temperature score, based on Embodiment 1.

[0030] In this embodiment, the data temperature score can be determined based on at least one of the following: access frequency, access recentity, business priority, latency sensitivity, write stability, migration cost, and access anomaly level. Specifically, the system can obtain the number of accesses, the most recent access time, the business type, the response latency requirement, the write changes, the resources required for migration, and the abnormal access behavior of the data object within a preset statistical period, and standardize the above features to obtain feature values ​​that can be used for comparison.

[0031] Among them, access frequency is used to characterize the activity level of data objects within the statistical period; the higher the access frequency, the higher the corresponding data temperature score. Access recentity is used to characterize how close the time since the last access to the data object is to the current time; the closer it is to the current time, the higher the corresponding data temperature score. Business priority is used to characterize the importance of the business to which the data object belongs; the higher the business priority, the higher the corresponding data temperature score. Latency sensitivity is used to characterize the data object's requirements for access response speed; the higher the latency sensitivity, the higher the corresponding data temperature score. Write stability is used to characterize the stability of the data object's access and write patterns; the more stable the access pattern, the higher the corresponding data temperature score.

[0032] Migration cost characterizes the system resources required to migrate data objects. For data objects with large volumes, long migration times, or significant business impact during migration, the system can lower their data temperature score or increase their migration trigger conditions to avoid wasting system resources due to excessive migration costs. Access anomaly level characterizes whether data objects experience abnormal bursts of access, scanning access, non-business routine access, or abnormal access distribution within a statistical period. For data objects with high levels of abnormal access, the system can reduce the impact of abnormal access behavior on the data temperature score, preventing temporary, non-business access peaks from causing data objects to be misclassified as hot data.

[0033] In its implementation, the system can assign corresponding influence weights to different access characteristics and comprehensively process each characteristic according to preset scoring rules. For characteristics that reflect the access value of data objects or business importance, the system can treat them as positive influencing factors; for characteristics that reflect migration overhead or the degree of access anomalies, the system can treat them as restrictive influencing factors. The system determines the data temperature score based on the comprehensive results of each characteristic.

[0034] Through the above method, this embodiment can avoid misjudgment caused by judging the hotness or coldness of data solely based on the number of accesses, and enable the data temperature score to simultaneously reflect the access activity level, business value, latency requirements, migration costs, and access stability of the data object.

[0035] Example 3: This embodiment describes the method of dynamically adjusting the stratification threshold based on embodiment 1.

[0036] In this embodiment, the system adjusts the tiering threshold based on the storage layer's capacity utilization and I / O load, and forms a hysteresis interval between the hot data threshold and the cold data threshold, wherein the hot data threshold is higher than the cold data threshold.

[0037] Specifically, the system pre-sets a basic threshold for thermal data, a target capacity utilization rate, and a target I / O load. During operation, the system acquires the current capacity utilization rate and current I / O load of the thermal data layer in real time, compares the current capacity utilization rate with the target capacity utilization rate, and compares the current I / O load with the target I / O load.

[0038] When the current capacity utilization rate of the hot data layer is higher than the target capacity utilization rate, it indicates that the storage space resources of the hot data layer are becoming strained. The system raises the hot data threshold to increase the barrier for data objects to enter the hot data layer, so that only data objects with high access frequency or high business value can enter or remain in the hot data layer.

[0039] When the current I / O load of the hot data layer is higher than the target I / O load, it indicates that the read and write pressure of the hot data layer is relatively large. The system increases the hot data threshold to reduce new data migration tasks and prevent migration tasks from further occupying the I / O resources of the high-performance storage layer.

[0040] When the current capacity utilization rate of the hot data layer is lower than the target capacity utilization rate, and the current I / O load is lower than the target I / O load, it means that the hot data layer still has available resources. The system can maintain the current hot data threshold or appropriately reduce the hot data threshold to allow more data objects with higher access value to enter the hot data layer, thereby improving the overall access efficiency.

[0041] After determining the hot data threshold, the system determines the cold data threshold based on a preset threshold interval, creating a hysteresis range between the hot and cold data thresholds. This threshold interval can be determined based on historical access data fluctuations, business access stability, or administrator configuration. When the data temperature score is within this hysteresis range, the system can maintain the data object's current storage layer. When the data temperature score consistently exceeds the hot data threshold, the system considers the data object as a candidate for migration to the high-performance storage layer. When the data temperature score consistently falls below the cold data threshold, the system considers the data object as a candidate for migration to the low-cost storage layer.

[0042] Through the above-described dynamic threshold adjustment method, this embodiment enables the tiered threshold to be adaptively adjusted according to the changes in the storage layer's operating status, avoiding inaccurate tiered judgment caused by fixed thresholds in scenarios of capacity shortage, increased load, or business fluctuations.

[0043] Example 4: This embodiment, based on embodiment 1, explains the method for determining the migration benefit value in step S400.

[0044] In tiered data storage scenarios, triggering migration solely based on changes in the hot / cold status of data objects can easily lead to a lack of benefit constraints in migration decisions. Even if a data object meets the hot / cold tiering criteria, if the performance improvements or storage cost savings after migration are insufficient to cover the system resources consumed during the migration process, the migration action will increase I / O bandwidth, network bandwidth, and computing resource consumption, impacting normal business operations. Therefore, this embodiment introduces a migration benefit determination mechanism before execution to assess whether the migration has actual execution value.

[0045] Specifically, the migration benefit value can be determined based on access benefit, storage cost benefit, and migration cost. Access benefit represents the improvement in access performance after migrating the data object to the target storage layer; storage cost benefit represents the savings in storage costs after migrating the data object to the target storage layer; and migration cost represents the system resources required during the migration process. The system determines the migration benefit value corresponding to this migration by comprehensively evaluating the above factors, and uses this migration benefit value as the basis for deciding whether to perform the migration.

[0046] The system determines access benefits based on the predicted access probability of a data object and the difference in access latency between the current storage layer and the target storage layer. The predicted access probability is determined based on the data object's historical access frequency, access periodicity, recent access time, and business access patterns, characterizing the likelihood of the data object being accessed within a future statistical period. The difference in access latency characterizes the difference in access response between the data object in the current storage layer and the target storage layer. When the probability of a data object being accessed in the future is high, and the target storage layer can significantly reduce access latency compared to the current storage layer, the system determines that the data object has high access benefits; when the probability of a data object being accessed in the future is low, even if the target storage layer has higher access performance, the system can still determine that its access benefits are low.

[0047] Regarding storage cost-benefit, the system can determine it based on the size of the data object and the difference in unit capacity cost between the current storage layer and the target storage layer. The difference in unit capacity cost can be determined based on the difference between the unit capacity cost of the current storage layer and the target storage layer. When a data object is migrated from a high-cost storage layer to a low-cost storage layer, if the data object occupies a large capacity, the migration will free up a significant amount of high-cost storage space, and the system can determine that it has a high storage cost-benefit. When the data object occupies a small capacity, or when the target storage layer does not have a significant cost advantage over the current storage layer, the system can determine that its storage cost-benefit is low. For the case of migrating a data object from a low-cost storage layer to a high-performance storage layer, the system can include the increased storage cost as a limiting factor in the migration benefit evaluation.

[0048] The migration cost can be determined based on the size of the data object, available migration bandwidth, current system load, and the impact of the migration on normal business access. Larger data objects require more data to be transferred during migration; lower available migration bandwidth results in longer migration times; and higher current system load increases the likelihood of disruption to normal business access during migration. Therefore, when determining the migration benefits, the system can comprehensively evaluate the migration costs by considering the I / O bandwidth, network bandwidth, computing resources, and migration duration.

[0049] In some embodiments, the system may further determine the migration benefit value by incorporating migration risk factors. These migration risk factors may include data consistency risk, business interruption risk, migration failure risk, rollback costs, and metadata switching risk during the data migration process. For data objects with high migration risk, the system may lower their migration benefit evaluation result or increase their migration triggering conditions to reduce the impact of high-risk migration on business stability.

[0050] In one specific implementation, the system can first determine the expected comprehensive benefits after migration, including access benefits and / or storage cost benefits; then determine the migration cost based on the system resources consumed during the migration process; and, if risk control needs to be considered, further determine the risk cost corresponding to the migration risk. The system determines the migration benefit value based on the comparison between the comprehensive benefits, migration costs, and risk costs. When the comprehensive benefits, after deducting the migration costs and risk costs, still meet the preset benefit conditions, the system determines that this migration is worthwhile and triggers the migration task; when the comprehensive benefits are insufficient to cover the migration costs and risk costs, or do not meet the preset benefit conditions, the system postpones or cancels this migration.

[0051] For example, when the system detects that the data temperature score of a historical order partition is lower than the cold data threshold and initially determines that its target storage layer is a cold data layer, the system continues to assess the migration benefit of that historical order partition. If the historical order partition has a low probability of future access, its access benefit is low; if the historical order partition occupies a large storage capacity, and migrating it from a high-cost storage layer to a low-cost storage layer can free up a lot of high-performance storage space, its storage cost benefit is high. The system further determines the migration cost by combining the capacity of the historical order partition, available migration bandwidth, and current system load. If the storage cost benefit can cover the migration cost and meet the preset benefit conditions, the system executes the migration; if the historical order partition meets the cold data determination conditions but occupies a small capacity, and the storage cost saved after migration is insufficient to cover the migration cost, the system postpones the migration and keeps it in the current storage layer.

[0052] It should be understood that the methods for determining access benefits, storage cost benefits, migration costs, and migration risk factors described above are merely illustrative and do not constitute a limitation on the scope of protection of this invention. In other embodiments, the system can set different evaluation rules for various benefit and cost factors based on business type, storage medium type, data access patterns, service level requirements, and administrator configuration. For example, for real-time transaction systems, the influence of access benefits in determining migration benefits can be increased; for archive storage systems, the influence of storage cost benefits in determining migration benefits can be increased. Through the above migration benefit determination mechanism, the system can reduce low-value migrations and ineffective migrations while meeting the requirements of hot and cold tiering, thereby improving storage resource scheduling efficiency and system operational stability.

[0053] Example 5: This embodiment, based on Embodiment 1, explains how to determine the migration benefit value.

[0054] In this embodiment, the migration benefit value is used to determine whether the data migration has practical value. After determining that the target storage layer of the data object is inconsistent with the current storage layer, the system does not immediately perform the migration, but first makes a comprehensive judgment on the potential benefits of the migration and the costs required for the migration process.

[0055] Specifically, the system can determine the migration benefit value based on access benefits, storage cost benefits, and migration costs. Access benefits characterize the improvement in access performance that can be achieved after migrating the data object; storage cost benefits characterize the savings in storage costs that can be achieved after migrating the data object; and migration costs characterize the I / O bandwidth, network bandwidth, computing resources, and migration time required during the migration process.

[0056] The system determines access benefits based on the likelihood of a data object being accessed within a future statistical period and the difference in access latency between the current and target storage layers. When a data object has a high probability of being accessed in the future, and migrating to the target storage layer significantly reduces access latency, the system determines that the data object has high access benefits. Conversely, when a data object has a low probability of being accessed in the future, even if the target storage layer offers higher access performance, the system can still determine that its access benefits are low.

[0057] Regarding storage cost-benefit, the system can determine it based on the size of the data object and the difference in unit capacity cost between the current storage layer and the target storage layer. When a data object is migrated from a high-cost storage layer to a low-cost storage layer, and the data object occupies a large capacity, the system determines that it has a high storage cost-benefit. When the data object occupies a small capacity, or when the target storage layer does not have a significant cost advantage over the current storage layer, the system determines that its storage cost-benefit is low.

[0058] Regarding migration costs, the system can determine them based on the size of the data object, available migration bandwidth, the current system load, and the impact of migration on normal business access. When the data object is large, the available migration bandwidth is low, or the current system load is high, the system determines the migration cost to be high; when the data object is small, the available migration bandwidth is sufficient, and the system is under low load, the system determines the migration cost to be low.

[0059] In some embodiments, the system may further consider migration risk factors. These risk factors may include data consistency risks, business interruption risks, migration failure risks, or rollback costs that may occur during the migration process. For data objects with high migration risks, the system may lower their migration benefit evaluation results or require them to meet higher migration triggering conditions.

[0060] The system will migrate the data objects when the access and / or storage costs cover the migration costs, and the overall assessment meets the preset benefit conditions. If the overall assessment does not meet the preset benefit conditions, the system will postpone or cancel the migration.

[0061] By using the above-described method for determining migration benefits, this embodiment can avoid migration operations that lack actual benefits, reduce the occupation of system resources by low-value migrations, and improve the operating efficiency of the data tiered storage system.

[0062] Example 6: like Figure 2As shown, this embodiment provides a data processing device for executing the data hierarchical storage control method described in any one of embodiments 1 to 5 above. The data processing device can be an independent storage controller, database server, storage array control node, or a management node in a cloud storage system; this embodiment does not limit this. The data processing device, through the collaborative work of hardware and software, achieves hierarchical control, migration judgment, and migration execution of data object storage locations.

[0063] Specifically, the data processing device includes a temperature scoring module 110, a threshold adjustment module 120, a target layer determination module 130, a hysteresis judgment module 140, a revenue calculation module 150, and a migration execution module 160.

[0064] The temperature scoring module 110 is used to determine a data temperature score based on the access characteristics of data objects. Specifically, the temperature scoring module 110 can obtain at least one access characteristic of the data object, including access frequency, access recentity, business priority, latency sensitivity, write stability, migration cost, and access anomaly degree, through access logs and monitoring interface 600, and determine the data temperature score of the data object according to preset scoring rules. The access logs and monitoring interface 600 can be connected to the business system 500, database system, file system, or object storage system to collect access status information of data objects within a preset statistical period. Through the temperature scoring module 110, the data processing device 100 can quantitatively identify the access popularity and business value of data objects, providing a data basis for subsequent hierarchical judgment.

[0065] The threshold adjustment module 120 is used to dynamically adjust the layering thresholds based on the current state parameters of the storage layer. Specifically, the threshold adjustment module 120 can acquire at least one of the following state parameters of the hot data layer 200, warm data layer 300, and cold data layer 400: capacity utilization, I / O load, bandwidth utilization, queue depth, and available migration bandwidth, and adjust the hot data threshold and cold data threshold according to the state parameters. When the capacity utilization of the hot data layer 200 is higher than the target utilization, or the I / O load of the hot data layer 200 is higher than the target load, the threshold adjustment module 120 increases the hot data threshold to improve the conditions for data objects to enter the hot data layer 200; when the resources of the hot data layer 200 are relatively idle, the threshold adjustment module 120 can maintain or decrease the hot data threshold. The threshold adjustment module 120 can also determine the cold data threshold based on the hot data threshold and a preset threshold interval to form a hysteresis interval between the hot data threshold and the cold data threshold.

[0066] The target layer determination module 130 is used to determine the target storage layer of the data object based on the data temperature score and the adjusted layering threshold. Specifically, the target layer determination module 130 receives the data temperature score output by the temperature scoring module 110 and the adjusted layering threshold output by the threshold adjustment module 120, and compares the data temperature score with the adjusted layering threshold. When the data temperature score is greater than or equal to the hot data threshold, the target layer determination module 130 can determine the hot data layer 200 as the target storage layer; when the data temperature score is less than or equal to the cold data threshold, the target layer determination module 130 can determine the cold data layer 400 as the target storage layer; when the data temperature score is between the hot data threshold and the cold data threshold, the target layer determination module 130 can determine the warm data layer 300 as the target storage layer, or keep the current storage layer of the data object unchanged.

[0067] The hysteresis judgment module 140 is used to determine whether the data object meets preset hysteresis conditions when the target storage layer and the current storage layer are inconsistent. Specifically, the hysteresis judgment module 140 can determine whether the data object meets the migration conditions in multiple consecutive statistical periods, or whether the number of times the data object meets the migration conditions within a preset time window reaches a preset proportion. When the data object does not meet the preset hysteresis conditions, the hysteresis judgment module 140 outputs a judgment result of temporarily not migrating, so that the data object remains in the current storage layer; when the data object meets the preset hysteresis conditions, the hysteresis judgment module 140 outputs a judgment result of allowing migration benefit determination. By setting the hysteresis judgment module 140, frequent data object migration caused by short-term access fluctuations can be reduced.

[0068] The benefit calculation module 150 is used to calculate the migration benefit value of the data object when it meets the preset hysteresis condition, based on the access benefit, storage cost benefit, and migration cost. Specifically, the benefit calculation module 150 can determine the migration benefit value based on access benefit, storage cost benefit, and migration cost. The access benefit characterizes the improvement in access performance after the data object is migrated to the target storage layer; the storage cost benefit characterizes the savings in storage cost after the data object is migrated to the target storage layer; and the migration cost characterizes the I / O bandwidth, network bandwidth, computing resources, and migration time consumed during the migration process. In some embodiments, the benefit calculation module 150 can also combine migration risk factors such as data consistency risk, business interruption risk, migration failure risk, and rollback cost to determine the migration benefit value.

[0069] The migration execution module 160 is used to execute the migration of the data object when the migration benefit value meets preset conditions. Specifically, when the migration benefit value determined by the benefit calculation module 150 is greater than a preset benefit threshold, or when other preset benefit conditions are met, the migration execution module 160 generates a migration task and controls the underlying data transport engine to perform data copying, integrity verification, metadata switching, and old data release operations. The underlying data transport engine may include a DMA controller, a network transmission module, a storage controller, or a distributed storage migration service. When the migration benefit value does not meet the preset conditions, the migration execution module 160 postpones or cancels the migration to reduce the occupation of system resources by low-value migrations.

[0070] In some embodiments, the data processing device 100 may further include a migration task scheduling module. This module, when multiple migration tasks are pending execution, determines the execution priority of each migration task based on migration benefit value, data object size, business strategy, and system load, and controls the execution order, execution time, and concurrency of the migration tasks according to the execution priority. The migration task scheduling module can be connected to the migration execution module 160 and is used to reduce the concurrency of migration tasks or pause low-priority migration tasks when the system load is high; and to perform data migration using idle resources when the system load is low.

[0071] At the hardware implementation level, the aforementioned temperature scoring module 110, threshold adjustment module 120, target layer determination module 130, hysteresis judgment module 140, profit calculation module 150, migration execution module 160, and migration task scheduling module can be implemented through a processor, memory, and communication interface. The processor can be a general-purpose CPU, microprocessor, application-specific integrated circuit, field-programmable gate array, or other hardware unit with data processing capabilities, used to execute computer program instructions stored in the memory. The memory is used to store program code, access feature data, data temperature scores, dynamic threshold parameters, hysteresis judgment results, migration profit values, and migration task queues. The communication interface is used to connect storage media of different performance levels, including hot data layers, warm data layers, and cold data layers. The communication interface can include NVMe SSD interfaces, SATA interfaces, SAS interfaces, network storage interfaces, or object storage access interfaces.

[0072] It should be understood that the above module division is only a functional division and does not require each module to correspond to independent physical hardware. In actual implementation, multiple modules can be integrated in the same processor, controller, or server node, or they can be distributed on different physical nodes to work together, as long as the above functions can be achieved.

[0073] Through the above structure, the data processing device 100 can sequentially complete the determination of data temperature score, dynamic adjustment of layer threshold, determination of target storage layer, judgment of hysteresis condition, calculation of migration benefit value and execution of data migration, so that the device structure and the processing flow of the data layer storage control method correspond to each other, thereby improving the accuracy, stability and resource utilization efficiency of cold and hot data layer control.

[0074] Example 7: This embodiment uses table partitions in a database system as data objects to illustrate in detail the operation process and technical effects of the data hierarchical storage control method provided by the present invention in a practical application scenario. It should be understood that this embodiment is merely illustrative, and the data object can also be a database table, data page, key-value object, file, or object storage file, etc., and the present invention does not limit it in this way.

[0075] In this application scenario, the data processing equipment manages a large e-commerce transaction database containing massive amounts of historical order data. To balance query performance and storage costs, the database administrator configured a three-tier storage architecture: the hot data tier uses NVMe SSDs, the warm data tier uses ordinary SATA SSDs, and the cold data tier uses high-capacity HDDs. The system needs to automatically allocate orders to the appropriate storage tier based on their access frequency.

[0076] Specifically, the data processing device performs the following control steps: Step S701: Determine the data temperature score based on the access characteristics of the data objects. The data processing equipment collects the access characteristics of each order table partition in real time. For the "last 7 days order partition," the system detects that its read count Rᵢ and write count Wᵢ within the statistical period are both high, and the recent access interval Tᵢ is extremely short, so the business priority Pᵢ is marked as "core transaction data." The system performs a weighted calculation based on the access frequency factor, access recentity factor, and business priority factor, and obtains a temperature score Hᵢ of 95 points (out of 100). Conversely, for the "180 days ago historical order partition," its access frequency is extremely low, and the recent access interval is long; the system calculates its temperature score Hᵢ to be only 15 points.

[0077] Step S702: Dynamically adjust the tiering threshold based on the current status parameters of the storage layer. The system detects that the current capacity utilization rate Uhot of the hot data layer (NVMe SSD) has reached 85%, higher than the target utilization rate Uhot* (set to 75%); at the same time, the I / O load Ihot of the hot data layer is also close to saturation. According to the dynamic threshold adjustment strategy, the system automatically increases the hot data threshold θhot. For example, the base threshold θhot0 is increased from 70 points to 80 points. This adjustment means that the threshold for entering the hot data layer is higher, and only truly high-value order partitions can occupy valuable NVMe resources, thereby preventing low-value data from crowding out high-performance storage space.

[0078] Step S703: Based on the data temperature score, the adjusted stratification threshold, and the preset hysteresis condition, the target storage layer of the data object is determined. For the "Recent 7 Days Orders Partition," its temperature score of 95 is higher than the adjusted hot data threshold of 80, and this condition is consistently met for three consecutive statistical periods, satisfying the preset hysteresis condition. Therefore, the system determines its target storage layer as the hot data layer. For the "180 Days Ago Historical Orders Partition," its temperature score of 15 is lower than the cold data threshold θcold (set to 40), and the hysteresis condition is also met for multiple consecutive periods. Therefore, the system determines its target storage layer as the cold data layer. Through hysteresis condition filtering, the system effectively avoids erroneous migration caused by short-term access fluctuations due to sudden queries.

[0079] Step S704: Calculate the migration benefit value of migrating data objects from the current storage layer to the target storage layer. Execute the migration when the migration benefit value meets preset conditions. For the "Recent 7 Days Order Partition," although its target storage layer is a hot data layer, the system does not immediately execute the migration but first calculates the migration benefit. The system predicts that the access probability of this partition is extremely high in the future period, and the access latency difference between the NVMe SSD and the current storage layer is significant, resulting in a high calculated access benefit. After deducting the bandwidth consumption cost during the migration process, the migration benefit value Bᵢ is much greater than the preset threshold, and the system determines to execute the migration. For the "180 Days Ago Historical Order Partition," although its access benefit is extremely low, migrating it to the HDD can save a significant amount of expensive SSD storage space, resulting in a significant storage cost benefit. The calculated net value of the storage cost benefit after deducting the migration cost is still positive and greater than the threshold, so the system determines to execute the migration.

[0080] During the migration execution phase, the system determines the execution priority of migration tasks based on migration benefits, data object size, and system load. Because the "Recent 7 Days Orders Partition" involves core transaction business, the system assigns it a higher business strategy coefficient and prioritizes its migration task. For migration tasks of historical order partitions, the system schedules them for execution during off-peak hours to avoid impacting normal transaction business.

[0081] Ultimately, the high-frequency order partition was successfully migrated to an NVMe SSD, resulting in a 10-fold improvement in query response speed; the historical order partition was migrated to a low-cost HDD, freeing up a significant amount of high-performance storage space. This embodiment, through the aforementioned control process, achieves synergistic optimization of storage resource utilization and business access performance, verifying the effectiveness of the invention in real-world application scenarios.

[0082] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data hierarchical storage control method, characterized in that, include: Determine data temperature scores based on the access characteristics of data objects; The tiering threshold is dynamically adjusted based on the current state parameters of the storage layer. Based on the data temperature score and the adjusted stratification threshold, the target storage layer of the data object is determined; When the target storage layer is inconsistent with the current storage layer, determine whether the data object meets the preset hysteresis condition; When the data object meets the preset hysteresis condition, calculate the migration benefit value of the data object from the current storage layer to the target storage layer; When the migration benefit value meets the preset conditions, the migration of the data object is executed.

2. The method according to claim 1, characterized in that, The step of dynamically adjusting the tiering threshold based on the current state parameters of the storage layer includes: Adjust the tiering threshold based on the storage tier's capacity utilization and I / O load; A hysteresis interval is formed between a hot data threshold and a cold data threshold, wherein the hot data threshold is higher than the cold data threshold.

3. The method according to claim 2, characterized in that, The adjustment of the tiering threshold based on the storage tier's capacity utilization and I / O load includes: When the storage layer's capacity utilization rate is higher than the target utilization rate, the hot data threshold is increased; When the I / O load of the storage layer is higher than the target load, the hot data threshold is increased.

4. The method according to claim 1, characterized in that, The preset hysteresis conditions include: The data object must meet a preset duration condition before migration can be triggered.

5. The method according to claim 4, characterized in that, The preset duration conditions include: The data object satisfies the migration conditions for multiple consecutive statistical periods.

6. The method according to claim 1, characterized in that, The migration benefit value is calculated based on access benefits and / or storage cost benefits, as well as migration costs.

7. The method according to claim 6, characterized in that, The access benefit is calculated based on the predicted access probability and the access latency difference between the current storage layer and the target storage layer. The storage cost benefit is calculated based on the size of the data object and the difference in unit storage cost between the current storage layer and the target storage layer. The migration cost is calculated based on the size of the data object and the available migration bandwidth.

8. The method according to claim 1, characterized in that, The method further includes: The execution priority of the migration task is determined based on the migration benefit value, the size of the data object, and the system load.

9. The method according to claim 1, characterized in that, The data temperature score is calculated based on access frequency, access recentity, business priority, latency sensitivity, write stability, migration cost, and access anomaly level.

10. A data processing device, characterized in that, include: The temperature rating module is used to determine the temperature rating of data based on the access characteristics of data objects. The threshold adjustment module is used to dynamically adjust the layering threshold based on the current state parameters of the storage layer. The target layer determination module is used to determine the target storage layer of the data object based on the data temperature score and the adjusted layer threshold. The hysteresis determination module is used to determine whether the data object meets the preset hysteresis condition when the target storage layer is inconsistent with the current storage layer. The revenue calculation module is used to calculate the migration revenue value of the data object from the current storage layer to the target storage layer when the data object meets the preset hysteresis condition. The migration execution module is used to perform the migration of the data object when the migration benefit value meets the preset conditions.

Images