Cold and hot data automatic migration method based on intelligent scheduling

Abstract

The invention discloses an intelligent scheduling-based cold and hot data automatic migration method, which comprises the following steps of: based on a migration strategy, carrying out cold and hot data migration between high-performance storage equipment and low-cost storage equipment; the migration strategy is dynamically adjusted based on an intelligent scheduling algorithm and equipment performance monitoring, and the cold and hot data automatic migration method based on intelligent scheduling performs dynamic adjustment through the intelligent scheduling algorithm and the equipment performance monitoring, so that the dynamic migration strategy adapts to the change of a data access mode; the cold and hot data can be identified more accurately, the migration accuracy and efficiency are improved, the storage cost of an enterprise is remarkably reduced, and the data access speed and the system response time are improved.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a method for automatic migration of cold and hot data based on intelligent scheduling. Background Art

[0002] With the advent of the digital age, the amount of data has exploded, and enterprises are facing huge challenges in data storage and management. Traditional data storage methods are often unable to efficiently cope with the different needs of hot and cold data, so partition management for hot and cold data has become an effective way to reduce costs and increase efficiency. However, current hot and cold data migration solutions usually rely on static rules or manual intervention, and the conversion of hot and cold data is judged by set fixed thresholds. The storage strategy for hot and cold data is single and does not adapt to the characteristics of the big data era. This is reflected in the failure to migrate cold data to low-cost storage devices in a timely manner, resulting in unnecessary storage costs, and the failure to migrate hot data back to high-performance storage devices in a timely manner, affecting data access speed and system response time. Summary of the invention

[0003] In order to address the deficiencies in the prior art, the present invention provides a method for automatic migration of hot and cold data based on intelligent scheduling. The method for automatic migration of hot and cold data based on intelligent scheduling is dynamically adjusted through an intelligent scheduling algorithm and equipment performance monitoring, and a dynamic migration strategy is used to adapt to changes in data access patterns, thereby achieving more accurate identification of hot and cold data, improving the accuracy and efficiency of migration, significantly reducing the storage costs of enterprises, and improving data access speed and system response time.

[0004] To achieve the above object, the present invention adopts the following technical solution:

[0005] The present invention provides a method for automatic migration of cold and hot data based on intelligent scheduling, comprising the following steps:

[0006] Based on the migration strategy, migrate hot and cold data between high-performance storage devices and low-cost storage devices;

[0007] The migration strategy is dynamically adjusted based on intelligent scheduling algorithms and device performance monitoring.

[0008] The automatic migration method of hot and cold data based on intelligent scheduling of the present invention first monitors various indicators of storage devices in real time during daily work, and then dynamically adjusts the migration strategy based on the performance indicators of the storage devices in conjunction with the intelligent scheduling algorithm. The hot and cold data are migrated based on this dynamic migration strategy, ensuring that the method can maintain a certain processing capability under diversified data access and storage modes, and maintain accurate identification and efficient migration of hot and cold data.

[0009] The automatic hot and cold data migration method based on intelligent scheduling of the present invention is dynamically adjusted through intelligent scheduling algorithms and equipment performance monitoring, and uses dynamic migration strategies to adapt to changes in data access patterns, thereby achieving more accurate identification of hot and cold data, improving the accuracy and efficiency of migration, significantly reducing the storage costs of enterprises, and improving data access speed and system response time.

[0010] In a further technical solution, the device performance monitoring includes monitoring the read and write speed, delay, and bandwidth of high-performance storage devices and low-cost storage devices.

[0011] Monitoring the read and write speed, latency, and bandwidth of the device helps the intelligent scheduling algorithm to accurately adjust the migration strategy dynamically, further improving the efficiency of migration.

[0012] In a further technical solution, the device performance monitoring also includes evaluating the load capacity and remaining capacity of high-performance storage devices and low-cost storage devices based on performance indicators to provide a basis for resource allocation for data migration.

[0013] Accurate evaluation is a prerequisite for accurate resource allocation. Evaluating load capacity and remaining capacity based on performance indicators helps the intelligent scheduling algorithm to accurately adjust the migration strategy dynamically and further improve the efficiency of migration.

[0014] In a further technical solution, the intelligent scheduling algorithm includes analyzing the optimal path for data migration based on the Dijkstra algorithm.

[0015] Data migration path planning based on the Dijkstra algorithm can complete data migration as quickly as possible and further improve the efficiency of data migration.

[0016] In a further technical solution, before and after the migration of cold and hot data, the data is checked for consistency by means of encrypted signatures.

[0017] Through real-time monitoring and error handling mechanisms, data consistency and integrity of the migration process are ensured, reducing the risk of data loss.

[0018] The beneficial effects are:

[0019] 1. The automatic migration method of hot and cold data based on intelligent scheduling of the present invention is dynamically adjusted through intelligent scheduling algorithms and equipment performance monitoring, and uses dynamic migration strategies to adapt to changes in data access patterns, thereby achieving more accurate identification of hot and cold data, improving the accuracy and efficiency of migration, significantly reducing the storage costs of enterprises, and improving data access speed and system response time.

[0020] 2. Monitoring the read and write speed, latency, and bandwidth of the device will help the intelligent scheduling algorithm to accurately adjust the migration strategy dynamically, further improving the efficiency of migration.

[0021] 3. Accurate evaluation is a prerequisite for accurate resource allocation. Evaluating load capacity and remaining capacity based on performance indicators is conducive to the intelligent scheduling algorithm to accurately adjust the dynamic migration strategy and further improve the efficiency of migration.

[0022] 4. Data migration path planning based on the Dijkstra algorithm can complete data migration as quickly as possible and further improve the efficiency of data migration.

[0023] 5. Ensure data consistency and integrity during the migration process and reduce the risk of data loss through real-time monitoring and error handling mechanisms. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the operation logic of the method for automatic migration of cold and hot data based on intelligent scheduling according to an embodiment of the present invention.

[0025] 10. High-performance storage devices; 20. Low-cost storage devices. DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with the accompanying drawings:

[0027] Example:

[0028] A method for automatic migration of hot and cold data based on intelligent scheduling, such as Figure 1 As shown, the following steps are included:

[0029] Based on the migration strategy, hot and cold data are migrated between the high-performance storage device 10 and the low-cost storage device 20;

[0030] The migration strategy is dynamically adjusted based on intelligent scheduling algorithms and device performance monitoring.

[0031] In this embodiment, the high-performance storage device 10 is a high-performance device with fast reading and writing speed, low latency, and large bandwidth, so its cost is relatively high, but due to its excellent performance, it can meet the high-frequency and large-scale data reading and writing needs, so it can be regarded as a storage unit for hot data; the performance of the low-cost storage device 20 is relatively reduced, but the advantages are low cost and large capacity, and it is a storage unit for cold data.

[0032] The automatic migration method of hot and cold data based on intelligent scheduling of the present invention first monitors various indicators of storage devices in real time during daily work, and then dynamically adjusts the migration strategy based on the performance indicators of the storage devices in conjunction with the intelligent scheduling algorithm. The hot and cold data are migrated based on this dynamic migration strategy, ensuring that the method can maintain a certain processing capability under diversified data access and storage modes, and maintain accurate identification and efficient migration of hot and cold data.

[0033] The automatic migration method of hot and cold data based on intelligent scheduling of the present invention is dynamically adjusted through intelligent scheduling algorithms and equipment performance monitoring, and uses dynamic migration strategies to adapt to changes in data access patterns, thereby achieving more accurate identification of hot and cold data and improving the accuracy and efficiency of migration.

[0034] In terms of cost, due to accurate identification and matching migration strategies, cold data can be migrated to low-cost storage devices 20 in a timely manner, which can significantly reduce the storage costs of the enterprise.

[0035] In terms of performance, since hot data is always stored on the high-performance storage device 10, the data access speed and system response time can be guaranteed to the greatest extent.

[0036] In another embodiment, device performance monitoring includes monitoring the read and write speed, latency, and bandwidth of the high-performance storage device 10 and the low-cost storage device 20 .

[0037] Monitoring the read and write speed, latency, and bandwidth of the device helps the intelligent scheduling algorithm to accurately adjust the migration strategy dynamically, further improving the efficiency of migration.

[0038] In another embodiment, the device performance monitoring further includes evaluating the load capacity and remaining capacity of the high-performance storage device 10 and the low-cost storage device 20 based on the performance indicators to provide a basis for resource allocation for data migration.

[0039] Accurate evaluation is a prerequisite for accurate resource allocation. Evaluating load capacity and remaining capacity based on performance indicators helps the intelligent scheduling algorithm to accurately adjust the migration strategy dynamically and further improve the efficiency of migration.

[0040] In another embodiment, the intelligent scheduling algorithm includes analyzing the best path for data migration based on the Dijkstra algorithm.

[0041] The Dijkstra algorithm is a shortest path algorithm from one vertex to the remaining vertices, and solves the shortest path problem in a weighted graph. In the present invention, it is used for path planning with high efficiency.

[0042] Data migration path planning based on the Dijkstra algorithm can complete data migration as quickly as possible and further improve the efficiency of data migration.

[0043] In another embodiment, before and after the hot and cold data migration, the data is checked for consistency by means of encrypted signatures.

[0044] Through real-time monitoring and error handling mechanisms, data consistency and integrity of the migration process are ensured, reducing the risk of data loss.

[0045] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A method for automatic migration of cold and hot data based on intelligent scheduling, characterized in that: The following steps are involved: Based on the migration strategy, migrate hot and cold data between high-performance storage devices and low-cost storage devices; The migration strategy is dynamically adjusted based on intelligent scheduling algorithms and device performance monitoring.

2. The method for automatic migration of cold and hot data based on intelligent scheduling according to claim 1 is characterized in that: The device performance monitoring includes monitoring the read and write speed, delay, and bandwidth of high-performance storage devices and low-cost storage devices.

3. The method for automatic migration of cold and hot data based on intelligent scheduling according to claim 2 is characterized in that: The device performance monitoring also includes evaluating the load capacity and remaining capacity of high-performance storage devices and low-cost storage devices based on performance indicators, providing a basis for resource allocation for data migration.

4. The method for automatic migration of cold and hot data based on intelligent scheduling according to claim 1 is characterized in that: The intelligent scheduling algorithm includes analyzing the optimal path of data migration based on the Dijkstra algorithm.

5. The method for automatic migration of cold and hot data based on intelligent scheduling according to claim 1 is characterized in that: Before and after hot and cold data migration, data consistency is verified through encrypted signatures.

Images