A data set native magneto-optical fusion hierarchical migration scheduling method

Abstract

This invention discloses a native magneto-optical-electrical fusion hierarchical migration scheduling method for datasets, belonging to the field of data storage technology. The method includes: using the dataset as the smallest scheduling unit, identifying the dataset's hot, warm, cold, or archived levels based on multi-dimensional indices such as access frequency, computing power status, data lifecycle, data value density, and security weight; performing cross-media hierarchical migration among three heterogeneous storage media—electrical storage, magnetic storage, and optical storage—based on the hierarchical identification results and a pre-migration decision mechanism, wherein the pre-migration decision mechanism jointly determines the migration timing based on cluster load, storage capacity, and business window; performing data consistency verification, breakpoint resumption, and fault rollback operations during the migration process, and performing batch sequential recording and physical disk locking operations on the archived datasets migrated to optical storage to form a compliant evidence storage chain. This invention enables overall migration scheduling at the dataset granularity, improving the efficiency and reliability of cross-media migration.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, specifically to a native magneto-optical-electric fusion hierarchical migration scheduling method for datasets, which is particularly suitable for localized deployment scenarios that need to balance data access performance, long-term archiving costs, data security, and migration reliability. Background Technology

[0002] With the deepening of digital transformation in government and enterprise sectors, the volume of business data is experiencing explosive growth. Single storage media (such as disks or optical discs alone) can no longer simultaneously meet the combined needs of high-performance access, low-cost archiving, and long-term data preservation. Therefore, a magneto-optical-electrical integrated storage architecture that combines multiple media such as solid-state drives (SSDs), hard disk drives (HDDs), and Blu-ray discs (BD-R) is gradually becoming the mainstream solution. This type of architecture typically stores frequently accessed, hot data on electrical storage media, while storing infrequently accessed, cold data on magnetic or optical storage media, aiming to achieve a balance between performance and cost.

[0003] However, existing cross-media data migration scheduling technologies still have several shortcomings in practical applications. First, most scheduling schemes use a single file or object as the smallest unit of operation, resulting in a relatively fine-grained scheduling granularity. When government and enterprise business uses complete datasets (such as a batch of related files generated from the same project) as carriers, fine-grained scheduling will lead to the frequent triggering of massive numbers of scattered migration tasks. This not only results in low scheduling efficiency but also easily causes the same business dataset to be stored in different media, thereby affecting the continuity and processing efficiency of subsequent data readings.

[0004] Secondly, the criteria for determining the "hot" or "cold" status of data are too simplistic. Current technologies generally rely solely on historical access frequency or recent access time to judge data popularity, neglecting factors such as the data's intrinsic value density, security weight, the status of associated computing tasks, and the data's lifecycle stage. This one-dimensional approach can easily lead to unreasonable scheduling decisions: high-value, high-security, low-frequency access data may be prematurely relegated to low-cost but low-performance storage media, while low-value, high-frequency redundant data occupies high-performance storage resources for extended periods, resulting in storage resource mismatch and failing to meet the actual needs of differentiated data management for government and enterprise clients.

[0005] Furthermore, the triggering mechanism for migration scheduling is often passive and delayed. The system often only starts the migration operation after the data access intensity has dropped significantly, and the migration timing does not fully consider the current load status of the cluster, the remaining storage capacity, and the business runtime. When large-scale data migration tasks are executed in a concentrated manner, they are prone to competing with front-end services for input / output (IO) bandwidth and computing resources, thereby causing business response delays or even service interruptions, affecting the stable operation of core businesses.

[0006] Furthermore, the data reliability assurance mechanisms during cross-media migration are still imperfect. When migrating datasets from electrical storage to magnetic or optical storage, existing solutions generally lack effective data consistency verification, breakpoint resumption, and fault rollback capabilities should network fluctuations, hardware failures, or abnormal task interruptions occur. This can lead to partial data loss, inconsistencies between the source and target data, or the inability to recover to the original state after a migration failure, failing to meet the stringent requirements for data integrity and recoverability in scenarios involving classified data.

[0007] The writing strategies for optical storage media also have shortcomings. Traditional solutions often employ random writing, which does not fully adapt to the physical characteristics of sequential recording on optical discs, easily accelerating disc wear and shortening their effective lifespan. Furthermore, once data is stored on optical storage, there is typically a lack of mandatory physical disk locking mechanisms and full-process audit log retention, failing to fully leverage the compliant evidence preservation advantages of the physically immutable nature of optical storage media and making it difficult to meet the tamper-proof and traceability requirements of long-term archiving scenarios.

[0008] In summary, existing cross-media migration scheduling methods in magneto-optical-electric integrated storage systems have significant technical deficiencies in terms of scheduling granularity, hot / cold determination dimensions, migration trigger timeliness, data reliability assurance, optical storage adaptation, and compliant evidence preservation. There is an urgent need for a hierarchical migration scheduling method that can use datasets as the granularity, integrate multi-dimensional identification, have a pre-migration decision mechanism, ensure migration consistency, and optimize optical storage writing. Summary of the Invention

[0009] This invention provides a native magneto-optical-electrical fusion hierarchical migration scheduling method for datasets, aiming to solve the problems existing in the cross-media data migration scheduling of the current magneto-optical-electrical fusion storage system, such as fragmented scheduling granularity, single cold and hot determination dimension, passive and delayed migration triggering, insufficient data reliability guarantee, poor optical storage write adaptability and lack of compliance evidence storage capability.

[0010] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: A native magneto-optical-electric fusion hierarchical migration scheduling method for datasets includes the following steps: using the dataset as the smallest scheduling unit, based on a multi-dimensional index of access popularity, computing power status, data lifecycle, data value density, and security weight, the corresponding dataset is identified as hot, warm, cold, or archived; based on the dataset hierarchy identification results and a pre-migration decision mechanism, cross-media hierarchical migration is performed between three heterogeneous storage media: electrical storage, magnetic storage, and optical storage. The pre-migration decision mechanism jointly determines the migration timing based on cluster load, storage capacity, and business window; during the cross-media migration of the dataset, data consistency verification, breakpoint resumption, and fault rollback operations are performed, and batch sequential recording and physical sealing and locking operations are performed on the archived datasets that are migrated to optical storage to form a compliant evidence storage chain.

[0011] Compared with the prior art, the present invention, by adopting the above technical solution, has the following beneficial effects: By replacing traditional file or object-level scheduling with datasets as the smallest scheduling unit, previously scattered files or objects are integrated into a complete business dataset for overall migration scheduling. This avoids the frequent triggering of numerous migration tasks and the fragmented storage of business data caused by fine-grained scheduling, improving the scheduling efficiency and read continuity of batch data. Simultaneously, a multi-dimensional index based on access frequency, computing power status, data lifecycle, data value density, and security weight identifies the hot / cold levels of datasets. This multi-dimensional identification mechanism incorporates data value and security level into the stratification criteria, helping to match appropriate storage media for high-value, high-security data and reducing resource mismatch caused by relying solely on access frequency.

[0012] Based on the hierarchical identification results of the dataset and the pre-migration decision mechanism, cross-media hierarchical migration is performed between electrical, magnetic, and optical media. The pre-migration decision mechanism determines the migration timing by jointly considering the current cluster IO load, the remaining capacity of each storage medium, and a preset business-allowed migration time window. The migration task is triggered when the cluster load is below a set threshold, the target medium has sufficient capacity, and the migration is within the business window. This avoids the migration task competing for IO resources with front-end services, reduces the impact on business response performance, and improves system stability during data transfer.

[0013] During the migration process, data consistency verification, breakpoint resumption, and fault rollback operations are performed to verify the integrity of data before and after the migration. If the transmission is interrupted, the transmission will continue from the breakpoint. If an anomaly occurs, the system will automatically restore the state to the state before the migration. This improves the reliability and fault tolerance of cross-media migration and reduces the risk of data loss or inconsistency caused by network fluctuations, hardware failures, or task interruptions.

[0014] Batch sequential recording and physical sealing of archived datasets stored in optical storage are performed, forming a compliant evidence chain. This adapts to the physical characteristics of sequential optical disc recording, reduces media wear caused by random writing, and extends the lifespan of optical storage devices. Physical sealing ensures that archived data is tamper-proof at the hardware level. Combined with the retention of the compliant evidence chain, this allows optical storage media to meet both long-term archiving cost requirements and the requirements for tamper-proofing and audit traceability of confidential data.

[0015] In summary, the native magneto-optical-electric fusion hierarchical migration scheduling method for datasets provided by this invention solves the problems of low scheduling efficiency, inaccurate hierarchical layering, business impact, unreliable migration, and weak compliance evidence storage in existing technologies by combining dataset granular scheduling, multi-dimensional index recognition, pre-migration decision mechanism, reliability assurance operations during migration, and optical storage write optimization with solidified evidence storage. It is suitable for localized, high-security, and large-scale magneto-optical-electric fusion storage scenarios in government and enterprises. Attached Figure Description

[0016] The accompanying drawings, which form part of this specification, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating a native magneto-optical-electric fusion hierarchical migration scheduling method for datasets provided in an embodiment of the present invention. Detailed Implementation

[0017] Before describing the technical solution of the present invention in detail, a brief explanation of the technical problem and design concept addressed by the present invention will be given first.

[0018] In the actual deployment of magneto-optical-electric converged storage systems, especially in localized, unmanaged scenarios for government and enterprise users, business data is typically organized and managed as complete datasets. Existing technologies mostly use files or objects as the basic scheduling unit, coupled with a hot / cold tiering strategy based on access frequency, to enable data transfer between different storage media. However, as mentioned in the background section, this scheduling method exposes a series of problems in large-scale, high-security, and data-compliant scenarios for government and enterprises, including low overall dataset scheduling efficiency, resource misallocation due to a single tiering judgment dimension, passively delayed migration affecting business stability, lack of consistency guarantees during cross-media migration, and insufficient compliance evidence storage capabilities due to a coarse optical storage write strategy. These shortcomings restrict the further promotion and application of magneto-optical-electric converged architecture in government and enterprise data management.

[0019] To address the aforementioned issues, this invention provides a native magneto-optical-electric fusion hierarchical migration scheduling method for datasets. The core design idea of ​​this method is: to use the complete dataset as the smallest operational unit for scheduling and migration, replacing traditional file or object-level scheduling; to construct a multi-dimensional index recognition model that integrates access popularity, computing power status, data lifecycle, data value density, and security weight to determine the hot / cold level of the dataset; to introduce a pre-migration decision mechanism that jointly determines the migration timing based on cluster load, storage capacity, and business window, avoiding migration during peak business periods; to integrate data consistency verification, breakpoint resumption, and fault rollback mechanisms during cross-media migration to ensure migration reliability; and to address the characteristics of optical storage media, to adopt batch sequential recording and physical disk locking methods to achieve hardware-level immutable archiving and form a traceable and compliant evidence storage chain.

[0020] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] See Figure 1 This invention provides a native magneto-optical-electric fusion hierarchical migration scheduling method for datasets, comprising the following steps: S1 uses the dataset as the smallest scheduling unit and identifies the hot, warm, cold, or archived level of the corresponding dataset based on a multi-dimensional index of access popularity, computing power status, data lifecycle, data value density, and security weight.

[0022] In practical applications, related data generated by the same business or process is usually used and managed as a whole. If a single file or object is used as the scheduling unit, a large amount of related data needs to trigger migration tasks separately, and may be stored in different media, affecting the efficiency of subsequent batch reads. Integrating multiple data objects with business relevance into a complete dataset serves as the basic unit for scheduling and migration.

[0023] Based on this, information is collected for each dataset across multiple dimensions, including access frequency, associated computing power task status, data lifecycle stage, data value density, and security weight. Through comprehensive calculations, the dataset is determined to belong to one of the following levels: hot data, warm data, cold data, or archived data. Different levels correspond to different storage media and access performance requirements.

[0024] S2, based on the dataset hierarchical identification results and the pre-migration decision mechanism, performs cross-media hierarchical migration among the three types of heterogeneous storage media: electrical storage, magnetic storage, and optical storage. The pre-migration decision mechanism jointly determines the migration timing based on cluster load, storage capacity, and business window.

[0025] Based on the dataset hierarchy determined in step S1, the dataset is migrated across three media types: electrical storage, magnetic storage, and optical storage. The timing of the migration is determined by a pre-migration decision mechanism, which is jointly determined based on the current I / O load of the cluster, the remaining capacity of each storage medium, and a preset business migration time window. Migration is triggered when the cluster load is below a set threshold, the remaining capacity of the target storage medium is sufficient, and the target storage medium is within the business window; otherwise, migration is delayed.

[0026] S3 performs data consistency verification, breakpoint resume and fault rollback operations during the cross-media migration of datasets, and performs batch sequential burning and physical sealing and locking operations on the archived datasets that are moved to optical storage, forming a compliant evidence storage chain.

[0027] Cross-media migration involves the transfer of data between different physical storage devices, and anomalies during the transfer process may lead to data corruption or loss. During cross-media migration, data consistency checks, breakpoint resumption, and fault rollback operations ensure the reliability of the migration. For archived datasets ultimately migrated to optical storage, batch sequential burning is used for writing, and physical disk sealing and locking are performed. Simultaneously, relevant archival records are retained, forming a traceable and compliant evidence chain.

[0028] The above-described technical solutions of the embodiments of the present invention have the following beneficial effects: (1) Using datasets as the smallest scheduling unit replaces the traditional fine-grained scheduling method at the file or object level, reducing the number of times migration tasks are triggered, avoiding the same business-related data being stored in different media, and improving the scheduling efficiency of batch data migration and the continuity of business data reading.

[0029] (2) Based on the multi-dimensional index of access frequency, computing power status, data life cycle, data value density and security weight, the hot and cold levels of the dataset are identified. Data value and security level are included in the judgment criteria, so that high-value and high-security data can obtain storage resources that match their needs, and reduce the resource mismatch caused by judging solely by access frequency.

[0030] (3) Cross-media layered migration is performed based on the pre-migration decision mechanism. The migration timing is determined jointly based on the cluster load, storage capacity and business window. The migration task is scheduled to be executed when the system load is low, the capacity is sufficient and it is within the business window, which reduces the interference of migration operation on the front-end business response performance and improves the stability of the system during data flow.

[0031] (4) Data consistency verification, breakpoint resume and fault rollback operations are performed during the migration process, which improves the reliability of cross-media migration and reduces the risk of data loss, data inconsistency or unrecoverable source data due to network fluctuations, hardware failures or task abnormalities.

[0032] (5) Batch sequential burning and physical sealing and locking operations are performed on the archived datasets that are buried in optical storage, which adapts to the physical characteristics of sequential writing of optical discs, reduces media wear caused by random writing, and extends the service life of optical storage devices; at the same time, a hardware-level immutable and traceable compliant evidence storage chain is formed.

[0033] According to some embodiments of the present invention, using a dataset as the smallest scheduling unit includes: integrating related data of the same business scenario, the same level of confidentiality, or the same generation cycle into a complete business dataset, and performing batch cross-media migration scheduling with the complete dataset as a whole, without splitting the dataset, modifying the internal structure of the data, or transferring user data ownership; at the same time, assigning a globally unique identifier to each dataset and binding version information for full lifecycle traceability, access control, and version backtracking.

[0034] In practical applications, data generated by the same business system is often naturally grouped according to project, time window, or security level. For example, all raw images and processing results generated from a remote sensing mapping task, all financial settlement vouchers for a quarter, or a set of design drawings of the same confidentiality level. If these related data are scattered into independent units for separate processing, it requires maintaining a large number of fragmented task states, and multiple address lookups are needed when reading data from different media, reducing overall efficiency. Therefore, a method is adopted to pre-integrate related data from the same business scenario, the same confidentiality level, or the same generation cycle into a complete business dataset. This dataset is logically considered an indivisible whole, and all subsequent hot / cold data identification, migration decisions, and media allocation are performed on the entire dataset. When performing batch cross-media migration, the migration of all related data is completed at once using this dataset as the operation unit, without splitting the internal file or object structure of the dataset, nor making any modifications to the storage format of the data content. Simultaneously, the data ownership information (such as the department, creator, access control list, etc.) is migrated along with the dataset and remains unchanged throughout the scheduling process, always belonging to the original user or business system. This approach avoids the problems of scattered data storage and ownership confusion caused by split scheduling.

[0035] When implementing dataset identification and version management, a globally unique identifier is assigned to each integrated business dataset. This identifier is unique throughout the entire storage system and does not change with data migration or media changes. Based on this identifier, a full lifecycle tracking record of the dataset can be established, including creation time, target media for each migration, access operation logs, and cold / hot tier change history. Simultaneously, version information is bound to the dataset. When a dataset undergoes content updates (e.g., adding files, modifying existing data, or deleting parts of content), a new version number is automatically generated, and the states before and after the change are saved separately. Version information allows users to trace back to any historical state of the dataset, such as viewing the original data version before archiving during compliance audits. The identifier and version information together constitute the foundation for dataset management. The access control system determines a user's access permissions to the dataset based on the dataset identifier, while the version rollback mechanism allows users to restore specific versions of data content as needed, ensuring that the dataset remains identifiable, traceable, and controllable throughout the entire cross-media migration process.

[0036] This embodiment integrates related data from the same business scenario, the same level of confidentiality, or the same generation cycle into a complete business dataset for overall batch migration. This reduces the number of tasks and management complexity, and avoids the related data being scattered across different media, thereby improving batch migration efficiency and data reading continuity. The operation method of not splitting, modifying, or transferring ownership ensures the integrity and ownership stability of data during cross-media flow, meeting the requirements of government and enterprise businesses for data sovereignty control. By assigning a globally unique identifier to each dataset and binding version information, the entire process of dataset traceability from creation to archiving and even destruction is realized. At the same time, it supports version-based permission control and historical status retrospection, enhancing the granularity of data management and compliance audit capabilities.

[0037] According to some embodiments of the present invention, the identification of the hot, warm, cold, or archived levels of a corresponding dataset based on a multi-dimensional index of access popularity, computing power status, data lifecycle, data value density, and security weight includes: automatically calculating an access popularity index based on the real-time access frequency and concurrent access count of the dataset; automatically calculating a computing power status index based on the real-time running status and computing power occupancy time of the computing power tasks associated with the dataset; automatically calculating a lifecycle index based on the creation time, business flow status, and preset archived time limit of the dataset; automatically calculating a data value density index based on the access frequency, growth rate, citation count, number of associated tasks, and data lifecycle position of the dataset; automatically calculating a security weight index based on the encryption level, de-identification status, access control complexity, and audit requirement level of the dataset; and calculating a comprehensive score of the dataset using a weighted fusion algorithm, wherein the comprehensive score is calculated as: Score = α×H + β×C + γ×L + δ×V + ε×S, where H is the access popularity index, C is the computing power status index, L is the data life cycle index, V is the data value density index, S is the security weight index, and α, β, γ, δ, and ε are the corresponding weight coefficients; the dataset is divided into four levels: hot, warm, cold, and archived based on the comprehensive score.

[0038] The collection of access popularity index is continuously performed during the operation of the method in this embodiment. Specifically, the frequency of access and the number of concurrent access requests for each dataset are monitored. A statistical time window can be set, such as the last 24 hours or the last 7 days. The access frequency and the number of concurrent accesses for the dataset within this time window are recorded, and the above values ​​are normalized into access popularity index values ​​according to preset conversion rules. The higher the access frequency and the denser the concurrent access for a dataset, the larger its access popularity index value, indicating that it is currently in a highly active state.

[0039] Collecting the computing power status index requires obtaining information about the computing tasks associated with the current dataset. In scenarios where computing power and storage work together, the dataset may be being used by an AI training task, a video transcoding task, or an industrial simulation task. Through the interface between the storage system and the upper-layer computing power scheduling platform, the real-time running status (e.g., running, queued, or completed) and computing power occupancy duration of the computing tasks associated with the dataset can be obtained. The longer the computing power is occupied and the higher the task priority of the dataset, the larger its computing power status index value, indicating that the data still has high importance in the current computing process.

[0040] The lifecycle index is primarily collected based on the dataset's creation time, business workflow status, and preset archiving timeframe. It records the dataset's creation timestamp, its stage in the business workflow (e.g., raw data collection, processing, analysis output, awaiting archiving), and the pre-configured archiving timeframe strategy. For example, video surveillance data can be configured to be automatically archived after 90 days; transaction logs can be configured to be archived after 180 days. The longer the dataset's creation time, the closer its workflow status is to completion, and the closer it is to the preset archiving timeframe, the smaller its lifecycle index value, indicating that the data is gradually transitioning to a "cold" state.

[0041] The data value density index is automatically calculated based on the dataset's access frequency, growth rate, citation count, number of associated tasks, and its position in the data lifecycle. For example, datasets with high access frequency, rapid growth rate, numerous citations from other datasets or tasks, a large number of associated computational tasks, and are in an active phase of their lifecycle have a higher data value density index; conversely, datasets that have not been accessed for a long time, have stagnant growth, no citation records, no associated tasks, and are nearing or at the end of their lifecycle have a lower data value density index. Through the comprehensive calculation of these multiple factors, a quantitative value density index can be generated for each dataset.

[0042] The security weight index is automatically calculated based on the dataset's encryption level, anonymization status, access control complexity, and audit requirement level. Datasets with higher encryption levels, stricter anonymization, more complex access control policies, and higher audit requirement levels have a higher security weight index. This index directly affects the dataset's scheduling decisions; datasets with higher security weights typically need to be stored on more secure storage media and subject to stricter access control.

[0043] After obtaining the specific values ​​of the five indices, a weighted fusion algorithm is used for comprehensive calculation. The weighted fusion algorithm assigns a weight coefficient to each index, which can be dynamically adjusted according to the operational phase or business strategy. For example, during peak business periods, the weight of the access popularity index can be appropriately increased to more sensitively respond to hot data migration; during periods of strict compliance audits, the coefficient of the security weight index can be increased. The comprehensive score of the dataset is obtained by multiplying each index value by its corresponding weight and summing the results. Based on the threshold range into which the comprehensive score falls, the dataset is classified into one of four levels: hot, warm, cold, and archived. For example, a score above the first threshold is hot data, between the first and second thresholds is warm data, between the second and third thresholds is cold data, and below the third threshold is archived data.

[0044] This embodiment of the method collects indices from five dimensions: access popularity, computing power status, data lifecycle, data value density, and security weight, and uses a weighted fusion algorithm for comprehensive calculation. This makes the determination of the hot and cold levels of the dataset more comprehensive and accurate, avoiding misjudgments caused by relying solely on a single access frequency indicator. At the same time, it provides multi-dimensional quantitative basis for the scheduling decision of the dataset.

[0045] According to some embodiments of the present invention, the cross-media hierarchical migration between the three heterogeneous storage media of electrical storage, magnetic storage, and optical storage includes: scheduling hot data to electrical storage media, scheduling warm data to magnetic storage media, and sinking cold or archived data to optical storage media according to the hierarchical identification results; and, in response to computing power task requests or business access needs, re-migrating archived datasets from optical storage media to electrical or magnetic storage media by heating them up.

[0046] To facilitate the scheduling of data migration to lower temperature thresholds, after determining the hot / cold level of the dataset, the appropriate target storage medium type is matched based on the determination result. For datasets identified as hot data, they are scheduled to electrical storage media, such as solid-state drives (SSDs) or non-volatile memory fast channel (NVMe) hard drives. Electrical storage media offer microsecond-level read / write latency and high input / output operations per second (IOPS), meeting the high-frequency access requirements of hot datasets. For datasets identified as warm data, they are scheduled to magnetic storage media, such as hard disk drives (HDDs) or linear open format (LTO) tape libraries. Magnetic storage media have lower unit storage costs than electrical storage and access latency in the millisecond range, making them suitable for warm datasets with moderate access frequency and less stringent response time requirements. For datasets identified as cold data or archived data, they are migrated to optical storage media, such as Blu-ray discs (BD-R) or write-once-read-many (WORM) discs. Optical storage media offer a lifespan of decades or more, extremely low offline storage power consumption, and the characteristic of being unmodifiable after physical writes, making them suitable for long-term preservation and compliant archiving. During the scheduling process described above, specific storage device instances are automatically selected based on the current capacity usage, health status, and priority of each storage medium.

[0047] When scheduling data warm-up recovery, upon receiving a request from a front-end computing task (e.g., starting an AI training task that requires access to historical archived data) or a user's direct business access request, the system first determines the current storage medium of the requested dataset. If the dataset has been moved to optical storage, the warm-up recovery process is automatically initiated. This process includes: locating the optical disc containing the dataset from the optical storage medium's disk library; loading the disc into the optical drive; reading the data and writing it to a temporary or target storage area; and completing data integrity verification. The target medium for warm-up recovery can be electrical storage (for scenarios requiring high-performance access) or magnetic storage (for scenarios requiring lower access latency but not necessarily expensive electrical storage). During the warm-up process, the dataset's metadata is simultaneously updated, changing its status from "archived" or "cold" to "warm" or "hot," and subsequent scheduling strategies are adjusted based on the new status. After warm-up, the requester accesses the data from the specified electrical or magnetic storage medium, achieving a faster response time than reading directly from optical storage. It allows setting data retention policies after warming up, such as retaining data in electrical storage for 7 days and then automatically relocating it back to optical storage if there is no further access, in order to avoid high-performance resources being occupied for a long time.

[0048] This embodiment's method schedules hot data to electrical storage, warm data to magnetic storage, and cold or archived data to optical storage based on hierarchical identification results. This forms a three-level media adaptation mechanism based on data access characteristics and value gradients, ensuring the performance of hot data access while reducing the long-term storage cost of cold data. In response to computing task requests or business access needs, archived datasets are warmed up and migrated from optical storage to electrical or magnetic storage, enabling archived data to be reactivated and enter the online computing process. This avoids the loss of reusability due to data sinking and enhances business support capabilities in historical data backtracking and secondary computing scenarios.

[0049] According to some embodiments of the present invention, the data consistency verification, breakpoint resumption, and fault rollback operations performed during the cross-media migration of the dataset specifically include: calculating the global hash value of the source dataset before migration, calculating and comparing the global hash value of the target dataset after migration to verify data integrity; when migration is interrupted, resuming the transmission of remaining data from the recorded breakpoint position; when migration is abnormal, clearing the abnormal data already transmitted at the target end and restoring the source dataset to its state before migration.

[0050] When performing data consistency verification, before initiating the cross-media migration task, all data blocks of the complete dataset from the source are first read, and a global hash value is calculated using a secure hash algorithm (e.g., SHA-256). This hash value represents the complete data state of the source dataset at the start of the migration. After the migration task is completed and all data has been written to the target storage medium, the dataset written to the target is read again, and the same hash algorithm is used to calculate the global hash value for the target. The source hash value and the target hash value are compared bit by bit. If they match, it indicates that no data corruption, loss, or damage occurred during the migration process, and the data was completely copied from the source to the target. If they do not match, the migration is marked as a failure, and subsequent failure rollback operations are triggered. This hash verification method does not rely on filenames or metadata; it directly verifies the data content and can detect any bit-level errors during data transmission.

[0051] To enable resumable downloads, the dataset is transmitted sequentially in blocks during the migration task. After each successful transmission of one or more data blocks, progress information, including the sequence number, byte offset, or fragment identifier of the transmitted data blocks, is recorded in persistent storage (e.g., a database or metadata service). If the migration is unexpectedly interrupted due to network outages, node restarts, hardware failures, or other reasons, the system saves this breakpoint information. Once the fault is resolved and the migration task is rescheduled, the breakpoint information is retrieved, the first untransmitted data block is located, and the remaining data is transmitted from that breakpoint forward, without retransmitting already transmitted portions. This approach avoids the time overhead of retransmitting the entire dataset due to interruption, making it particularly suitable for migration scenarios across slow networks or involving massive datasets.

[0052] To enable fault rollback, when anomalies are detected during migration (such as hash verification failure, target-side write errors, source-side data read errors, or task timeouts), the current migration operation is first stopped, releasing occupied transmission resources. Then, a clear command is sent to the target storage medium to delete all written abnormal data blocks. If necessary, the storage space is reformatted or reclaimed to ensure that no incomplete data remains on the target. Simultaneously, the original state of the source dataset is maintained. If source data has been locked or marked during migration (e.g., setting a migration flag to prevent concurrent modifications), these locks are released, and the source data is restored to its state before migration began. In some scenarios where source data may be deleted after migration (e.g., a cut operation), the rollback mechanism cancels the deletion operation, ensuring the source data remains intact and usable. Through these operations, migration failure will not result in unusable source data or inconsistent split states between the source and target data.

[0053] This embodiment uses a global hash value comparison method to verify data integrity before and after migration, which can accurately detect data errors during cross-media transmission and ensure the correctness of the migration results. When migration is interrupted, transmission is resumed from the recorded breakpoint, avoiding the time waste caused by retransmitting all data and improving the efficiency and recovery capability of large-scale data migration. When migration is abnormal, abnormal data on the target end is automatically cleared and the source end state is restored, ensuring the cleanliness and consistency of data at both ends after migration failure and reducing the risk of data irrecoverability or data pollution caused by migration failure.

[0054] According to some embodiments of the present invention, the migration process also includes parallel acceleration and traffic control operations, specifically including: splitting the ultra-large-scale dataset into multiple sub-data blocks and using multi-threaded or multi-node parallel transmission; simultaneously monitoring the overall IO load and network bandwidth usage of the cluster, dynamically adjusting the migration traffic threshold and the number of parallel threads, limiting the flow during high-load periods and migrating at full speed during low-load periods.

[0055] For migrating extremely large datasets, traditional serial transmission methods send data blocks one by one, with transmission time directly proportional to the data volume. When the dataset reaches TB or even PB levels, a single migration can last for days or even weeks, during which network links and storage I / O ports are occupied for extended periods. To improve migration efficiency, the extremely large dataset is first logically divided into multiple equal-sized sub-data blocks. The size of each sub-data block can be dynamically set based on network conditions, storage device performance, and the total size of the dataset (e.g., 64MB, 256MB, or 1GB). After partitioning, an independent transmission task is assigned to each sub-data block. These transmission tasks are executed concurrently using multi-threading (using multiple CPU cores on the same server for parallel transmission) or multi-node methods (distributing the transmission load across multiple servers). The transmission of multiple sub-data blocks overlaps in time, significantly improving the overall migration throughput and drastically reducing the completion time.

[0056] While transmitting data in parallel, the overall I / O load and network bandwidth usage of the cluster are continuously monitored. Specifically, by collecting metrics such as disk read / write queue length of storage nodes, I / O latency, network interface send / receive rates, and switch port congestion status, the current resource utilization is calculated in real time. Based on this monitoring data, flow control parameters during the migration process are dynamically adjusted. Flow control parameters mainly include two dimensions: first, the maximum bandwidth threshold that the migration task can occupy (e.g., not exceeding 30% of the total bandwidth); and second, the number of threads or concurrent tasks for parallel transmission. When the cluster is detected to be under high load (e.g., I / O latency exceeding the set threshold or bandwidth utilization exceeding 80%), the migration flow threshold is automatically lowered, and the number of parallel threads is reduced, thereby limiting the resource consumption of the migration task and ensuring that front-end business requests receive sufficient I / O and network resources. Conversely, when the cluster is detected to be under low load (e.g., bandwidth utilization below 20% during late-night hours), the migration flow threshold is raised, and the number of parallel threads is increased, allowing the migration task to transmit at a higher rate and making full use of idle resources. This dynamic adjustment process is continuous and adaptive, requiring no manual intervention.

[0057] This embodiment of the method splits the ultra-large-scale dataset into multiple sub-data blocks and uses multi-threaded or multi-node parallel transmission, which significantly shortens the migration time of massive data and improves the overall throughput. At the same time, it monitors the cluster IO load and bandwidth usage and dynamically adjusts the traffic threshold and the number of parallel threads, so that the migration task can adaptively adjust the resource usage according to the current load status, reduce the impact on business under high load, and make full use of idle resources under low load, thus achieving a balance between migration efficiency and business stability.

[0058] According to some embodiments of the present invention, a global cache bypass acceleration operation is also performed during the migration process, specifically including: constructing a three-level cache architecture of node local cache, cluster shared cache and hot spot bypass cache, and preloading frequently accessed hot data into the cache; during the dataset migration process, read and write requests for front-end hot data are responded to through the bypass cache channel, which is isolated from the IO channel of the background migration task.

[0059] The three-tier caching architecture consists of three layers from bottom to top: node-local cache, cluster shared cache, and hotspot bypass cache. The node-local cache is deployed on each storage node, using the node's memory or high-speed SSD as the caching medium. It stores frequently accessed data copies for that node, reducing network overhead from cross-node data reads. The cluster shared cache is independent of specific storage nodes and is typically composed of a distributed memory cluster or high-speed storage devices. It is shared by all nodes in the cluster and stores frequently accessed datasets. The hotspot bypass cache is located at the front end of the three-tier architecture, directly facing business access requests. It is implemented using a high-performance in-memory database or key-value store system and is specifically used to store the most frequently accessed hotspot datasets at any given time. Data flow exists between these three tiers: the node-local cache and the cluster shared cache can synchronize with each other based on access patterns, and the hotspot bypass cache loads data from the cluster shared cache or direct backend storage. By analyzing statistical information such as the access frequency and recent access time of the dataset, the system automatically identifies frequently accessed hot data and loads this data in advance from the backend storage medium (electric, magnetic, or optical storage) into the cache at each level, so that subsequent accesses can directly retrieve the data from the cache without triggering I / O operations on the backend storage.

[0060] During dataset migration, the background migration task needs to read data from the source storage medium and write it to the target storage medium, which consumes the storage system's I / O channels and network bandwidth. To mitigate the impact of migration operations on front-end services, read / write requests for hot data on the front end are isolated from the I / O channels of the background migration task. The specific implementation is as follows: When a business system initiates a read or write request for a dataset, the request is first routed to a hotspot bypass cache. If the requested data is already in the cache, the result is returned directly through the bypass cache channel; the entire process does not pass through the back-end storage's I / O channel and does not share any resources with the migration task. If the requested data is not in the cache (cache miss), the request is forwarded to the back-end storage, but at this time, a separate business I / O channel is still used, logically or physically separated from the channel occupied by the migration task. The migration task is restricted to execution in a dedicated background I / O channel, whose bandwidth and queue depth are independently managed, preventing it from preempting channel resources used by business requests. Through this I / O channel isolation, even if the migration task transfers large amounts of data at high speed, front-end business requests can still obtain fast responses through the bypass cache channel, minimizing the impact of the migration process on business access performance.

[0061] This embodiment constructs a three-level caching architecture consisting of node-local caching, cluster-shared caching, and hotspot bypass caching. It preloads frequently accessed hot data into the cache, significantly improving the access speed of hot data and reducing the load pressure on the backend storage. During the migration process, read and write requests for frontend hot data are responded to through the bypass cache channel, which is isolated from the IO channel of the backend migration task. This avoids the migration task and business requests competing for IO resources, achieving low or even no awareness of the migration operation on the frontend business, and ensuring the overall response performance during data transfer.

[0062] According to some embodiments of the present invention, the step of performing batch sequential recording and physical disk locking operations on the archived datasets sinking to optical storage to form a compliant evidence storage chain specifically includes: periodically collecting cold datasets to be archived, and sequentially recording them to the optical storage medium in batches; triggering physical disk locking after recording is completed, so that the archived datasets are in a hardware-level immutable state; and retaining the archiving time, disk location information, recording logs and disk locking status to form a traceable compliant evidence storage chain.

[0063] When implementing periodic data collection and batch sequential burning, a collection cycle is set, such as once every 24 hours or once a week. At the end of each cycle, all datasets identified as cold or archived data that have not yet been written to optical storage are scanned and collected into a single burning queue. The datasets in the queue are arranged according to a preset sorting rule, typically in ascending or descending order based on archiving priority, generation timestamp, or data size. After sorting, multiple datasets are continuously written to the same one or more optical discs in sequential burning mode, starting from the beginning address of the disc and writing sequentially without gaps, until all collected datasets are written or the disc capacity is full. Compared to random writing, batch sequential burning reduces the number of seek operations and start / stop operations of the optical drive, reducing mechanical wear on the optical disc media and the aging speed of the laser head. Simultaneously, batch burning combines multiple writes into a single centralized operation, reducing the overhead of burning task scheduling.

[0064] The physical sealing and locking operation is triggered after batch burning is complete. A sealing command is sent to the optical disc drive, putting the disc into a read-only state. Any subsequent write, modification, or deletion requests will be rejected by the drive hardware. For Blu-ray Discs (BD-R) or Write-once-Read-Many (WORM) discs, the sealing operation is irreversible; once sealed, the disc content is permanently fixed. This hardware-level immutability stems from the physical structure of the optical storage medium: during the burning process, the laser leaves permanent marks on the dye layer or phase change layer, which cannot be altered by any software means after sealing. Therefore, archived datasets gain compliance protection against tampering and deletion after sealing, making them suitable for confidential or credential-related data that requires long-term preservation and cannot be modified.

[0065] The construction of a compliant evidence storage chain relies on the retention of information throughout the entire archiving process. During each batch burning and sealing operation, the following information is automatically recorded: archiving time (timestamp accurate to milliseconds); optical disc location information (including the physical slot number, volume label, barcode, or RFID tag); burning log (including a list of datasets written, the size of each dataset, the starting write address, and verification results); and sealing status (whether sealing was successful, sealing time, and the executor or caller of the sealing operation). This information is written to an independent audit database or blockchain evidence storage system, and hash chain or digital signature technology is used to prevent log tampering. When a compliance audit is required, auditors can query the corresponding evidence storage records based on the dataset identifier or archiving time period to verify the original content and sealing status of the archived data, forming a complete evidence loop.

[0066] This embodiment of the method reduces the number of random writes and mechanical wear on optical storage media by periodically collecting cold datasets to be archived and performing batch sequential burning in sequence, thus extending the lifespan of the optical discs. After burning is completed, a physical disk sealing lock is triggered, making the archived datasets in a hardware-level immutable state, meeting the anti-tampering compliance requirements for long-term preservation of classified data. The archiving time, disk location information, burning logs, and disk sealing status are all retained in full, forming a traceable and compliant evidence storage chain, providing government and enterprise users with complete data archiving audit evidence, and enhancing the credibility and legal effect of data evidence storage.

[0067] According to some embodiments of the present invention, the method further includes a distributed sharding scheduling step: splitting the metadata of the massive dataset into multiple data shards and distributing each shard in multiple cluster nodes; distributing the migration scheduling task to each cluster node for parallel execution through a cluster load balancing mechanism; and horizontally increasing the number of cluster nodes in response to the growth of the dataset size or scheduling load.

[0068] As business data volumes grow to petabyte (PB) levels or even 10 PB or more, the centralized storage of metadata on a single node becomes a performance bottleneck. On one hand, the number of dataset entries managed by the metadata service may reach hundreds of millions or even billions. With all query and update operations concentrated on a single node, the CPU and memory resources of that node quickly become exhausted. On the other hand, if that node fails, the entire scheduling scheme becomes unavailable. To address this issue, this embodiment employs a metadata sharding storage strategy. Specifically, the metadata of all datasets is evenly divided into multiple independent data shards according to a certain sharding key (e.g., the hash value of the dataset ID). Each shard contains a portion of the dataset's metadata records. These shards are distributed across multiple nodes in the cluster, with each node responsible for managing its own shards. For metadata read / write requests, the shard to which the request belongs is first calculated based on the dataset identifier in the request, and then the request is routed to the corresponding node for processing. This approach distributes the metadata management load across multiple nodes, eliminating the performance limitations of a single node and the risk of single-point-of-failure.

[0069] At the task execution scheduling level, a cluster load balancing mechanism is introduced. When cross-media migration tasks need to be performed (e.g., migrating a batch of datasets from magnetic storage to optical storage), the scheduler does not process all tasks serially on a single node. Instead, it encapsulates the migration operation for each dataset as an independent subtask. These subtasks are placed in a global task queue. Multiple nodes in the cluster periodically pull subtasks to be executed from the queue and execute them in parallel. To balance the load across nodes, a weighted round-robin, least-connection, or dynamic load balancing algorithm based on the real-time CPU / memory utilization of nodes is used to preferentially allocate newly generated subtasks to nodes with lighter current loads. In this way, the execution throughput of migration tasks increases linearly with the number of nodes, avoiding scheduling latency caused by tasks piling up on a single node.

[0070] This embodiment supports horizontal scaling to adapt to the continuous growth of dataset size and scheduling load. When the resource utilization of existing cluster nodes reaches a preset threshold (e.g., average CPU utilization exceeds 70% or the storage capacity of a single node for metadata shards is close to its limit), administrators or automated operation and maintenance systems can add new physical machines or virtual machines to the cluster. After the new node is added, a rebalancing operation is automatically performed: some metadata shards are migrated from the original nodes to the new nodes, and the original global task queue is also opened to pull permissions for the new nodes. The entire process does not require stopping the ongoing migration task, nor does it require modifying the configuration of the upper-layer business application. By horizontally increasing the number of cluster nodes, both metadata storage capacity and scheduling task processing capacity can be scaled approximately linearly, thereby supporting the normalized scheduling needs of datasets ranging from TB to over 10PB.

[0071] This embodiment of the method splits the metadata of massive datasets into multiple shards and distributes them across multiple cluster nodes, eliminating the performance bottlenecks and single-point-of-failure risks of centralized metadata management, and improving the scalability and availability of the overall storage architecture. By distributing migration scheduling tasks to each node for parallel execution through a cluster load balancing mechanism, the throughput efficiency of large-scale migration tasks can be improved and the completion time of batch scheduling can be shortened. The number of cluster nodes can be increased in response to the growth level of the dataset or the scheduling load, giving the entire solution horizontal elastic scaling capability. It can smoothly adapt to government and enterprise data storage scenarios from small and medium scale to ultra-large scale (10PB and above), avoiding the dilemma of needing to redesign the architecture due to capacity growth.

[0072] Additionally, according to some embodiments of the present invention, the method further includes the step of configuring a visual scheduling strategy engine, supporting timed triggering, business event triggering, storage space threshold triggering, and data popularity threshold triggering modes, and automatically executing the hierarchical, migration, archiving, or warm-up callback operations of the dataset as needed.

[0073] The configuration of the visual scheduling strategy engine is achieved through a graphical interface. After logging into the scheduling management platform, operations and maintenance personnel can access the strategy configuration page and set scheduling rules for different datasets or dataset categories. Rules include two basic elements: trigger conditions and execution actions. Trigger conditions can be a point in time or a period (timed trigger), a change in specific business status (business event trigger), the remaining capacity of the storage cluster falling below a set percentage (storage space threshold trigger), or a change in dataset access popularity exceeding a set range (data popularity threshold trigger). Execution actions include migrating the dataset from one level to another (e.g., reducing hot data to lukewarm data), cross-media migration (from electrical storage to magnetic storage), archiving (sinking to optical storage), or warm-up callback (restoring from optical storage to electrical / magnetic storage). After configuration, the engine persists the rules and monitors the system's operating status in real time, automatically initiating the corresponding scheduling task when the trigger conditions are met.

[0074] The timed trigger mode relies on a time scheduler. Users can set a single execution time (e.g., 23:00 on December 31, 2026) or a periodic execution plan (e.g., every Sunday at 2:00 AM) in the visual interface. At the preset time, the scheduler automatically generates the corresponding migration or archiving task and submits it to the task queue for execution. This mode is suitable for batch operations with a clear time window, such as end-of-month archiving and quarterly data cleanup.

[0075] Implementing a business event-triggered model requires listening to status change signals from upper-layer business systems. For example, when a data processing task is completed, the business system can notify the scheduling engine via an API call that "Dataset XXX has been processed." Upon receiving this event, the scheduling engine determines, based on preset rules, whether the dataset needs to be degraded from hot data to warm data or archived. Common events include: dataset creation, dataset version update, completion of computation tasks, and changes in data lifecycle status. By binding business events, scheduling operations can be seamlessly integrated with business processes, avoiding the lag inherent in fixed-time triggering.

[0076] The storage space threshold triggering mode is implemented based on real-time monitoring of the capacity utilization of each storage medium in the cluster. Operations personnel can set high-water level thresholds and emergency-water level thresholds for each type of storage medium (electric, magnetic, optical) in a visual interface. For example, when the electric storage utilization reaches 80%, a cooling migration is triggered, migrating some warm data to magnetic storage; when it reaches 90%, an emergency migration is triggered, prioritizing the migration of the coldest data to optical storage. When the monitoring module detects that the current utilization of a certain type of medium exceeds the set threshold, it automatically triggers the corresponding data migration operation to release space on that medium, preventing business write failures due to insufficient capacity.

[0077] The implementation of the data popularity threshold trigger mode relies on the multi-dimensional hot / cold identification model established in step S1. The storage system continuously calculates hot / cold scores for each dataset and monitors the trend of score changes. Users can set popularity increase and decrease thresholds in the visualization interface. For example, when the access frequency of a cold dataset suddenly increases and its hot / cold score rises above the set increase threshold, a heat-up callback operation is automatically triggered, restoring the dataset from optical or magnetic storage to electrical storage. Conversely, when the hot / cold score of a hot dataset continues to decrease and falls below the set cooling threshold, a cooling migration is automatically triggered, transferring it to magnetic or optical storage. This mode allows data tiering to dynamically adapt to changes in access patterns without manual intervention.

[0078] This embodiment of the method, through the configuration of a visual scheduling strategy engine, allows operations and maintenance personnel to flexibly configure four trigger modes: timed, business event, storage space threshold, and data heat threshold, meeting the scheduling needs of diverse business scenarios in government and enterprises. It automatically executes the hierarchical, migration, archiving, or heat-up callback operations of datasets on demand, reducing manual intervention and improving the timeliness and accuracy of scheduling. Multiple trigger modes can be used in combination (e.g., configuring timed triggers and threshold triggers simultaneously), making the scheduling strategy more refined and enhancing its adaptability to different business loads and storage resource conditions.

[0079] According to some embodiments of the present invention, the method further includes a cross-media migration cost assessment step: calculating the comprehensive cost of migrating the dataset from the current storage medium to the target storage medium, the comprehensive cost including the cost of the electrical storage medium, the cost of the magnetic storage medium, the remaining lifetime loss of the optical storage medium, the energy consumption and carbon emissions during the migration process; and selecting the target medium and execution time with the lowest comprehensive cost based on the comprehensive cost assessment results, and performing the cross-media migration.

[0080] In magneto-optical-electrical integrated storage systems, data migration between different media involves not only balancing access performance and storage costs, but also factors such as the lifespan of the media themselves, energy consumption during migration, and carbon emissions. Electrical storage media (such as SSDs) offer high performance but have high unit costs; magnetic storage media (such as HDDs) offer moderate costs and balanced performance; and optical storage media (such as BD-R optical discs) are low-cost and have long lifespans, but cannot be modified after writing and each write causes some wear and tear on the media. In actual operation, the same dataset may face multiple optional migration target media and execution times, with different choices leading to different overall costs. For example, migrating data to lower-cost optical storage when the cluster load is low may incur additional lifespan reduction costs due to the write wear and tear of the optical storage media; performing large-scale migrations during peak electricity price periods will result in higher energy costs. Therefore, it is necessary to comprehensively evaluate all costs when making migration decisions.

[0081] In the migration cost assessment step, the overall cost includes several quantifiable components. The cost of electrical storage media is calculated as the storage cost per unit of data based on the purchase cost, lifespan, and expected write volume of the target electrical storage device. The cost of magnetic storage media is similarly calculated based on the unit capacity cost and lifespan of the magnetic storage device. The remaining lifespan loss of optical storage media is quantified by calculating the remaining usable lifespan based on the rated number of records and the accumulated number of records, converting the lifespan loss of a single record operation into a corresponding cost value. The energy consumption during the migration process is calculated by estimating the total power consumption of the migration operation based on the data volume of the dataset to be migrated, the read / write power consumption characteristics of the source and target media, network transmission power consumption, and the estimated migration duration, and then calculating the energy cost based on the current unit electricity price. Carbon emissions are calculated by estimating the carbon dioxide emission equivalent generated by the migration operation based on the total power consumption during the migration process and the carbon emission factor of the local power grid.

[0082] After obtaining the quantified values ​​of the above-mentioned costs, the components are added together to obtain the comprehensive cost of migrating to each candidate target medium. Different target media and execution times correspond to different comprehensive costs. For example, migrating to optical storage during off-peak electricity prices at night, although the write loss of optical storage remains the same, will result in lower energy costs and carbon emissions compared to peak electricity prices during the day. Similarly, migrating warm data to magnetic storage instead of optical storage may have a slightly higher unit storage cost, but it avoids the write loss costs associated with optical storage. When making migration decisions, the system calculates the comprehensive cost for each candidate target medium and its available execution times, selects the scheme with the lowest comprehensive cost, and performs the cross-media migration according to the target medium and execution time determined by that scheme.

[0083] This embodiment introduces a cross-media migration cost assessment step, incorporating the costs of electrical storage media, magnetic storage media, remaining lifetime loss of optical storage media, energy consumption during migration, and carbon emissions into the calculation of the comprehensive cost. Based on this, the target medium and execution time with the lowest comprehensive cost are selected, thus achieving comprehensive optimization of migration scheduling in terms of economy, energy consumption, and environmental protection. At the same time, by considering the lifetime loss of optical storage, excessive consumption of optical storage devices caused by frequent writing is avoided.

[0084] In summary, the native magneto-optical-electric fusion hierarchical migration scheduling method for datasets provided in this embodiment of the invention improves cross-media migration scheduling in magneto-optical-electric fusion storage systems in terms of scheduling efficiency, resource matching accuracy, business stability, data reliability, compliance and evidence preservation capabilities, and massive scalability by combining multiple technical features such as dataset granular scheduling, multi-dimensional index recognition, pre-migration decision mechanism, cross-media hierarchical migration, batch burning and sealing of optical storage, parallel acceleration and flow control, global cache bypass, multi-mode triggering, cross-media migration cost evaluation, and distributed sharding scheduling.

[0085] Specifically, compared with the prior art, the main technical features of the present invention have the following beneficial effects: 1) Replacing file or object-level scheduling with datasets as the smallest scheduling unit, related data from the same business scenario, the same security level, or the same generation cycle are integrated into a complete business dataset for overall batch migration. This reduces the number of scheduling tasks and management complexity, and avoids related data being scattered across different media. Assigning a globally unique identifier to each dataset and binding version information enables full lifecycle traceability, access control, and version rollback for the dataset.

[0086] 2) A multi-dimensional index system is used to identify the hot / cold levels of datasets based on access frequency, computing power status, data lifecycle, data value density, and security weight. A weighted fusion algorithm (Score = α×H + β×C + γ×L + δ×V + ε×S) is used to calculate a comprehensive score, classifying the datasets into four levels: hot, warm, cold, and archived. Compared to single-dimensional methods that rely solely on access frequency, this multi-dimensional index system more comprehensively reflects the true state of the dataset, reducing the premature relegation of high-value, high-security data to cold storage and preventing low-value data from occupying high-performance resources for extended periods.

[0087] 3) Based on the dataset hierarchy identification results and the pre-migration decision mechanism, cross-media hierarchical migration is performed between three heterogeneous storage media: electrical, magnetic, and optical. Hot data is scheduled to electrical storage, warm data to magnetic storage, and cold or archived data to optical storage, forming a three-tiered media adaptation mechanism based on data access characteristics and value gradients. The pre-migration decision mechanism determines the migration timing based on the current cluster IO load, the remaining capacity of each storage medium, and a preset business-allowed migration time window. Migration is triggered when the cluster load is below a set threshold, the target medium has sufficient capacity, and the data is within the business window, avoiding interference with front-end business response performance. In response to computing task requests or business access needs, archived datasets are warmed up and migrated from optical storage to electrical or magnetic storage, enabling archived data to re-enter the online computing process.

[0088] 4) During the migration process, data consistency verification, breakpoint resumption, and fault rollback operations are performed. Data integrity is verified by comparing global hash values ​​before and after migration, enabling the detection of data errors during cross-media transmission; if migration is interrupted, transmission resumes from the recorded breakpoint, avoiding the need to retransmit all data; in case of migration anomalies, abnormal data on the target end is automatically cleared and the source end is restored to its pre-migration state. These operations collectively reduce the risk of data loss, data inconsistency, or unrecoverable data at the source end.

[0089] 5) Perform batch sequential burning and physical disk locking operations on the archived datasets that have been moved to optical storage. Periodically collect cold datasets to be archived and perform batch sequential burning in order, reducing the wear and tear on the optical storage media caused by random writing; after burning is completed, trigger physical disk locking to make the archived datasets in a hardware-level immutable state; retain the archiving time, disk location information, burning logs and disk locking status to form a traceable and compliant evidence storage chain.

[0090] 6) Parallel acceleration and flow control operations are also performed during the migration process. The massive dataset is split into multiple sub-data blocks and multi-threaded or multi-node parallel transmission is used to shorten the migration time of massive data. At the same time, the overall IO load and network bandwidth usage of the cluster are monitored, and the migration flow threshold and the number of parallel threads are dynamically adjusted. The flow is limited during high load periods and the migration is carried out at full speed during low load periods, so that the resource usage of the migration task is adaptively adjusted with the system load.

[0091] 7) Global cache bypass acceleration is also performed during the migration process. A three-level caching architecture is built, consisting of node local cache, cluster shared cache, and hotspot bypass cache. Frequently accessed hot data is preloaded into the cache, reducing the access pressure on the backend storage. During the dataset migration, read and write requests for frontend hot data are responded to through the bypass cache channel, which is isolated from the IO channel of the backend migration task, reducing the impact of the migration operation on business access performance.

[0092] 8) By employing a distributed sharding scheduling process, the metadata of massive datasets is split into multiple data shards and distributed across multiple cluster nodes, eliminating the performance bottlenecks and single-point-of-failure risks of centralized metadata management. A cluster load balancing mechanism distributes migration scheduling tasks to each node for parallel execution, improving the throughput efficiency of large-scale migration tasks. The number of cluster nodes can be increased in response to growth in dataset size or scheduling load, giving the entire solution horizontal elastic scalability, adaptable to data storage scenarios ranging from small to medium scale (10PB and above).

[0093] 9) By configuring a visual scheduling strategy engine, it supports timed triggering, business event triggering, storage space threshold triggering, and data popularity threshold triggering modes. It automatically executes dataset tiering, migration, archiving, or warm-up callback operations as needed, reducing manual intervention and improving the timeliness and accuracy of scheduling. Multiple triggering modes can be combined to adapt scheduling strategies to changes in different business loads and storage resource conditions.

[0094] 10) Through cross-media migration cost assessment, calculate the comprehensive cost of migrating the dataset from the current storage medium to the target storage medium. The comprehensive cost includes the cost of electrical storage medium, the cost of magnetic storage medium, the remaining lifetime loss of optical storage medium, the energy consumption and carbon emissions during the migration process, and select the target medium and execution time with the lowest comprehensive cost based on the comprehensive cost assessment results to perform the migration, so that the migration scheduling achieves comprehensive optimization in terms of storage cost, energy consumption and environmental protection.

[0095] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A native magneto-optical-electric fusion hierarchical migration scheduling method for datasets, characterized in that, Includes the following steps: Using datasets as the smallest scheduling unit, and based on multi-dimensional indices such as access popularity, computing power status, data lifecycle, data value density, and security weight, the corresponding datasets are identified as hot, warm, cold, or archived. Based on the hierarchical identification results of the dataset and the pre-migration decision mechanism, cross-media hierarchical migration is performed among three types of heterogeneous storage media: electrical storage, magnetic storage, and optical storage. The pre-migration decision mechanism jointly determines the migration timing based on cluster load, storage capacity, and business window. During the cross-media migration of the dataset, data consistency verification, breakpoint resume and fault rollback operations are performed, and batch sequential burning and physical sealing and locking operations are performed on the archived datasets that are moved to optical storage to form a compliant evidence storage chain.

2. The method according to claim 1, characterized in that, The use of datasets as the smallest scheduling unit includes: Data related to the same business scenario, the same level of confidentiality, or the same generation cycle are integrated into a complete business dataset. This complete dataset is used as a whole to perform batch cross-media migration scheduling without splitting the dataset, modifying the internal structure of the data, or transferring user data ownership. Assign a globally unique identifier to each dataset and bind version information for full lifecycle traceability, access control, and version rollback.

3. The method according to claim 1, characterized in that, The multi-dimensional index, based on access popularity, computing power status, data lifecycle, data value density, and security weight, identifies the hot, warm, cold, or archived levels of the corresponding dataset, including: The access popularity index is automatically calculated based on the real-time access frequency and concurrent access count of the dataset. The computing power status index is automatically calculated based on the real-time running status and computing power occupancy time of the associated computing power tasks in the dataset. The lifecycle index is automatically calculated based on the dataset's creation time, business flow status, and preset archiving time. The data value density index is automatically calculated based on the access frequency, growth rate, number of references, number of associated tasks, and data lifecycle position of the dataset. The security weight index is automatically calculated based on the encryption level, de-identification status, access control complexity, and audit requirement level of the dataset. The comprehensive score of the dataset is calculated using a weighted fusion algorithm. The comprehensive score is calculated as follows: Score = α×H + β×C + γ×L + δ×V + ε×S, where H is the access popularity index, C is the computing power status index, L is the data life cycle index, V is the data value density index, S is the security weight index, and α, β, γ, δ, and ε are the corresponding weight coefficients. The dataset is divided into four levels: hot, warm, cold, and archived, based on the comprehensive score.

4. The method according to claim 1, characterized in that, The cross-media hierarchical migration between the three heterogeneous storage media—electrical storage, magnetic storage, and optical storage—includes: Based on the hierarchical identification results, hot data is scheduled to electrical storage media, warm data to magnetic storage media, and cold or archived data is sent to optical storage media; furthermore... In response to computing power task requests or business access needs, archived datasets are migrated from optical storage media to electrical or magnetic storage media by heating and restoring them.

5. The method according to claim 1, characterized in that, The process of migrating datasets across media includes performing data consistency checks, resuming interrupted data transfers, and rollback operations. Specifically, this includes: Before migration, calculate the global hash value of the source dataset; after migration, calculate the global hash value of the target dataset and compare them to verify data integrity. If the migration is interrupted, the remaining data will be transmitted from the recorded breakpoint. In the event of a migration anomaly, clear the abnormal data already transmitted on the target end and restore the source dataset to its state before the migration.

6. The method according to claim 1, characterized in that, The migration process also includes parallel acceleration and flow control operations: the massive dataset is split into multiple sub-data blocks and transmitted in parallel using multiple threads or multiple nodes; at the same time, the overall IO load and network bandwidth usage of the cluster are monitored, and the migration flow threshold and the number of parallel threads are dynamically adjusted to limit the flow during high load periods and migrate at full speed during low load periods.

7. The method according to claim 1, characterized in that, The migration process also includes a global cache bypass acceleration operation: a three-level cache architecture consisting of node local cache, cluster shared cache, and hotspot bypass cache is constructed, and frequently accessed hot data is preloaded into the cache; During the dataset migration process, read and write requests for hot data in the foreground are responded to through a bypass cache channel, which is isolated from the IO channel of the background migration task.

8. The method according to claim 1, characterized in that, The process of performing batch sequential burning and physical sealing / locking operations on the archived datasets stored in optical storage to form a compliant evidence preservation chain specifically includes: Periodically collect cold datasets to be archived and sequentially burn them to optical storage media in batches. After burning is complete, a physical disk sealing and locking mechanism is triggered, making the archived dataset in a hardware-level immutable state; The archiving time, disk location information, burning logs, and disk sealing status are retained to form a traceable and compliant evidence storage chain.

9. The method according to claim 1, characterized in that, The method also includes a distributed sharding scheduling step: The metadata of a massive dataset is split into multiple data shards, and each shard is distributed and stored on multiple cluster nodes. The migration scheduling task is distributed to each cluster node for parallel execution through a cluster load balancing mechanism. In response to an increase in dataset size or scheduling load, the number of cluster nodes can be increased horizontally.

10. The method according to claim 1, characterized in that, The method also includes a cross-media migration cost assessment step: Calculate the overall cost of migrating a dataset from the current storage medium to the target storage medium, including the cost of the electrical storage medium, the cost of the magnetic storage medium, the remaining lifetime loss of the optical storage medium, the energy consumption and carbon emissions during the migration process; Based on the comprehensive cost assessment results, the target medium and execution time with the lowest comprehensive cost are selected to perform cross-media migration.

Images