Data migration method and device based on task offloading, equipment and medium

Abstract

The invention relates to the technical field of data processing, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a data migration method, device, equipment and medium based on task offload, which comprises the following steps: scanning a source end directory hierarchy and a target end directory hierarchy, and generating source end file metadata and target end file metadata; files are classified into source-end unique files, target-end unique files and two-end shared files based on file paths, and the files are transmitted to a comparison processing module; generating a source end newly-added migration task, a difference coverage migration task and a target end deletion migration task; and transmitting the migration task to a migration processing module and executing the migration task to generate a migration result. According to the method, source-end unique files, difference files and target-end redundant files are processed in a classified mode, full-amount migration is disassembled into incremental tasks to be executed in a split mode, and the invalid data transmission amount is reduced; a difference coverage task is generated based on double comparison of metadata and content, repeated coverage of homonymous files is avoided, and redundant data is automatically cleaned.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data migration method, device, equipment and storage medium based on task offloading. Background Art

[0002] In the evolution of data centers, industry platforms, and business systems, data migration between file storage systems has become a common and critical basic operation, especially when performing storage architecture replacements, equipment updates, or cross-system data integration. In actual applications, especially data migration between heterogeneous file storage systems, the lack of unified standards between the source and target sides in terms of file format, path structure, interface protocol, etc. makes incremental data identification and migration a technical difficulty. Existing systems generally lack mechanisms for rapid identification and efficient processing of incremental files in heterogeneous environments, especially when faced with complex directory structures and large data volumes.

[0003] Currently, most data migration tools employ a strategy of first performing a directory scan and metadata extraction on the source file system, then performing the same operations on the target, and then loading all metadata information from both ends into memory for centralized comparison. This centralized comparison model can perform basic functions when the data volume is small, but it can easily lead to the following problems when the number of files is large or the file size is large: First, as the file size grows, the space required to load the metadata structure into memory increases exponentially, which can easily lead to system memory exhaustion or a severe impact on computing resources; second, the migration process cannot be started before all metadata has been extracted and compared, resulting in a long migration preparation time, idle network and storage resources, and a significant reduction in overall system efficiency.

[0004] In the fintech sector, customer transaction records, financial statements, and compliance records are often stored in heterogeneous systems as a mix of structured and unstructured files. Regulatory audits and disaster recovery migrations necessitate frequent migration of this data to new platforms. Faced with millions of archival accounting documents or transaction voucher images, existing centralized comparison-based migration strategies result in unmanageable memory usage, slow migration response times, and significant resource waste, significantly hindering business continuity.

[0005] In the healthcare sector, patient imaging data, surgical videos, and medical records are stored in a distributed manner across multiple hospital campuses or systems. To achieve unified data access or data retention and archiving, regular or asynchronous migration between these various medical file storage systems is required. Due to the complex types and varying sizes of medical files, the current comparison model consumes excessive computing resources when generating the difference data list, often leading to migration interruptions, resource deadlocks, or system overloads. This makes it difficult to meet the healthcare industry's requirements for high-availability and real-time data migration.

[0006] In summary, existing heterogeneous file system data migration tools have limitations in metadata loading methods, comparison strategies, and resource scheduling mechanisms, lacking scalable, high-performance, and low-resource solutions. This is especially true for scenarios involving large-scale, batch-based migrations of business data that require uninterrupted business operations. Traditional migration tools face significant reliability and performance bottlenecks. Summary of the Invention

[0007] The main purpose of the present invention is to provide a data migration method, device, equipment and storage medium based on task offloading, aiming to solve the technical problem that the existing technology cannot efficiently complete the metadata comparison and task distribution of large-scale files under memory-constrained conditions when processing data migration of heterogeneous file storage systems, resulting in uncontrollable resource usage and serious migration response delays.

[0008] To achieve the above objectives, the present invention provides a data migration method based on task offloading, comprising:

[0009] Scan the source directory hierarchy and the target directory hierarchy to generate source file metadata and target file metadata respectively;

[0010] Classify the source-end file and the target-end file into source-end unique files, target-end unique files, and files shared by both ends according to the file paths in the source-end file metadata and the target-end file metadata;

[0011] The source-side unique file, the target-side unique file, and the files shared by both ends are transferred to a comparison processing module;

[0012] The comparison processing module generates a source-side new migration task for the source-side unique files, generates a difference-overwrite migration task after performing metadata and data content comparison on the files shared by both ends, and generates a target-side delete migration task for the target-side unique files according to the deletion policy;

[0013] Delivering the source-side newly added migration task, the difference-covering migration task, and the target-side deleted migration task to a migration processing module;

[0014] The migration processing module executes the source-side new addition migration task, the difference overwriting migration task, and the target-side deletion migration task, and generates a migration result.

[0015] Furthermore, to achieve the above-mentioned purpose, the present invention provides a data migration device based on task offloading, comprising:

[0016] A metadata generation module is used to scan the source directory hierarchy and the target directory hierarchy to generate source file metadata and target file metadata respectively;

[0017] A file classification module, configured to classify source-end files and target-end files into source-end unique files, target-end unique files, and files shared by both ends according to the file paths in the source-end file metadata and the target-end file metadata;

[0018] A file transfer scheduling module, configured to transfer the source-side unique files, the target-side unique files, and the files shared by both ends to the comparison processing module;

[0019] A migration task generation module is configured to generate a source-side new migration task for the source-side unique files using the comparison processing module, generate a difference-overwrite migration task after performing a metadata and data content comparison on the files shared by both ends, and generate a target-side delete migration task for the target-side unique files according to the deletion policy;

[0020] A task distribution module is used to transfer the source-side newly added migration task, the difference-covering migration task, and the target-side deleted migration task to the migration processing module;

[0021] The migration processing module is used to execute the source-side new addition migration task, the difference overwriting migration task and the target-side deletion migration task through the migration processing module, and generate a migration result.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a task-offloading-based data migration program stored in the memory and runnable on the processor, and when the task-offloading-based data migration program is executed by the processor, the steps of the task-offloading-based data migration method as described above are implemented.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a data migration program based on task offloading is stored. When the data migration program based on task offloading is executed by a processor, the steps of the data migration method based on task offloading as described above are implemented.

[0024] Beneficial effects: The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. It discloses a data migration method based on task diversion, including: scanning the source-side directory hierarchy and the target-side directory hierarchy to generate source-side file metadata and target-side file metadata; classifying files into source-side unique files, target-side unique files and files shared by both ends based on file paths, and passing them to the comparison processing module; generating source-side new migration tasks, difference overwriting migration tasks and target-side deletion migration tasks; passing the above-mentioned migration tasks to the migration processing module and executing them to generate migration results. The present invention classifies and processes source-side unique files, difference files and target-side redundant files, and decomposes the full migration into incremental tasks for diversion execution, thereby reducing the amount of invalid data transmission; generates difference overwriting tasks based on dual comparison of metadata and content to avoid repeated overwriting of files with the same name, and automatically cleans up redundant data on the target end in combination with deletion strategies, thereby reducing the cost of manual intervention; optimizes migration resource allocation, shortens the migration cycle, and improves the efficiency and accuracy of data synchronization between heterogeneous storage systems by independently processing addition, difference and deletion operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0026] Figure 1 A schematic diagram of an application environment of a data migration method based on task offloading in an embodiment of the present invention;

[0027] Figure 2 This is a flow chart of an embodiment of a data migration method based on task offloading according to the present invention;

[0028] Figure 3 This is a functional module diagram of a preferred embodiment of a data migration device based on task offloading according to the present invention;

[0029] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0030] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] The data migration method based on task offloading provided by the embodiment of the present invention can be applied in the following situations: Figure 1In an application environment, the user end communicates with the server end through a network. The server end can scan the source end directory hierarchy and the target end directory hierarchy through the user end, generate source end file metadata and target end file metadata; classify the files into source end unique files, target end unique files and target end common files based on the file path, and pass them to the comparison processing module; generate source end new migration tasks, difference overwrite migration tasks and target end deletion migration tasks; pass the above migration tasks to the migration processing module and execute them to generate migration results. The present invention classifies and processes source end unique files, difference files and target end redundant files, and decomposes the full migration into incremental tasks for diversion execution, thereby reducing the amount of invalid data transmission; generates difference overwrite tasks based on double comparison of metadata and content, avoids repeated overwriting of files with the same name, and automatically cleans up redundant data on the target end in combination with the deletion strategy, thereby reducing the cost of manual intervention; optimizes migration resource allocation, shortens the migration cycle, and improves the efficiency and accuracy of data synchronization between heterogeneous storage systems by independently processing addition, difference and deletion operations. Among them, the user end can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server side can be implemented by an independent server or a server cluster composed of multiple servers. The present invention will be described in detail below through specific embodiments.

[0033] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a data migration method based on task offloading provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the data migration method based on task offloading proposed by the present invention includes the following steps:

[0035] S10, scanning the source-side directory hierarchy and the target-side directory hierarchy to generate source-side file metadata and target-side file metadata respectively;

[0036] In this embodiment, scanning the source-side directory hierarchy and the target-side directory hierarchy is a key preparatory operation before implementing data migration across heterogeneous storage systems, where the "directory hierarchy" refers to a file system path structure recursively organized in a tree structure, typically containing multiple subdirectories and file nodes. By scanning this hierarchical structure, the physical location, ownership, and storage depth of the file in the file system can be accurately obtained, thereby establishing a structural foundation for subsequent incremental data comparison and migration. In heterogeneous systems, due to differences in file system formats, naming rules, access protocols, etc., it is necessary to perform independent and consistent scanning operations on the directory hierarchies of the source and target ends respectively to ensure the consistency and comparability of the metadata structure.

[0037] The process of generating source-side file metadata and target-side file metadata is based on the directory hierarchy obtained by scanning, and extracting the core attribute information of each file, including the absolute path of the file, file size, last modification time, and hash value of the data content. The absolute path is the unique identifier of the file in the system and is used to accurately match the correspondence between the source and target ends; the file size reflects the storage resource usage of the file, which helps to evaluate the migration cost and bandwidth scheduling; the last modification time can be used to identify the version status of the file and is the core judgment basis for implementing incremental migration; the hash value of the data content is generated by a consistent hashing algorithm (such as MD5, SHA256, etc.) and is used to compare whether the file content has changed. Even if the file name and timestamp are consistent, it can determine whether the data has been tampered with or damaged. The above fields together constitute the file metadata entry, which is the basic input for subsequent task generation, data verification, and other links.

[0038] In the actual implementation process, the scanning operation uses a multi-threaded concurrent traversal method to improve processing efficiency. The system establishes traversal task queues for the root directories of the source and target ends respectively. Each thread takes out the directory nodes to be processed from the queue and performs depth-first or breadth-first traversal. During the traversal, the file attributes are extracted and metadata entries are constructed in real time. In the hash value generation link, the computing cost can be reduced by reading part of the file's data blocks or using a fast summary algorithm. At the same time, it supports on-demand configuration of whether to enable the generation logic of the data content hash. When the number of files is extremely large, the metadata file can be written while scanning through an incremental cache mechanism to avoid the risk of resource overflow caused by one-time memory loading. The generated file metadata will be constructed into a hierarchical tree data structure according to the directory hierarchy, supporting subsequent difference analysis and migration path control based on the structural path.

[0039] When processing high-concurrency directory scans, the thread pool size can be set based on the file distribution density of the actual storage system. For example, for IO-intensive storage, more threads can be set to increase traversal speed, while for devices with limited computing resources, the number of threads can be reduced to reduce resource competition. The hash value calculation method can also be flexibly adjusted based on security and performance requirements. For example, for scenarios requiring high accuracy in financial systems, high-security hash functions such as SHA256 or SHA3 can be selected; in scenarios such as medical and health data archiving, if performance is more sensitive, CRC32 can be used or hash digests can be generated only for key data segments. In addition, to address the problem of excessive tree depth caused by large-scale directory structures, the maximum directory scan depth can be configured or a mechanism for filtering subdirectories by rules can be enabled to achieve on-demand scanning. Tree-structured metadata storage can be implemented using nested JSON, a hierarchical key-value database, or a lightweight file indexing system. In a high-concurrency environment, a write buffer mechanism can be further combined to improve metadata writing efficiency and reliability.

[0040] Example: In the healthcare business, the information systems of large hospitals often involve the migration needs of multiple PACS imaging systems or electronic medical record systems. Since the file naming methods and directory hierarchical structures of old systems are often inconsistent, the file structures of different imaging systems can be scanned layer by layer and metadata in a unified format can be generated, which helps to efficiently migrate historical inspection records to the new platform while retaining the complete patient data path and modification traces. In the financial technology business, regulatory reports, business images, contract documents, etc. need to be incrementally backed up between heterogeneous systems. The directory structure between the production system and the backup system can be scanned regularly to quickly locate new and changed documents, achieve high-frequency and high-reliability data replication, reduce the system memory and CPU usage of full comparison, and improve task scheduling efficiency and system availability.

[0041] By building complete source and target directory structures and generating file metadata containing key attributes respectively, the problem of heterogeneous file storage systems having difficulty accurately locating file differences in the absence of unified protocol support is solved. This avoids the problem of uncontrollable memory resources caused by the traditional method of relying on full metadata loading, and provides a structured and verifiable data foundation for subsequent accurate classification, difference identification, and migration scheduling, effectively improving the system's stability and processing efficiency in large-scale file migration.

[0042] S20, classifying the source-end file and the target-end file into source-end unique files, target-end unique files, and files shared by both ends according to the file paths in the source-end file metadata and the target-end file metadata;

[0043] In this embodiment, file classification based on the file paths in the source-side file metadata and the target-side file metadata is a key prerequisite operation for realizing incremental data identification between heterogeneous storage systems. The file path refers to the address of each file that is uniquely identified by a hierarchical structure in the file system to which it belongs, and is generally expressed as a complete path string starting from the root directory, such as / data / user / a / report.docx. Since there may be differences in the file system structures of the source and target ends, a global comparison based on the absolute path is required to ensure the accuracy and consistency of the classification. The essence of this operation is to perform set operations on file sets from two different sources through a path matching algorithm, thereby distinguishing between files that only exist on the source end (i.e., source-only files), files that only exist on the target end (i.e., target-only files), and files that exist on both ends and have the same path (i.e., files shared by both ends). This classification operation will provide a structural basis for subsequent task diversion, difference identification, and precise migration.

[0044] In the actual implementation, it is first necessary to build an index table for the file metadata of the source and target ends according to the file path, such as using data structures such as hash tables or balanced trees for fast path search. The source file path set is compared with the target file path set, and the division of three types of files can be achieved by using symmetric difference and intersection. The part that exists in the source file path set but does not exist on the target side is the source-only file; the part that exists on the target side but does not exist on the source side is the target-only file; the intersection of the two sets is the file shared by both ends. In order to improve processing efficiency, when processing large-scale data, a parallel computing framework (such as MapReduce structure) can be used to compare paths in batches and divide each type of file set in parallel. If the path contains soft links or symbolic link files, it is also necessary to combine the path resolution rules of the file system to determine the physical file that the path actually corresponds to, so as to avoid the risk of misclassification caused by the same logical path but different physical objects.

[0045] In the process of processing file path comparison, a variety of optimization methods can be used. The source and target file paths can be compared hierarchically by directory using a partition comparison method, and local indexes can be established in the sub-directory dimension to accelerate matching. For cases where there are dynamic identifiers (such as timestamps, user IDs) in the file path, regular matching rules or path normalization strategies can be introduced to unify the processing format before comparison to avoid misclassification due to naming differences. In addition, if the system is deployed in a multi-node environment, the path comparison task can also be sharded and distributed to reduce the pressure on a single node and speed up the comparison. In some scenarios with extremely high precision requirements, the path comparison results can be cross-validated for path existence and metadata integrity to ensure that the situation where the path exists but the file has been deleted or damaged is accurately identified as an anomaly classification.

[0046] Example: In healthcare scenarios, different hospitals or medical data centers typically deploy multiple heterogeneous file storage systems, which are used to store electronic medical records, medical images, test reports, and other data. Due to different data management strategies, file structures and naming methods vary, and data synchronization has strong real-time requirements. By comparing the file paths of the source and target ends, it is possible to accurately identify newly generated pathological images or emergency records in the local medical system, avoiding repeated downloading or overwriting of archived data in the cloud. At the same time, it is also possible to promptly discover and clear discarded old versions of medical order files on the target end, thereby ensuring the consistency of patient data across multiple systems and improving the reliability of telemedicine and intelligent decision-making.

[0047] In the financial sector, incremental data synchronization is often used to synchronize cash flows, risk control logs, and audit records in local core transaction systems to cloud data lakes or disaster recovery systems. During the path comparison phase, absolute path matching can be used to accurately locate newly added customer credential files or abnormal record analysis data due to business expansion, and prevent the accidental overwriting of file versions that already contain supplementary information on the target side. For some financial message systems that use dynamic path structures (such as date and transaction number as path levels), data attribution batches can also be identified based on path matching rules to ensure the timely collection and version confirmation of clearing, compliance, and audit data, thereby significantly reducing the cost of manual review and data consistency verification.

[0048] By precisely comparing file paths in source and target file metadata, we categorize files based on their structural differences, effectively distinguishing three types of file collections and avoiding the computational and storage pressures associated with full scans and content comparisons. This path classification mechanism provides clear task boundaries for subsequent incremental migrations, enabling the system to specifically handle new, updated, and redundant files, improving the efficiency and stability of data migration and providing an input foundation for optimizing task scheduling strategies.

[0049] S30, transmitting the source-side unique file, the target-side unique file, and the files shared by both ends to a comparison processing module;

[0050] In this embodiment, the source-side unique files, the target-side unique files, and the files shared by both ends are passed to the comparison processing module, which involves the organization of file collections, the construction of channel structures, resource quota control, and cross-module data flow mechanisms. Source-side unique files refer to a collection of files that exist in the source-side file metadata but cannot find a corresponding path in the target-side file metadata, usually indicating newly added content to be migrated. Target-side unique files refer to a collection of files that only appear on the target side, often representing redundant objects or objects that need to be determined whether to be deleted. Files shared by both ends are a collection of file paths that exist on both the source and target sides, and are the core source of difference coverage judgment. Inputting these three types of file collections into subsequent modules must not only ensure the accurate classification of files, but also ensure the order and controllability during data transmission.

[0051] Before the source and target ends complete the comparison of file paths and content, they must first conduct a stable, continuous, and controllable data flow for the three types of classified files to ensure that the comparison process has an accurate input source. In the data interaction architecture, this process not only involves the triggering mechanism of data transmission, but also the structured expression and identification method of the data in the transmission path. In order to achieve efficient organization of data from different sources, the directory path can be used as the primary index benchmark to give each file item to be transferred a unique identifier, which contains structural fields such as its category information, source system identifier, directory node, and data reading priority. This identification structure facilitates the subsequent processing logic to locate, classify, and operate on the data without relying on the overall context.

[0052] From the perspective of data structure design, there are significant differences in processing logic, timeliness requirements, and system resource consumption between different types of file collections. In order to ensure the overall flow efficiency and stability of the system, the three types of files can be configured in independent data channels. This configuration method avoids blocking and mutual exclusion problems during data processing. For example, for new types of files with large file content but simple processing logic, it is suitable to set up high-throughput, wide-bandwidth asynchronous channels at the transport layer; and for common types of files with more complex processing, they can be allocated higher-frequency scheduling resources and smaller-granularity batch transmission rules to cooperate with precise identification processes such as hash verification.

[0053] The stability of the transmission path and the sequential nature of the input directly determine the deterministic execution behavior of downstream modules. If the incoming data contains classification errors, omissions, or duplications, this will lead to biased subsequent content judgments and even the risk of inadvertent data migration or deletion. Therefore, in addition to the necessary structural tagging and transmission boundary control mechanisms, a batch archiving management strategy for each type of data set can be formed by combining transmission numbers and transmission batch indexes. This allows the system to restore contextual state by reconstructing transmission batches in the face of distributed network fluctuations, abnormal node responses, or transmission interruptions.

[0054] Furthermore, during the transfer process, dynamic priority can be set based on the depth of the directory structure or path length, prioritizing frequently accessed files closer to the root path for processing while deferring files in deeper directories with less frequent data access and a low probability of change. This improves overall system throughput in resource-constrained computing environments. This mechanism can also achieve tenant-level isolation through path prefixes in multi-tenant storage environments, improving resource controllability and accuracy during the comparison process.

[0055] By passing the three classified file collections to the comparison processing module and introducing mechanisms such as batch scheduling, independent channels, and cache threshold control, a stable transition from file classification to comparison is achieved, avoiding memory overflows caused by loading the entire file collection at once. This effectively alleviates the resource bottlenecks faced by traditional comparison tools when processing large-scale heterogeneous file systems. Combined with a high-concurrency task pipeline mechanism, this also ensures the real-time and continuous data acquisition of the comparison module, providing stable data input for subsequent task generation steps.

[0056] S40, generating a source-side new migration task for the source-side unique files by the comparison processing module, generating a difference-overwrite migration task after performing metadata and data content comparison on the files shared by both ends, and generating a target-side delete migration task for the target-side unique files according to the deletion policy;

[0057] In this embodiment, after the file classification information flows to the comparison processing module, a differentiated task generation mechanism needs to be constructed around the processing characteristics of each type of file collection to support the consistency of data distribution, scheduling and status management in the subsequent migration process. In the processing logic of files unique to the source end, their file paths are not recognized on the target end, and there are no alternatives with the same name or at the same level. Therefore, key metadata such as path, size, modification time and hash value are extracted from such files to form a minimum complete migration unit without performing additional content comparison operations. This type of file is usually defined as static new data, and its main processing goal is to generate write instructions with complete structure and clear fields, so that the target end can restore the corresponding structure and complete the corresponding data reconstruction accordingly.

[0058] For files shared by both ends, path consistency does not mean content consistency, so a refined metadata and data content joint comparison mechanism needs to be introduced. The last modification time field recorded in the metadata serves as the first reference dimension for change judgment, and can quickly distinguish whether the version has been iterated in most scenarios. However, considering that some file systems have limitations on time accuracy or there are cache delays in data synchronization, the hash value of the data content still needs to be used as a strong consistency comparison method to avoid misjudgment. By extracting the hash values ​​of the corresponding files on the source and target ends for comparison, it is possible to further determine whether the content has changed. If the content is inconsistent, it is determined to be a valid difference, and a difference overwriting migration task is generated. This task retains the original path and hash information in structure, and marks the task as an overwriteable reconstruction type for subsequent migration stages to perform targeted overwriting and replacement operations.

[0059] The processing logic for target-side unique files is reflected in the recycling judgment logic. This type of file has not been identified on the source side during the current scanning cycle and needs to be further processed in combination with a predefined cleanup strategy. This strategy can be configured based on the file's most recent access time, creation time, naming pattern, or business identifier. It supports multiple mechanisms such as static rule matching and dynamic script filtering to filter out the set of files that need to be retained and recyclable. When it is determined that a target-side unique file meets the deletion conditions, a deletion-type migration task can be generated based on its path, directory, and file type. Its execution behavior will not involve source-side access, but only initiates precise positioning and deletion instructions on the target side to clean up data silos, compress storage space, and reduce data redundancy.

[0060] In order to improve the generation efficiency of the three types of tasks and the scalability of the system, a task template and field filling mechanism can be constructed to decouple the construction process of the task object from the memory comparison and avoid the formation of a comparison processing bottleneck. The task template can contain the task type identifier, field definition structure and verification constraints, and cooperate with the task field filler to extract the corresponding content from the metadata on demand to quickly construct a standardized task object and output it to the migration processing flow. This structure has natural compatibility and extensibility, which facilitates the subsequent implementation of control strategies such as priority scheduling, transmission optimization and status feedback at the task level. Task generation is not only a static data output, but also a key hub logic in the flow data system, which directly affects the efficiency, accuracy and controllability of the overall migration link.

[0061] In one specific embodiment, the classified source-side unique files can be directly mapped as objects to be migrated based on the task scheduling engine, and a unique migration task identifier can be assigned to each file. The absolute path, file size and last modification time of the file are synchronously recorded, and a migration task data structure in a standard format is further generated and written into the new task queue. For files shared by both ends, the system will call the file difference comparison module, read the last modification time field of the corresponding file from the source and target metadata, and generate an overwrite task if the source time is newer. If the modification time is the same, the hash value comparison process is further triggered, and the hash value of the current data content of the file is calculated through a hash algorithm (such as SHA-256). If there is inconsistency, it is considered that the content has mutated, and an overwrite task is also generated. In the overwrite task, the source hash value and the target hash value can be attached as tracking identifiers to improve the efficiency of subsequent verification. For the processing of files unique to the target end, a policy engine can be integrated to determine whether it is a temporary file, a cache file, a historical archive file, etc. If it matches the set cleanup rules, a deletion type task is generated and the task is marked as a logical deletion or physical deletion. All migration tasks are encapsulated as migration task description information in a unified format and diverted according to task type in the internal task transmission pipeline for subsequent module parsing and execution.

[0062] By accurately mapping the file classification results to different types of migration tasks and determining the differences of common files based on the dual standards of modification time and data content, it is possible to effectively reduce redundant migration operations and improve overall migration efficiency. The generation of deletion tasks is triggered by the policy control module to avoid accidental deletion of critical files and improve the level of data integrity protection. The unified task description structure and diversion mechanism enhance the system's module collaboration capabilities and make the scheduling of tasks in the pipeline more controllable. By distinguishing the status differences of files at the source and target ends and generating tasks on demand, accurate processing of new, updated and redundant files is achieved, which reduces the continuous occupation of system memory and unnecessary network bandwidth consumption. At the same time, the tracking and rollback capabilities of migration tasks are enhanced, and it has good adaptability and scalability in heterogeneous storage environments.

[0063] S50, delivering the source-side newly added migration task, the difference-covering migration task, and the target-side deleted migration task to a migration processing module;

[0064] In this embodiment, the source-side addition migration task, the difference overwrite migration task, and the target-side deletion migration task represent responsive processing instructions for three types of file state changes in a heterogeneous file storage system. Although task generation is triggered by the comparison processing module, task execution must rely on receipt and scheduling by the subsequent migration processing module. Therefore, task transfer becomes a critical step before the migration logic is activated. During this process, tasks are not only moved as data structures but also logically complete the contextual transition from the comparison domain to the migration domain. This transition must maintain data integrity and identifiability, and ensure high efficiency during the transfer process.

[0065] When designing task delivery, it is necessary to first clarify the processing objectives and data structures of each type of task. For example, a new migration task on the source side usually contains complete file metadata and migration instructions, and it is necessary to clarify the absolute path, data size, creation time, and subsequent transmission priority of the source file; the difference overwrite migration task emphasizes the smallest difference unit of content-level changes, so it is necessary to transfer hash digests, change intervals, or comparison-based difference block descriptions; the target-side deletion migration task is relatively lightweight, usually only containing the absolute path and identification instructions for the deletion action. These three types of tasks have completely different data volumes, processing priorities, and system impacts, so they should not be mixed in the delivery mechanism. A parallel classification mechanism must be introduced for decoupling.

[0066] The task delivery mechanism involves not only the design and management of task queues, but also communication structures, bandwidth control, and concurrent scheduling strategies. In a large-scale data migration environment, the task diversion mechanism should be able to cope with sudden data peaks. For example, to avoid congestion of the entire transmission link due to the high frequency of a certain type of task, an independent data channel and cache mechanism can be configured for each type of task, and a maximum capacity threshold and dynamic adjustment strategy can be set. When the task volume of a channel approaches its threshold, the system can temporarily reduce the receiving rate of this task type, while maintaining normal processing for other task types, thereby achieving dynamic and balanced allocation of resources. This type of control mechanism requires not only that the transmission module be highly configurable, but also that the task delivery structure support polymorphic recognition, asynchronous writing, and state-aware flow control.

[0067] Furthermore, the structural design of task delivery should incorporate integrity verification and state traceability capabilities. For example, when tasks are passed to the migration processing module, they should be accompanied by a transfer ID, a task generation timestamp, and a task source tag. This supports task link tracking and scheduling rollback. This is crucial for ensuring system operability and consistency during fault-tolerant retries, task priority changes, and interruption recovery.

[0068] By building a task-type-based queuing structure and independent transmission channels, we not only improve the efficiency of task data transfer between modules, but also significantly reduce the risk of system resource congestion caused by task congestion. The orderly input of tasks ensures that the migration processing module can execute smoothly within the available resources, while also providing support for task priority control and error fallback.

[0069] S60: Execute the source-side new migration task, the difference-covering migration task, and the target-side deletion migration task through the migration processing module, and generate a migration result.

[0070] In this embodiment, the migration processing module assumes the core responsibility for executing tasks and generating results, serving as the key execution engine for incremental data migration. Upon receiving new migration tasks from the source, differential overwrite migration tasks, and target deletion migration tasks, the module invokes different sub-processes to complete the actual transfer, overwrite, or cleanup of file data, based on the metadata information, operational instructions, and processing priorities contained in the tasks. The entire migration process involves not only reading, writing, and deleting data, but also the closed-loop execution of a complete chain, including exception identification, status recording, and result archiving.

[0071] For new migration tasks on the source side, the migration processing module must first locate the source file in the source storage system based on the absolute path recorded in the task, and read the file in segments based on the file size and the preset data block granularity. During the transmission process, these blocks need to be written to the newly created files in the target storage system through the data channel. After the writing is completed on the target side, the last modification time of the newly created file must be synchronously updated to ensure that its metadata status is consistent with the source side to support the accuracy of subsequent incremental comparisons. After completing the task, the module records relevant indicators, including the file path of the successful migration, the actual number of bytes transferred, and the timestamp of the completion of the operation. This information constitutes the basic entries of the migration result.

[0072] For differential overwriting migration tasks, the execution process focuses more on the accurate extraction and efficient updating of incremental content. The migration processing module needs to read the files under the same path on the source and target sides respectively, and calculate the hash value of its current data content. If there is still inconsistency in the hash value comparison, it means that the target side has not been correctly overwritten or the file has changed again after the last migration. After confirming the data difference, the module extracts the changed data fragments in the source file through the differential positioning algorithm, replaces these different parts with the corresponding positions of the target file, and updates the modification time of the target file to the current operation time to complete the synchronous correction of the content. This process not only improves bandwidth utilization, but also significantly reduces the load on the storage system.

[0073] For target-side deletion migration tasks, the migration processing module locates the file on the target side according to the absolute path recorded in the task description and performs the deletion operation. To ensure operational security, the module quickly verifies the path validity, access permissions, and lock status before execution to ensure that the target file is indeed deletable. After the deletion is completed, the corresponding path, execution time, and deletion status are also recorded and stored in the migration result structure to facilitate subsequent verification and audit operations.

[0074] In addition, the migration processing module must also monitor the status and manage errors of all operational behaviors during the task processing process. If any abnormal situation occurs during the file transfer process, such as network interruption, source file loss, insufficient space on the target end, or restricted access rights, the module must promptly mark the corresponding task as failed, record the cause of the failure, and generate an error log. The failed task will then be re-entered into the corresponding type of task queue for subsequent retry scheduling. After the migration is completed, the module will classify and integrate all successfully migrated task records and failed task records by task type to form a unified structured migration result for subsequent data consistency verification and system execution evaluation.

[0075] In different scenarios, a multi-threaded task scheduling pool can be built based on task type, mapping the source-side new migration tasks, difference-overwrite migration tasks, and target-side deletion migration tasks to corresponding processing sub-threads. Each type of sub-thread is bound to an independent resource channel and adopts an asynchronous scheduling strategy, allowing the batch writing of new files, the rewriting of file difference content, and the execution of deletion instructions to be performed concurrently, maximizing the execution throughput of the migration processing module. For situations where network resources are limited or the target end is remote distributed storage, a transmission delay-aware strategy can also be introduced to dynamically adjust the task write window size based on the current network conditions to avoid write blocking or cache overflow.

[0076] When migration tasks involve specific security and compliance requirements, such as transaction credentials in financial services or patient imaging data in healthcare, the migration processing module can be enhanced with a data tag-based access control policy. This policy verifies access permissions and transmission encryption rules based on the file's tag before executing the task, and introduces an encrypted transmission channel (such as TLS) to facilitate cross-system transmission. When creating files on the target end, the system call layer's fsync mechanism is used to synchronize and persist metadata and content, ensuring write consistency.

[0077] To improve data verification efficiency, after the differential overlay task is completed, the real-time hash value of the target file can be pre-generated and cached in the result structure, which can be used to collaborate with the verification module to complete subsequent consistency verification; for migration tasks that frequently fail, the aggregation characteristics of the failure type and path can be recorded to drive the adaptive retry interval and task jump strategy, thereby improving the system's error recovery efficiency.

[0078] In large-scale migration scenarios, you can also configure strategic migration plans based on business time windows, for example, scheduling coverage tasks only during off-peak hours to avoid disrupting online service performance. The task scheduling module allows for configurable priority parameters, prioritizing files with high target space usage and frequently changing hash values ​​before batch execution, effectively controlling system resource consumption.

[0079] Through the task type-driven execution diversion strategy, the three types of migration tasks generated by the comparison of the source and target end states are passed to the migration processing module and executed respectively, thereby realizing the decoupling of differentiated migration execution paths. The addition, overwriting and deletion of files are no longer processed with a unified logic, but the minimum operation path is matched according to their actual data status. This not only greatly reduces the redundant data transmission bandwidth, but also significantly improves the overall migration efficiency. Through the result recording mechanism, the execution status and data indicators of each migration task can be traced, providing structured basic information for subsequent data consistency verification. Combined with the failed task re-entry mechanism, the system has high fault tolerance and stability, and is adapted to the refined migration needs in massive heterogeneous file environments.

[0080] The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. A data migration method based on task diversion is disclosed, including: scanning the source-side directory hierarchy and the target-side directory hierarchy to generate source-side file metadata and target-side file metadata; classifying files into source-side unique files, target-side unique files, and files shared by both ends based on file paths, and passing them to a comparison processing module; generating source-side new migration tasks, difference-overwrite migration tasks, and target-side deletion migration tasks; passing the above-mentioned migration tasks to a migration processing module and executing them to generate migration results. The present invention classifies and processes source-side unique files, difference files, and target-side redundant files, decomposing the full migration into incremental tasks for diversion execution, thereby reducing the amount of invalid data transmission; generating difference-overwrite tasks based on dual comparison of metadata and content to avoid repeated overwriting of files with the same name, automatically cleaning up redundant data on the target end in combination with deletion strategies, and reducing manual intervention costs; optimizing migration resource allocation, shortening the migration cycle, and improving the efficiency and accuracy of data synchronization between heterogeneous storage systems by independently processing addition, difference, and deletion operations.

[0081] In one embodiment, the above step S10 includes:

[0082] S101, traversing each subdirectory of the source directory hierarchy in a multi-threaded concurrent manner, and recording the absolute path, file size, and last modification time of each file in each subdirectory of the source directory hierarchy;

[0083] S102, traversing each subdirectory of the target directory hierarchy in a multi-threaded concurrent manner, and recording the absolute path, file size, and last modification time of each file in each subdirectory of the target directory hierarchy;

[0084] S103, generating a data content hash value for each file in each subdirectory in the source directory hierarchy, and combining the absolute path, file size, last modification time, and data content hash value of each file into a metadata entry for the corresponding file;

[0085] S104, generating a data content hash value for each file in each subdirectory in the target directory hierarchy, and combining the absolute path, file size, last modification time, and data content hash value of each file into a metadata entry for the corresponding file;

[0086] S105, storing metadata entries of all files in the source directory hierarchy according to the tree structure of the source directory hierarchy to form source file metadata;

[0087] S106 , storing metadata entries of all files in the target-end directory hierarchy according to the tree structure of the target-end directory hierarchy to form target-end file metadata.

[0088] In this embodiment, before performing efficient incremental data migration between heterogeneous file systems, a comparable metadata set must be constructed. The source directory hierarchy and the target directory hierarchy each represent two independent file trees without native synchronization mechanisms. The directory hierarchy not only reflects the logical organization of files but also determines the hierarchical read strategy of the data structure. The core task of scanning is to convert static storage structures into structured data records.

[0089] Multi-threaded concurrent traversal is a key approach to addressing the bottleneck of large-scale file system enumeration. Each thread can be bound to a subdirectory task and independently complete the reading of file information under that path, improving I / O scheduling parallelism. The recorded absolute path is the unique location value resolved during the path resolution process. The file size is used for subsequent block migration resource allocation, and the last modification time is used to identify the basis for incremental changes.

[0090] The data content hash value is an irreversible summary of the entire file data obtained by applying a hash function. Typical implementations include MD5 and SHA-256. This value can be used to determine content changes and perform integrity verification after migration. The hash value should be calculated during the file read phase to avoid duplicate reads and is typically tied to the modification time to form a composite consistency identifier.

[0091] Metadata entries are constructed using a combined approach, structuring absolute paths, sizes, timestamps, and hash values ​​into a unified format for easy processing by the subsequent comparison module. Tree-structured storage restores directory logic. Common formats include Trie-based trees, prefix path mapping tables, or nested Map structures, supporting fast retrieval and batch operations by directory hierarchy.

[0092] This two-sided mirroring metadata generation process ensures an aligned comparison basis before migration. It does not rely on any unified protocol or shared database and is adaptable to the concurrent scanning and incremental comparison requirements of multiple heterogeneous file storage systems.

[0093] A thread pool model can be used to divide the source and target directory structures into multiple task units for concurrent execution, with each thread responsible for reading files from one or more directory nodes. During file reading, underlying operating system APIs (such as Linux's stat or Windows' FindFirstFile) are called to extract the file's absolute path, size, and last modification time. Hash values ​​can be generated by reading the file contents and streaming them to the hash algorithm processing module, implementing simultaneous calculation while reading, making it suitable for low-memory processing of large files.

[0094] When constructing metadata entries, you can use JSON objects or custom structures to encapsulate each field, providing a unified interface for subsequent comparison processing modules. To facilitate recursive maintenance of the storage structure and rapid retrieval, you can use directory paths as hierarchical keys, organizing them into nested hash tables or ordered tree structures. For example, the path / a / b / c.txt can be mapped to a tree hierarchy of root->a->b->c.txt, supporting precise comparisons and sub-directory-level migrations.

[0095] In large-scale directory scenarios, a breakpoint-resume mechanism can be enabled for scan tasks. This mechanism persists the scan status of each directory level, allowing the task to resume from the last completed position in the event of an abnormal interruption, enhancing stability. Furthermore, path mapping strategies can be adapted to multiple storage backends (such as distributed file systems and object storage) to unify metadata extraction logic.

[0096] This embodiment constructs a metadata set under a directory tree structure by multi-threaded concurrent scanning, which not only improves the parallel efficiency of file reading, but also realizes the preparation of basic data for change identification without loading the full file content. The standardized structure of metadata entries enables subsequent classification and comparison operations to have high scalability and automated processing capabilities. Through the dual identification mechanism of hash value and timestamp, it can not only efficiently identify modification behavior, but also reduce the error rate. Under the tree structure storage, the logical position of each file is consistent with the order of operations, which effectively supports local positioning, structural reconstruction and task generation operations in the incremental migration process, and fundamentally solves the problems of memory overflow and uncontrollable resource competition caused by one-time loading in the existing technology.

[0097] In one embodiment, the above step S30 includes:

[0098] S301, setting a cache capacity threshold of a pipeline, wherein the cache capacity threshold limits the maximum memory usage of the pipeline;

[0099] S302, writing the source-side unique files, the target-side unique files, and the files shared by both ends into the pipeline in batches according to the tree structure order of the source-side directory hierarchy and the target-side directory hierarchy;

[0100] S303, when the number of files cached in the pipeline reaches the cache capacity threshold, suspending writing files to the pipeline until the comparison processing module reads and processes files from the pipeline and the number of files cached in the pipeline falls below the cache capacity threshold, and then resuming writing;

[0101] S304: The comparison processing module reads files from the pipeline in batches according to the directory hierarchy, and calls corresponding comparison processing logic according to the read file types.

[0102] In this embodiment, the source-side unique files, the target-side unique files, and the files shared by both ends are passed to the subsequent processing logic, which requires an intermediate communication mechanism that is highly coordinated in terms of file system structure, processing queue order, and resource usage. The "pipeline" used here not only refers to the memory buffer in the traditional sense, but also emphasizes its central role in asynchronous flow control and task scheduling. This structure supports the handover between source data and target data in the form of intermediate data objects, and has data isolation and dynamic regulation capabilities. Its core purpose is to isolate producer and consumer modules, alleviate instantaneous loads, and control resource occupancy. It is not limited to the traditional operating system pipeline (such as Linux's pipe), but can choose different forms of implementation according to the technology stack and scenario.

[0103] The cache capacity threshold defines the upper limit of data entries that can be written at a time. It is a soft resource boundary set dynamically based on storage and processing capabilities. Setting this threshold effectively avoids memory preemption and scheduling blockages caused by task backlogs in large-scale file systems. Essentially, it introduces predictability in memory load, ensuring that the system operates within a stable range. This threshold is typically adjustable based on historical processing throughput or target hardware conditions and should allow for dynamic reconfiguration.

[0104] Files are written in batches based on the directory hierarchy tree structure on both the source and target ends. This structure inherits the metadata organization from the previous stage and maintains the continuity of directory mappings during the transfer process. This helps maintain logical consistency and facilitates hierarchical grouping in the comparison module, improving cache hit rates and path-level recognition efficiency. "Batching" refers to the pacing of data push, often based on directory units or batch processing windows.

[0105] The system's suspension of writing after the number of files reaches a threshold demonstrates respect for soft resource boundaries, ensuring that tasks do not backlog or pipeline overflow before subsequent processing is complete. Once the comparison module processes some files, freeing up pipeline space and bringing the cache count back below a safe range, the system resumes writing. This type of mechanism typically relies on a producer-consumer model supplemented by a signaling mechanism to control state synchronization.

[0106] The comparison processing module's reading method follows the same directory sequential reading strategy as the writing method, maintaining symmetry between the source and target logical structures in the intermediate processing chain. Furthermore, the module dynamically invokes different processing logic based on each file type (e.g., source-only, target-only, or both). This allows for completely separate execution of classification tasks, forming a mutually exclusive multi-channel data processing structure. This type-aware decoupling structure ensures the accuracy of subsequent task generation logic.

[0107] In actual deployment, the transmission structure "pipeline" can be constructed through a multi-threaded buffer queue, so that a decoupled asynchronous communication relationship is formed between the data production module and the comparison module. Taking a typical implementation based on the Java language as an example, a blocking queue with capacity control capabilities can be built in the memory. Whenever the file classification processing module generates a batch of files unique to the source end, unique to the target end, or files shared by both ends, it is encapsulated as a file metadata unit according to the corresponding classification label and written to the buffer queue. When the number of data entries in the queue reaches the preset capacity threshold, the upstream writing is automatically suspended, and the writing is resumed after the downstream processing completes part of the data, thereby ensuring the stability of the overall memory usage.

[0108] For scenarios requiring cross-service module communication between different processing nodes, the transport architecture can be built using lightweight messaging middleware systems, such as Redis Stream or Kafka, to construct data stream channels with guaranteed order and capacity control. The file classification module writes the classified data metadata into the streaming channel as key-value pairs, appending directory path labels and type identifiers. The comparison module, acting as a data consumer, reads the file classification data from the streaming channel and selects different comparison strategies based on the identifiers. Given that the target system is deployed across different network nodes, this mechanism can effectively circumvent the limitations of traditional in-memory pipelines in distributed systems.

[0109] In certain operating system environments, the transfer structure can be implemented by constructing an intermediate cache directory on the local disk. After the file classification module completes grouping the source and target files, it generates a series of intermediate cache files according to the directory structure and writes these cache files to a temporary directory at the specified path. This intermediate cache directory acts as a logical "pipeline" that is read sequentially by the comparison module. Before each read, the module decides whether to continue loading based on the current disk space usage, thereby indirectly achieving capacity control. This implementation is suitable for data synchronization between heterogeneous operating systems, avoiding compatibility issues caused by directly sharing memory.

[0110] In data processing systems with extremely high performance requirements, the transmission structure can also be constructed using a memory-mapped mechanism. File classification results are written to a pre-allocated memory-mapped file area, avoiding the overhead of frequent heap memory allocations and allowing downstream modules to read the data content with zero copy. This mechanism is particularly suitable for systems that are extremely sensitive to transmission latency, such as those processing large-scale business logs, transaction vouchers, or raw clinical imaging data in real time.

[0111] This embodiment successfully implements the throttled transmission of file classification data in a large-scale directory structure by introducing a data handover strategy based on a pipeline cache mechanism, making the task generation logic more controllable and resource-adaptable. The sequential batch writing and reading logic under the tree structure eliminates the path dislocation and naming conflict problems brought about by traditional linear loading, while the decoupling of the processing logic based on file type realizes the complete separation of logical classification and behavioral classification. Under the dynamic flow control mechanism, the system can automatically adjust the task flow during peak processing periods, effectively avoiding the risk of memory overload caused by sudden accumulation, while ensuring the sustainable advancement of high-priority tasks. The overall strategy guarantees resource utilization efficiency and processing stability at the architectural level.

[0112] In one embodiment, the above step S40 includes:

[0113] S401, extracting the absolute path, file size, and last modification time recorded in the source-side file metadata of the source-side unique file, and generating a new source-side migration task containing the absolute path, file size, and last modification time of the source-side unique file;

[0114] S402, extracting the last modification time and data content hash value from the source-side file metadata of the shared file at both ends, and extracting the last modification time and data content hash value from the target-side file metadata corresponding to the shared file at both ends;

[0115] S403: When the last modification time recorded in the source-side file metadata of the shared file is later than the corresponding last modification time recorded in the target-side file metadata, a difference-overwriting migration task is generated, including the absolute path, file size, and last modification time recorded in the source-side file metadata of the shared file;

[0116] S404: When the last modification time recorded in the source-side file metadata of the shared file at both ends is the same as the corresponding last modification time recorded in the target-side file metadata, and the data content hash value recorded in the source-side file metadata is inconsistent with the corresponding data content hash value recorded in the target-side file metadata, a difference overwriting migration task is generated, which includes the absolute path and file size of the source-side file of the shared file at both ends, and the data content hash value recorded in the source-side file metadata;

[0117] S405, determining whether to retain or delete the target-side unique file according to a predefined deletion policy;

[0118] S406: When it is determined that the target-side unique file is to be deleted, a target-side deletion migration task is generated that includes the absolute path recorded in the target-side file metadata and a deletion operation instruction.

[0119] In this embodiment, when performing task generation operations through the comparison processing module, differentiated processing logic needs to be adopted for different types of file collections. Source-unique files refer to files that only exist in the source directory structure, and their task generation behavior is based on incremental appending. Extracting metadata fields of source-unique files, including absolute path, file size, and last modification time, is the basis for constructing the task structure. The absolute path is used to indicate the target location of the migration, the file size reflects the data load, and the last modification time is used for migration validity in subsequent processes or integrity judgment in recovery scenarios. The above fields are combined to form a new migration task on the source side.

[0120] For files shared by both ends, the core processing idea is to determine whether the file content has changed substantially. This process requires extracting the last modification time of the source and target ends, supplemented by the content hash value for additional verification. If the modification time of the source end is later than that of the target end, it means that the updated version of the source file takes precedence, and the target end data needs to be replaced by overwriting. The difference overwriting migration task constructed at this time must include path, size, and time information for accurate replacement by the migration execution module. If the time is consistent but the hash value is different, it means that the content has been modified but the time has not been updated. The system will use the hash difference as the criterion to trigger the forced overwriting logic. The constructed task must carry a hash field as a unique identifier of data consistency.

[0121] To handle files unique to the target, a deletion policy is required to control data cleanup behavior. This policy can be set in a configuration file, and the policy's results determine whether a target-side deletion migration task is generated. If a deletion condition is triggered, the absolute path of the file in the target's metadata is extracted and appended with a deletion instruction, forming a structured deletion task description unit. This task is executed during subsequent migration processing to ensure consistency between the source and target states.

[0122] In one deployment, the comparison processing module can be designed as a task generation engine that supports multi-threaded concurrent processing, continuously receiving data structure inputs for source-specific files, target-specific files, and files shared by both ends from independent transmission channels. This module implements path mapping by parsing the absolute path field of the file, reads the file size field to generate the task load indicator, and reads the last modification time and data content hash value to construct the comparison judgment criteria. When comparing files shared by both ends, the system first determines the timestamp difference. If the difference is significant, it prioritizes the generation of an overwrite task according to the time strategy; if the time is consistent, it triggers the hash calculation judgment to supplement the ability to identify subtle changes in content. Task description information is generated in a structured format and encapsulated as a migration task structure unit.

[0123] In another implementation, the comparison processing module supports a deletion policy mechanism driven by a rules engine. Policies can be defined using DSL rule expressions, configuration files, or database tables. For example, a target file can be deleted if its existence in a specified directory exceeds a threshold and there is no corresponding file path on the source. When executing a deletion decision, the system matches metadata entries against the policy expression; if a match is found, a deletion task is generated.

[0124] When implementing this logic across different storage systems, a cross-platform path mapping mechanism can be introduced to accommodate differences in path syntax across operating systems. For example, when migrating between Windows and Linux, a path format conversion table can be introduced to ensure that the path field in the task structure is recognizable to the execution end.

[0125] Example: In a healthcare data center, patient imaging data is frequently updated daily. The source is a PACS server, and the target is a historical imaging archive system. Because there is significant path overlap between newly added images and historical images, this task generation mechanism generates new migration tasks for newly added image data and overwrite migration tasks for existing images with modified versions. Furthermore, it automatically identifies and policy-based cleanup of discarded image files on the target due to changes in patient information. This prevents network congestion caused by duplicate transmissions and improves archiving system consistency.

[0126] In the financial sector, compliance document archiving uses transaction system file directories as the source storage and compliance audit storage as the target storage. Daily transaction documents are named and archived by transaction number, and some files may be overwritten during consolidation and reconciliation. This mechanism generates overwrite tasks for transaction documents based on both time and content, ensuring that the latest documents replace older records. Excess target-side test documents can be cleaned up based on deletion policies, improving data security and audit consistency.

[0127] This embodiment introduces differentiated construction logic driven by task type, allowing the system to precisely categorize addition, overwrite, and deletion migration behaviors at the file level. Through fine-grained metadata field extraction and rule-driven judgment logic, it significantly reduces unnecessary migration operations and the repeated transmission of redundant data. Furthermore, the generated structure of classified tasks provides a clear control foundation for subsequent pipeline distribution, priority scheduling, and execution feedback, enhancing the determinism and manageability of data migration.

[0128] In one embodiment, the above step S50 includes:

[0129] S501, classifying the source-side newly added migration tasks, the difference-covering migration tasks, and the target-side deleted migration tasks into a newly added task queue, a covered task queue, and a deleted task queue according to task type;

[0130] S502, setting independent transmission pipes for the newly added task queue, the overwritten task queue, and the deleted task queue, respectively, wherein the cache capacity threshold of the independent transmission pipe of the newly added task queue is a first threshold, the cache capacity threshold of the independent transmission pipe of the overwritten task queue is a second threshold, and the cache capacity threshold of the independent transmission pipe of the deleted task queue is a third threshold;

[0131] S503, writing the migration tasks in the newly added task queue, the overwritten task queue, and the deleted task queue into corresponding independent transmission channels in batches according to a preset task priority order;

[0132] S504: When the number of cached files in any independent transmission pipeline reaches the cache capacity threshold of the corresponding independent transmission pipeline, suspending writing of migration tasks to the independent transmission pipeline that has reached the cache capacity threshold until the migration processing module reads and processes the migration tasks in the independent transmission pipeline, reducing the number of cached files in the independent transmission pipeline to below the corresponding cache capacity threshold, and then resuming writing;

[0133] S505: The migration processing module reads migration tasks from each independent transmission pipeline according to the preset task priority order.

[0134] In this embodiment, after the source-side new migration tasks, difference-overwrite migration tasks, and target-side deletion migration tasks are output from the comparison processing module, they need to be transmitted to the migration processing module in an orderly, efficient, and isolated manner. In order to improve the controllability of the overall scheduling and the processing stability of the system, these tasks need to be categorized and organized according to task type. Task type refers to the specific operational semantics of the migration task, namely, adding, overwriting, or deleting. Different types correspond to significantly different data volumes, latency tolerance, and execution costs, so independent isolation is required in the transmission structure.

[0135] After classification, new task queues, overwrite task queues, and deletion task queues are formed, with each type of queue bound to an independent transmission channel. This transmission channel adopts a structure with cache control capabilities, and each channel sets a unique cache capacity threshold. The first threshold applies to new tasks, which usually have large data volumes, so the cache threshold can be set to a higher value to reduce scheduling jitter caused by frequent write pauses; the second threshold corresponds to overwrite tasks, which have relatively medium data volumes and certain real-time requirements; the third threshold applies to deletion tasks. Deletion operations are usually metadata-level actions that occupy less system resources, so the threshold can be set appropriately lower.

[0136] When tasks are written to independent transmission channels, they are processed according to a pre-set task priority order. This order can be based on business rules or system policies. For example, overwrite tasks can be processed first to maintain data consistency on the target end, followed by new tasks to supplement data, and finally delete operations. The writing process is carried out in batches to avoid sudden resource congestion.

[0137] In actual operation, when the number of cached files in any channel reaches the corresponding cache capacity threshold, write operations to that channel are suspended to prevent memory risks caused by buffer overflows. Writes are suspended until the migration processing module reads tasks from the channel and processes the cached data to below the threshold, ensuring that the system operates in a dynamically controlled state.

[0138] The migration processing module reads tasks from each transport channel in the same priority order, maintaining consistent system behavior and enabling consistent control of the migration execution process. Task reading can be accomplished through priority queues, scheduling policies, or event-driven mechanisms, ensuring that high-priority tasks enter the migration execution path first.

[0139] Example: In healthcare data systems, hospitals often face the need to centrally migrate multiple file types, including historical imaging data, newly added examination files, and erroneous image copies, when switching PACS (Picture Archiving System) storage. The system classifies newly generated examination files as new tasks and transmits them to the disaster recovery system via a dedicated transmission channel. These tasks are prioritized and require channels with higher cache thresholds to support high-speed transmission. Historical examination records may exist in both the source and target systems. Version differences identified through metadata comparison are considered differential overwrite tasks, receiving lower priority and using dedicated channels to ensure accurate transmission. Temporary image caches or duplicated images are marked as target-side deletion tasks based on rules. These tasks have low data volume and low business urgency, so they use the transmission channel with the smallest cache threshold and are assigned the lowest priority. By mapping these task types to channel configurations, the system optimizes network bandwidth and memory resources while maintaining image data continuity, ensuring prioritized migration of frequently added files and stable overwrite of historical data differences.

[0140] In electronic archive storage optimization projects in the financial industry, it is often necessary to migrate contract scans, approval records, and daily reports in the original production system to a structured data lake. Newly added tasks are usually the latest generated compliance documents, and the migration priority is high. The system allocates high-bandwidth channels and sets a large cache capacity to ensure fast delivery. Difference overwrite tasks are mainly used to update supplementary pages or revised versions of historical approval records. The migrated content needs to be accurately identified and overwritten, so its channel priority is slightly lower, but the reading and writing accuracy must be guaranteed. Deletion tasks correspond to cleaning up documents that are mistakenly synchronized or duplicated in the target system. These tasks enter the low-speed channel and are set to the lowest priority. When any channel reaches the cache capacity limit, by suspending writing and waiting for the migration processing module to read, memory exhaustion during peak hours is effectively avoided, and the overall task scheduling efficiency and system security and stability are improved.

[0141] In the control system synchronization scenario of a smart manufacturing factory, the configuration files, operation logs, and temporary program segments generated by different production nodes need to be regularly aggregated and migrated to the central control warehouse. The system categorizes daily new operation logs as new tasks and places them in the high-speed channel. Configuration files modified by engineers are identified as differential overwrite tasks and executed in the secondary channel. Invalid or obsolete temporary programs are scheduled as deletion tasks in the lowest channel. The cache capacity of each channel is dynamically configured based on the task volume and processing cycle. The priority scheduling mechanism ensures that control strategy changes are quickly synchronized to the central node, while avoiding command delays caused by excessive transmission load during network peak hours.

[0142] This embodiment classifies migration tasks into fine-grained categories according to operation types and adopts an independent transmission channel structure to establish a logical isolation and resource control mechanism between task scheduling and execution, which significantly improves the robustness of system operation and its high concurrency bearing capacity. The introduction of cache capacity thresholds enables the system to control overall memory usage through dynamic current limiting and on-demand suspension in the face of large-scale data bursts, avoiding uncontrollable risks caused by migration tool crashes or resource preemption. At the same time, a unified priority order ensures the behavioral consistency of migration tasks throughout the entire process from generation to execution, improving the predictability of system behavior and the stability of business docking. The system can customize execution strategies based on task types, making the migration path clearer and significantly improving execution efficiency.

[0143] In one embodiment, the above step S60 includes:

[0144] S601, the migration processing module parses the absolute path, file size, and last modification time in the newly added migration task on the source side, reads the corresponding source file from the source storage system according to the absolute path, transfers the source file to the target storage system in blocks according to a preset data block size and the file size, creates a target file corresponding to the source file in the target storage system, sets the last modification time of the target file to the last modification time of the source file, and records the file path, number of bytes transferred, and completion timestamp of the successful transfer;

[0145] S602, the migration processing module parses the absolute path and data content hash value in the difference overwrite migration task, reads the current data content of the source file and the target file from the source storage system and the target storage system respectively based on the absolute path, analyzes and compares the data content hash values ​​between the source file and the target file, determines the data difference between the source file and the target file, transfers the data difference of the source file to the target storage system to overwrite the target file, updates the last modification time of the target file to the current timestamp, and records the file path of the successful overwrite, the number of bytes of the difference data, and the completion timestamp;

[0146] S603: The migration processing module parses the absolute path and deletion operation instruction in the target-side deletion migration task, locates the target-side file in the target-side storage system according to the absolute path, deletes the target file, and records the file path and completion timestamp of the successful deletion.

[0147] S604, the migration processing module classifies and integrates the records of successful transfer, successful overwrite, and successful deletion by task type, and generates a structured migration result file including task type, file path, operation status, data volume, and timestamp;

[0148] S605: The migration processing module marks the migration task that fails during the migration task execution as a failed state, records the failure reason and failure timestamp, generates a migration error log, and adds the failed migration task to a corresponding migration task queue.

[0149] In this embodiment, the migration processing module undertakes the core tasks of the data migration execution phase. Different types of migration tasks are uniformly scheduled and processed in this module, and finally summarized into a structured migration result file. When processing a new migration task on the source side, the migration processing module first parses the absolute path, file size and last modification time of the file contained in the task. The path information is used to accurately locate the file entity to be migrated in the source storage system. The file size is used as a parameter of the transmission strategy to determine the block strategy, and the last modification time is used to restore the original file attributes on the target side. After the file is read, it will be divided into blocks according to the preset data block size and the actual file size. Each block is sequentially transmitted to the target side, and a new target side file is created in the target side storage system. The target side file is not only consistent with the source file in structure, but its last modification time will also be set to be consistent with the source side, thereby providing metadata basis for the subsequent verification mechanism. After the transfer is completed, the file path, the total number of bytes transferred and the completion timestamp are output as record information.

[0150] When processing the differential overwrite migration task, the migration processing module still locates the source and target files based on the absolute path, but unlike the newly added task, this task requires comparing the current data content hash values ​​of the source and target files to further analyze the data differences. The "data content hash value" here is not directly used as a judgment criterion, but is used to assist in confirming whether an overwrite transfer is required. If there are differences, only the compared and identified differential data blocks will be transferred from the source to the target to complete the overwriting of the original file content, and the last modification time of the target file will be updated to the current timestamp at the time of migration processing to form a traceable change record. After the overwrite is successful, the relevant fields will be structured and recorded, including the file path, the number of bytes of differential data transmission, and the time the operation was completed.

[0151] When processing a target-side deletion migration task, the migration processing module locates and deletes target files based on the absolute path and deletion instructions specified in the task. This operation primarily targets unique files identified earlier on the target side. Not all files are deleted directly; instead, they are filtered based on the deletion policy. After the deletion is complete, the file path and completion time are recorded for subsequent auditing or reconstruction operations.

[0152] To ensure that task execution results are fully tracked and archived, the migration processing module categorizes all successfully executed migration task records by task type and outputs them in a unified format as a structured migration result file. This result file contains fields such as the task type (add, overwrite, delete), the corresponding file path, the operation status (success or failure), the actual amount of data processed, and the completion timestamp. Furthermore, failed tasks during the migration process are explicitly marked with a failure status, and the corresponding failure cause and time are recorded. Such tasks are re-added to the corresponding migration task queue for retry, ensuring the closed-loop nature of task execution and system stability.

[0153] In different scenarios, the implementation of the migration processing module can adjust parameters according to the file system protocol, transmission network capabilities, and task scheduling requirements. For example, the transmission block size can be automatically adjusted according to the link throughput capacity, thereby achieving efficient migration of large blocks in high-bandwidth scenarios and gradually completing tasks in small blocks with low latency in low-bandwidth environments. The transmission process of newly added tasks can integrate a breakpoint resume mechanism to prevent task interruptions due to network fluctuations during large file transfers. Differential data transmission can identify changed blocks through a hash block comparison algorithm to transmit only necessary content, reduce duplicate writes, and reduce the I / O load on the target end. In the deletion task processing link, a batch deletion mechanism for file groups based on policy tags is supported, such as screening according to path rules, file naming specifications, or creation time intervals, to achieve controllable cleanup.

[0154] In terms of failed task management, you can configure parameters such as the number of automatic retries, retry intervals, and failure classification labels. It also supports reporting to the system management backend or alarm center through an asynchronous notification mechanism to assist manual intervention in handling irrecoverable failures.

[0155] Example description: In the active-active data center deployment scenario of the fintech system, the data migration processing module is often used to synchronize contract documents, account reconciliation sheets, and business logs in the production environment from the primary node to the disaster recovery node on a daily basis. The newly added task processing module pushes the new documents generated daily to the disaster recovery storage at high speed to ensure the continuous availability of key business information. The difference overwrite task is used to transmit change records or regenerated historical data. The system identifies the scope of changes based on hash differences and only synchronizes the changed blocks, thereby avoiding large amounts of unnecessary data retransmission. In some regular cleanup windows, the deletion task module will delete invalid files on the target side that have expired or have been identified as mistransmitted, freeing up storage space. The above tasks are all summarized as structured migration results and automatically generate logs to support business compliance audits and operation center status tracking.

[0156] In healthcare data systems, a new task module migrates X-ray images and ultrasound videos generated that day directly to the target device while synchronizing them with a remote disaster recovery center, preserving their original time attributes. If a patient's medical record image is retaken due to equipment error, a difference overwrite task is triggered to update the original image, with the updated file timestamp reflecting the overwrite time. Discarded or expired images identified by the system are deleted according to established rules, and logs are retained to ensure data governance and compliance.

[0157] Through the above steps, this embodiment enables precise scheduling, efficient execution, and tracking of tasks within the migration processing module. This addresses issues inherent in traditional data migration, such as confusing overwriting logic for new tasks, lack of feedback for failed tasks, overly coarse-grained difference identification, and uncontrollable deletion operations. Throughout the migration process, migration strategies are differentiated by clearly defining task types. Different migration tasks are handled differently, and a closed-loop migration path is formed through structured logging and failure management mechanisms, improving data integrity, consistency, and migration efficiency.

[0158] In one embodiment, after the above step S60, the method further includes:

[0159] S701, transmitting the migration result to a verification processing module;

[0160] S702: The verification processing module reads the current data content of the target file from the target storage system according to the file path in the migration result, and generates a hash value of the current data content of the target file according to the current data content;

[0161] S703: The verification processing module extracts a data content hash value of the corresponding file from the source-side file metadata according to the file path in the migration result, and compares the data content hash value of the corresponding file with the current data content hash value of the target-side file;

[0162] S704: When the current data content hash value of the target file is inconsistent with the data content hash value recorded in the source file metadata, the verification processing module adds the migration task corresponding to the file path to the migration processing module and marks the corresponding migration task as a migration task to be retried.

[0163] S705, when the current data content hash value of the target-side file is consistent with the data content hash value recorded in the source-side file metadata, the verification processing module adds a verification status field to the migration result and sets the verification status field to verification success.

[0164] In this embodiment, the migration results are transmitted to the verification processing module as a subsequent processing action after the migration processing module completes all tasks and generates the migration results. This independent module verifies the consistency of the file data in the target storage system. During the transmission process, the migration result file is a structured output containing core fields such as the file path, operation type, data volume, and completion time. The verification processing module directly relies on these fields to locate and schedule verification tasks.

[0165] After receiving the migration results, the verification module reads the file path field in each record and uses it to locate the target file in the target storage system. To ensure real-time and representative data comparison results, this module does not read the hash value generated in the migration record. Instead, it rereads the data content of the current target file and regenerates a hash value of the target file's current data content using a standard hash algorithm (such as SHA-256 or MD5). This hash value truly reflects the data content of the file in its current state on the target.

[0166] The verification module then extracts the file metadata entry corresponding to the path from the source file metadata and obtains the hash value of the data content recorded therein. This hash value is pre-generated and saved based on the source file content during the file metadata generation phase and represents the original data state of the source file before migration. By comparing the current hash value on the target side with the hash value recorded on the source side, we can accurately determine the success of the migration process and ensure that the file has not been modified or damaged on the target side.

[0167] If the comparison results show an inconsistency, indicating migration failure, data corruption, or changes to the target, the verification module will re-add the migration task corresponding to the file path to the migration module and explicitly mark it as a migration task to be retried. This mechanism not only improves system fault tolerance but also forms a self-healing closed-loop data processing mechanism.

[0168] If the comparison results are consistent, indicating the migration file is complete and correct, the verification module will add a new field to the migration results to indicate the verification status of the record and set it to "Verified Successful." This field will serve as an important reference for subsequent migration audits, data statistics, and automated archiving systems.

[0169] The verification processing module can adopt a variety of technical strategies in different deployment environments. For example, in a high-performance cluster, hash value verification can be performed on multiple migration result items in parallel, with each verification task assigned to a separate thread or container to avoid I / O blocking. On resource-constrained edge computing nodes, a batch verification strategy can be enabled, which divides verification by file size or migration time to reduce the one-time load. When generating the target-side hash value, a lightweight hashing algorithm can be used to speed up processing, or a caching mechanism can be introduced to avoid repeated calculations.

[0170] If deployed in scenarios with strict compliance requirements, a dual hash check strategy can be configured, generating and comparing hash values ​​using two different algorithms to further enhance data integrity. For the management of failed retry tasks, priority tags, retry counters, and failure classification tags can be introduced to enhance the ability to handle abnormal tasks.

[0171] Example: In a healthcare business scenario, a city-level tertiary hospital is undergoing a data center upgrade and migration. They need to migrate various historical files (such as electronic medical records, imaging data, examination reports, and patient label documents) from the original in-hospital storage system to a newly deployed distributed heterogeneous file system to achieve centralized data management and remote disaster recovery. Due to differences in protocols and structures between the old and new storage systems, and the fact that in-hospital data is constantly updated daily, the hospital wants to complete the data migration without affecting business continuity and ensure the integrity, accuracy, and verifiability of the migrated data.

[0172] The system first scans the directory structure from both the old storage system (i.e., the source) and the new target system (i.e., the target), extracting the absolute path, file size, and last modification time of each file. It also generates content hashes for all files and constructs file metadata for both the source and target in a tree-like structure. Since new records are added daily to the electronic medical record system, and some historical files may change due to patient reviews and revisions, file path identity alone is not sufficient to determine data identity. Therefore, the system accurately compares duplicated files in the two directory structures based on hash values ​​and modification times.

[0173] After receiving the scanned file collection, the comparison processing module identifies new files, modified files, and expired files, and generates corresponding migration tasks: for CT image files that only exist on the source side, a new migration task is created; for image report files whose modification time on the source side is later than that on the target side or whose content hash is inconsistent, a difference overwrite task is generated; and for those backup reports that have been deleted from the source side but still remain on the target side, the hospital's data retention policy automatically determines whether to generate a deletion task.

[0174] The above migration tasks are dynamically divided into three queues: new addition, overwrite, and delete, and enter the transmission pipeline with independent cache control thresholds. Based on the task priority strategy preset by the hospital IT administrator, the system gives priority to the rapid execution of overwrite tasks to ensure the consistency of patient data during the migration. After each type of task is written into the pipeline, the migration processing module reads the task content from it and triggers specific file reading and writing actions. For large-volume data such as image files, the system will adopt a block transmission method and build new files according to the corresponding path on the target end, while retaining the metadata attributes of the source end to support consistency verification.

[0175] After the migration is complete, the verification module rereads the latest contents of all migrated files on the target side. By generating hash values ​​in real time and comparing them with the hash records generated on the source side, it verifies that each file has maintained its integrity during the migration process. If any files (such as inspection reports) are found to have content deviations due to transmission interruptions, they will be automatically marked as failed and the corresponding migration task will be rescheduled. A migration result report will be generated, including transfer success, overwrite completion, and failed retry status, for subsequent auditing.

[0176] In a fintech business scenario, a national joint-stock bank planned to migrate the data backup platform for its credit approval system to a next-generation hybrid cloud architecture to accommodate higher-frequency incremental backup needs and improve cross-regional access efficiency. The system was originally deployed on a closed intranet file storage platform, while the new platform utilizes a unified object-based file system. The data organization structure, directory hierarchy rules, and file naming conventions differed between the two platforms, making direct synchronization impossible. Furthermore, bank approval documents such as scanned loan contracts, credit reports, loan disbursement certificates, and customer signature images are constantly added and updated during processing, placing extremely high demands on the data migration's incremental recognition capabilities and content consistency verification.

[0177] The migration tool first concurrently scans each business subdirectory on both the legacy and target platforms, performing directory-level parsing for files under different credit products (such as micro-loans, mortgages, and auto loans), generating structured file metadata. This metadata includes not only file path, size, and time, but also incorporates content hash values ​​as a key feature for subsequent decision-making. This ensures that even if the target platform's file path differs from the original platform, duplicate migrations can be avoided as long as the content is the same.

[0178] After scanning is complete, the system categorizes and transfers all identified files to the comparison module through a pipeline mechanism. New files on the source side, such as loan image packages from newly signed customers, are identified as "source-unique" and will directly generate new migration tasks. Files updated in historical contracts due to customer-requested revisions (such as re-signed guarantee documents) are identified as differential overwrite files due to their hash values ​​not matching the original files on the target side. If risk warning letters deleted from revoked business processes still remain on the target platform, the system will automatically generate deletion tasks based on the bank's compliance cleanup policy.

[0179] After all tasks enter the queue, the system sets a higher-priority task pipeline cache threshold based on financial regulatory requirements. For example, the transmission channel for differential overwrite tasks is given the highest processing priority, ensuring that historical revision files can be quickly synchronized to the target system for subsequent audit review. Each migration task records complete read and write logs and result markers during execution. Files on the target side not only match the source side in path length but also retain the precise last modification time for business logic inheritance.

[0180] After the migration is complete, the verification module activates and re-compares the hash values ​​of all migrated files. If any credit report attachments are tampered with or interrupted during transmission, the system automatically rolls the task back to the retry queue and marks it as "pending re-migration." Finally, the bank's compliance audit platform can access the structured task execution status, data volume, and verification flags in the migration results to generate a complete, traceable migration report to meet regulatory requirements for data consistency and verifiability.

[0181] This embodiment introduces an independent verification module to verify each migration result, reconfirming from the target end that data has been successfully migrated and that no content discrepancies have occurred. This mechanism significantly overcomes the "write-only, no-verification" flaw in traditional data migration systems, establishing a complete closed-loop migration system. Verification results can be used to generate final audit reports or trigger automatic error recovery processes, thereby improving system reliability, data consistency, and security.

[0182] In one embodiment, a data migration device based on task offloading is provided, and the data migration device based on task offloading corresponds one-to-one with the data migration method based on task offloading in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the data migration device based on task offloading according to the present invention. It includes a metadata generation module 10, a file classification module 20, a file transfer scheduling module 30, a migration task generation module 40, a task distribution module 50, and a migration processing module 60. Each functional module is described in detail below:

[0183] The metadata generation module 10 is used to scan the source directory hierarchy and the target directory hierarchy to generate source file metadata and target file metadata respectively;

[0184] A file classification module 20, configured to classify source files and target files into source-only files, target-only files, and files shared by both ends based on the file paths in the source-end file metadata and the target-end file metadata;

[0185] A file transfer scheduling module 30 is used to transfer the source-side unique files, the target-side unique files and the files shared by both ends to the comparison processing module;

[0186] Migration task generation module 40, configured to generate a source-side new migration task for the source-side unique files using the comparison processing module, generate a difference-overwrite migration task after performing metadata and data content comparison on the files shared by both ends, and generate a target-side delete migration task for the target-side unique files according to the deletion policy;

[0187] A task distribution module 50 is used to transfer the source-side newly added migration task, the difference-covering migration task, and the target-side deleted migration task to the migration processing module;

[0188] The migration processing module 60 is configured to execute the source-side new migration task, the difference-overwriting migration task, and the target-side deletion migration task through the migration processing module, and generate a migration result.

[0189] In one embodiment, the metadata generation module 10 is specifically configured to:

[0190] Traverse each subdirectory of the source directory hierarchy in a multi-threaded concurrent manner, and record the absolute path, file size, and last modification time of each file in each subdirectory of the source directory hierarchy;

[0191] Traverse each subdirectory of the target directory hierarchy in a multi-threaded concurrent manner, and record the absolute path, file size, and last modification time of each file in each subdirectory of the target directory hierarchy;

[0192] Generate a data content hash value for each file in each subdirectory in the source directory hierarchy, and combine the absolute path, file size, last modification time, and data content hash value of each file into a metadata entry for the corresponding file;

[0193] Generate a data content hash value for each file in each subdirectory in the target directory hierarchy, and combine the absolute path, file size, last modification time, and data content hash value of each file into a metadata entry for the corresponding file;

[0194] Storing metadata entries of all files in the source directory hierarchy according to the tree structure of the source directory hierarchy to form source file metadata;

[0195] The metadata entries of all files in the target-side directory hierarchy are stored according to the tree structure of the target-side directory hierarchy to form target-side file metadata.

[0196] In one embodiment, the file transfer scheduling module 30 is specifically configured to:

[0197] Setting a cache capacity threshold for a pipeline, wherein the cache capacity threshold limits the maximum memory usage of the pipeline;

[0198] Writing the source-side unique files, the target-side unique files, and the files shared by both ends into the pipeline in batches according to the tree structure order of the source-side directory hierarchy and the target-side directory hierarchy;

[0199] When the number of files cached in the pipeline reaches the cache capacity threshold, suspending writing files to the pipeline until the comparison processing module reads and processes files from the pipeline and resumes writing after the number of files cached in the pipeline falls below the cache capacity threshold;

[0200] The comparison processing module reads files in batches from the pipeline according to the directory hierarchy, and calls corresponding comparison processing logic according to the read file types.

[0201] In one embodiment, the migration task generation module 40 is specifically configured to:

[0202] Extracting the absolute path, file size, and last modification time recorded in the source-side file metadata of the source-side unique file, and generating a new source-side migration task containing the absolute path, file size, and last modification time of the source-side unique file;

[0203] Extracting the last modification time and data content hash value from the source-side file metadata of the shared file at both ends, and extracting the last modification time and data content hash value from the target-side file metadata corresponding to the shared file at both ends;

[0204] When the last modification time recorded in the source-side file metadata of the shared file is later than the corresponding last modification time recorded in the target-side file metadata, a difference-overwriting migration task is generated, which includes the absolute path, file size, and last modification time recorded in the source-side file metadata of the shared file;

[0205] When the last modification time recorded in the source-side file metadata of the shared file at both ends is the same as the corresponding last modification time recorded in the target-side file metadata, and the data content hash value recorded in the source-side file metadata is inconsistent with the corresponding data content hash value recorded in the target-side file metadata, a difference overwriting migration task is generated, which includes the absolute path, file size, and data content hash value recorded in the source-side file metadata of the shared file at both ends.

[0206] Determine whether to retain or delete the target-side unique file according to a predefined deletion policy;

[0207] When it is determined that the target-side unique file is to be deleted, a target-side deletion migration task is generated that includes the absolute path recorded in the target-side file metadata and a deletion operation instruction.

[0208] In one embodiment, the task distribution module 50 is specifically configured to:

[0209] Classify the source-side new migration tasks, difference-coverage migration tasks, and target-side deletion migration tasks into a new task queue, an overwriting task queue, and a deletion task queue according to task type;

[0210] Independent transmission pipelines are respectively set for the newly added task queue, the overwritten task queue, and the deleted task queue, wherein the cache capacity threshold of the independent transmission pipeline of the newly added task queue is a first threshold, the cache capacity threshold of the independent transmission pipeline of the overwritten task queue is a second threshold, and the cache capacity threshold of the independent transmission pipeline of the deleted task queue is a third threshold;

[0211] Writing the migration tasks in the newly added task queue, the overwritten task queue, and the deleted task queue into corresponding independent transmission pipelines in batches according to a preset task priority order;

[0212] When the number of cached files in any independent transmission pipeline reaches the cache capacity threshold of the corresponding independent transmission pipeline, the writing of migration tasks to the independent transmission pipeline that has reached the cache capacity threshold is suspended until the migration processing module reads and processes the migration tasks in the independent transmission pipeline, and the number of cached files in the independent transmission pipeline is reduced to below the corresponding cache capacity threshold, and then the writing is resumed;

[0213] The migration processing module reads the migration tasks from each independent transmission pipeline according to the preset task priority order.

[0214] In one embodiment, the migration processing module 60 is specifically configured to:

[0215] The migration processing module parses the absolute path, file size, and last modification time in the newly added migration task on the source side, reads the corresponding source file from the source storage system according to the absolute path, transfers the source file to the target storage system in blocks according to a preset data block size and the file size, creates a target file corresponding to the source file in the target storage system, sets the last modification time of the target file to the last modification time of the source file, and records the file path, number of bytes transferred, and completion timestamp of the successful transfer;

[0216] The migration processing module parses the absolute path and data content hash value in the difference overwrite migration task, reads the current data content of the source file and the target file from the source storage system and the target storage system respectively based on the absolute path, analyzes and compares the data content hash values ​​between the source file and the target file, determines the data difference between the source file and the target file, transfers the data difference of the source file to the target storage system to overwrite the target file, updates the last modification time of the target file to the current timestamp, and records the file path of the successful overwrite, the number of bytes of the difference data, and the completion timestamp;

[0217] The migration processing module parses the absolute path and deletion operation instruction in the target-side deletion migration task, locates the target-side file in the target-side storage system according to the absolute path, deletes the target-side file, and records the file path and completion timestamp of the successful deletion;

[0218] The migration processing module classifies and integrates the records of successful transfer, successful overwriting, and successful deletion according to task type, and generates a structured migration result file including task type, file path, operation status, data volume, and timestamp;

[0219] The migration processing module marks a migration task that fails during the execution of the migration task as a failed state, records the failure reason and failure timestamp, generates a migration error log, and adds the failed migration task to a corresponding migration task queue.

[0220] In one embodiment, the migration processing module 60 is specifically configured to:

[0221] Transmitting the migration result to a verification processing module;

[0222] The verification processing module reads the current data content of the target-side file from the target-side storage system according to the file path in the migration result, and generates a current data content hash value of the target-side file according to the current data content;

[0223] The verification processing module extracts a data content hash value of the corresponding file from the source-side file metadata according to the file path in the migration result, and compares the data content hash value of the corresponding file with the current data content hash value of the target-side file;

[0224] When the current data content hash value of the target-side file is inconsistent with the data content hash value recorded in the source-side file metadata, the verification processing module adds the migration task corresponding to the file path to the migration processing module and marks the corresponding migration task as a migration task to be retried;

[0225] When the current data content hash value of the target-side file is consistent with the data content hash value recorded in the source-side file metadata, the verification processing module adds a verification status field to the migration result and sets the verification status field to verification success.

[0226] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a data migration method based on task offloading.

[0227] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a data migration method based on task offloading.

[0228] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0229] Scan the source directory hierarchy and the target directory hierarchy to generate source file metadata and target file metadata respectively;

[0230] Classify the source-end file and the target-end file into source-end unique files, target-end unique files, and files shared by both ends according to the file paths in the source-end file metadata and the target-end file metadata;

[0231] The source-side unique file, the target-side unique file, and the files shared by both ends are transferred to a comparison processing module;

[0232] The comparison processing module generates a source-side new migration task for the source-side unique files, generates a difference-overwrite migration task after performing metadata and data content comparison on the files shared by both ends, and generates a target-side delete migration task for the target-side unique files according to the deletion policy;

[0233] Delivering the source-side newly added migration task, the difference-covering migration task, and the target-side deleted migration task to a migration processing module;

[0234] The migration processing module executes the source-side new addition migration task, the difference overwriting migration task, and the target-side deletion migration task, and generates a migration result.

[0235] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0236] Scan the source directory hierarchy and the target directory hierarchy to generate source file metadata and target file metadata respectively;

[0237] Classify the source-end file and the target-end file into source-end unique files, target-end unique files, and files shared by both ends according to the file paths in the source-end file metadata and the target-end file metadata;

[0238] The source-side unique file, the target-side unique file, and the files shared by both ends are transferred to a comparison processing module;

[0239] The comparison processing module generates a source-side new migration task for the source-side unique files, generates a difference-overwrite migration task after performing metadata and data content comparison on the files shared by both ends, and generates a target-side delete migration task for the target-side unique files according to the deletion policy;

[0240] Delivering the source-side newly added migration task, the difference-covering migration task, and the target-side deleted migration task to a migration processing module;

[0241] The migration processing module executes the source-side new addition migration task, the difference overwriting migration task, and the target-side deletion migration task, and generates a migration result.

[0242] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0243] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0244] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0245] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A data migration method based on task offloading, characterized in that: The following steps are involved: Scan the source directory hierarchy and the target directory hierarchy to generate source file metadata and target file metadata respectively; Classify the source-end file and the target-end file into source-end unique files, target-end unique files, and files shared by both ends according to the file paths in the source-end file metadata and the target-end file metadata; The source-side unique file, the target-side unique file, and the files shared by both ends are transferred to a comparison processing module; The comparison processing module generates a source-side new migration task for the source-side unique files, generates a difference-overwrite migration task after performing metadata and data content comparison on the files shared by both ends, and generates a target-side delete migration task for the target-side unique files according to the deletion policy; Delivering the source-side newly added migration task, the difference-covering migration task, and the target-side deleted migration task to a migration processing module; The migration processing module executes the source-side new addition migration task, the difference overwriting migration task, and the target-side deletion migration task, and generates a migration result.

2. The data migration method based on task offloading according to claim 1, characterized in that: Scan the source directory hierarchy and the target directory hierarchy to generate source file metadata and target file metadata, including: Traverse each subdirectory of the source directory hierarchy in a multi-threaded concurrent manner, and record the absolute path, file size, and last modification time of each file in each subdirectory of the source directory hierarchy; Traverse each subdirectory of the target directory hierarchy in a multi-threaded concurrent manner, and record the absolute path, file size, and last modification time of each file in each subdirectory of the target directory hierarchy; Generate a data content hash value for each file in each subdirectory in the source directory hierarchy, and combine the absolute path, file size, last modification time, and data content hash value of each file into a metadata entry for the corresponding file; Generate a data content hash value for each file in each subdirectory in the target directory hierarchy, and combine the absolute path, file size, last modification time, and data content hash value of each file into a metadata entry for the corresponding file; Storing metadata entries of all files in the source directory hierarchy according to the tree structure of the source directory hierarchy to form source file metadata; The metadata entries of all files in the target-side directory hierarchy are stored according to the tree structure of the target-side directory hierarchy to form target-side file metadata.

3. The data migration method based on task offloading according to claim 1, characterized in that: The source-side unique file, the target-side unique file, and the files shared by both ends are transferred to a comparison processing module, including: Setting a cache capacity threshold for a pipeline, wherein the cache capacity threshold limits the maximum memory usage of the pipeline; Writing the source-side unique files, the target-side unique files, and the files shared by both ends into the pipeline in batches according to the tree structure order of the source-side directory hierarchy and the target-side directory hierarchy; When the number of files cached in the pipeline reaches the cache capacity threshold, suspending writing files to the pipeline until the comparison processing module reads and processes files from the pipeline and resumes writing after the number of files cached in the pipeline falls below the cache capacity threshold; The comparison processing module reads files in batches from the pipeline according to the directory hierarchy, and calls corresponding comparison processing logic according to the read file types.

4. The data migration method based on task offloading according to claim 1, characterized in that: The comparison processing module generates a source-side new migration task for the source-side unique files, generates a difference-overwrite migration task after performing metadata and data content comparison on the files shared by both ends, and generates a target-side deletion migration task for the target-side unique files according to the deletion policy, including: Extracting the absolute path, file size, and last modification time recorded in the source-side file metadata of the source-side unique file, and generating a new source-side migration task containing the absolute path, file size, and last modification time of the source-side unique file; Extracting the last modification time and data content hash value from the source-side file metadata of the shared file at both ends, and extracting the last modification time and data content hash value from the target-side file metadata corresponding to the shared file at both ends; When the last modification time recorded in the source-side file metadata of the shared file is later than the corresponding last modification time recorded in the target-side file metadata, a difference-overwriting migration task is generated, which includes the absolute path, file size, and last modification time recorded in the source-side file metadata of the shared file; When the last modification time recorded in the source-side file metadata of the shared file at both ends is the same as the corresponding last modification time recorded in the target-side file metadata, and the data content hash value recorded in the source-side file metadata is inconsistent with the corresponding data content hash value recorded in the target-side file metadata, a difference overwriting migration task is generated, which includes the absolute path, file size, and data content hash value recorded in the source-side file metadata of the shared file at both ends. Determine whether to retain or delete the target-side unique file according to a predefined deletion policy; When it is determined that the target-side unique file is to be deleted, a target-side deletion migration task is generated that includes the absolute path recorded in the target-side file metadata and a deletion operation instruction.

5. The data migration method based on task offloading according to claim 1, characterized in that: The source-side newly added migration task, the difference-coverage migration task, and the target-side deleted migration task are delivered to the migration processing module, including: Classify the source-side newly added migration tasks, the difference-covering migration tasks, and the target-side deleted migration tasks into a newly added task queue, a covered task queue, and a deleted task queue according to task type; Independent transmission pipelines are respectively set for the newly added task queue, the overwritten task queue, and the deleted task queue, wherein the cache capacity threshold of the independent transmission pipeline of the newly added task queue is a first threshold, the cache capacity threshold of the independent transmission pipeline of the overwritten task queue is a second threshold, and the cache capacity threshold of the independent transmission pipeline of the deleted task queue is a third threshold; Writing the migration tasks in the newly added task queue, the overwritten task queue, and the deleted task queue into corresponding independent transmission pipelines in batches according to a preset task priority order; When the number of cached files in any independent transmission pipeline reaches the cache capacity threshold of the corresponding independent transmission pipeline, the writing of migration tasks to the independent transmission pipeline that has reached the cache capacity threshold is suspended until the migration processing module reads and processes the migration tasks in the independent transmission pipeline, and the number of cached files in the independent transmission pipeline is reduced to below the corresponding cache capacity threshold, and then the writing is resumed; The migration processing module reads the migration tasks from each independent transmission pipeline according to the preset task priority order.

6. The data migration method based on task offloading according to claim 1, characterized in that: The migration processing module executes the source-side new migration task, the difference-coverage migration task, and the target-side deletion migration task, and generates a migration result, including: The migration processing module parses the absolute path, file size, and last modification time in the newly added migration task on the source side, reads the corresponding source file from the source storage system according to the absolute path, transfers the source file to the target storage system in blocks according to a preset data block size and the file size, creates a target file corresponding to the source file in the target storage system, sets the last modification time of the target file to the last modification time of the source file, and records the file path, number of bytes transferred, and completion timestamp of the successful transfer; The migration processing module parses the absolute path and data content hash value in the difference overwrite migration task, reads the current data content of the source file and the target file from the source storage system and the target storage system respectively based on the absolute path, analyzes and compares the data content hash values ​​between the source file and the target file, determines the data difference between the source file and the target file, transfers the data difference of the source file to the target storage system to overwrite the target file, updates the last modification time of the target file to the current timestamp, and records the file path of the successful overwrite, the number of bytes of the difference data, and the completion timestamp; The migration processing module parses the absolute path and deletion operation instruction in the target-side deletion migration task, locates the target-side file in the target-side storage system according to the absolute path, deletes the target-side file, and records the file path and completion timestamp of the successful deletion; The migration processing module classifies and integrates the records of successful transfer, successful overwriting, and successful deletion according to task type, and generates a structured migration result file including task type, file path, operation status, data volume, and timestamp; The migration processing module marks a migration task that fails during the execution of the migration task as a failed state, records the failure reason and failure timestamp, generates a migration error log, and adds the failed migration task to a corresponding migration task queue.

7. The data migration method based on task offloading according to claim 1, characterized in that: After the migration processing module executes the source-side new migration task, the difference-coverage migration task, and the target-side deletion migration task and generates a migration result, the method further includes: Transmitting the migration result to a verification processing module; The verification processing module reads the current data content of the target-side file from the target-side storage system according to the file path in the migration result, and generates a current data content hash value of the target-side file according to the current data content; The verification processing module extracts a data content hash value of the corresponding file from the source-side file metadata according to the file path in the migration result, and compares the data content hash value of the corresponding file with the current data content hash value of the target-side file; When the current data content hash value of the target-side file is inconsistent with the data content hash value recorded in the source-side file metadata, the verification processing module adds the migration task corresponding to the file path to the migration processing module and marks the corresponding migration task as a migration task to be retried; When the current data content hash value of the target-side file is consistent with the data content hash value recorded in the source-side file metadata, the verification processing module adds a verification status field to the migration result and sets the verification status field to verification success.

8. A data migration device based on task offloading, characterized in that: The data migration device based on task offloading includes: A metadata generation module is used to scan the source directory hierarchy and the target directory hierarchy to generate source file metadata and target file metadata respectively; A file classification module, configured to classify source-end files and target-end files into source-end unique files, target-end unique files, and files shared by both ends according to the file paths in the source-end file metadata and the target-end file metadata; A file transfer scheduling module, configured to transfer the source-side unique files, the target-side unique files, and the files shared by both ends to the comparison processing module; A migration task generation module is configured to generate a source-side new migration task for the source-side unique files using the comparison processing module, generate a difference-overwrite migration task after performing a metadata and data content comparison on the files shared by both ends, and generate a target-side delete migration task for the target-side unique files according to the deletion policy; A task distribution module is used to transfer the source-end newly added migration task, the difference-covering migration task, and the target-end deleted migration task to the migration processing module; The migration processing module is used to execute the source-side new addition migration task, the difference overwriting migration task and the target-side deletion migration task through the migration processing module, and generate a migration result.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a task-offloading-based data migration program stored in the memory and runnable on the processor. When the task-offloading-based data migration program is executed by the processor, the steps of the task-offloading-based data migration method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a data migration program based on task offloading, and when the data migration program based on task offloading is executed by the processor, the steps of the data migration method based on task offloading according to any one of claims 1 to 7 are implemented.

Images