Distributed cluster-based cross-cloud storage platform data migration method and device

Abstract

The application provides a cross-cloud storage platform data migration method and device based on a distributed cluster, the method comprising: through a single distributed file collection cluster, performing concurrent data scanning on a source end cloud storage platform, and writing scanned metadata records into an intermediate database subtable; according to a preset routing rule, encapsulating file information of the metadata records in the intermediate database subtable into a to-be-migrated task message through a plurality of distributed file migration clusters, and placing the to-be-migrated task message into a preset Redis task pool, comparing metadata features according to the file information, matching a corresponding target migration mode, and performing data migration to a target end cloud storage platform according to the target migration mode, thereby greatly saving bandwidth resources and guaranteeing data consistency; and adopting an asymmetric cluster architecture of 'one collection and multiple migration' to realize decoupling of collection and migration, supporting independent horizontal expansion of migration capacity, and efficiently supporting concurrent processing of massive files.

Description

Technical Field

[0001] This invention relates to the field of cloud computing and big data processing technology, and in particular to a data migration method and apparatus for cross-cloud storage platforms based on distributed clusters. Background Technology

[0002] With the rapid development of cloud computing platforms, cloud storage has become a fundamental core service. Due to business migration, cost optimization, or security backup needs (such as cross-vendor disaster recovery), data transfer between cloud storage platforms has become a frequent and essential requirement. In practice, enterprises often need to perform large-scale data migrations between different cloud service providers to cope with business layout adjustments under a multi-cloud architecture.

[0003] Currently, data migration across cloud storage platforms mainly relies on two types of tools: one is dedicated tools provided by specific service providers. These tools are usually only optimized for their own platforms, have poor universality, and are difficult to meet the complex migration needs in multi-cloud hybrid environments; the other is general migration tools, which support data migration between multiple cloud platforms based on a unified framework (such as related patents CN106953893A, CN107291750B, etc.). However, when faced with petabyte-scale (hundreds of millions) files or high-concurrency online business migration scenarios, existing technical solutions generally suffer from the following technical bottlenecks: First, the transmission mechanism is crude, often employing a mechanical "download-upload" full migration logic. When the content of the source file remains unchanged, the entire data flow still occurs, resulting in a significant waste of network bandwidth and IO resources. Second, data consistency is difficult to guarantee. During large-scale data migration cycles (lasting several days or even months), the source business data often changes continuously. Existing solutions lack a real-time change awareness mechanism, which can easily lead to data lag or missed migration at the target end. Third, the system's concurrency and scalability are insufficient. Faced with hundreds of millions of file indexes, traditional monolithic database architectures cannot support the huge read and write pressure. Moreover, existing solutions typically use a single cluster architecture, which cannot cope with the concurrent throughput requirements of massive data through asymmetric horizontal scaling. Fourth, the fault tolerance mechanism and state management granularity are crude. Faced with network jitter or long-tail tasks, existing systems are prone to deadlock or task loss.

[0004] In summary, existing cross-cloud storage platform data migration technologies suffer from problems such as low transmission efficiency, poor data consistency, insufficient scalability, and imperfect fault tolerance mechanisms when dealing with petabyte-scale massive file transfer scenarios and high-concurrency online migration scenarios. These issues make it difficult to meet the actual needs of enterprises for large-scale cross-cloud business data transfer.

[0005] This section is intended to provide background or context for the embodiments of this application set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0006] One objective of this invention is to provide a data migration method for cross-cloud storage platforms based on a distributed cluster. This method utilizes multi-dimensional feature comparison of metadata and intelligent matching of various refined migration modes, prioritizing metadata operations over physical data stream transmission to significantly save bandwidth resources and ensure data consistency. It employs an asymmetric cluster architecture of "one-collection-multiple-migration" to decouple data collection and migration, supporting independent horizontal scaling of migration capabilities and efficiently supporting concurrent processing of petabyte-scale massive files. A closed-loop scheduling mechanism based on Redis's atomic task state pool enables automatic recycling of abnormal tasks and breakpoint resumption, significantly improving the system's reliability and fault tolerance in large-scale migration scenarios. Another objective of this invention is to provide a data migration device for cross-cloud storage platforms based on a distributed cluster. A further objective of this invention is to provide a computer-readable medium. A final objective of this invention is to provide a computer device.

[0007] To achieve the above objectives, this invention discloses a data migration method for cross-cloud storage platforms based on a distributed cluster, comprising: A single distributed file collection cluster performs concurrent data scanning on the source cloud storage platform and writes the scanned metadata records into a partitioned table in an intermediate database. According to the preset routing rules, the file information of the metadata records in the intermediate database partition table is encapsulated into a task message to be migrated through multiple distributed file migration clusters, and the task message to be migrated is placed into the preset Redis task pool. The metadata features of the tasks to be migrated in the Redis task pool and the file information of the target cloud storage platform are compared to match the corresponding target migration mode, and the data is migrated to the target cloud storage platform according to the target migration mode.

[0008] Preferably, a single distributed file collection cluster performs concurrent data scanning on the source cloud storage platform, and writes the scanned metadata records into a partitioned table in an intermediate database, including: Call the application programming interface of the source cloud storage platform to concurrently collect metadata records from multiple collection nodes within a single distributed file collection cluster; The metadata records are calculated according to the preset database sharding and table partitioning algorithm to obtain the table partitioning calculation results; Write the metadata records to the intermediate database shard table corresponding to the sharding calculation result.

[0009] Preferably, the method further includes: If no response message from the application programming interface of the source cloud storage platform is detected within the preset response delay threshold, the number of concurrent collection nodes will be reduced.

[0010] Preferably, according to preset routing rules, the file information of the metadata records in the intermediate database sub-tables is encapsulated into a task message to be migrated through multiple distributed file migration clusters, and the task message to be migrated is placed into a preset Redis task pool, including: According to the routing rules, the metadata records of the initial state corresponding to each distributed file migration cluster are retrieved in batches from the intermediate database tables. The file information of the initial state metadata record is encapsulated to generate a task message to be migrated, and the task message to be migrated is placed into a preset Redis task pool. The file information includes cluster identifier and file identifier.

[0011] Preferably, the metadata features of the tasks to be migrated in the Redis task pool and the file information of the target cloud storage platform are compared to match the corresponding target migration mode, and the data is migrated to the target cloud storage platform according to the target migration mode, including: Retrieve the task message to be migrated from the Redis task pool and set the status of the task message to be migrated to the running status in the intermediate database partition table; Based on the task message to be migrated, obtain the content verification value of the file to be migrated, the path of the file to be migrated, the name of the file to be migrated, the last modification time of the file to be migrated, and the existence status of the object to be migrated from the intermediate database sub-table. In addition, based on the file information of the target cloud storage platform, obtain the key content verification value, key file path, key file name, key file last modification event, and key object existence status from the intermediate database sub-table. According to the preset migration mode triggering conditions, the verification value of the content of the file to be migrated, the path of the file to be migrated, the name of the file to be migrated, the last modification time of the file to be migrated, and the existence status of the object to be migrated are compared with the key content verification value, key file path, key file name, key file last modification time, and key object existence status to match the corresponding target migration mode. According to the target migration mode, the task message to be migrated is migrated to the target cloud storage platform, and the task message to be migrated is identified as the file information of the target cloud storage platform in the intermediate database sub-table, and the status is updated to the success status.

[0012] Preferably, the method further includes: Count the duration for which tasks awaiting migration in the Redis task pool remain in a running state; The Redis task pool is polled at preset time intervals, and migration messages are selected for recycling based on the duration and preset timeout threshold. Record the location information and task identifier of the recovered migration message, and update the status of the recovered migration message to the initial status in the intermediate database sub-table.

[0013] This invention also discloses a data migration device for cross-cloud storage platforms based on a distributed cluster, comprising: The concurrent data scanning unit is used to perform concurrent data scanning on the source cloud storage platform through a single distributed file collection cluster, and write the scanned metadata records into the intermediate database partition table. The task message generation unit is used to encapsulate the file information of the metadata records in the intermediate database sub-tables into a task message according to the preset routing rules through multiple distributed file migration clusters, and put the task message into the preset Redis task pool. The data migration unit is used to compare the metadata features of the task messages to be migrated in the Redis task pool and the file information of the target cloud storage platform, match the corresponding target migration mode, and migrate the data to the target cloud storage platform according to the target migration mode.

[0014] The present invention also discloses a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0015] The present invention also discloses a computer device, including a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, wherein the processor executes the program to implement the method described above.

[0016] The present invention also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method described above.

[0017] This invention uses a single distributed file acquisition cluster to perform concurrent data scanning on the source cloud storage platform, writing the scanned metadata records into an intermediate database partition table. Following preset routing rules, multiple distributed file migration clusters encapsulate the file information of the metadata records in the intermediate database partition table into migration task messages, which are then placed into a preset Redis task pool. Metadata features are compared between the migration task messages in the Redis task pool and the file information on the target cloud storage platform to match the corresponding target migration mode. Data migration is then performed to the target cloud storage platform according to the target migration mode. Through multi-dimensional metadata feature comparison and intelligent matching of various refined migration modes, metadata operations are prioritized over physical data stream transmission, significantly saving bandwidth resources and ensuring data consistency. An asymmetric cluster architecture of "one acquisition, multiple migrations" decouples acquisition and migration, supports independent horizontal scaling of migration capabilities, and efficiently supports concurrent processing of petabyte-scale massive files. A closed-loop scheduling mechanism based on the Redis atomic task state pool enables automatic recycling of abnormal tasks and breakpoint resumption, significantly improving the system's reliability and fault tolerance in large-scale migration scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a cross-cloud storage platform data migration method based on a distributed cluster, provided as an embodiment of the present invention; Figure 2 A flowchart illustrating another data migration method across cloud storage platforms based on a distributed cluster, provided as an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a cross-cloud storage platform data migration device based on a distributed cluster, provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] To facilitate understanding of the technical solution provided in this application, the relevant content of the technical solution will be described below. Based on the design concept of decoupling the "management control plane" and the "distributed execution plane," this invention constructs an asymmetric cluster architecture of "one-collection-multiple-migration," breaking the limitations of traditional linear migration processes. Simultaneously, by combining a highly reliable scheduling model based on Redis with various refined transmission modes, it effectively solves the problems of bandwidth resource waste, system concurrency bottlenecks, and data consistency during large-scale data migration.

[0022] This invention provides a cross-cloud storage platform data migration system based on a distributed cluster. The system is logically divided into a management control unit and a distributed execution unit.

[0023] The management and control unit includes a user cloud storage information management module and a task information management module: The user cloud storage information management module is responsible for managing authentication information in a multi-cloud environment. Specifically, the module receives and encrypts the user's access keys (Access Key / Secret Key) on both the source and target cloud storage platforms. Simultaneously, this module defines the database sharding and table partitioning rules for file metadata, including the number of tables and the hash algorithm for the sharding keys. The user cloud storage information management module ensures the security of authentication information through encrypted storage and provides a storage foundation for concurrent writing and querying of billions of data entries through the design of database sharding and table partitioning rules.

[0024] The Task Information Management module provides a visual interface for maintaining the full lifecycle status of migration tasks. Specific functions include displaying the current task status (e.g., initial, running, successful, failed), and allowing operators to manually reset the status of failed tasks to trigger retries. Furthermore, this module provides file transfer routing configuration, allowing operators to establish mapping relationships between logical database tables and the physical migration cluster (e.g., assigning tables 0-100 to cluster A). The Task Information Management module enables visualized operation and maintenance of migration tasks and provides routing control capabilities for task distribution.

[0025] The distributed execution unit is responsible for the high-concurrency execution of specific migration business. In order to solve the resource competition problem between two types of tasks with different resource characteristics, namely metadata traversal (input / output intensive) and entity data transmission (bandwidth intensive), the distributed execution unit adopts a business decoupling design and an asymmetric cluster configuration of "one acquisition and multiple migrations".

[0026] This invention logically decouples the entire data migration lifecycle into two independent stages: the file metadata acquisition stage and the physical file migration stage. The file metadata acquisition stage is responsible for obtaining the file list and its metadata information from the source. The physical file migration stage is responsible for performing the actual file data transfer based on the metadata information.

[0027] The file metadata acquisition phase and the entity file migration phase interact asynchronously through an intermediate database. The acquisition phase writes metadata to the intermediate database, and the migration phase reads metadata from the intermediate database and performs the transmission; the two phases do not block each other. This invention separates input / output intensive tasks from bandwidth-intensive tasks, avoiding resource contention between the two types of tasks within the same cluster.

[0028] Based on the above decoupling design, two types of distributed clusters are constructed: distributed file acquisition clusters and distributed file migration clusters.

[0029] The distributed file acquisition cluster is configured as a single distributed cluster focused on performing metadata acquisition tasks. Specific functions include: calling the application programming interface (API) of the source cloud storage platform to traverse the file list, identifying newly added or modified files, and building a file metadata index. During operation, this cluster primarily consumes computing resources (CPU) and disk I / O resources.

[0030] The distributed file migration cluster is configured as multiple physically isolated distributed clusters, focusing on performing physical file transfer tasks. Specific functions include: reading metadata records from an intermediate database and calling the application programming interfaces (APIs) of the source and target cloud storage platforms to perform file upload, download, or metadata operations. This cluster primarily consumes network bandwidth resources during operation.

[0031] Based on the characteristics of fast acquisition speed, slow transmission speed, and different resource consumption types of the two types of clusters, this invention implements asymmetric configuration: Data collection configuration: The system typically deploys a distributed file collection cluster. Due to the high efficiency of metadata scanning operations, a single cluster is sufficient to support the rapid generation and real-time updating of massive file lists.

[0032] Migration endpoint configuration: The system deploys multiple distributed file migration clusters to horizontally scale and distribute the data transmission load in parallel. When an increase in overall transmission speed is needed, only the number of migration clusters needs to be increased; there is no need to expand the collection clusters.

[0033] Metadata generated by a single file acquisition cluster is stored in sharded tables within an intermediate database, forming a common data source. Multiple file migration clusters can simultaneously connect to this common data source through routing rules configured in the management control unit (i.e., the mapping relationship between logical shards and physical clusters). Each migration cluster identifies the shard range it handles based on the routing rules, batches metadata records from the corresponding shards, encapsulates them into migration tasks, and executes them in parallel. This invention achieves independent and elastic scheduling of acquisition and migration capabilities, eliminating the need for redundant expansion of the acquisition cluster as migration load increases, thus maximizing resource utilization efficiency.

[0034] The internal task scheduling of the aforementioned file acquisition cluster and file migration cluster is built on the same distributed scheduling model, namely: a Redis-based highly reliable distributed scheduling model, which includes a Redis-based task pool and a four-role collaboration mechanism.

[0035] Each distributed execution unit cluster uses Redis components to build its own dedicated task state pool. Redis's atomic operation features are used to maintain the flow of task state to four collaborative roles for processing, ensuring that the state does not become chaotic in a distributed environment.

[0036] The Redis-based highly reliable distributed scheduling model defines four collaborative roles: Publisher, Contractor, Worker, and Recycler.

[0037] The task publisher is responsible for pulling raw records to be processed in batches from the data source (such as intermediate database partitions), encapsulating them into standard task messages, and pushing them to the Redis task pool. The initial state is set to "initial".

[0038] The Contractor is responsible for pulling tasks in the "Initial" state from the Redis task pool, updating the task status to "Running", and distributing them to specific task handlers. The Contractor is also responsible for recording the task distribution relationships and timeout periods.

[0039] The task handler (Worker) receives tasks distributed by the contractor and executes specific business logic (such as metadata comparison, file transfer, etc.). During task execution, the handler periodically reports heartbeat information to Redis (to maintain the running status for a certain duration); after execution, the task status is updated to "success" or "failure" based on the execution result.

[0040] The task recycler runs independently, periodically scanning the Redis task pool for tasks in the "running" state. For tasks that haven't been updated after a preset heartbeat time (i.e., zombie tasks), the recycler determines that the node to which it belongs is abnormal and forcibly resets the task's state to "initial" in the intermediate database. The reset task can then be reclaimed and executed by other healthy nodes. The technical advantages are: automatically detecting and reclaiming tasks left behind by abnormal nodes, avoiding task deadlocks or loss, and achieving breakpoint resumption and automated fault tolerance.

[0041] It is worth noting that the cross-cloud storage platform data migration system based on distributed clusters is also suitable for... Figure 1 or Figure 2 The data migration method based on distributed clusters across cloud storage platforms will not be elaborated here.

[0042] The following uses a data migration device based on a distributed cluster for cross-cloud storage platforms as an example to illustrate the implementation process of the data migration method based on a distributed cluster for cross-cloud storage platforms provided in this embodiment of the invention. It is understood that the execution entity of the data migration method based on a distributed cluster for cross-cloud storage platforms provided in this embodiment of the invention includes, but is not limited to, a data migration device based on a distributed cluster for cross-cloud storage platforms.

[0043] Figure 1 A flowchart illustrating a cross-cloud storage platform data migration method based on a distributed cluster, as provided in this embodiment of the invention, is shown below. Figure 1 As shown, the method includes: Step 101: Perform concurrent data scanning on the source cloud storage platform through a single distributed file collection cluster, and write the scanned metadata records into the intermediate database sub-table.

[0044] In this embodiment of the invention, a distributed file acquisition cluster comprising multiple acquisition nodes is deployed, with each node operating in parallel. The acquisition nodes call the application programming interface (API) provided by the source cloud storage platform to traverse and scan the file list of the source cloud storage platform.

[0045] In this embodiment of the invention, the cloud storage platform is preferably a cloud disk service platform with a file identifier (File ID) attribute.

[0046] During the scanning process, the acquisition nodes obtain key metadata information for each file, including but not limited to: file logical path, content checksum (such as ETag, MD5, SM3, SHA256, etc.), file size, and last modification time. After obtaining the above information, the acquisition nodes determine which physical table in the intermediate database the metadata record should be written to based on preset database sharding rules (such as hashing the file path and then taking the modulo), and write the record to the corresponding table. At the same time, the system initializes the migration status field of the metadata record to initial, indicating that the file has not yet been migrated.

[0047] Furthermore, a single distributed file acquisition cluster performs concurrent data scans on the source cloud storage platform at preset time intervals to update metadata records, add files, or modify existing files.

[0048] This invention efficiently completes the metadata traversal and index construction of massive files through concurrent scanning of a single distributed collection cluster; through a database sharding and table partitioning mechanism, hundreds of millions of metadata are distributed and stored in multiple database tables, laying the foundation for subsequent high-concurrency reading; the collection process runs independently and is not affected by migration speed, and can detect changes in source data in real time and update metadata records, ensuring data consistency.

[0049] Step 102: According to the preset routing rules, the file information of the metadata records in the intermediate database sub-tables is encapsulated into a task message to be migrated through multiple distributed file migration clusters, and the task message to be migrated is placed into the preset Redis task pool.

[0050] In this embodiment of the invention, routing rules are configured in the management control unit. These routing rules establish a mapping relationship between logical database sub-tables and physical migration clusters. For example, sub-tables 0-100 are designated to be processed by distributed file migration cluster A, and sub-tables 101-200 are designated to be processed by distributed file migration cluster B.

[0051] Each file migration cluster has a task publisher component deployed within it. Based on the aforementioned routing rules, the task publisher actively connects to the physical tables in the intermediate database that belong to the cluster's processing scope, and batches pull metadata records in the initial state. After pulling the records, the task publisher encapsulates the file information of each metadata record into a standardized migration task message. The file information includes the cluster identifier and file identifier, and the message content includes at least the file identifier, source file path, target cloud storage platform target path, and file content checksum.

[0052] The task publisher pushes the packaged task message to be migrated to the pre-set Redis task pool within the cluster and updates the task status to published in the intermediate database.

[0053] This invention enables multiple migration clusters to consume files to be migrated from different databases or tables in parallel through routing rules, thereby improving the overall migration throughput; it centrally manages task status through a Redis task pool, providing a foundation for subsequent fine-grained scheduling and fault-tolerant processing; and each migration cluster runs independently, achieving physical resource isolation and fine-grained allocation of network egress bandwidth.

[0054] Step 103: Compare the metadata features of the task messages to be migrated in the Redis task pool and the file information of the target cloud storage platform to match the corresponding target migration mode, and migrate the data to the target cloud storage platform according to the target migration mode.

[0055] In this embodiment of the invention, the task contractor pulls tasks in the initial state from the intermediate database page by page, writes them into the Redis task pool, and updates them to published in the database.

[0056] In this embodiment of the invention, a task message to be migrated is obtained from the Redis task pool, and the status of the task message to be migrated is set to running. Based on the task message to be migrated, the content verification value of the file to be migrated, the path of the file to be migrated, the name of the file to be migrated, the last modification time of the file to be migrated, and the existence status of the object to be migrated are obtained from the intermediate database sub-table. Furthermore, based on the file information of the target cloud storage platform, the key content verification value, the key file path, the key name, the last modification event of the key file, and the existence status of the key object are obtained.

[0057] Specifically, the task handler cross-compares the verification values ​​of the file content to be migrated, the file path to be migrated, the file name to be migrated, the last modification time of the file to be migrated, and the existence status of the object to be migrated with the verification values ​​of key content, key file path, key file name, the last modification event of key files, and the existence status of key objects. The comparison dimensions include: file existence, consistency of content verification values, consistency of file path, consistency of file name, and last modification time. Based on the comparison results, the system automatically matches the corresponding execution logic from a variety of preset migration modes. Migration modes include, but are not limited to, upload mode, update mode, rename mode, move mode, delete mode, rename and move mode, skip mode, and delete and upload mode. According to the matched migration mode, the system calls the API of the corresponding cloud storage platform to execute the specific operation. For modes such as upload and update that require the transmission of physical data, data stream transmission is performed; for modes such as rename and move that only involve metadata changes, the target cloud storage platform's internal metadata operation interface is directly called, without the need for cross-network physical data transmission. After the task is completed, the task handler updates the task status in Redis to "Success" or "Failed".

[0058] This invention uses multi-dimensional feature comparison and refined pattern matching to replace physical data transmission with metadata operations when only metadata changes such as renaming or moving occur in a file, significantly saving cross-cloud bandwidth resources; it achieves idempotency of migration operations through skip mode to avoid duplicate transmissions; and it maintains data consistency at both ends through deletion mode, effectively solving the problems of data migration omissions and dirty data residues.

[0059] In the technical solution provided by this invention, a single distributed file acquisition cluster performs concurrent data scanning on the source cloud storage platform, writing the scanned metadata records into an intermediate database partition table. According to preset routing rules, multiple distributed file migration clusters encapsulate the file information of the metadata records in the intermediate database partition table into migration task messages, which are then placed into a preset Redis task pool. Based on the metadata feature comparison between the migration task messages in the Redis task pool and the file information of the target cloud storage platform, a corresponding target migration mode is matched, and data migration is performed to the target cloud storage platform according to the target migration mode. Through multi-dimensional feature comparison of metadata and intelligent matching of various refined migration modes, metadata operations are prioritized to replace physical data stream transmission, significantly saving bandwidth resources and ensuring data consistency. An asymmetric cluster architecture of "one acquisition, multiple migrations" is adopted to decouple acquisition and migration, supporting independent horizontal scaling of migration capabilities and efficiently supporting concurrent processing of PB-level massive files. Based on the closed-loop scheduling mechanism of the Redis atomic task state pool, automatic recycling of abnormal tasks and breakpoint resumption are achieved, significantly improving the reliability and fault tolerance of the system in large-scale migration scenarios.

[0060] Figure 2 A flowchart illustrating another data migration method for cross-cloud storage platforms based on a distributed cluster, as provided in this embodiment of the invention, is shown below. Figure 2 As shown, the method includes: Step 201: Call the API of the source cloud storage platform to collect metadata records concurrently from multiple collection nodes within a single distributed file collection cluster.

[0061] In this embodiment of the invention, each step is performed by a cross-cloud storage platform data migration device based on a distributed cluster.

[0062] In this embodiment of the invention, a single distributed file collection cluster (e.g., 3 nodes) containing multiple collection nodes is deployed. Each collection node is configured with access credentials (such as Access Key and SecretKey) of the source cloud storage platform and has permission to call the platform's API.

[0063] After the data collection task is initiated, each collection node runs in parallel, collectively traversing the file list of the source cloud storage platform. Specifically, each node is responsible for scanning a portion of the file path range (e.g., splitting by filename hash value or directory prefix) and calling the enumeration object interface provided by the source cloud storage platform to obtain file metadata and generate metadata records.

[0064] Through the aforementioned concurrent calls, a single distributed file collection cluster can complete the metadata scanning of massive amounts of files in a relatively short time. The acquired metadata records include at least: the file's logical path (object key), content checksum (such as ETag), file size, and last modification time.

[0065] Furthermore, if no response message from the application programming interface of the source cloud storage platform is detected within the preset response delay threshold, the number of concurrent collection nodes is reduced.

[0066] In this embodiment of the invention, during the process of the collection node calling the source API, the system's built-in flow control algorithm continuously monitors the response status of each API request. Specifically, it records the time interval from sending a request to receiving a response (i.e., response latency) and compares it with a preset response latency threshold. If no response message is received from the source API within the preset response latency threshold, or if the response latency continues to exceed the threshold, the system determines that the current request frequency may put pressure on the source service or that network congestion has occurred. At this time, the flow control algorithm automatically triggers a degradation mechanism: reducing the number of concurrent requests of the current collection node (e.g., reducing the number of concurrent threads from 10 to 5, or pausing some scanning tasks).

[0067] Once the response latency is monitored and found to have returned to normal and remained stable for a period of time, the system gradually restores the number of concurrent requests to the initial level. Through this dynamic feedback adjustment, the data collection cluster can adaptively adjust the data collection intensity, avoiding the triggering of rate limiting mechanisms by the source cloud storage service due to instantaneous high concurrency requests, thereby ensuring the normal operation of the source service.

[0068] This invention significantly improves the efficiency of collecting massive amounts of file metadata by scanning concurrently through multiple collection nodes, thus building a complete file index foundation for subsequent migration. The built-in adaptive flow control algorithm monitors API response latency in real time and dynamically adjusts the number of concurrent processes, effectively avoiding rate limiting of the source cloud storage service due to excessive collection pressure and ensuring the stability of the source business.

[0069] Step 202: Calculate the metadata records according to the preset database sharding and table partitioning algorithm to obtain the table partitioning calculation result.

[0070] As an optional approach, after acquiring the metadata records (including file paths, content checksums, sizes, modification times, etc.) from the source cloud storage platform, the acquisition node extracts a key value for sharding from these records. Typically, the logical path of the file (object key) is chosen as the sharding key because file paths are unique and random, making them suitable as input to a hash function. The acquisition node calls a preset hash function to calculate the sharding key, generating a fixed-length hash value. This hash value serves as the result of subsequent sharding calculations to determine the sharding table number. To ensure sharding uniformity, the hash function should be chosen so that the hash results for different file paths are relatively evenly distributed across the value range.

[0071] It is worth noting that database sharding and table partitioning algorithms can also use other calculation methods, not limited to hash functions.

[0072] Step 203: Write the metadata records into the intermediate database shard table corresponding to the sharding calculation result.

[0073] In this embodiment of the invention, the total number of intermediate database tables is predefined (e.g., 1024 tables). Specifically, the acquisition node performs a modulo operation on the total number of intermediate database tables using the table partitioning calculation result, obtaining an integer between 0 and 1023, which is the number of the target table.

[0074] The data acquisition node, based on the table partition number, performs an insert operation via database connection, writing the complete metadata record to the corresponding intermediate database partition table. Simultaneously, it initializes the migration status field of this record to "Pending," indicating that the file has not yet been migrated.

[0075] Furthermore, if a database connection failure or write conflict occurs during the write process, the collection node can be configured with a retry mechanism to ensure that the metadata is eventually written to the database. In addition, each collection node independently executes the database sharding algorithm and write operations without the need for coordination between them, thereby ensuring write efficiency in high-concurrency scenarios.

[0076] This invention performs hash operations and modulo operations on metadata records to evenly distribute massive amounts of file metadata across multiple database tables, effectively avoiding single-table data overload and improving the overall storage and read / write capabilities of the system. At the same time, based on the results of the database sharding algorithm, it provides precise sharding granularity for subsequent multiple migration clusters to pull tasks in parallel according to routing rules, realizing distributed parallel processing of tasks and physical resource isolation.

[0077] Step 204: According to the routing rules, retrieve the metadata records of the initial state of each distributed file migration cluster in batches from the intermediate database tables.

[0078] In this embodiment of the invention, the routing rules are the mapping relationships between intermediate database table partitions and distributed file migration clusters. These mapping relationships are stored in the configuration center of the management control unit. Each distributed file migration cluster loads the table partition range it is responsible for from the configuration center upon startup or periodically and caches it in local memory. The routing rules support dynamic adjustment. When load distribution needs to be adjusted, operators can update the configuration in the management control unit, and each cluster automatically detects and re-fetches the task range.

[0079] As an optional solution, the routing rules are as follows: table 0~200 corresponds to distributed file migration cluster A; table 201~400 corresponds to distributed file migration cluster B; and table 401~600 corresponds to distributed file migration cluster C.

[0080] In this embodiment of the invention, one or more task publishers are deployed within each distributed file migration cluster. The task publisher determines the range of shards to connect to based on the routing rules loaded in the cluster. The task publisher establishes a connection with an intermediate database and performs batch query operations on the shard range. Specifically, it iterates through the shard range and selects metadata records with the migration status field set to "Pending". The maximum number of records retrieved at once can be set as needed (e.g., 1000 records per query) to avoid memory pressure caused by loading too much data at once. The task publisher supports multi-threaded concurrent queries on multiple shards to improve retrieval efficiency.

[0081] This invention maps massive amounts of metadata shards to multiple migration clusters for parallel retrieval through routing rules, achieving linear scaling of migration throughput with the number of clusters and effectively supporting high-concurrency processing of petabyte-level data. Each distributed file migration cluster only processes tasks within a specified table range, achieving physical resource isolation and load balancing, and avoiding mutual interference between business processes. A batch retrieval mechanism is adopted to reduce frequent database access and ensure the stable operation of the intermediate database.

[0082] Step 205: Encapsulate the file information of the initial state metadata record, generate the task message to be migrated, and place the task message to be migrated into the preset Redis task pool.

[0083] In this embodiment of the invention, the task publisher extracts file information, including a cluster identifier and a file identifier. A unique task identifier (Task ID) is generated for each task to be migrated, which can be a UUID or a combination of file path and timestamp to ensure the task's uniqueness globally. The cluster identifier (fixed prefix) and file identifier are assembled into a standardized task message to be migrated according to a preset data structure. This task message has a unique task identifier. The message format can be JSON or other lightweight data exchange formats for easy subsequent parsing and processing.

[0084] In this embodiment of the invention, after the task message to be migrated is encapsulated, the task publisher pushes the task message to be migrated to the pre-defined Redis task pool in the cluster. Specifically, the task publisher writes the task message to be migrated into the task pool through Redis atomic operations.

[0085] This invention encapsulates the file information of metadata records into standard task messages and places them into a Redis task pool to achieve unified centralized scheduling and full lifecycle status management of migration tasks. By leveraging the memory storage and atomic operation characteristics of Redis, it supports concurrent push from multiple task publishers and concurrent pull from multiple task processors, ensuring the efficiency of task flow and consistency of status under high load scenarios.

[0086] Step 206: Obtain the task message to be migrated from the Redis task pool, and set the status of the task message to be migrated to running status in the intermediate database partition table.

[0087] In this embodiment of the invention, an idle task handler obtains a task message in its initial state for migration through atomic operations in Redis, and updates the task's status to "Running" in the intermediate database shard table. To ensure state consistency, this operation is typically implemented using Redis transactions, for example, by simultaneously performing atomic operations to obtain the task and update its status, thus preventing multiple contractors from obtaining the same task at the same time.

[0088] To prevent tasks from remaining stuck in the "running" state (i.e., zombie tasks) for extended periods due to node failures, network fluctuations, or other reasons, a dedicated task reclaimer component is deployed within the cluster, specifically responsible for detecting and recovering abnormal tasks. Furthermore, the duration for which task messages awaiting migration in the Redis task pool remain in the running state is statistically analyzed; the Redis task pool is polled at preset time intervals, and migration messages are selected for reclamation based on their duration and a preset timeout threshold; the location information of the reclaimed migration messages is recorded, and their status is updated to the initial state.

[0089] Specifically, for each task message to be migrated, once its status is set to "Running," a startup timestamp is recorded in Redis. The task handler periodically updates the heartbeat timestamp during execution, indicating that the task is still running normally. The task reclaimer polls all tasks in Redis with a "Running" status at preset time intervals (e.g., every 30 seconds). For each task, the reclaimer calculates the difference between the current time and the startup timestamp to determine the duration the task has been running. The task reclaimer compares the calculated duration with a preset timeout threshold. If the duration exceeds the threshold and the task's heartbeat timestamp has not been updated for an extended period (or there is no heartbeat mechanism), the task is considered an abnormal task, i.e., a zombie task.

[0090] It is worth noting that the timeout threshold can be dynamically configured based on file size or network conditions (e.g., 5 minutes for small files and 30 minutes for large files).

[0091] After determining that a task is abnormal, the task reclaimer needs to read the task's execution progress information (i.e., position information) and task identifier from Redis. Position information is periodically written to Redis by the original task handler during execution (e.g., recording the number of bytes transferred or the number of completed shards). After obtaining this position information, the task reclaimer temporarily stores it or saves it in association with the task message so that subsequent task handlers can continue execution from the breakpoint. The task reclaimer uses atomic operations to forcibly reset the task's status from "running" to "initial" in the intermediate database shard table, while clearing the task's handler identifier and start timestamp, but retaining the position information in the task context. The reset task will re-enter the Redis task pool's pending queue, waiting for the task contractor to redistribute it. When a new task handler acquires the task, it reads the task identifier and position information, and calls the cloud storage's breakpoint resumption interface based on this information to continue execution from the already transmitted position, avoiding starting the transmission from the beginning.

[0092] This invention ensures strong consistency between task acquisition and state updates through atomic operations, avoiding duplicate task processing caused by multi-node concurrent conflicts; the built-in task reclaimer automatically detects and reclaims zombie tasks, eliminating task deadlocks caused by node failures, and combines position information retention to achieve breakpoint resume, significantly improving reliability and execution efficiency in large-scale migration scenarios.

[0093] Step 207: Based on the task message to be migrated, obtain the content verification value of the file to be migrated, the path of the file to be migrated, the name of the file to be migrated, the last modification time of the file to be migrated, and the existence status of the object to be migrated from the intermediate database sub-table. In addition, based on the file information of the target cloud storage platform, obtain the key content verification value, key file path, key name, key file last modification event, and key object existence status from the intermediate database sub-table.

[0094] Specifically, the metadata features corresponding to the task message to be migrated are obtained from the intermediate database sub-tables through the cluster identifier and file identifier. The metadata features corresponding to the task message to be migrated include, but are not limited to, the content verification value of the file to be migrated, the path of the file to be migrated, the name of the file to be migrated, the last modification time of the file to be migrated, and the existence status of the object to be migrated.

[0095] In this embodiment of the invention, the file information of the target cloud storage platform is the migrated file information. The metadata features corresponding to the file information of the target cloud storage platform are obtained from the intermediate database sub-table through the migrated file information. The metadata features corresponding to the file information of the target cloud storage platform include, but are not limited to, key content verification values, key file paths, key file names, key file last modification events, and key object existence status.

[0096] Step 208: According to the preset migration mode triggering conditions, perform multi-dimensional feature comparison between the verification value of the content of the file to be migrated, the path of the file to be migrated, the name of the file to be migrated, the last modification time of the file to be migrated, and the existence status of the object to be migrated, and the verification value of the key content, the path of the key file, the name of the key file, the last modification time of the key file, and the existence status of the key object, and match the corresponding target migration mode.

[0097] In this embodiment of the invention, the object existence status is whether the file exists; the content check value is used to determine whether the file content has changed; the file path is the complete object key; the file name is the last-level name parsed from the path; and the last modification time is used to help determine whether the file has been updated.

[0098] In this embodiment of the invention, the migration modes include, but are not limited to, upload mode, update mode, rename mode, move mode, delete mode, rename and move mode, skip mode, and delete and upload mode.

[0099] Specifically, if the existence status of the object to be migrated is "existent" and the existence status of the key object is "non-existent", the triggering conditions for the upload mode are met, and the target migration mode is determined to be the upload mode.

[0100] Specifically, the consistency of the filename to be migrated with the critical filename, the consistency of the file path to be migrated with the critical file path, and the consistency of the content verification value of the file to be migrated with the critical content verification value are compared. If the comparison results show that there are files with the same name and path on the source and target ends, but the content verification values ​​are inconsistent, then the triggering conditions for the update mode are met, and the target migration mode is determined to be the update mode.

[0101] Specifically, the consistency of the verification values ​​of the file content to be migrated with the verification values ​​of the key content, the consistency of the parent directory path in the path of the file to be migrated with the parent directory path in the path of the key file, and the consistency of the file name to be migrated with the file name of the key file are compared. If the comparison results show that the verification values ​​of the content are consistent and the parent directory paths are the same but only the file names are different, then the triggering conditions of the renaming mode are met, and the target migration mode is determined to be the renaming mode.

[0102] Specifically, the consistency of the verification values ​​of the files to be migrated with the verification values ​​of the critical content, the consistency of the parent directory paths in the paths of the files to be migrated with the parent directory paths in the paths of the critical files, and the consistency of the filenames to be migrated with the filenames of the critical files are compared. If the comparison results show that the verification values ​​of the content are consistent, the filenames are the same but the parent directory paths are different, then the triggering conditions for the migration mode are met, and the target migration mode is determined to be the migration mode.

[0103] Specifically, if the existence status of the object to be migrated is non-existent, and the existence status of the key object is present, then the triggering conditions for the deletion mode are met, and the target migration mode is determined to be the deletion mode.

[0104] Specifically, the consistency between the verification values ​​of the file content to be migrated and the verification values ​​of the key content, as well as the consistency between the file name to be migrated and the file name of the key, are compared. If the comparison results show that there are objects with the same name on the source and target cloud storage platforms but with inconsistent file content verification values, then the triggering conditions for the delete-and-upload mode are met, and the target migration mode is determined to be the delete-and-upload mode.

[0105] Specifically, the consistency of the verification values ​​of the files to be migrated with the verification values ​​of the critical content, the consistency of the parent directory paths in the paths of the files to be migrated with the parent directory paths in the paths of the critical files, and the consistency of the filenames to be migrated with the filenames of the critical files are compared. If the comparison results show that the verification values ​​of the files on the source and target cloud storage platforms are consistent, but the filenames and parent directory paths have changed, then the triggering conditions for the rename and move mode are met, and the target migration mode is determined to be the rename and move mode.

[0106] Specifically, the consistency of the verification values ​​of the file content to be migrated with the verification values ​​of the key content, the consistency of the file path to be migrated with the path of the key file, and the consistency of the file name to be migrated with the file name of the key file are compared. If the comparison results show that the path, file name, and content verification values ​​of the source and target cloud storage platforms are completely consistent, that is, the triggering conditions of the no-operation mode are met, and the target migration mode is determined to be the no-operation mode.

[0107] This invention uses multi-dimensional metadata feature comparison and refined pattern matching to replace physical data transmission with internal operations when only metadata changes such as renaming or moving occur in a file, significantly saving cross-cloud bandwidth resources. At the same time, based on existence and content consistency judgment, it realizes differentiated processing of upload, update, and deletion, ensuring that the data on the source and target cloud storage platforms is ultimately consistent, avoiding invalid transmission and data redundancy.

[0108] Step 209: According to the target migration mode, migrate the task message to be migrated to the target cloud storage platform, and identify the task message to be migrated as the file information of the target cloud storage platform in the intermediate database sub-table, and update the status to success.

[0109] In this embodiment of the invention, if an upload mode is matched, the task handler calls the target cloud storage platform's storage interface to perform a standard data stream upload operation, writing the entity file to the target cloud storage platform.

[0110] If an update pattern is matched, the task handler performs an overwrite upload operation, writing the latest data entity from the source to the target cloud storage platform to update the content.

[0111] If a renaming pattern is matched, the task handler directly calls the metadata operation interface of the target cloud storage platform's storage system to complete the file name change within the target cloud storage platform, without needing to download or upload physical data throughout the process.

[0112] If a move mode is matched, the task handler calls the target cloud storage platform's metadata operation interface to move the file's logical location within the target cloud storage platform, without needing to transmit physical data.

[0113] If a deletion pattern is matched, the task handler calls the target cloud storage platform interface to delete the corresponding redundant files to maintain data consistency between the two ends.

[0114] If a delete and upload pattern is matched, the task handler will first delete the conflicting objects on the target cloud storage platform, and then perform the upload operation on the source file to resolve the content conflict issue.

[0115] If a rename and move pattern is matched, the task handler combines and calls the target cloud storage platform's metadata operation interface to synchronously change the file name and path without transmitting physical data across the network.

[0116] If a no-operation mode is matched, the task handler determines that the file is already synchronized and skips the current task.

[0117] In this embodiment of the invention, for a successfully migrated task, the message of the task to be migrated is identified as the file information of the target cloud storage platform in the intermediate database sub-table. That is, the migrated file information is updated so that the metadata features can be compared during the next round of data migration.

[0118] After the above operations are completed, regardless of success or failure, the task handler will update the status of the corresponding task in the intermediate database shard to a success or failure status through Redis atomic operations, thus completing the closed-loop processing of the migration task.

[0119] Furthermore, if a task fails, the status of the failed task is updated to the initial status in the intermediate database partition table so that it can be retrieved during the next round of data migration.

[0120] This invention significantly saves cross-cloud bandwidth resources by refining the execution of various migration modes and replacing physical transmission with internal operations when only the metadata of the file changes. At the same time, it ensures the idempotency and traceability of migration tasks through accurate status updates.

[0121] This invention fundamentally solves the resource conflict between IO-intensive and bandwidth-intensive tasks by decoupling metadata collection from entity migration and adopting an asymmetric cluster architecture of "one collection, multiple migrations." Users can independently expand and migrate the cluster as needed to improve transmission speed and maximize resource efficiency. Simultaneously, the multi-migration cluster architecture supports binding physically isolated network egress points to different services, ensuring fine-grained allocation of bandwidth resources and transmission stability for high-priority services. Based on this, the task information management module provides full lifecycle visual monitoring and precise retry capabilities. Operations personnel can reset specific failed tasks individually without full migration, significantly reducing operational costs. Regarding transmission efficiency, combined with eight fine-grained migration modes, internal operations replace entity data transmission when only metadata changes such as file renaming or moving occur, reducing invalid bandwidth consumption by over 90%. Combined with adaptive dynamic concurrency control at the collection and migration ends, this ensures maximized network utilization without causing congestion. Furthermore, the distributed scheduling model based on Redis atomic operations achieves automatic detection, recycling, and breakpoint resumption of abnormal tasks through a four-role collaboration mechanism, completely solving the problem of task deadlock and loss in a distributed environment and providing high reliability assurance for large-scale data migration.

[0122] It is worth noting that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. The user information in the embodiments of this application was obtained through legal and compliant means, and the acquisition, storage, use, and processing of user information have been authorized and agreed upon by the client.

[0123] It is worth noting that the information collected in this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation portals are provided for users to choose to authorize or refuse.

[0124] It is worth noting that the technical solution provided in this application provides users with a corresponding operation entry point, allowing users to choose to agree to or reject the automated decision-making result; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0125] The technical solution of the cross-cloud storage platform data migration method based on distributed clusters provided in this invention involves using a single distributed file acquisition cluster to perform concurrent data scanning on the source cloud storage platform, writing the scanned metadata records into an intermediate database partition table; according to preset routing rules, multiple distributed file migration clusters encapsulate the file information of the metadata records in the intermediate database partition table into a migration task message, and place the migration task message into a preset Redis task pool; based on the metadata feature comparison between the migration task message in the Redis task pool and the file information of the target cloud storage platform, the corresponding target migration is matched. The system migrates data to the target cloud storage platform according to the target migration mode. Through multi-dimensional feature comparison of metadata and intelligent matching of various refined migration modes, metadata operations are prioritized to replace physical data stream transmission, which greatly saves bandwidth resources and ensures data consistency. The asymmetric cluster architecture of "one collection, multiple migrations" is adopted to decouple collection and migration, supports independent horizontal expansion of migration capabilities, and efficiently supports concurrent processing of petabyte-level massive files. Based on the closed-loop scheduling mechanism of Redis atomic task state pool, abnormal tasks are automatically recycled and breakpoint resumed, which significantly improves the reliability and fault tolerance of the system in large-scale migration scenarios.

[0126] Figure 3 This is a schematic diagram of a data migration device for cross-cloud storage platforms based on a distributed cluster, provided in an embodiment of the present invention. This device is used to execute the aforementioned data migration method for cross-cloud storage platforms based on a distributed cluster. Figure 3 As shown, the device includes: a concurrent data scanning unit 11, a task message generation unit 12 to be migrated, and a data migration unit 13.

[0127] The concurrent data scanning unit 11 is used to perform concurrent data scanning on the source cloud storage platform through a single distributed file collection cluster, and write the scanned metadata records into the intermediate database sub-table.

[0128] The task message generation unit 12 is used to encapsulate the file information of the metadata records in the intermediate database sub-tables into a task message according to the preset routing rules through multiple distributed file migration clusters, and put the task message into the preset Redis task pool.

[0129] The data migration unit 13 is used to compare the metadata features of the task messages to be migrated in the Redis task pool and the file information of the target cloud storage platform, match the corresponding target migration mode, and migrate the data to the target cloud storage platform according to the target migration mode.

[0130] In this embodiment of the invention, the concurrent data scanning unit 11 is specifically used to call the application programming interface of the source cloud storage platform, and concurrently collect metadata records through multiple collection nodes in a single distributed file collection cluster; calculate the metadata records according to the preset database sharding algorithm to obtain the sharding calculation result; and write the metadata records into the intermediate database sharding table corresponding to the sharding calculation result.

[0131] In this embodiment of the invention, the device further includes a monitoring unit 14.

[0132] The monitoring unit 14 is used to reduce the number of concurrent collection nodes if no response message from the application programming interface of the source cloud storage platform is detected within a preset response delay threshold.

[0133] In this embodiment of the invention, the task message generation unit 12 is specifically used to obtain the metadata records of the initial state corresponding to each distributed file migration cluster in batches from the intermediate database sub-tables according to the routing rules; encapsulate the file information of the metadata records of the initial state to generate the task message to be migrated, and place the task message to be migrated into a preset Redis task pool. The file information includes cluster identifier and file identifier.

[0134] In this embodiment of the invention, the data migration unit 13 is specifically used to obtain the task message to be migrated from the Redis task pool and set the status of the task message to be migrated to the running state in the intermediate database sub-table; according to the task message to be migrated, obtain the content verification value of the file to be migrated, the path of the file to be migrated, the file name of the file to be migrated, the last modification time of the file to be migrated, and the existence status of the object to be migrated from the intermediate database sub-table; and according to the file information of the target cloud storage platform, obtain the key content verification value, key file path, key file name, key file last modification event, and key object existence status from the intermediate database sub-table; according to the preset migration mode triggering conditions, perform multi-dimensional feature comparison between the content verification value of the file to be migrated, the path of the file to be migrated, the file name of the file to be migrated, the last modification time of the file to be migrated, and the existence status of the object to be migrated and the key content verification value, key file path, key file name, key file last modification time, and key object existence status to match the corresponding target migration mode; according to the target migration mode, migrate the task message to be migrated to the target cloud storage platform, and determine the task message to be migrated as the file information of the target cloud storage platform in the intermediate database sub-table, and update the status to the success state.

[0135] In this embodiment of the invention, the device further includes: a statistics unit 15, a screening unit 16, and a recycling unit 17.

[0136] The statistics unit 15 is used to count the duration for which the task messages to be migrated in the Redis task pool are in the running state.

[0137] The filtering unit 16 is used to poll the Redis task pool at preset time intervals and filter out recycling migration messages based on the duration and preset timeout threshold.

[0138] The recycling unit 17 is used to record the location information and task identifier of the recycling migration message, and update the status of the recycling migration message to the initial status in the intermediate database sub-table.

[0139] In this embodiment of the invention, a single distributed file acquisition cluster performs concurrent data scanning on the source cloud storage platform, writing the scanned metadata records into an intermediate database partition table. Following preset routing rules, multiple distributed file migration clusters encapsulate the file information of the metadata records in the intermediate database partition table into migration task messages, which are then placed into a preset Redis task pool. Metadata features are compared between the migration task messages in the Redis task pool and the file information on the target cloud storage platform to match the corresponding target migration mode. Data migration is then performed to the target cloud storage platform according to the target migration mode. Through multi-dimensional metadata feature comparison and intelligent matching of various refined migration modes, metadata operations are prioritized over physical data stream transmission, significantly saving bandwidth resources and ensuring data consistency. An asymmetric cluster architecture of "one acquisition, multiple migrations" decouples acquisition and migration, supports independent horizontal scaling of migration capabilities, and efficiently supports concurrent processing of PB-level massive files. A closed-loop scheduling mechanism based on the Redis atomic task state pool enables automatic recycling of abnormal tasks and breakpoint resumption, significantly improving the system's reliability and fault tolerance in large-scale migration scenarios.

[0140] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer device, specifically, a computer device can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0141] This invention provides a computer device including a memory and a processor. The memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, they implement the steps of the above-described embodiment of the cross-cloud storage platform data migration method based on a distributed cluster. For a detailed description, please refer to the above-described embodiment of the cross-cloud storage platform data migration method based on a distributed cluster.

[0142] The following is for reference. Figure 4It shows a schematic diagram of the structure of a computer device 600 suitable for implementing the embodiments of this application.

[0143] like Figure 4 As shown, the computer device 600 includes a central processing unit (CPU) 601, which can perform various appropriate tasks and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer device 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0144] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal feedback (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed in storage section 608 as needed.

[0145] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611.

[0146] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0147] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0148] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0152] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0153] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0154] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0156] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0157] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data migration method for cross-cloud storage platforms based on distributed clusters, characterized in that, The method includes: A single distributed file collection cluster performs concurrent data scanning on the source cloud storage platform and writes the scanned metadata records into a partitioned table in an intermediate database. According to the preset routing rules, the file information of the metadata records in the intermediate database sub-tables is encapsulated into a task message to be migrated through multiple distributed file migration clusters, and the task message to be migrated is placed into a preset Redis task pool. The metadata features of the task messages to be migrated in the Redis task pool and the file information of the target cloud storage platform are compared to match the corresponding target migration mode, and the data is migrated to the target cloud storage platform according to the target migration mode.

2. The data migration method for cross-cloud storage platforms based on distributed clusters according to claim 1, characterized in that, The process of concurrently scanning the source cloud storage platform using a single distributed file collection cluster and writing the scanned metadata records into a partitioned table in an intermediate database includes: The application programming interface of the source cloud storage platform is invoked to concurrently collect metadata records through multiple collection nodes within the single distributed file collection cluster. The metadata records are calculated according to a preset database sharding and table partitioning algorithm to obtain the table partitioning calculation results; The metadata record is written into the intermediate database sub-table corresponding to the sub-table calculation result.

3. The data migration method for cross-cloud storage platforms based on distributed clusters according to claim 2, characterized in that, The method further includes: If no response message from the application programming interface of the source cloud storage platform is detected within the preset response delay threshold, the number of concurrent acquisition nodes will be reduced.

4. The data migration method for cross-cloud storage platforms based on distributed clusters according to claim 1, characterized in that, The process of encapsulating file information of metadata records in the intermediate database tables into migration task messages according to preset routing rules through multiple distributed file migration clusters, and placing the migration task messages into a preset Redis task pool, includes: According to the routing rules, the metadata records of the initial state corresponding to each distributed file migration cluster are obtained in batches from the intermediate database tables; The file information of the metadata record of the initial state is encapsulated to generate the task message to be migrated, and the task message to be migrated is placed into a preset Redis task pool. The file information includes cluster identifier and file identifier.

5. The data migration method for cross-cloud storage platforms based on distributed clusters according to claim 1, characterized in that, The step of comparing metadata features of the task messages to be migrated in the Redis task pool and the file information of the target cloud storage platform to match the corresponding target migration mode, and then migrating the data to the target cloud storage platform according to the target migration mode, includes: Obtain the task message to be migrated from the Redis task pool, and set the status of the task message to be migrated to the running status in the intermediate database table; Based on the task message to be migrated, the verification value of the file content to be migrated, the path of the file to be migrated, the name of the file to be migrated, the last modification time of the file to be migrated, and the existence status of the object to be migrated are obtained from the intermediate database sub-table. Furthermore, based on the file information of the target cloud storage platform, the verification value of key content, the path of key file, the name of key file, the last modification event of key file, and the existence status of key object are obtained from the intermediate database sub-table. According to the preset migration mode triggering conditions, the verification value of the content of the file to be migrated, the path of the file to be migrated, the name of the file to be migrated, the last modification time of the file to be migrated, and the existence status of the object to be migrated are compared with the key content verification value, key file path, key file name, key file last modification time, and key object existence status to match the corresponding target migration mode. According to the target migration mode, the task message to be migrated is migrated to the target cloud storage platform, and the task message to be migrated is identified as the file information of the target cloud storage platform in the intermediate database sub-table, and the status is updated to success.

6. The data migration method for cross-cloud storage platforms based on distributed clusters according to claim 1, characterized in that, The method further includes: The duration for which the tasks to be migrated in the Redis task pool remain in the running state is recorded. The Redis task pool is polled at preset time intervals, and migration messages are selected for recycling based on the duration and preset timeout threshold. Record the location information and task identifier of the recovered migration message, and update the status of the recovered migration message to the initial state in the intermediate database sub-table.

7. A data migration device for cross-cloud storage platforms based on a distributed cluster, characterized in that, The device includes: The concurrent data scanning unit is used to perform concurrent data scanning on the source cloud storage platform through a single distributed file collection cluster, and write the scanned metadata records into the intermediate database partition table. The task message generation unit is used to encapsulate the file information of the metadata records in the intermediate database sub-table into a task message according to the preset routing rules through multiple distributed file migration clusters, and put the task message into the preset Redis task pool. The data migration unit is used to compare the metadata features of the task messages to be migrated in the Redis task pool and the file information of the target cloud storage platform, match the corresponding target migration mode, and migrate the data to the target cloud storage platform according to the target migration mode.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the cross-cloud storage platform data migration method based on a distributed cluster as described in any one of claims 1 to 6.

9. A computer device comprising a memory and a processor, the memory for storing information including program instructions, and the processor for controlling the execution of the program instructions, characterized in that, When the program instructions are loaded and executed by the processor, they implement the cross-cloud storage platform data migration method based on a distributed cluster as described in any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the cross-cloud storage platform data migration method based on a distributed cluster as described in any one of claims 1 to 6.

Images