Object storage thermal migration method and device

Abstract

The invention relates to the field of cloud storage, and particularly provides an object storage live migration method and device, and the method comprises the following steps: S1, an event-driven automatic triggering mechanism; s2, exporting a multi-dimensional parallel full-quantity list; s3, an intelligent timestamp comparison migration strategy; and S4, dynamic redirection downloading service implementation. Compared with the prior art, the method has the advantages that the problems of low efficiency, difficulty in guaranteeing data consistency, business interruption risk and the like in traditional object storage live migration can be solved, and the method is of great significance in promoting data management efficiency and business agility in a cloud computing environment.

Description

Technical Field

[0001] The present invention relates to the field of cloud storage, and specifically provides an object storage hot migration method and system. Background Art

[0002] As digital transformation accelerates, the management and migration of enterprise data assets have become a key link in the construction of information technology infrastructure. As a distributed storage solution, object storage has become the preferred choice for storing massive amounts of unstructured data due to its high scalability, low cost and flexible data management capabilities. However, with the rapid development of business and changing needs, enterprises may face challenges such as data center migration, cloud service provider replacement or storage resource optimization. In these cases, how to efficiently transfer data in object storage while ensuring business continuity, the so-called "hot migration", has become an urgent problem to be solved.

[0003] Traditional data migration methods often involve downtime maintenance windows, which is unacceptable for many online services that require 24 / 7 uninterrupted operation. Downtime not only affects the user experience, but may also lead to revenue loss and loss of customer trust. Therefore, developing technologies that can complete object storage migration without service interruption is critical to maintaining business continuity, optimizing resource utilization, and promoting business agility.

[0004] Most of the hot migration solutions on the market currently have problems such as low efficiency, complex migration process, high resource usage, or difficulty in ensuring data consistency. Especially when facing PB-level or even EB-level data scale, how to quickly locate and migrate incremental data while ensuring data consistency and integrity has become a major obstacle in technical implementation. In addition, efficient use of network bandwidth between source storage and target storage during the migration process, as well as fine management and monitoring of migration tasks, are also key factors in improving migration efficiency and success rate.

[0005] In view of this, developing an efficient, secure, and automated object storage hot migration system without affecting existing business operations has become an urgent task in the current technology field. Such a system needs to integrate advanced data processing technology, intelligent migration strategies, and highly scalable architecture design to meet the comprehensive requirements of modern enterprises for data migration speed, economy, and reliability. Summary of the invention

[0006] The present invention aims at the above-mentioned deficiencies of the prior art and provides a highly practical object storage hot migration method.

[0007] A further technical task of the present invention is to provide an object storage hot migration device that is reasonably designed, safe and applicable.

[0008] The technical solution adopted by the present invention to solve its technical problem is:

[0009] A method for hot migration of object storage includes the following steps:

[0010] S1, event-driven automatic triggering mechanism;

[0011] S2, multi-dimensional parallel full list export;

[0012] S3, intelligent timestamp comparison migration strategy;

[0013] S4. Implementation of dynamic redirection download service.

[0014] Furthermore, in step S1, it includes:

[0015] S1-1. Configure event monitoring service;

[0016] S1-2, triggering the processor of the migration system.

[0017] Furthermore, in step S1-1, it includes:

[0018] S1-1.1. Log in to the management console of the source object storage service provider and select the bucket to be migrated.

[0019] S1-1.2. In the bucket properties or security settings, configure the event notification service, add a new rule, and specify the event type and target object.

[0020] S1-1.3. Set the callback URL to the API gateway address of the migration system and ensure that the address has read and write permissions to the source bucket.

[0021] Furthermore, in step S1-2, it includes:

[0022] S1-2.1. Deploy the migration program on the migration node. The monitoring module is responsible for monitoring event notifications of the source bucket.

[0023] S1-2.2. After the monitoring module monitors the event notification of the source bucket, it converts the received event into a specific migration task and merges it into the message queue. The scheduling module is responsible for obtaining the migration task from the message queue and using a suitable scheduling algorithm to assign the task to the migration execution module. The migration module performs specific actions.

[0024] Furthermore, in step S2, it includes:

[0025] S2-1. Before the formal migration, you need to analyze the structure and distribution of files in the source bucket, and obtain basic statistical information of the source bucket by calling the management API or SDK of the source object storage to prepare for the next task segmentation.

[0026] S2-2, based on the statistical information of the source storage bucket in the previous step, divide the tasks according to the object name prefix and the time range, and convert the tasks into multiple task lists, each list containing query instructions for a part of the objects, and the query instructions including the starting position of the file to be queried;

[0027] S2-3. Execute the parallel export file list task, start multiple modules for listing file lists, each module subscribes to a specific task queue, and after receiving the task, the module for listing file lists uses ListObjects or similar APIs of the source object storage to batch query and collect the object metadata within the specified range, and aggregate the collected metadata to the central database or file to prepare for the next migration.

[0028] Furthermore, in step S3, it includes:

[0029] S3-1, obtain target storage object information;

[0030] (1) Intelligent filtering and deduplication: Before migration, check the hash table of the migrated files and calculate the hash value of the file to be migrated. If the calculated hash value is in the above hash table, skip it, otherwise execute the next step;

[0031] (2) For each object to be migrated, first check whether there is an object with the same name in the target bucket. If it does not exist in the target bucket, perform the migration. If it does exist, it is necessary to continue to compare the timestamps or version IDs of the source and target objects. Migration is performed only when the upload timestamp of the file in the source bucket is later than the timestamp or version ID of the target object and is greater than the version ID of the target bucket. Otherwise, skip it.

[0032] S3-2, execute migration;

[0033] (1) The migration module starts multiple threads to perform the migration action. First, it traverses the list of files to be migrated, downloads the files to be migrated from the source bucket one by one, and uploads the files to the target bucket.

[0034] (2) After the file is successfully uploaded to the target bucket, the migration log is recorded;

[0035] Specifically, it includes the name of the successfully migrated file, the timestamp of the source bucket file, the timestamp of the target bucket file, and the hash value for subsequent verification;

[0036] If the file upload fails, it will be recorded in the failure log file and a second round of migration will be performed later.

[0037] Furthermore, in step S4, it includes:

[0038] S4-1. Build an intelligent routing layer and deploy a load balancing module on the gateway node as the entry point for user requests to receive and forward user requests to the backend object storage service component.

[0039] S4-2, implement dynamic download logic;

[0040] S4-3, monitoring and optimization, adopt full-link monitoring, adjust the number of threads and queues according to monitoring indicators, and continuously optimize migration efficiency and user experience.

[0041] Furthermore, in step S4-2, it includes:

[0042] S4-2.1. After receiving the request, the backend object storage service determines the request type. If it is a download request, it checks whether the file to be downloaded exists in the target bucket. If the file to be downloaded is not found in the target bucket, it immediately initiates a download request to the source bucket.

[0043] S4-2.2. When downloading files from the source bucket, the downloaded file stream is split into two, one is returned to the user, and the other is uploaded to the target bucket asynchronously.

[0044] An object storage hot migration device includes: at least one memory and at least one processor;

[0045] The at least one memory is used to store a machine-readable program;

[0046] The at least one processor is used to call the machine-readable program to execute an object storage hot migration method.

[0047] Compared with the prior art, the object storage hot migration method and system of the present invention have the following outstanding beneficial effects:

[0048] The present invention significantly enhances the efficiency, reliability and business continuity of data migration through various designed implementation steps, and its beneficial effects are specifically reflected in the following aspects:

[0049] (1) Significantly improve migration efficiency: The event-driven automatic trigger mechanism ensures real-time response to data changes and reduces manual intervention, while the multi-dimensional parallel full list export technology greatly shortens the preparatory work time by processing the generation of a large number of data lists in parallel. Combined with intelligent timestamp comparison and differentiated migration strategies, only necessary data updates are migrated, further saving migration time and resources.

[0050] (2) Ensure data consistency and integrity: Through intelligent timestamp comparison, the system can accurately identify and migrate only the data that is newer on the source side or missing on the target side, avoiding data redundancy and version conflicts, and effectively ensuring data consistency and integrity.

[0051] (3) Seamless business continuity guarantee: The dynamic redirection download service design allows user requests to be processed transparently during the migration process. Regardless of whether the data is located in the source storage bucket or is being migrated to the target storage bucket, service continuity and user experience can be ensured, avoiding business interruptions caused by migration.

[0052] (4) Resource optimization and cost saving: Through sophisticated parallel processing and intelligent filtering mechanisms, the present invention effectively utilizes computing resources and reduces storage and network resource consumption during data migration, thereby helping users save costs.

[0053] (5) High flexibility and scalability: The system design fully considers the needs of different application scenarios, supports parallel task division strategies in multiple dimensions, and dynamically adjusts strategies based on monitoring data during the migration process, making this method flexibly adaptable to object storage migration projects of different sizes and types.

[0054] (6) Enhanced monitoring and optimization capabilities: The full-link monitoring system and continuous optimization mechanism based on data feedback ensure the controllability and predictability of the migration process, and can promptly detect and resolve potential problems, thereby continuously improving migration efficiency and user experience.

[0055] In summary, the present invention solves the problems of low efficiency, difficulty in ensuring data consistency, and risk of business interruption in traditional object storage hot migration through a series of innovative technologies and strategies, and provides users with an efficient, stable, and easy-to-use object storage migration solution, which is of great significance for promoting data management efficiency and business agility in cloud computing environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0057] Attached Figure 1 The present invention is a flowchart of an object storage hot migration method. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] A best embodiment is given below:

[0060] like Figure 1 As shown, a method for hot migration of object storage in this embodiment has the following steps:

[0061] S1, event-driven automatic triggering mechanism;

[0062] During the data migration process, in order to ensure that any additions, deletions, and changes to files in the source bucket can be synchronized to the target bucket in real time to avoid data loss or inconsistency, an efficient method based on event notification is proposed.

[0063] Here are the specific steps:

[0064] S1-1. Configure event monitoring service;

[0065] include:

[0066] S1-1.1. Log in to the management console of the source object storage service provider and select the bucket to be migrated.

[0067] S1-1.2. In the bucket properties or security settings, configure the event notification service, add a new rule, specify the event type (such as PutObject, DelObject) and the target object (such as the object with the prefix and suffix xxx).

[0068] S1-1.3. Set the callback URL to the API gateway address of the migration system and ensure that the address has read and write permissions to the source bucket.

[0069] S1-2. Configure event monitoring service;

[0070] include:

[0071] S1-2.1. Deploy the migration program on the migration node. The monitoring module is responsible for monitoring the event notifications of the source bucket.

[0072] S1-2.2, after the monitoring module monitors the event notification of the source bucket, it converts the received event into a specific migration task and merges it into the message queue. The scheduling module is responsible for obtaining the migration task from the message queue and using a suitable scheduling algorithm to assign the task to the migration execution module, which performs specific actions;

[0073] If an event of uploading a file from the source bucket is received, the file will be downloaded from the source bucket and uploaded to the target bucket; if an event of deleting a file from the source bucket is received, the corresponding file in the target bucket will be deleted.

[0074] S2. Multi-dimensional parallel full inventory export - preparation for migration;

[0075] include:

[0076] S2-1. Before the formal migration, you need to analyze the structure and distribution of the files in the source bucket. By calling the management API or SDK of the source object storage, you can obtain basic statistical information of the source bucket, including the total number of objects, size distribution, etc., to prepare for the next task segmentation.

[0077] S2-2. Based on the statistical information of the source storage bucket in the previous step, tasks can be divided according to object name prefix, time range or other logical dimensions. The tasks are converted into multiple task lists, each of which contains query instructions for a part of objects, which mainly include the starting position of the file to be queried.

[0078] S2-3. Execute the parallel export file list task:

[0079] Start multiple modules that list files, and each module subscribes to a specific task queue. After receiving the task, the module that lists files uses the ListObjects or similar API of the source object storage to batch query and collect the metadata of objects within the specified range. The collected metadata is aggregated into a central database or file to prepare for the next migration.

[0080] S3, intelligent timestamp comparison migration strategy;

[0081] include:

[0082] S3-1. Get the target storage object information:

[0083] (1) Intelligent filtering and deduplication: To avoid repeated migration of existing files or invalid migration due to false positives, the hash table of migrated files is checked before migration, and the hash value of the file to be migrated is calculated. If the calculated hash value is in the above hash table, it is skipped; otherwise, the next step is executed.

[0084] (2) For each object to be migrated, first check whether there is an object with the same name in the target bucket. If it does not exist in the target bucket, migration can be performed; if it does exist, it is necessary to continue to compare the timestamps (or version IDs) of the source and target objects. Migration is performed only when the upload timestamp of the file in the source bucket is later than the timestamp or version ID of the target object and is greater than the version ID of the target bucket. Otherwise, it is skipped.

[0085] S3-2, execute migration;

[0086] (1) In order to improve the migration efficiency, the migration module starts multi-threading to perform the migration action. First, it traverses the list of files to be migrated, downloads the files to be migrated from the source bucket one by one, and uploads the files to the target bucket.

[0087] (2) After the file is successfully uploaded to the target bucket, the migration log is recorded. Specifically, it includes the name of the successfully migrated file, the timestamp of the source bucket file, the timestamp of the target bucket file, the hash value and other information for subsequent verification. If the file upload fails, it is recorded in the failure log file and the second round of migration is performed later.

[0088] S4, dynamic redirection download service implementation;

[0089] include:

[0090] S4-1. Build an intelligent routing layer and deploy a load balancing module on the gateway node as the entry point for user requests to receive and forward user requests to the backend object storage service component.

[0091] S4-2. Implement dynamic download logic:

[0092] (1) After receiving the request, the backend object storage service determines the request type. If it is a download request, it checks whether the file to be downloaded exists in the target bucket. If the file to be downloaded is not found in the target bucket, it immediately initiates a download request to the source bucket.

[0093] (2) When downloading a file from the source bucket, the downloaded file stream is split into two, one is returned to the user, and the other is uploaded to the target bucket asynchronously, ensuring that the next time the file is requested, it is directly returned to the user from the target bucket.

[0094] S4-3, monitoring and optimization, using full-link monitoring to track key indicators such as migration progress, download request success rate, object synchronization delay, etc. Adjust parameters such as the number of threads and queues according to monitoring indicators to continuously optimize migration efficiency and user experience.

[0095] Based on the above method, an object storage hot migration device in this embodiment includes: at least one memory and at least one processor;

[0096] The at least one memory is used to store a machine-readable program;

[0097] The at least one processor is used to call the machine-readable program to execute an object storage hot migration method.

[0098] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0099] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for hot migration of object storage, characterized in that: The steps are as follows: S1, event-driven automatic triggering mechanism; S2, multi-dimensional parallel full list export; S3, intelligent timestamp comparison migration strategy; S4. Implementation of dynamic redirection download service.

2. The object storage hot migration method according to claim 1, characterized in that: In step S1, it includes: S1-1. Configure event monitoring service; S1-2, triggering the processor of the migration system.

3. The object storage hot migration method according to claim 2, characterized in that: In step S1-1, it includes: S1-1.

1. Log in to the management console of the source object storage service provider and select the bucket to be migrated. S1-1.

2. In the bucket properties or security settings, configure the event notification service, add a new rule, and specify the event type and target object. S1-1.

3. Set the callback URL to the API gateway address of the migration system and ensure that the address has read and write permissions to the source bucket.

4. The object storage hot migration method according to claim 3, characterized in that: In step S1-2, it includes: S1-2.

1. Deploy the migration program on the migration node. The monitoring module is responsible for monitoring event notifications of the source bucket. S1-2.

2. After the monitoring module monitors the event notification of the source bucket, it converts the received event into a specific migration task and merges it into the message queue. The scheduling module is responsible for obtaining the migration task from the message queue and using a suitable scheduling algorithm to assign the task to the migration execution module. The migration module performs specific actions.

5. The object storage hot migration method according to claim 4, characterized in that: In step S2, it includes: S2-1. Before the formal migration, you need to analyze the structure and distribution of files in the source bucket, and obtain basic statistical information of the source bucket by calling the management API or SDK of the source object storage to prepare for the next task segmentation. S2-2, based on the statistical information of the source storage bucket in the previous step, divide the tasks according to the object name prefix and the time range, and convert the tasks into multiple task lists, each list containing query instructions for a part of the objects, and the query instructions including the starting position of the file to be queried; S2-3. Execute the parallel export file list task, start multiple modules for listing file lists, each module subscribes to a specific task queue, and after receiving the task, the module for listing file lists uses ListObjects or similar APIs of the source object storage to batch query and collect the object metadata within the specified range, and aggregate the collected metadata to the central database or file to prepare for the next migration.

6. The object storage hot migration method according to claim 5, characterized in that: In step S3, it includes: S3-1, obtain target storage object information; (1) Intelligent filtering and deduplication: Before migration, check the hash table of the migrated files and calculate the hash value of the file to be migrated. If the calculated hash value is in the above hash table, skip it, otherwise execute the next step; (2) For each object to be migrated, first check whether there is an object with the same name in the target bucket. If it does not exist in the target bucket, perform the migration. If it does exist, it is necessary to continue to compare the timestamps or version IDs of the source and target objects. Migration is performed only when the upload timestamp of the file in the source bucket is later than the timestamp or version ID of the target object and is greater than the version ID of the target bucket. Otherwise, skip it. S3-2, execute migration; (1) The migration module starts multiple threads to perform the migration action. First, it traverses the list of files to be migrated, downloads the files to be migrated from the source bucket one by one, and uploads the files to the target bucket. (2) After the file is successfully uploaded to the target bucket, the migration log is recorded; Specifically, it includes the name of the successfully migrated file, the timestamp of the source bucket file, the timestamp of the target bucket file, and the hash value for subsequent verification; If the file upload fails, it will be recorded in the failure log file and a second round of migration will be performed later.

7. The object storage hot migration method according to claim 6, characterized in that: In step S4, it includes: S4-1. Build an intelligent routing layer and deploy a load balancing module on the gateway node as the entry point for user requests to receive and forward user requests to the backend object storage service component. S4-2, implement dynamic download logic; S4-3, monitoring and optimization, adopt full-link monitoring, adjust the number of threads and queues according to monitoring indicators, and continuously optimize migration efficiency and user experience.

8. The object storage hot migration method according to claim 6, characterized in that: In step S4-2, it includes: S4-2.

1. After receiving the request, the backend object storage service determines the request type. If it is a download request, it checks whether the file to be downloaded exists in the target bucket. If the file to be downloaded is not found in the target bucket, it immediately initiates a download request to the source bucket. S4-2.

2. When downloading files from the source bucket, the downloaded file stream is split into two, one is returned to the user, and the other is uploaded to the target bucket asynchronously.

9. An object storage hot migration device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 8.

Images