Object storage index metadata live migration method and device

Abstract

The invention relates to the technical field of cloud computing, and particularly provides an object storage index metadata live migration method and device, and the method comprises the following steps: S1, deploying index metadata migration modules on a migration control node and a migration execution node; s2, performing addition and deletion operation on the object storage record index metadata to a log; s3, a bucket enumeration module and an object enumeration module enumerate full bucket names and object names in the current cluster; s4, a total migration module migrates total index metadata in the cluster; s5, the log analysis module generates an index metadata increment migration task; s6, migrating incremental index metadata in the cluster by an incremental migration module; s7, the index metadata in the distributed database and the index metadata in the object storage are consistent; s8, redirecting the index metadata storage of the object storage into the distributed database, and restarting the object storage service to enable the configuration item change to take effect; and S9, executing the last incremental migration index metadata. Compared with the prior art, the method can promote the healthy development of cloud computing and big data technologies.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and specifically provides a method and device for hot migration of object storage index metadata. Background Art

[0002] In today's era of accelerated digital transformation, big data has become a key factor driving corporate decision-making, innovation and efficiency. With the exponential growth of data volume, object storage technology has become the preferred data storage solution for many enterprises and cloud service providers due to its massive data storage capacity, high scalability and low cost. However, as the scale of data expands, the management and access efficiency of index metadata has become a key bottleneck restricting system performance. Index metadata contains the attribute information of all data objects in object storage, such as location, size, access rights, etc., and is the basis for data retrieval, analysis and migration.

[0003] Traditional index metadata management methods usually store metadata together with data objects in the object storage system. Although simple, it often exposes problems such as high response delay and poor scalability when facing large-scale concurrent requests, complex queries, and frequent data migration needs. Especially when data needs to be migrated to a new storage system or replicated across regions, the lack of an efficient and transparent migration mechanism can easily lead to risks such as data inconsistency, long migration windows, and service interruptions, which seriously affect business continuity and user experience.

[0004] In addition, as data protection regulations become increasingly stringent, the requirements for data real-time, integrity, and compliance are also increasing, further exacerbating the complexity of index metadata management. Most existing migration technologies rely on full backup and recovery, which not only takes up a lot of network bandwidth and storage resources, but also makes it difficult to ensure uninterrupted service operation during the migration process, bringing huge challenges to enterprise operation and maintenance. Summary of the invention

[0005] The present invention aims at the above-mentioned deficiencies of the prior art and provides a highly practical object storage index metadata hot migration method.

[0006] A further technical task of the present invention is to provide a reasonably designed, safe and applicable object storage index metadata hot migration device.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A method for hot migration of object storage index metadata comprises the following steps:

[0009] S1. Deploy the index metadata migration module on the migration control node and the migration execution node;

[0010] S2: Object storage records the addition and deletion operations of index metadata in the log;

[0011] S3, bucket enumeration module and object enumeration module enumerate all bucket names and object names in the current cluster;

[0012] S4, the full migration module migrates all index metadata in the cluster;

[0013] S5, the log analysis module generates an index metadata incremental migration task;

[0014] S6, the incremental migration module migrates the incremental index metadata in the cluster;

[0015] S7, repeating step S5 and step S6 until the index metadata in the distributed database is consistent with the index metadata in the object storage;

[0016] S8. Redirect the index metadata storage of the object storage to the distributed database, and restart the object storage service to make the configuration item changes take effect;

[0017] S9. Perform the last incremental migration of index metadata.

[0018] Further, in step S1, the migration control node and the migration execution node are two types of nodes in the index metadata hot migration system, and the migration control node refers to a node that retrieves the cluster bucket list and analyzes the index metadata operation log and issues the index metadata migration task;

[0019] The migration execution node refers to a node that runs the index metadata migration program;

[0020] The index metadata migration module is responsible for specific metadata enumeration, log analysis, and index metadata migration in the index metadata hot migration system. Specifically, it includes a bucket enumeration module, an object enumeration module, a log analysis module, a full migration module, and an incremental migration module.

[0021] The bucket enumeration module is responsible for requesting object storage to obtain a list of all bucket names in the current cluster;

[0022] The object enumeration module is responsible for requesting object storage and obtaining a list of all object names of a specified bucket name;

[0023] The log analysis module is responsible for analyzing the index metadata addition and deletion operation logs of the object storage and the migration failure logs of the migration module, and parsing the bucket names and object names therein;

[0024] The full migration module is responsible for migrating all index metadata in the current cluster to the distributed database;

[0025] The incremental migration module is responsible for migrating the index metadata updated after the last migration to the distributed database.

[0026] Furthermore, in step S2, before migrating the index metadata, the object storage is configured to print the addition and deletion operations of the index metadata into a log, which specifically includes the index metadata operation type, the index metadata operation time, the bucket name and the object name of the index metadata operation.

[0027] Further, in step S3, first, the bucket enumeration module requests the object storage to obtain the full bucket name list in the current cluster, then the object enumeration module carries the obtained bucket name to access the object storage to obtain the full object name list in the current cluster, and finally passes the full bucket name and object name list to the full migration module of the migration execution node. After the transfer is completed, a signal is sent to the full migration module to notify the full migration module to start full migration of index metadata, and the start time of the full migration is recorded to ensure that the log analysis module can accurately find the index metadata operation log;

[0028] When passing the bucket name and object name list, the bucket name and object name are hashed to obtain the address of the migration execution node, and the incremental migration tasks are evenly distributed to different migration execution nodes.

[0029] Further, in step S4, after receiving the signal from the migration control node, the full migration module reads the index metadata corresponding to the full bucket name list and the object name list in the cluster from the object storage and then writes it into the distributed database;

[0030] At the same time, the bucket name and object name of the failed migration and the failure time are recorded in the log of the full migration module, and the migration failure log is passed to the log analysis module.

[0031] Further, in step S5, the log analysis module analyzes the index metadata operation log of the object storage, parses the bucket name list and the object name list from the log after the start time of the last migration, and parses the bucket name list and the object name list of the last failed migration from the migration failure log of the migration execution node, sends the two parts of the bucket name list and the object name list to the incremental migration module of the migration execution node, and sends a signal to the migration execution node after the sending is completed;

[0032] When sending, the bucket name and object name are hashed to obtain the address of the migration execution node, and the incremental migration tasks are evenly distributed to different migration execution nodes.

[0033] Furthermore, in step S6, the start time of the incremental migration is recorded to ensure that the log analysis module accurately searches for the index metadata operation log. After the incremental migration module receives the signal from the migration control node, it reads the index metadata corresponding to the incremental bucket name list and the object name list in the cluster from the object storage and then writes them into the distributed database.

[0034] The bucket name and object name of the failed migration are recorded in the log of the incremental migration module, and the migration failure log is passed to the log analysis module.

[0035] Further, in step S7, step S5 and step S6 are repeatedly executed, and the execution time of the incremental migration decreases as the number of executions increases, and the bucket list and object list of the incremental migration decrease as the interval time of the incremental migration shortens, until the bucket list and object list screened by the log analysis module are empty, or the number of the bucket list and the object list is basically the same for three consecutive times;

[0036] At this point, the index metadata in the object storage is completely or basically consistent with the index metadata in the distributed database. The current index metadata consistency time is recorded to ensure that after the index metadata storage is redirected, the log analysis module accurately screens the last incremental migration task.

[0037] Further, in step S9, the log analysis module analyzes the index metadata operation log of the object storage, parses the bucket name list and the object name list from the log after the index metadata consistency time, and parses the bucket name list and the object name list of the last failed migration from the migration failure log of the migration execution node, and sends the two parts of the bucket name list and the object name list to the migration execution node, and sends a signal to the migration execution node after the sending is completed;

[0038] After receiving the signal from the migration control node, the incremental migration module reads the index metadata corresponding to the incremental bucket name list and object name list in the cluster from the object storage and writes it to the distributed database. After the writing is completed, the distributed database has the latest version of the index metadata.

[0039] An object storage index metadata hot migration device includes: at least one memory and at least one processor;

[0040] The at least one memory is used to store a machine-readable program;

[0041] The at least one processor is used to call the machine-readable program to execute a method for hot migration of object storage index metadata.

[0042] Compared with the prior art, the object storage index metadata hot migration method and device of the present invention have the following outstanding beneficial effects:

[0043] The present invention significantly improves migration efficiency and system performance: by separating control and execution nodes, combined with intelligent task allocation and load balancing mechanisms, the system can efficiently process data migration tasks in parallel, greatly shortening the time required for migration. At the same time, the strategy of combining full and incremental migration ensures the real-time and integrity of data migration, avoids ineffective occupation of resources, and improves overall system performance;

[0044] Optimize resource management and reduce operation and maintenance costs: Intelligent resource management and allocation mechanism enables the system to dynamically adjust according to the actual load, effectively avoiding excessive resource consumption and reducing the difficulty and cost of operation and maintenance. In the case of tight resources, the ability to merge control nodes and execution nodes further improves resource utilization efficiency;

[0045] Ensure business continuity and user experience: The smooth data storage path switching design, combined with the incremental migration strategy in the final stage, ensures that the continuity of external services and user experience are not affected during the data migration and system switching process, reduces service interruption time caused by data migration, and improves user satisfaction.

[0046] Enhanced data management and compliance: Through centralized index metadata management, the system enhances control over data, facilitates advanced data analysis, audit tracking and compliance checks, and provides a solid foundation for meeting increasingly stringent privacy protection and data management regulations;

[0047] Promote flexible expansion of technology and business: The system's design concept and architecture are highly flexible and scalable, and can easily cope with the continued growth of data volume and changes in business needs in the future, providing strong technical support for the digital transformation of enterprises and the in-depth application of cloud services;

[0048] In summary, the present invention not only solves many challenges of data migration in the big data era through an efficient, intelligent and reliable index metadata hot migration solution, but also provides a powerful tool for achieving efficient data management, business continuity and cost optimization, and promotes the healthy development of cloud computing and big data technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Attached Figure 1 The present invention is a flowchart of a method for hot migration of object storage index metadata. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] A best embodiment is given below:

[0053] like Figure 1 As shown, a method for hot migration of object storage index metadata in this embodiment has the following steps:

[0054] S1. Deploy the index metadata migration module on the migration control node and the migration execution node;

[0055] Migration control nodes and migration execution nodes are two types of nodes in the index metadata hot migration system: migration control nodes are nodes that retrieve cluster bucket lists, analyze index metadata operation logs, and issue index metadata migration tasks; migration execution nodes are nodes that run index metadata migration programs.

[0056] There is one migration control node and multiple migration execution nodes in the index metadata hot migration system. Because the migration control node consumes small network resources and computing resources, when node resources are tight, the migration control node and the migration execution node can be the same node.

[0057] The index metadata migration module is responsible for specific metadata enumeration, log analysis, and index metadata migration in the index metadata hot migration system: specifically, it includes the bucket enumeration module, object enumeration module, log analysis module, full migration module, and incremental migration module. Among them:

[0058] The bucket enumeration module is responsible for requesting object storage to obtain a list of all bucket names in the current cluster;

[0059] The object enumeration module is responsible for requesting object storage and obtaining a list of all object names for a specified bucket name;

[0060] The log analysis module is responsible for analyzing the index metadata addition and deletion operation logs of the object storage and the migration failure logs of the migration module, and resolving the bucket names and object names therein;

[0061] The full migration module is responsible for migrating all index metadata in the current cluster to the distributed database;

[0062] The incremental migration module is responsible for migrating the index metadata updated after the last migration to the distributed database.

[0063] S2: Object storage records the addition and deletion operations of index metadata in the log;

[0064] Before migrating index metadata, configure object storage to print the addition and deletion operations of index metadata to the log. The log contains the index metadata operation type, index metadata operation time, bucket name, and object name of the index metadata operation.

[0065] S3, bucket enumeration module and object enumeration module enumerate all bucket names and object names in the current cluster;

[0066] First, the bucket enumeration module requests the object storage to obtain the full bucket name list in the current cluster. Then the object enumeration module accesses the object storage with the obtained bucket name to obtain the full object name list in the current cluster. Finally, the full bucket name and object name list are passed to the full migration module of the migration execution node. After the transmission is completed, a signal is sent to the full migration module to notify the full migration module to start the full migration of index metadata. At the same time, the start time of the full migration is recorded to ensure that the log analysis module can accurately find the index metadata operation log.

[0067] When passing the bucket name and object name list, the bucket name and object name are hashed to obtain the address of the migration execution node, ensuring that the incremental migration tasks are evenly distributed to different migration execution nodes.

[0068] S4, the full migration module migrates all index metadata in the cluster;

[0069] After receiving the signal from the migration control node, the full migration module reads the index metadata corresponding to the full bucket name list and object name list in the cluster from the object storage and writes it to the distributed database.

[0070] At the same time, the bucket name and object name of the failed migration and the failure time are recorded in the log of the full migration module, and the migration failure log is passed to the log analysis module.

[0071] S5, the log analysis module generates an index metadata incremental migration task;

[0072] The log analysis module analyzes the index metadata operation log of the object storage, parses the bucket name list and object name list from the log after the start time of the last migration, and parses the bucket name list and object name list of the last failed migration from the migration failure log of the migration execution node, and sends the two parts of the bucket name list and object name list to the incremental migration module of the migration execution node. After the sending is completed, a signal is sent to the migration execution node.

[0073] When sending, the bucket name and object name are hashed to obtain the address of the migration execution node, ensuring that the incremental migration tasks are evenly distributed to different migration execution nodes.

[0074] S6, the incremental migration module migrates the incremental index metadata in the cluster;

[0075] Record the start time of incremental migration to ensure that the log analysis module can accurately find the index metadata operation log;

[0076] After receiving the signal from the migration control node, the incremental migration module reads the index metadata corresponding to the incremental bucket name list and object name list in the cluster from the object storage and writes it to the distributed database. The bucket name and object name that failed to migrate are recorded in the incremental migration module log, and the migration failure log is passed to the log analysis module.

[0077] S7, repeating step S5 and step S6 until the index metadata in the distributed database is consistent with the index metadata in the object storage;

[0078] Repeat step S5 and step S6. The execution time of the incremental migration decreases as the number of executions increases, and the bucket list and object list of the incremental migration become smaller as the interval between incremental migrations becomes shorter, until the bucket list and object list screened by the log analysis module are empty, or the number of bucket lists and object lists is basically the same for three consecutive times (the coefficient of variation does not exceed 5%).

[0079] At this time, the index metadata in the object storage is completely consistent or basically consistent with the index metadata in the distributed database. The current index metadata consistency time is recorded to ensure that after the index metadata storage is redirected, the log analysis module can accurately screen the last incremental migration task.

[0080] S8. Modify the object storage configuration item, convert the index metadata storage address of the object storage to the distributed database, and restart the object storage service to make the configuration item change take effect.

[0081] S9. Perform the last incremental migration of index metadata.

[0082] The log analysis module analyzes the index metadata operation log of the object storage, parses the bucket name list and object name list from the log after the index metadata consistency time, and parses the bucket name list and object name list of the last failed migration from the migration failure log of the migration execution node, and sends the two parts of the bucket name list and object name list to the migration execution node. After the sending is completed, a signal is sent to the migration execution node.

[0083] After receiving the signal from the migration control node, the incremental migration module reads the index metadata corresponding to the incremental bucket name list and object name list in the cluster from the object storage and writes it to the distributed database. After writing is completed, the distributed database has the latest version of the index metadata.

[0084] Based on the above method, an object storage index metadata hot migration device in this embodiment includes: at least one memory and at least one processor;

[0085] The at least one memory is used to store a machine-readable program;

[0086] The at least one processor is used to call the machine-readable program to execute a method for hot migration of object storage index metadata.

[0087] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0088] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for hot migration of object storage index metadata, characterized in that: The steps are as follows: S1. Deploy the index metadata migration module on the migration control node and the migration execution node; S2: Object storage records the addition and deletion operations of index metadata in the log; S3, bucket enumeration module and object enumeration module enumerate all bucket names and object names in the current cluster; S4, the full migration module migrates all index metadata in the cluster; S5, the log analysis module generates an index metadata incremental migration task; S6, the incremental migration module migrates the incremental index metadata in the cluster; S7, repeating step S5 and step S6 until the index metadata in the distributed database is consistent with the index metadata in the object storage; S8. Redirect the index metadata storage of the object storage to the distributed database, and restart the object storage service to make the configuration item changes take effect; S9. Perform the last incremental migration of index metadata.

2. The object storage index metadata hot migration method according to claim 1, characterized in that: In step S1, the migration control node and the migration execution node are two types of nodes in the index metadata hot migration system. The migration control node refers to a node that retrieves the cluster bucket list and analyzes the index metadata operation log and issues the index metadata migration task; The migration execution node refers to a node that runs the index metadata migration program; The index metadata migration module is responsible for specific metadata enumeration, log analysis, and index metadata migration in the index metadata hot migration system. Specifically, it includes a bucket enumeration module, an object enumeration module, a log analysis module, a full migration module, and an incremental migration module. The bucket enumeration module is responsible for requesting object storage to obtain a list of all bucket names in the current cluster; The object enumeration module is responsible for requesting object storage and obtaining a list of all object names of a specified bucket name; The log analysis module is responsible for analyzing the index metadata addition and deletion operation logs of the object storage and the migration failure logs of the migration module, and parsing the bucket names and object names therein; The full migration module is responsible for migrating all index metadata in the current cluster to the distributed database; The incremental migration module is responsible for migrating the index metadata updated after the last migration to the distributed database.

3. The object storage index metadata hot migration method according to claim 2, characterized in that: In step S2, before migrating the index metadata, the object storage is configured to print the addition and deletion operations of the index metadata into a log, which specifically includes the index metadata operation type, the index metadata operation time, the bucket name and the object name of the index metadata operation.

4. The object storage index metadata hot migration method according to claim 3, characterized in that: In step S3, first, the bucket enumeration module requests the object storage to obtain the full bucket name list in the current cluster. Then, the object enumeration module carries the obtained bucket name to access the object storage to obtain the full object name list in the current cluster. Finally, the full bucket name and object name list are passed to the full migration module of the migration execution node. After the transmission is completed, a signal is sent to the full migration module to notify the full migration module to start full migration of index metadata. At the same time, the start time of the full migration is recorded to ensure that the log analysis module can accurately find the index metadata operation log. When passing the bucket name and object name list, the bucket name and object name are hashed to obtain the address of the migration execution node, and the incremental migration tasks are evenly distributed to different migration execution nodes.

5. The object storage index metadata hot migration method according to claim 4, characterized in that: In step S4, after receiving the signal from the migration control node, the full migration module reads the index metadata corresponding to the full bucket name list and object name list in the cluster from the object storage and writes them into the distributed database; At the same time, the bucket name and object name of the failed migration and the failure time are recorded in the log of the full migration module, and the migration failure log is passed to the log analysis module.

6. The object storage index metadata hot migration method according to claim 5, characterized in that: In step S5, the log analysis module analyzes the index metadata operation log of the object storage, parses the bucket name list and the object name list from the log after the start time of the last migration, and parses the bucket name list and the object name list of the last failed migration from the migration failure log of the migration execution node, and sends the two parts of the bucket name list and the object name list to the incremental migration module of the migration execution node, and sends a signal to the migration execution node after the sending is completed; When sending, the bucket name and object name are hashed to obtain the address of the migration execution node, and the incremental migration tasks are evenly distributed to different migration execution nodes.

7. The object storage index metadata hot migration method according to claim 6, characterized in that: In step S6, the start time of the incremental migration is recorded to ensure that the log analysis module accurately finds the index metadata operation log. After the incremental migration module receives the signal from the migration control node, it reads the index metadata corresponding to the incremental bucket name list and object name list in the cluster from the object storage and writes them into the distributed database. The bucket name and object name of the failed migration are recorded in the log of the incremental migration module, and the migration failure log is passed to the log analysis module.

8. The object storage index metadata hot migration method according to claim 7, characterized in that: In step S7, step S5 and step S6 are repeatedly executed, and the execution time of the incremental migration decreases as the number of executions increases, and the bucket list and object list of the incremental migration become smaller as the interval time of the incremental migration becomes shorter, until the bucket list and object list screened by the log analysis module are empty, or the number of bucket lists and object lists is basically the same for three consecutive times; At this point, the index metadata in the object storage is completely or basically consistent with the index metadata in the distributed database. The current index metadata consistency time is recorded to ensure that after the index metadata storage is redirected, the log analysis module accurately screens the last incremental migration task.

9. The object storage index metadata hot migration method according to claim 8, characterized in that: In step S9, the log analysis module analyzes the index metadata operation log of the object storage, parses the bucket name list and object name list from the log after the index metadata consistency time, and parses the bucket name list and object name list of the last failed migration from the migration failure log of the migration execution node, and sends the two parts of the bucket name list and object name list to the migration execution node. After the sending is completed, a signal is sent to the migration execution node; After receiving the signal from the migration control node, the incremental migration module reads the index metadata corresponding to the incremental bucket name list and object name list in the cluster from the object storage and writes it to the distributed database. After the writing is completed, the distributed database has the latest version of the index metadata.

10. An object storage index metadata hot migration device, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 9.

Images