A method for managing object storage index metadata

Abstract

The present invention discloses an object index metadata management method, which relates to the technical field of cloud computing. The method comprises the following steps: step 1: designing object index metadata of a distributed database system; step 11: sorting and storing the object storage index metadata according to the lexicographic order of the key when managing the index metadata; step 12: writing the index metadata by a database instance in a pure append-only writing mode; step 13: grouping adjacent index metadata by a scheduler instance; step 14: making each region save a copy on each of three database instances; step 15: storing and managing the database instance locations of all copies of the index metadata by the scheduler instance; step 16: checking whether the database instance is down by a heartbeat message, and performing recovery management on the down database instance; and step 2: reading and writing the index metadata in the distributed database system by an object storage gateway.

Description

Technical Field

[0001] The invention discloses an object storage index metadata management method, which relates to the technical field of cloud computing. Background Art

[0002] In the current data storage landscape, with the rapid development of cloud computing and big data technologies, object storage has become the preferred solution for storing massive amounts of unstructured data due to its high scalability, low cost, and ease of management. However, as storage scale continues to expand, efficiently managing and retrieving the index metadata for this data has become a critical issue that needs to be addressed. Given this technical background, while various object storage solutions already exist on the market, they still face challenges in managing large-scale index metadata, such as low retrieval efficiency, insufficient data consistency, inefficient resource utilization, complex operations and maintenance, and limited scalability. Summary of the Invention

[0003] In order to solve the problems in the prior art, the present invention provides an object storage index metadata management method. The specific solution proposed by the present invention is:

[0004] The present invention provides an object index metadata management method, comprising:

[0005] Step 1: Design the object index metadata of the distributed database system:<key,value> The key is the object ID, and the value is the object index metadata. The object index metadata includes the object creation time, object size, object storage type, object permissions, and object data OSD distribution. The management of index metadata involves the database instance and scheduler instance of the distributed database system.

[0006] Step 11: When managing index metadata, sort and store the object storage index metadata in lexicographic order of the key;

[0007] Step 12: Use the database instance to write index metadata using a pure append-only write method;

[0008] Step 13: Group adjacent index metadata using the scheduler instance: Mark the regions marked with index metadata according to the preset size;

[0009] Step 14: Each region stores a replica on each of the three database instances. Set only one replica as the leader replica and the other two replicas as follower replicas.

[0010] Step 15: The scheduler instance stores and manages the database instance locations of all replicas of the index metadata, and uses the Raft protocol to ensure distributed consistency.

[0011] Step 16: Use heartbeat messages to check whether the database instance is down, and perform recovery management on the downed database instance;

[0012] Step 2: The object storage gateway reads and writes index metadata in the distributed database system:

[0013] Step 21: When the object storage gateway uploads object data, it stores the object data in the OSD and records the object creation time, object size, object storage type, object permissions, and object data OSD distribution in the value of the index metadata.

[0014] Step 22: When the object storage gateway downloads object data, it reads the value of the index metadata from the distributed database system, obtains the OSD distribution of the object data, and then requests the OSD to obtain the object data.

[0015] Furthermore, in step 11 of the object index metadata management method, the key is the object ID, which is composed of the bucket ID followed by the object name. The database instance supports querying a specified number of index metadata from the starting point of the specified key, and the database instance supports querying the ordered list of all objects in the specified bucket through the object storage gateway. The specific steps are as follows:

[0016] Step 111: Obtain the bucket ID based on the bucket name through the object storage gateway, and append an empty string after the bucket ID to obtain the query starting point key.

[0017] Step 112: Request the database instance with the query starting point key to obtain the specified number of index metadata entries.

[0018] Step 113: After completing the request for the database instance, if the number of index metadata returned is less than the specified number, the query is successful and the query ends; if the number of index metadata returned is equal to the specified number, the key in the last item of the returned index metadata list is used as the query starting key, and step 112 is executed in a loop.

[0019] Furthermore, writing index metadata in step 12 of the object index metadata management method includes:

[0020] Step 121: When writing index metadata, it is directly written to the memory cache, and the write log is recorded to the flash device in an appended manner. Only after the writing of the memory and the flash device is completed, the object storage gateway is notified that the index metadata is successfully written.

[0021] Step 122: When the index metadata in the memory exceeds 1GB, the index metadata in the memory is flushed to the back-end flash memory device and compressed at the same time to improve the utilization rate of the flash memory device.

[0022] Step 123: After the index metadata in the memory is flushed to the backend flash memory device, the write log in the flash memory is cleared to release excess storage space.

[0023] Furthermore, in step 13 of the object index metadata management method, grouping adjacent index metadata includes:

[0024] Mark the adjacent 150MB index metadata as a region.

[0025] When the database instance is initialized, five empty regions are generated. As the storage capacity of index metadata increases, regions larger than 150MB are split into two regions of equal size. At the same time, as index metadata is deleted, two adjacent regions smaller than 25MB are merged into one region.

[0026] Furthermore, in step 14 of the object index metadata management method, managing the copy of the index metadata includes:

[0027] When reading index metadata, a replica is randomly selected to complete the reading task. When writing index metadata, only the leader replica is written. The scheduler instance asynchronously synchronizes the index metadata changes to the follower replica. The distribution of the three replicas complies with the principle of maximum isolation domain. The maximum isolation domain principle means that the maximum network topology distance is maintained between replicas. The priority across centers is greater than across computer rooms, the priority across computer rooms is greater than across racks, the priority across racks is greater than across physical machines, and the priority across physical machines is greater than across disks. At the same time, the database instance has an additional weight attribute. The weight decreases as the available disk space is consumed. While complying with the maximum isolation domain principle, data replicas tend to be stored in database instances with large weights.

[0028] Furthermore, in step 15 of the object index metadata management method, when an error occurs in the leader copy, the scheduler instance initiates an election to select a new leader copy and generates a new follower copy; when an error occurs in the follower copy, the scheduler instance initiates an election to select a new leader copy and generates a new follower copy.

[0029] Furthermore, step 16 of the object index metadata management method specifically includes:

[0030] Heartbeat messages are used to check whether the database instance is down. When a database instance is down, other database instances will not be able to sense the heartbeat from the down database for 30 minutes. Then, a database instance down signal is sent to the scheduler instance. After receiving the down signal, the caller instance waits for 30 minutes. If the down database instance does not resume heartbeat after 30 minutes, the down database instance is kicked out of the distributed database topology, and all leader and follower copies on the down database instance are migrated to normal database instances.

[0031] The present invention also provides an object index metadata management device, including an object index metadata management module and an object storage gateway management module.

[0032] Object index metadata management module design distributed database system object index metadata: index metadata entries according to<key,value> The key is the object ID, and the value is the object index metadata. The object index metadata includes the object creation time, object size, object storage type, object permissions, and object data OSD distribution. The management of index metadata involves the database instance and scheduler instance of the distributed database system.

[0033] Step 11: When managing index metadata, sort and store the object storage index metadata in lexicographic order of the key;

[0034] Step 12: Use the database instance to write index metadata using a pure append-only write method;

[0035] Step 13: Group adjacent index metadata using the scheduler instance: Mark the regions marked with index metadata according to the preset size;

[0036] Step 14: Each region stores a replica on each of the three database instances. Set only one replica as the leader replica and the other two replicas as follower replicas.

[0037] Step 15: The scheduler instance stores and manages the database instance locations of all replicas of the index metadata, and uses the Raft protocol to ensure distributed consistency.

[0038] Step 16: Use heartbeat messages to check whether the database instance is down, and perform recovery management on the downed database instance;

[0039] The object storage gateway management module manages the object storage gateway to read and write index metadata in the distributed database system:

[0040] Step 21: When the object storage gateway uploads object data, the object storage gateway management module stores the object data in the OSD and records the object creation time, object size, object storage type, object permissions, and object data OSD distribution in the value of the index metadata.

[0041] Step 22: When the object storage gateway downloads the object data, the object storage gateway management module reads the value of the index metadata from the distributed database system, obtains the object data OSD distribution, and then requests the OSD to obtain the object data.

[0042] The benefits of the present invention are:

[0043] By designing a distributed database system to store object storage index metadata, performance is optimized: users can experience faster object retrieval speeds and higher write throughput, especially when processing large numbers of concurrent requests and massive data sets, significantly reducing latency. It also saves costs: through refined data grouping and compression strategies, the demand for storage resources is effectively reduced, while intelligent replica layout strategies reduce the operating costs of data centers. Enhanced reliability: Even in the event of partial component failure, the system can continue to provide services and ensure business continuity, which is critical for businesses that rely on highly reliable data storage. Simplified operation and maintenance: Automated management and scheduling functions reduce the need for manual intervention, making the system easier to maintain and expand, and adapt to the needs of future data growth and technological evolution. Improved user experience: The fast response and high availability of object storage services directly translate into improved end-user satisfaction, especially in application scenarios such as cloud storage, content distribution networks (CDNs), and big data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0046] Example 1

[0047] The present invention provides an object index metadata management method, comprising:

[0048] Step 1: Design the object index metadata of the distributed database system:<key,value> The key is the object ID, and the value is the object index metadata. The object index metadata includes the object creation time, object size, object storage type, object permissions, and object data OSD distribution. The management of index metadata involves the database instance and scheduler instance of the distributed database system.

[0049] Step 11: When managing index metadata, in order to ensure the query efficiency of index metadata, the object storage index metadata is sorted and stored in the lexicographic order of the key.

[0050] The key is the object ID, which is the bucket ID followed by the object name. The database instance supports querying a specified number of index metadata from a specified key starting point. Therefore, the database instance supports the Object Storage Gateway query for an ordered list of all objects in a specified bucket. The specific steps are as follows:

[0051] Step 111: The object storage gateway obtains the bucket ID based on the bucket name and appends an empty string to the bucket ID to obtain the query starting point key.

[0052] Step 112: Request the database instance with the query starting key to obtain 1,000 index metadata entries.

[0053] Step 113: After completing the request for the database instance, if the number of returned index metadata is less than 1000, the query is successful and the query ends; if the number of returned index metadata is equal to 1000, the key in the last item of the returned index metadata list is used as the query starting key, and step 112 is executed.

[0054] Step 12: To ensure the writing efficiency of index metadata, write index metadata through the database instance using a pure append-only writing method.

[0055] The specific steps for writing index metadata are as follows:

[0056] Step 121: When writing index metadata, it is written directly to the memory cache, and the write log is recorded to the flash device in an appended manner. Only after the writing to the memory and flash device is completed, the object storage gateway is notified that the index data is written successfully.

[0057] Step 122: When the index metadata in the memory exceeds 1 GB, the index metadata in the memory is flushed to the backend flash memory device and compressed at the same time to improve the utilization rate of the flash memory device.

[0058] Step 123: After the index metadata in the memory is flushed to the backend flash memory device, the write log in the flash memory is cleared to release excess storage space.

[0059] If the database instance crashes before the index metadata in memory is flushed to the backend flash device, the index metadata in memory can be rebuilt after restart by replaying the write logs in the flash device, ensuring data security.

[0060] Step 13: To facilitate the management and scheduling of the object storage index metadata in the database instance, the scheduler instance groups adjacent index metadata: the regions marked with index metadata are marked according to the preset size.

[0061] Grouping adjacent index metadata may further include:

[0062] Mark the adjacent 150MB index metadata as a region.

[0063] When the database instance is initialized, five empty regions are generated. As the storage capacity of index metadata increases, regions larger than 150MB are split into two regions of equal size. At the same time, as index metadata is deleted, two adjacent regions smaller than 25MB are merged into one region.

[0064] Step 14: To ensure high availability of object storage index metadata, each region stores a copy on three database instances. Set one and only one copy as the leader copy, and the other two copies as follower copies.

[0065] Step 14, managing the copy of the index metadata, may include:

[0066] When reading index metadata, a replica is randomly selected to complete the reading task. When writing index metadata, only the leader replica is written. The scheduler instance asynchronously synchronizes the index metadata changes to the follower replica. The distribution of the three replicas complies with the principle of maximum isolation domain. The maximum isolation domain principle means that the maximum network topology distance is maintained between replicas. The priority across centers is greater than across computer rooms, the priority across computer rooms is greater than across racks, the priority across racks is greater than across physical machines, and the priority across physical machines is greater than across disks. At the same time, the database instance has an additional weight attribute. The weight decreases as the available disk space is consumed. While complying with the maximum isolation domain principle, data replicas tend to be stored in database instances with large weights.

[0067] Step 15: The scheduler instance stores and manages the database instance locations of all replicas of the index metadata, and uses the Raft protocol to ensure distributed consistency. If a leader replica fails, the scheduler instance initiates an election to select a new leader replica and generates a new follower replica. If a follower replica fails, the scheduler instance initiates an election to select a new leader replica and generates a new follower replica.

[0068] Step 16: Use heartbeat messages to verify whether the database instance is down, and perform recovery management on the downed database instance. This may include:

[0069] Heartbeat messages are used to check whether the database instance is down. When a database instance is down, other database instances will not be able to sense the heartbeat from the down database for 30 minutes. Then, a database instance down signal is sent to the scheduler instance. After receiving the down signal, the caller instance waits for 30 minutes. If the down database instance does not resume heartbeat after 30 minutes, the down database instance is kicked out of the distributed database topology, and all leader and follower copies on the down database instance are migrated to normal database instances.

[0070] Step 2: The object storage gateway reads and writes index metadata in the distributed database system:

[0071] Step 21: When the object storage gateway uploads object data, it stores the object data in the OSD and records the object creation time, object size, object storage type, object permissions, and object data OSD distribution in the value of the index metadata.

[0072] Step 22: When the object storage gateway downloads object data, it reads the value of the index metadata from the distributed database system, obtains the OSD distribution of the object data, and then requests the OSD to obtain the object data.

[0073] When the object storage gateway lists the object names in the bucket, it obtains the index metadata list in batches multiple times and returns the keys to the client.

[0074] Example 2

[0075] The present invention also provides an object index metadata management device, including an object index metadata management module and an object storage gateway management module.

[0076] Object index metadata management module design distributed database system object index metadata: index metadata entries according to<key,value> The key is the object ID, and the value is the object index metadata. The object index metadata includes the object creation time, object size, object storage type, object permissions, and object data OSD distribution. The management of index metadata involves the database instance and scheduler instance of the distributed database system.

[0077] Step 11: When managing index metadata, sort and store the object storage index metadata in lexicographic order of the key;

[0078] Step 12: Use the database instance to write index metadata using a pure append-only write method;

[0079] Step 13: Group adjacent index metadata using the scheduler instance: Mark the regions marked with index metadata according to the preset size;

[0080] Step 14: Each region stores a replica on each of the three database instances. Set only one replica as the leader replica and the other two replicas as follower replicas.

[0081] Step 15: The scheduler instance stores and manages the database instance locations of all replicas of the index metadata, and uses the Raft protocol to ensure distributed consistency.

[0082] Step 16: Use heartbeat messages to check whether the database instance is down, and perform recovery management on the downed database instance;

[0083] The object storage gateway management module manages the object storage gateway to read and write index metadata in the distributed database system:

[0084] Step 21: When the object storage gateway uploads object data, the object storage gateway management module stores the object data in the OSD and records the object creation time, object size, object storage type, object permissions, and object data OSD distribution in the value of the index metadata.

[0085] Step 22: When the object storage gateway downloads the object data, the object storage gateway management module reads the value of the index metadata from the distributed database system, obtains the object data OSD distribution, and then requests the OSD to obtain the object data.

[0086] Since the information interaction, execution process and other contents between the modules of the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0087] Similarly, the device of the present invention optimizes performance by designing a distributed database system to store object storage index metadata: users can experience faster object retrieval speeds and higher write throughput, especially when processing a large number of concurrent requests and massive data sets, significantly reducing latency. It also saves costs: through refined data grouping and compression strategies, the demand for storage resources is effectively reduced, while the intelligent replica layout strategy reduces the operating costs of the data center. It enhances reliability: even in the event of partial component failure, the system can continue to provide services and ensure business continuity, which is critical for businesses that rely on highly reliable data storage. It simplifies operation and maintenance: automated management and scheduling functions reduce the need for manual intervention, making the system easier to maintain and expand, and adapt to the needs of future data growth and technological evolution. It improves user experience: fast object storage service response and high availability directly translate into improved end-user satisfaction, especially in application scenarios such as cloud storage, content distribution network (CDN) and big data processing.

[0088] The above embodiments are merely preferred embodiments for the purpose of fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention. The scope of protection of the present invention shall be subject to the claims.

Claims

1. A method for managing object index metadata, characterized by include: Step 1: Design the object index metadata of the distributed database system:<key,value> The key is the object ID, and the value is the object index metadata. The object index metadata includes the object creation time, object size, object storage type, object permissions, and object data OSD distribution. The management of index metadata involves the database instance and scheduler instance of the distributed database system. Step 11: When managing index metadata, sort and store the object storage index metadata in lexicographic order of the key; Step 12: Use the database instance to write index metadata using a pure append-only write method; Step 13: Group adjacent index metadata using the scheduler instance: Mark the regions marked with index metadata according to the preset size; Step 14: Each region stores a replica on each of the three database instances. Set only one replica as the leader replica and the other two replicas as follower replicas. Step 15: The scheduler instance stores and manages the database instance locations of all replicas of the index metadata, and uses the Raft protocol to ensure distributed consistency. Step 16: Use heartbeat messages to check whether the database instance is down, and perform recovery management on the downed database instance; Step 2: The object storage gateway reads and writes index metadata in the distributed database system: Step 21: When the object storage gateway uploads object data, it stores the object data in the OSD and records the object creation time, object size, object storage type, object permissions, and object data OSD distribution in the value of the index metadata. Step 22: When the object storage gateway downloads object data, it reads the value of the index metadata from the distributed database system, obtains the OSD distribution of the object data, and then requests the OSD to obtain the object data.

2. The object index metadata management method according to claim 1, characterized in that In step 11, the key is the object ID, which is the bucket ID followed by the object name. The database instance supports querying a specified number of index metadata from the starting point of the specified key. The database instance supports querying the ordered list of all objects in the specified bucket through the object storage gateway. The specific steps are as follows: Step 111: Obtain the bucket ID based on the bucket name through the object storage gateway, and append an empty string after the bucket ID to obtain the query starting point key. Step 112: Request the database instance with the query starting point key to obtain the specified number of index metadata entries. Step 113: After completing the request for the database instance, if the number of index metadata returned is less than the specified number, the query is successful and the query ends; if the number of index metadata returned is equal to the specified number, the key in the last item of the returned index metadata list is used as the query starting key, and step 112 is executed in a loop.

3. The object index metadata management method according to claim 1, characterized in that In step 12, index metadata is written, including: Step 121: When writing index metadata, it is directly written to the memory cache, and the write log is recorded to the flash device in an appended manner. Only after the writing of the memory and the flash device is completed, the object storage gateway is notified that the index metadata is successfully written. Step 122: When the index metadata in the memory exceeds 1GB, the index metadata in the memory is flushed to the back-end flash memory device and compressed at the same time to improve the utilization rate of the flash memory device. Step 123: After the index metadata in the memory is flushed to the backend flash memory device, the write log in the flash memory is cleared to release excess storage space.

4. The object index metadata management method according to claim 1, characterized in that In step 13, adjacent index metadata are grouped, including: Mark the adjacent 150MB index metadata as a region. When the database instance is initialized, five empty regions are generated. As the storage capacity of index metadata increases, regions larger than 150MB are split into two regions of equal size. At the same time, as index metadata is deleted, two adjacent regions smaller than 25MB are merged into one region.

5. The object index metadata management method according to claim 1, characterized in that Managing copies of index metadata in step 14 includes: When reading index metadata, a replica is randomly selected to complete the reading task. When writing index metadata, only the leader replica is written. The scheduler instance asynchronously synchronizes the index metadata changes to the follower replica. The distribution of the three replicas complies with the principle of maximum isolation domain. The maximum isolation domain principle means that the maximum network topology distance is maintained between replicas. The priority across centers is greater than across computer rooms, the priority across computer rooms is greater than across racks, the priority across racks is greater than across physical machines, and the priority across physical machines is greater than across disks. At the same time, the database instance has an additional weight attribute. The weight decreases as the available disk space is consumed. While complying with the maximum isolation domain principle, data replicas tend to be stored in database instances with large weights.

6. The object index metadata management method according to claim 1, characterized in that In step 15, when an error occurs in the leader replica, the scheduler instance initiates an election to select a new leader replica and generates a new follower replica. When an error occurs in a follower replica, the scheduler instance initiates an election to select a new leader replica and generates a new follower replica.

7. The object index metadata management method according to claim 1, characterized in that Step 16 specifically includes: Heartbeat messages are used to check whether the database instance is down. When a database instance is down, other database instances will not be able to sense the heartbeat from the down database for 30 minutes. Then, a database instance down signal is sent to the scheduler instance. After receiving the down signal, the caller instance waits for 30 minutes. If the down database instance does not resume heartbeat after 30 minutes, the down database instance is kicked out of the distributed database topology, and all leader and follower copies on the down database instance are migrated to normal database instances.

8. An object index metadata management device, characterized by Including object index metadata management module and object storage gateway management module, Object index metadata management module design distributed database system object index metadata: index metadata entries according to<key,value> The key is the object ID, and the value is the object index metadata. The object index metadata includes the object creation time, object size, object storage type, object permissions, and object data OSD distribution. The management of index metadata involves the database instance and scheduler instance of the distributed database system. Step 11: When managing index metadata, sort and store the object storage index metadata in lexicographic order of the key; Step 12: Use the database instance to write index metadata using a pure append-only write method; Step 13: Group adjacent index metadata using the scheduler instance: Mark the regions marked with index metadata according to the preset size; Step 14: Each region stores a replica on each of the three database instances. Set only one replica as the leader replica and the other two replicas as follower replicas. Step 15: The scheduler instance stores and manages the database instance locations of all replicas of the index metadata, and uses the Raft protocol to ensure distributed consistency. Step 16: Use heartbeat messages to check whether the database instance is down, and perform recovery management on the downed database instance; The object storage gateway management module manages the object storage gateway to read and write index metadata in the distributed database system: Step 21: When the object storage gateway uploads object data, the object storage gateway management module stores the object data in the OSD and records the object creation time, object size, object storage type, object permissions, and object data OSD distribution in the value of the index metadata. Step 22: When the object storage gateway downloads the object data, the object storage gateway management module reads the value of the index metadata from the distributed database system, obtains the object data OSD distribution, and then requests the OSD to obtain the object data.

Images