Object storage index metadata management method

Abstract

The invention discloses an object index metadata management method, and relates to the technical field of cloud computing. Comprising the following steps: step 1, designing object index metadata of a distributed database system, and step 11, when the index metadata is managed, sorting and storing the object index metadata according to the lexicographical order of a key; 12, the index metadata is written in through the database instance in a pure appending type write-in mode; the method comprises the following steps of: (1) dividing index metadata into three database instances, (2) dividing the index metadata into three database instances, (13) grouping adjacent index metadata through the scheduler instances, (14) enabling each region to store a copy on each of the three database instances, (15) storing and managing database instance positions of all the copies of the index metadata through the scheduler instances, and (16) checking whether the database instances are crashed or not through heartbeat messages. And step 2, the object storage gateway reads and writes the index metadata in the distributed database system.

Description

Technical Field

[0001] The invention discloses an object storage index metadata management method, and relates to the technical field of cloud computing. Background Art

[0002] In the current data storage field, with the rapid development of cloud computing and big data technology, object storage has become the preferred solution for massive unstructured data storage due to its high scalability, low cost and easy management. However, as the storage scale continues to expand, how to efficiently manage and retrieve the index metadata of this data has become a key issue that needs to be solved urgently. In view of the above technical background, although there are many object storage solutions on the market, they still have problems such as low retrieval efficiency, insufficient data consistency guarantee, unreasonable resource utilization, complex operation and maintenance management, and limited scalability when dealing with large-scale index metadata management. Summary of the invention

[0003] In view of the problems of the prior art, the present invention provides an object storage index metadata management method. The specific solution proposed by the present invention is:

[0004] The present invention provides an object index metadata management method, comprising:

[0005] Step 1: Design the object index metadata of the distributed database system:<key,value> The key is the object ID, and the value is the object index metadata. The object index metadata includes the creation time of the object, the size of the object, the storage type of the object, the object's permissions, and the distribution of the object data on the osd. The management of the index metadata involves the database instance and scheduler instance of the distributed database system.

[0006] Step 11: When managing index metadata, sort and store the object storage index metadata in the lexicographic order of the key;

[0007] Step 12: Use the database instance to write index metadata in a pure append-only writing mode;

[0008] Step 13: Group adjacent index metadata through the scheduler instance: mark the region marked by the index metadata according to the preset size;

[0009] Step 14: Make each region save a copy on each of the three database instances, set one and only one copy as the leader copy, and the other two copies as follower copies;

[0010] Step 15: The database instance locations of all replicas of the index metadata are stored and managed through the scheduler instance, and the distributed consistency is ensured through the raft protocol;

[0011] Step 16: Use heartbeat messages to check whether the database instance is down, and perform recovery management on the downed database instance;

[0012] Step 2: The object storage gateway reads and writes index metadata in the distributed database system:

[0013] Step 21: When the object storage gateway uploads object data, it stores the object data in the osd and records the object creation time, object size, object storage type, object permissions, and object data osd distribution in the value of the index metadata.

[0014] Step 22: When the object storage gateway downloads object data, it reads the value of the index metadata from the distributed database system, obtains the osd distribution of the object data, and then requests the osd to obtain the object data.

[0015] Further, in step 11 of the object index metadata management method, the key is the object ID, and the object ID is composed of the bucket ID followed by the object name. The database instance supports querying the specified number of index metadata from the starting point of the specified key, and the database instance supports querying the ordered list of all objects in the specified bucket by the object storage gateway. The specific steps are:

[0016] Step 111: Obtain the bucket ID based on the bucket name through the object storage gateway, and append an empty string after the bucket ID to obtain the query starting point key.

[0017] Step 112: Request the database instance with the query starting point key to obtain the specified number of index metadata entries.

[0018] Step 113: After completing the request for the database instance, if the number of returned index metadata is less than the specified number, the query is successful and the query ends; if the number of returned index metadata is equal to the specified number, the key in the last item of the returned index metadata list is used as the query starting key, and step 112 is executed in a loop.

[0019] Furthermore, writing index metadata in step 12 of the object index metadata management method includes:

[0020] Step 121: When writing index metadata, it is directly written to the memory cache, and the write log is recorded to the flash device in an appended manner. Only after the writing of the memory and the flash device is completed, the object storage gateway is notified that the index metadata is written successfully.

[0021] Step 122: When the index metadata in the memory exceeds 1 GB, the index metadata in the memory is flushed to the backend flash memory device and compressed at the same time to improve the utilization rate of the flash memory device.

[0022] Step 123: After the index metadata in the memory is flushed to the backend flash memory device, the write log in the flash memory is cleared to release excess storage space.

[0023] Furthermore, in step 13 of the object index metadata management method, adjacent index metadata are grouped, including:

[0024] Mark the adjacent 150MB index metadata as a region.

[0025] When the database instance is initialized, 5 empty regions are generated. As the storage capacity of index metadata increases, the region exceeding 150MB is split into two regions of the same size. At the same time, as the index metadata is deleted, two adjacent regions smaller than 25MB are merged into one region.

[0026] Furthermore, in step 14 of the object index metadata management method, managing the copy of the index metadata includes:

[0027] When reading index metadata, a replica is randomly selected to complete the reading task. When writing index metadata, only the leader replica is written. The scheduler instance asynchronously synchronizes the index metadata changes to the follower replica. The distribution of the three replicas complies with the principle of maximum isolation domain, where the maximum isolation domain principle means that the maximum network topology distance is maintained between replicas. The priority across centers is greater than that across computer rooms, the priority across computer rooms is greater than that across racks, the priority across racks is greater than that across physical machines, and the priority across physical machines is greater than that across disks. At the same time, a weight attribute is added to the database instance. The weight decreases as the available disk space is consumed. While complying with the maximum isolation domain principle, data replicas tend to be stored in database instances with large weights.

[0028] Furthermore, in step 15 of the object index metadata management method, when an error occurs in the leader copy, the scheduler instance initiates an election to select a new leader copy and generates a new follower copy; when an error occurs in the follower copy, the scheduler instance initiates an election to select a new leader copy and generates a new follower copy.

[0029] Further, step 16 of the object index metadata management method specifically includes:

[0030] Use heartbeat messages to check whether the database instance is down. When a database instance is down, other database instances cannot sense the heartbeat from the down database for 30 minutes, and then send a database instance down signal to the scheduler instance. After receiving the down signal, the caller instance waits for 30 minutes. If the down database instance does not restore the heartbeat after 30 minutes, the down database instance will be kicked out of the distributed database topology, and all the leader and follower copies on the down database instance will be migrated to normal database instances.

[0031] The present invention also provides an object index metadata management device, comprising an object index metadata management module and an object storage gateway management module.

[0032] Object index metadata management module design distributed database system object index metadata: index metadata entries according to<key,value> The key is the object ID, and the value is the object index metadata. The object index metadata includes the creation time of the object, the size of the object, the storage type of the object, the object's permissions, and the distribution of the object data on the osd. The management of the index metadata involves the database instance and scheduler instance of the distributed database system.

[0033] Step 11: When managing index metadata, sort and store the object storage index metadata in the lexicographic order of the key;

[0034] Step 12: Use the database instance to write index metadata in a pure append-only writing mode;

[0035] Step 13: Group adjacent index metadata through the scheduler instance: mark the region marked by the index metadata according to the preset size;

[0036] Step 14: Make each region save a copy on each of the three database instances, set one and only one copy as the leader copy, and the other two copies as follower copies;

[0037] Step 15: The database instance locations of all replicas of the index metadata are stored and managed through the scheduler instance, and the distributed consistency is ensured through the raft protocol;

[0038] Step 16: Use heartbeat messages to check whether the database instance is down, and perform recovery management on the downed database instance;

[0039] The object storage gateway management module manages the object storage gateway to read and write index metadata in the distributed database system:

[0040] Step 21: When the object storage gateway uploads object data, the object storage gateway management module stores the object data in the osd and records the object creation time, object size, object storage type, object permissions, and object data osd distribution in the value of the index metadata.

[0041] Step 22: When the object storage gateway downloads the object data, the object storage gateway management module reads the value of the index metadata from the distributed database system, obtains the osd distribution of the object data, and then requests the osd to obtain the object data.

[0042] The benefits of the present invention are:

[0043] By designing a distributed database system to store object storage index metadata, performance is optimized: users can experience faster object retrieval speeds and higher write throughput, especially when dealing with large numbers of concurrent requests and massive data sets, significantly reducing latency. It also saves costs: through refined data grouping and compression strategies, the demand for storage resources is effectively reduced, while intelligent replica layout strategies reduce the operating costs of data centers. Enhanced reliability: Even in the event of partial component failure, the system can continue to provide services and ensure business continuity, which is critical for businesses that rely on highly reliable data storage. Simplified operation and maintenance: Automated management and scheduling functions reduce the need for manual intervention, making the system easier to maintain and expand, and adapting to the needs of future data growth and technological evolution. Improved user experience: Fast object storage service response and high availability directly translate into improved end-user satisfaction, especially in application scenarios such as cloud storage, content distribution network (CDN), and big data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic flow chart of the method of the present invention. DETAILED DESCRIPTION

[0045] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0046] Example 1

[0047] The present invention provides an object index metadata management method, comprising:

[0048] Step 1: Design the object index metadata of the distributed database system:<key,value> The key is the object ID, and the value is the object index metadata. The object index metadata includes the creation time of the object, the size of the object, the storage type of the object, the object's permissions, and the distribution of the object data on the osd. The management of the index metadata involves the database instance and scheduler instance of the distributed database system.

[0049] Step 11: When managing index metadata, in order to ensure the query efficiency of index metadata, the object storage index metadata is sorted and stored according to the lexicographical order of the key.

[0050] The key is the object ID, which is the bucket ID followed by the object name. The database instance supports querying a specified number of index metadata from the starting point of a specified key, so the database instance supports the object storage gateway to query the ordered list of all objects in a specified bucket. The specific steps are as follows:

[0051] Step 111: The object storage gateway obtains the bucket ID based on the bucket name, and appends an empty string after the bucket ID to obtain the query starting point key.

[0052] Step 112: Request the database instance with the query starting point key to obtain 1,000 index metadata entries.

[0053] Step 113: After completing the request for the database instance, if the number of returned index metadata is less than 1000, the query is successful and the query ends; if the number of returned index metadata is equal to 1000, the key in the last item of the returned index metadata list is used as the query starting key and step 112 is executed.

[0054] Step 12: To ensure the writing efficiency of index metadata, write index metadata through the database instance in a pure append-only writing mode.

[0055] The specific steps for writing index metadata are as follows:

[0056] Step 121: When writing index metadata, it is directly written to the memory cache, and the write log is recorded to the flash device in an appended manner. Only after the writing of the memory and the flash device is completed, the object storage gateway is notified that the index data is written successfully.

[0057] Step 122: When the index metadata in the memory exceeds 1 GB, the index metadata in the memory is flushed to the backend flash memory device and compressed at the same time to improve the utilization rate of the flash memory device.

[0058] Step 123: After the index metadata in the memory is flushed to the backend flash memory device, the write log in the flash memory is cleared to release excess storage space.

[0059] If the database instance crashes before the index metadata in the memory is flushed to the backend flash device, the index metadata in the memory can be rebuilt after the restart by replaying the write logs in the flash device, ensuring data security.

[0060] Step 13: To facilitate the management and scheduling of the object storage index metadata in the database instance, the adjacent index metadata are grouped through the scheduler instance: the region marked with the index metadata is marked according to the preset size.

[0061] The grouping of adjacent index metadata may further include:

[0062] Mark the adjacent 150MB index metadata as a region.

[0063] When the database instance is initialized, 5 empty regions are generated. As the storage capacity of index metadata increases, the region exceeding 150MB is split into two regions of the same size. At the same time, as the index metadata is deleted, two adjacent regions smaller than 25MB are merged into one region.

[0064] Step 14: To ensure high availability of object storage index metadata, each region stores a copy on three database instances. Only one copy is set as the leader copy, and the other two copies are follower copies.

[0065] Wherein step 14 manages the copy of the index metadata, which may include:

[0066] When reading index metadata, a replica is randomly selected to complete the reading task. When writing index metadata, only the leader replica is written. The scheduler instance asynchronously synchronizes the index metadata changes to the follower replica. The distribution of the three replicas complies with the principle of maximum isolation domain, where the maximum isolation domain principle means that the maximum network topology distance is maintained between replicas. The priority across centers is greater than that across computer rooms, the priority across computer rooms is greater than that across racks, the priority across racks is greater than that across physical machines, and the priority across physical machines is greater than that across disks. At the same time, a weight attribute is added to the database instance. The weight decreases as the available disk space is consumed. While complying with the maximum isolation domain principle, data replicas tend to be stored in database instances with large weights.

[0067] Step 15: The database instance locations of all copies of the index metadata are stored and managed through the scheduler instance, and the distributed consistency is ensured through the raft protocol. When an error occurs in the leader copy, the scheduler instance initiates an election to select a new leader copy and generates a new follower copy; when an error occurs in the follower copy, the scheduler instance initiates an election to select a new leader copy and generates a new follower copy.

[0068] Step 16: Use heartbeat messages to check whether the database instance is down, and perform recovery management on the downed database instance. This may include:

[0069] Use heartbeat messages to check whether the database instance is down. When a database instance is down, other database instances cannot sense the heartbeat from the down database for 30 minutes, and then send a database instance down signal to the scheduler instance. After receiving the down signal, the caller instance waits for 30 minutes. If the down database instance does not restore the heartbeat after 30 minutes, the down database instance will be kicked out of the distributed database topology, and all the leader and follower copies on the down database instance will be migrated to normal database instances.

[0070] Step 2: The object storage gateway reads and writes index metadata in the distributed database system:

[0071] Step 21: When the object storage gateway uploads object data, it stores the object data in the osd and records the object creation time, object size, object storage type, object permissions, and object data osd distribution in the value of the index metadata.

[0072] Step 22: When the object storage gateway downloads object data, it reads the value of the index metadata from the distributed database system, obtains the osd distribution of the object data, and then requests the osd to obtain the object data.

[0073] When the object storage gateway lists the object names in the bucket, it obtains the index metadata list in batches multiple times and returns the keys to the client.

[0074] Example 2

[0075] The present invention also provides an object index metadata management device, comprising an object index metadata management module and an object storage gateway management module.

[0076] Object index metadata management module design distributed database system object index metadata: index metadata entries according to<key,value> The key is the object ID, and the value is the object index metadata. The object index metadata includes the creation time of the object, the size of the object, the storage type of the object, the object's permissions, and the distribution of the object data on the osd. The management of the index metadata involves the database instance and scheduler instance of the distributed database system.

[0077] Step 11: When managing index metadata, sort and store the object storage index metadata in the lexicographic order of the key;

[0078] Step 12: Use the database instance to write index metadata in a pure append-only writing mode;

[0079] Step 13: Group adjacent index metadata through the scheduler instance: mark the region marked by the index metadata according to the preset size;

[0080] Step 14: Make each region save a copy on each of the three database instances, set one and only one copy as the leader copy, and the other two copies as follower copies;

[0081] Step 15: The database instance locations of all replicas of the index metadata are stored and managed through the scheduler instance, and the distributed consistency is ensured through the raft protocol;

[0082] Step 16: Use heartbeat messages to check whether the database instance is down, and perform recovery management on the downed database instance;

[0083] The object storage gateway management module manages the object storage gateway to read and write index metadata in the distributed database system:

[0084] Step 21: When the object storage gateway uploads object data, the object storage gateway management module stores the object data in the osd and records the object creation time, object size, object storage type, object permissions, and object data osd distribution in the value of the index metadata.

[0085] Step 22: When the object storage gateway downloads the object data, the object storage gateway management module reads the value of the index metadata from the distributed database system, obtains the osd distribution of the object data, and then requests the osd to obtain the object data.

[0086] As the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.

[0087] Similarly, the device of the present invention optimizes performance by designing a distributed database system to store object storage index metadata: users can experience faster object retrieval speeds and higher write throughput, especially when processing a large number of concurrent requests and massive data sets, significantly reducing latency. It also saves costs: through refined data grouping and compression strategies, the demand for storage resources is effectively reduced, while the intelligent replica layout strategy reduces the operating costs of the data center. Enhanced reliability: Even in the event of partial component failure, the system can continue to provide services and ensure business continuity, which is critical for businesses that rely on highly reliable data storage. Simplified operation and maintenance: Automated management and scheduling functions reduce the need for manual intervention, making the system easier to maintain and expand, and adapting to the needs of future data growth and technological evolution. Improved user experience: Fast object storage service response and high availability directly translate into improved end-user satisfaction, especially in application scenarios such as cloud storage, content distribution network (CDN), and big data processing.

[0088] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. A method for managing object index metadata, characterized in that include: Step 1: Design the object index metadata of the distributed database system:<key,value> The key is the object ID, and the value is the object index metadata. The object index metadata includes the creation time of the object, the size of the object, the storage type of the object, the object's permissions, and the distribution of the object data on the osd. The management of the index metadata involves the database instance and scheduler instance of the distributed database system. Step 11: When managing index metadata, sort and store the object storage index metadata in the lexicographic order of the key; Step 12: Use the database instance to write index metadata in a pure append-only writing mode; Step 13: Group adjacent index metadata through the scheduler instance: mark the region marked by the index metadata according to the preset size; Step 14: Make each region save a copy on each of the three database instances, set one and only one copy as the leader copy, and the other two copies as follower copies; Step 15: The database instance locations of all replicas of the index metadata are stored and managed through the scheduler instance, and the distributed consistency is ensured through the raft protocol; Step 16: Use heartbeat messages to check whether the database instance is down, and perform recovery management on the downed database instance; Step 2: The object storage gateway reads and writes index metadata in the distributed database system: Step 21: When the object storage gateway uploads object data, it stores the object data in the osd and records the object creation time, object size, object storage type, object permissions, and object data osd distribution in the value of the index metadata. Step 22: When the object storage gateway downloads object data, it reads the value of the index metadata from the distributed database system, obtains the osd distribution of the object data, and then requests the osd to obtain the object data.

2. The object index metadata management method according to claim 1, characterized in that In step 11, the key is the object ID, which is composed of the bucket ID followed by the object name. The database instance supports querying the specified number of index metadata from the starting point of the specified key. The database instance supports querying the ordered list of all objects in the specified bucket through the object storage gateway. The specific steps are as follows: Step 111: Obtain the bucket ID based on the bucket name through the object storage gateway, and append an empty string after the bucket ID to obtain the query starting point key. Step 112: Request the database instance with the query starting point key to obtain the specified number of index metadata entries. Step 113: After completing the request for the database instance, if the number of returned index metadata is less than the specified number, the query is successful and the query ends; if the number of returned index metadata is equal to the specified number, the key in the last item of the returned index metadata list is used as the query starting key, and step 112 is executed in a loop.

3. The object index metadata management method according to claim 1, characterized in that In step 12, the index metadata is written, including: Step 121: When writing index metadata, it is directly written to the memory cache, and the write log is recorded to the flash device in an appended manner. Only after the writing of the memory and the flash device is completed, the object storage gateway is notified that the index metadata is written successfully. Step 122: When the index metadata in the memory exceeds 1 GB, the index metadata in the memory is flushed to the backend flash memory device and compressed at the same time to improve the utilization rate of the flash memory device. Step 123: After the index metadata in the memory is flushed to the backend flash memory device, the write log in the flash memory is cleared to release excess storage space.

4. The object index metadata management method according to claim 1, characterized in that In step 13, adjacent index metadata are grouped, including: Mark the adjacent 150MB index metadata as a region. When the database instance is initialized, 5 empty regions are generated. As the storage capacity of index metadata increases, the region exceeding 150MB is split into two regions of the same size. At the same time, as the index metadata is deleted, two adjacent regions smaller than 25MB are merged into one region.

5. The object index metadata management method according to claim 1, characterized in that Step 14 manages a copy of the index metadata, including: When reading index metadata, a replica is randomly selected to complete the reading task. When writing index metadata, only the leader replica is written. The scheduler instance asynchronously synchronizes the index metadata changes to the follower replica. The distribution of the three replicas complies with the principle of maximum isolation domain, where the maximum isolation domain principle means that the maximum network topology distance is maintained between replicas. The priority across centers is greater than that across computer rooms, the priority across computer rooms is greater than that across racks, the priority across racks is greater than that across physical machines, and the priority across physical machines is greater than that across disks. At the same time, a weight attribute is added to the database instance. The weight decreases as the available disk space is consumed. While complying with the maximum isolation domain principle, data replicas tend to be stored in database instances with large weights.

6. The object index metadata management method according to claim 1, characterized in that In step 15, when an error occurs in the leader replica, the scheduler instance initiates an election to select a new leader replica and generates a new follower replica; When an error occurs in a follower replica, the scheduler instance initiates an election to select a new leader replica and generates a new follower replica.

7. The object index metadata management method according to claim 1, characterized in that Step 16 specifically includes: Use heartbeat messages to check whether the database instance is down. When a database instance is down, other database instances cannot sense the heartbeat from the down database for 30 minutes, and then send a database instance down signal to the scheduler instance. After receiving the down signal, the caller instance waits for 30 minutes. If the down database instance does not restore the heartbeat after 30 minutes, the down database instance will be kicked out of the distributed database topology, and all the leader and follower copies on the down database instance will be migrated to normal database instances.

8. An object index metadata management device, characterized in that Including object index metadata management module and object storage gateway management module, Object index metadata management module design distributed database system object index metadata: index metadata entries according to<key,value> The key is the object ID, and the value is the object index metadata. The object index metadata includes the creation time of the object, the size of the object, the storage type of the object, the object's permissions, and the distribution of the object data on the osd. The management of the index metadata involves the database instance and scheduler instance of the distributed database system. Step 11: When managing index metadata, sort and store the object storage index metadata in the lexicographic order of the key; Step 12: Use the database instance to write index metadata in a pure append-only writing mode; Step 13: Group adjacent index metadata through the scheduler instance: mark the region marked by the index metadata according to the preset size; Step 14: Make each region save a copy on each of the three database instances, set one and only one copy as the leader copy, and the other two copies as follower copies; Step 15: The database instance locations of all replicas of the index metadata are stored and managed through the scheduler instance, and the distributed consistency is ensured through the raft protocol; Step 16: Use heartbeat messages to check whether the database instance is down, and perform recovery management on the downed database instance; The object storage gateway management module manages the object storage gateway to read and write index metadata in the distributed database system: Step 21: When the object storage gateway uploads object data, the object storage gateway management module stores the object data in the osd and records the object creation time, object size, object storage type, object permissions, and object data osd distribution in the value of the index metadata. Step 22: When the object storage gateway downloads the object data, the object storage gateway management module reads the value of the index metadata from the distributed database system, obtains the osd distribution of the object data, and then requests the osd to obtain the object data.

Images