A metadata storage method and device based on a distributed key-value database

Abstract

The application discloses a metadata storage method and device based on a distributed key-value database, and relates to the field of data processing.The method comprises the following steps: determining the metadata of an object stored in a bucket, and determining the preset type and key-value pair of the metadata; based on the determined preset type and key-value pair of the metadata, saving the metadata into a corresponding hash table and an ordered list; each bucket in the distributed storage system stores metadata through at least one hash table and at least one ordered list, the hash table is used for storing metadata, and the ordered list is used for storing retrieval information of the metadata.The application reduces the complexity of I / O operation, improves the single-bucket storage scale while ensuring efficiency, and realizes efficient, low-space-usage metadata retrieval applicable to multiple ordered lists.

Description

Technical Field

[0001] This invention relates to the field of data processing, and specifically to a method and apparatus for storing metadata based on a distributed key-value database. Background Technology

[0002] In distributed object storage systems, each bucket maintains an index table that stores the mapping relationship between the bucket and the metadata of all objects within it. When a user accesses an object within a bucket, the specific data of the object is retrieved through the bucket index. However, when the number of objects stored in a bucket is too large, an excessively large index can cause performance and reliability issues. To address the limitation of single-bucket object size, existing technologies employ dynamic sharding mechanisms by creating multiple index tables. However, during bucket sharding, the object metadata within the bucket is rebalanced. If a bucket stores a large number of objects, the data rebalancing operation during sharding becomes extremely time-consuming, blocking front-end writes and resulting in a very unfriendly user experience.

[0003] Therefore, how to solve the problems of limited single-bucket object storage size and business blockage caused by sharding mechanism in distributed object storage systems is an important issue that the industry urgently needs to address. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a metadata storage method and apparatus based on a distributed key-value database to solve the problems of limited single-bucket object storage scale and business blocking caused by sharding mechanisms in distributed object storage systems.

[0005] According to a first aspect, embodiments of the present invention provide a metadata storage method based on a distributed key-value database, the method being applied to a server, the server having a distributed object storage system installed, the method comprising:

[0006] Determine the metadata of the objects stored in the bucket, and determine the preset type and key-value pairs of the metadata;

[0007] Based on the predetermined type and key-value pairs of the metadata, the metadata is saved to the corresponding hash table and ordered list; each bucket in the distributed storage system stores metadata through at least one hash table and at least one ordered list, the hash table is used to store metadata, and the ordered list is used to store metadata retrieval information.

[0008] In conjunction with the first aspect, in the first embodiment of the first aspect, the preset type includes a first, a second, a third, and a fourth type, wherein the metadata of the first type is basic metadata, the metadata of the second type is object attribute metadata, the metadata of the third type is index metadata, and the metadata of the fourth type is index order metadata.

[0009] The step of saving the metadata to the corresponding hash table and ordered list based on the predetermined type and key-value pairs of the metadata specifically includes:

[0010] Based on the determined preset type, the metadata is stored in the container group corresponding to the preset type; each preset type of metadata corresponds to one container group, each container group corresponds to at least one hash slot, and the number of hash slots corresponding to each container group is equal to each other;

[0011] Based on the key-value pairs of the metadata, determine the CRC16 value of the metadata; based on the total number of hash slots and the determined CRC16 value, determine the hash slot corresponding to the metadata.

[0012] The metadata is mapped and stored in the hash table corresponding to the determined hash slot; each hash table corresponds to at least one hash slot;

[0013] Assign a corresponding score value to the metadata, determine the ordered list corresponding to the metadata based on the assigned score value, and store the metadata in the corresponding ordered list; the metadata items in the ordered list are sorted in order according to the score value.

[0014] In conjunction with the first embodiment of the first aspect, in the second embodiment of the first aspect, the number of hash tables is determined based on a preset number and the total number of hash slots, wherein the number of hash tables exceeds the preset number and is a factor of the total number.

[0015] In conjunction with the first aspect, in the third embodiment of the first aspect, the method further includes the following steps:

[0016] Determine the client's metadata retrieval request; the metadata retrieval request contains retrieval information, including the index order and the number of index elements for each of the ordered lists;

[0017] Based on the retrieval information, the corresponding metadata is retrieved from each of the ordered lists;

[0018] The metadata is stored in a preset map table. Based on the preset map table, the metadata stored in the preset map table is sorted sequentially. The preset map table, which has completed the storage and sorting of the metadata, is then returned to the client.

[0019] In conjunction with the first aspect, in the fourth embodiment of the first aspect, the method further includes the following steps:

[0020] Determine the client's metadata retrieval request; the metadata retrieval request contains retrieval information, including baseline metadata and the number of index elements;

[0021] Determine the storage location of the baseline metadata in each of the ordered lists;

[0022] Based on the retrieval information and the storage location, the corresponding metadata is retrieved from each of the ordered lists;

[0023] The metadata is stored in a preset map table. Based on the preset map table, the metadata stored in the preset map table is sorted sequentially. The preset map table, which has completed the storage and sorting of the metadata, is then returned to the client.

[0024] In conjunction with the third or fourth embodiment of the first aspect, in the fifth embodiment of the first aspect, the step of storing the metadata in a preset map table, sorting the metadata stored in the preset map table according to the preset map table, and returning the preset map table with the metadata stored and sorted to the client further includes:

[0025] If it is determined that the preset map table has reached its storage limit and there are unstored metadata, then based on the preset map table, the map table sorting value of the unstored metadata and the metadata at the end of the preset map table is determined.

[0026] If the sort value of the map table corresponding to the last metadata exceeds the sort value of the map table corresponding to the metadata that has not been stored, delete the last metadata and store the metadata that has not been stored into the preset map table.

[0027] According to a second aspect, embodiments of the present invention also provide a metadata storage device based on a distributed key-value database, the device being applied to a server, the server having a distributed object storage system installed, the device comprising:

[0028] The first determining module is used to determine the metadata of the objects stored in the bucket, and to determine the preset type and key-value pairs of the metadata;

[0029] The data storage module is used to save the metadata to the corresponding hash table and ordered list based on the predetermined type and key-value pairs of the metadata; each bucket in the distributed storage system stores metadata through at least one hash table and at least one ordered list, the hash table is used to store metadata, and the ordered list is used to store metadata retrieval information.

[0030] According to a third aspect, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the metadata storage method based on a distributed key-value database as described above.

[0031] According to a fourth aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the metadata storage method based on a distributed key-value database as described above.

[0032] According to a fifth aspect, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the metadata storage method based on a distributed key-value database as described above.

[0033] The metadata storage method and apparatus based on a distributed key-value database provided by this invention stores the metadata of objects in the form of key-value pairs in a distributed key-value cluster. It adopts a table partitioning approach, using multiple hash tables and multiple ordered lists to carry the metadata of objects within a bucket. Storing metadata through hash tables reduces the complexity of I / O operations, increasing the storage scale of a single bucket while ensuring efficiency. Storing metadata through ordered lists provides interfaces for adding, deleting, modifying, and querying object metadata. Without increasing storage costs, it achieves high-efficiency, low-space-utilization metadata retrieval applicable to multiple ordered lists. Attached Figure Description

[0034] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0035] Figure 1 A flowchart illustrating the metadata storage method based on a distributed key-value database provided by the present invention is shown.

[0036] Figure 2 The flowchart of step S20 in the metadata storage method based on a distributed key-value database provided by the present invention is shown.

[0037] Figure 3 A schematic diagram of the structure of the metadata storage device based on a distributed key-value database provided by the present invention is shown.

[0038] Figure 4 A schematic diagram of the structure of the electronic device provided by the present invention is shown. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Currently, distributed object storage systems based on distributed storage architecture have become a preferred solution for cloud computing, with file storage offering advantages such as shareability and low cost. Object metadata in a distributed object storage system is stored in two parts: one part is the object's index metadata, called omap (object map), which is used to store key-value pair map data. omap objects play a crucial role in distributed storage; their performance directly impacts the cluster's storage performance in the file and object services provided by distributed storage. omap is stored in an independent key-value storage system outside the local file system; when using Filestore, this is LevelDB, and when using Bluestore, it's RocksDB. The other part is the object's extended attributes, called xattr, which typically stores object version information, etc. xattr is stored in the RADOS (Reliable Autonomic Distributed Object Store) objects within the bucket. RADOS objects are stored within the local file system, and their size is limited by the file system, thus limiting the number of objects they can support. This results in the need to use two separate I / O paths to read the relevant data when reading the object's metadata, corresponding to the local file system and the key-value storage system respectively.

[0041] In distributed object storage systems, each bucket maintains an index table that stores the mapping relationship between the bucket and the metadata of all objects within it. When a user accesses an object within a bucket, the specific data of the object is retrieved through the bucket index. However, when the number of objects stored in a bucket is too large, an excessively large index can cause performance and reliability issues. To address the limitation of single-bucket object size, existing technologies employ dynamic sharding mechanisms. This involves using multiple RADOS objects to maintain the bucket index, creating multiple index tables to solve the problem of a single index object becoming too large to meet business needs such as data growth. However, bucket sharding involves rebalancing the object metadata within the bucket, recalculating and organizing the index data from the old RADOS objects and migrating it to new RADOS objects. Therefore, if a bucket stores a large number of objects, the data rebalancing operation during sharding can be very time-consuming, blocking front-end writes and resulting in a very unfriendly user experience.

[0042] To address the above issues, the following will be discussed in conjunction with... Figure 1 This invention describes a metadata storage method based on a distributed key-value database. This method aims to solve the problems of limited single-bucket object storage size and business blocking caused by sharding mechanisms in native distributed object storage. The method is applied to a server-side application with a distributed object storage system installed. The method includes:

[0043] S10. Determine the metadata of the objects stored in the bucket, and determine the preset type and key-value pairs of the metadata. Specifically, determine the metadata stored in each bucket on the server, as well as the preset type and key-value pairs of the metadata.

[0044] In this application, the preset types include the first, second, third, and fourth types. The first type is basic metadata (object metadata), which stores basic information about the latest version of the object, including size, instance, and last version. The second type is object attribute metadata (xattrs), which is the object's own attribute information. The third type is index metadata (omap), which stores the bucket object index information, i.e., omap information. The fourth type is index order metadata (omap order), which stores the order list of all omaps for objects within the bucket.

[0045] S20. Based on the predetermined type and key-value pairs of the metadata, the metadata is saved to the corresponding hash table and sorted list (zset). In this application, each bucket in the distributed storage system stores metadata through at least one hash table and at least one sorted list. Specifically, in this application, the hash table is used to store metadata, and the sorted list is used to store metadata retrieval information.

[0046] In this application, a single-bucket, multi-table metadata storage model is constructed based on a distributed key-value data structure. The native distributed object storage system only includes an object gateway and backend data storage. This invention adds a dedicated distributed key-value module to store object metadata, achieving separate storage of object data and metadata. Simultaneously, this application reorganizes the object metadata, removing redundant data types and reclassifying the metadata into four types. Since object operations do not simultaneously manipulate dozens of metadata types found in the native distributed object storage system, the granularity of object metadata can be reduced, making metadata read / write operations more flexible and efficient.

[0047] For the four preset types of metadata in the same bucket, this method uses at least one hash table and at least one ordered list for storage. When storing metadata, it also ensures that the metadata is evenly distributed within the container. How to achieve the even distribution of metadata in the container will be explained below.

[0048] In this application, the metadata of objects is stored in a distributed key-value cluster in the form of key-value pairs. The method adopts a table partitioning approach, using multiple hash tables to carry the metadata of objects in a bucket. Each hash table can store up to 4.2 billion key-value pairs. Multiple hash tables can easily support the metadata of tens of billions of objects. By uniformly storing the metadata of objects in a distributed key-value hash table, the time complexity of its read and write operations is 0 or 1, reducing the complexity of I / O operations. Furthermore, it increases the storage scale of a single bucket while ensuring efficiency, and eliminates the need to load RADOS object data from the local file system.

[0049] The metadata storage method based on a distributed key-value database provided in this application stores the metadata of objects in the form of key-value pairs in a distributed key-value cluster. It adopts a table partitioning approach, using multiple hash tables and multiple ordered lists to carry the metadata of objects within a bucket. Storing metadata through hash tables reduces the complexity of I / O operations, increasing the storage scale of a single bucket while ensuring efficiency. Storing metadata through ordered lists provides interfaces for adding, deleting, modifying, and querying object metadata. Without increasing storage costs, it achieves high-efficiency, low-space-utilization metadata retrieval applicable to multiple ordered lists.

[0050] The following is combined Figure 2 The metadata storage method based on a distributed key-value database of the present invention is described below, and step S20 specifically includes:

[0051] S21. Based on the determined preset type, the metadata is stored in the container group corresponding to the preset type. In this application, the metadata of each preset type corresponds to a container group, and each container group corresponds to at least one hash slot. Moreover, the number of hash slots corresponding to each container group is equal to that of each container group, that is, the number of hash slots corresponding to each container group is consistent.

[0052] S22. Based on the key-value pairs of the metadata, determine the CRC16 value of the metadata, and based on the total number of hash slots and the determined CRC16 value, determine the hash slot corresponding to the metadata.

[0053] Specifically, the CRC16 value of the metadata is first calculated based on the key of the metadata, and then the hash slot corresponding to the metadata is obtained by taking the modulo of the total number of hash slots.

[0054] S23. Map and store the metadata to the hash table corresponding to the determined hash slot. In this method, each hash table corresponds to at least one hash slot.

[0055] To ensure that metadata is evenly distributed across the hash tables in the container group, this application requires pre-assigning slot partitions to each hash table. For each container group, the total number of slots in the entire container group is 16384. The number of hash tables is configured to exceed a preset number (e.g., 5) and be a factor of 16384. Then, these 16384 slots are evenly distributed among the hash tables in the container group.

[0056] Therefore, the hash slot corresponding to the metadata can be determined in the following way:

[0057] HASH_SLOT=CRC16(key)mod 16384

[0058] The HASH_SLOT algorithm can be used to map the metadata of bucket objects to the corresponding hash tables, so that the metadata is evenly distributed in the corresponding container group.

[0059] S24. Assign a corresponding score to the metadata. Based on the assigned score, determine the ordered list corresponding to the metadata and store the metadata in the corresponding ordered list. In this method, the metadata items in the ordered list are sorted in order according to the score. For example, based on the score assigned / assigned to the metadata, the metadata in a single ordered list is sorted in order from low to high score.

[0060] In this embodiment, a score value is assigned to the key of the metadata.

[0061] Compared to the process of uploading, downloading, and deleting individual object metadata operations, the business of ordered object lists is undoubtedly more complex. This application uses multiple ordered lists to maintain the ordered metadata (omap key) of objects within the bucket. These omap keys are ordered within a single ordered list, but in order to achieve a uniform distribution of metadata, omap keys across multiple ordered lists cannot be arranged in an ordered manner. To solve the problem of low data retrieval efficiency, existing technologies often separate and store metadata on high-performance disks, such as solid-state drives (SSDs). However, although the read and write efficiency of objects is improved, the required storage cost is also significantly increased.

[0062] In some possible embodiments, the method further includes the following steps, designed to achieve efficient, low-space-utilization retrieval of metadata across multiple ordered lists without increasing storage costs:

[0063] A10. Determine the client's metadata retrieval request. In these embodiments, the metadata retrieval request includes retrieval information, such as the index order and the number of index elements for each ordered list.

[0064] In this embodiment, the index order and the number of index elements of different ordered lists are consistent. For example, the metadata retrieval request retrieves the first 10 elements of each ordered list. That is, the index order is to perform a continuous sequential retrieval starting from the first element, and the number of index elements is 10.

[0065] A20. Based on the retrieved information, extract the corresponding metadata from each ordered list. For example, extract the first, last, and n consecutive elements (metadata) from each ordered list, including the first, last, and i-th to i+n-1th elements.

[0066] A30. Store the metadata in a preset map table. Based on the preset map table, sort the metadata stored in the preset map table in order. Return the preset map table with completed metadata storage and sorting to the client.

[0067] Because map tables can sort elements alphabetically, when metadata is stored in a preset map table, the preset map table can sort the stored metadata sequentially. The preset map table is essentially a list that reorders the retrieved metadata, allowing users to perform efficient, space-efficient metadata retrieval across multiple ordered lists.

[0068] In some possible embodiments, the method further includes the following steps:

[0069] A40. Determine the client's metadata retrieval request. Similarly, in these embodiments, the metadata retrieval request also includes retrieval information, similar to step A10. The difference is that the retrieval information includes base metadata and the number of index elements, etc. The base metadata is the start key.

[0070] A50. Determine the storage location of the baseline metadata in each ordered list.

[0071] It should be noted that the start key is metadata that is definitively present within the bucket. This means that the start key metadata will be stored in one and only one ordered list. Therefore, step A50 will determine the specific storage location, or storage order, of the start key metadata within its corresponding ordered list, which is the element number / sequence number in that ordered list. For ordered lists that do not store start key metadata, the storage location, or storage order, of the start key metadata if it were stored in these ordered lists will be determined first. For example, based on the letters of the start key metadata and the letters of metadata already stored in other ordered lists, the pre-storage location of the start key metadata in the ordered lists that do not store start key metadata will be determined.

[0072] A60. Based on the retrieved information and storage location, including the actual storage location and the pre-storage location, retrieve the corresponding metadata from each ordered list. For example, starting from the pre-storage location, retrieve n consecutive elements (metadata) from the ordered list that has not stored the baseline metadata.

[0073] A70. Store the metadata in a preset map table. Based on the preset map table, sort the metadata stored in the preset map table in order. Return the preset map table with completed metadata storage and sorting to the client.

[0074] Because map tables can sort elements alphabetically, when metadata is stored in a preset map table, the preset map table can sort the stored metadata sequentially. The preset map table is essentially a list that reorders the retrieved metadata, allowing users to perform efficient, space-efficient metadata retrieval across multiple ordered lists.

[0075] The preset map table has a storage limit, which can be set by the user, for example, the storage limit is N elements. When the storage limit of the preset map table is not reached and metadata needs to be stored, the metadata will be stored directly. When the preset map table reaches the storage limit, a corresponding retrieval algorithm needs to be designed. Therefore, steps A30 and A70 will also include:

[0076] A80. Determine that the preset map table has reached its storage limit and there is unstored metadata, for example, the storage limit N is less than the total number of metadata retrieved S. Based on the preset map table, determine the map table sorting value of the unstored metadata and the metadata at the end of the preset map table. In this embodiment, the sorting value can be understood as storage location / storage order.

[0077] A90. If the sort value of the map table corresponding to the last / last metadata element exceeds the sort value of the map table corresponding to the metadata not yet stored, delete the last metadata element and store the metadata not yet stored in the preset map table. This method determines whether each piece of metadata not yet stored needs to be stored in the preset map table. It should be noted that after deleting the last element and storing new metadata in the preset map table, all stored metadata in the preset map table will be re-sorted according to their map table sort values ​​to ensure that the sort value of the last metadata element in the preset map table is the maximum value among all element sort values.

[0078] The metadata storage device based on a distributed key-value database provided by the present invention will be described below. The metadata storage device based on a distributed key-value database described below can be referred to in correspondence with the metadata storage method based on a distributed key-value database described above.

[0079] The following is combined Figure 3 The present invention describes a metadata storage device based on a distributed key-value database, which aims to solve the problems of limited single-bucket object storage size and business blocking caused by sharding mechanisms in native distributed object storage. This device is applied to a server-side application, where a distributed object storage system is installed. The device includes:

[0080] The first determining module 10 is used to determine the metadata of objects stored in a bucket, and to determine the preset type and key-value pairs of the metadata. Specifically, it determines the metadata stored in each bucket on the server, as well as the preset type and key-value pairs of the metadata.

[0081] In this application, the preset types include the first, second, third, and fourth types. The first type is basic metadata (object metadata), which stores basic information about the latest version of the object, including size, instance, and last version. The second type is object attribute metadata (xattrs), which is the object's own attribute information. The third type is index metadata (omap), which stores the bucket object index information, i.e., omap information. The fourth type is index order metadata (omap order), which stores the order list of all omaps for objects within the bucket.

[0082] The data storage module 20 is used to save metadata to corresponding hash tables and sorted lists (zsets) based on the predetermined type and key-value pairs of the metadata. In this application, each bucket in the distributed storage system stores metadata through at least one hash table and at least one sorted list. Specifically, in this application, the hash table is used to store metadata, and the sorted list is used to store metadata retrieval information.

[0083] In this application, a single-bucket, multi-table metadata storage model is constructed based on a distributed key-value data structure. The native distributed object storage system only includes an object gateway and backend data storage. This invention adds a dedicated distributed key-value module to store object metadata, achieving separate storage of object data and metadata. Simultaneously, this application reorganizes the object metadata, removing redundant data types and reclassifying the metadata into four types. Since object operations do not simultaneously manipulate dozens of metadata types found in the native distributed object storage system, the granularity of object metadata can be reduced, making metadata read / write operations more flexible and efficient.

[0084] For the four preset types of metadata in the same bucket, the device uses at least one hash table and at least one ordered list for storage, and ensures that the metadata is evenly distributed within the container when storing it. How the metadata is evenly distributed within the container will be explained below.

[0085] In this application, the metadata of objects is stored in a distributed key-value cluster in the form of key-value pairs. The device adopts a table partitioning approach, using multiple hash tables to carry the metadata of objects in a bucket. Each hash table can store up to 4.2 billion key-value pairs. Multiple hash tables can easily support the metadata of tens of billions of objects. By uniformly storing the metadata of objects in a distributed key-value hash table, the time complexity of its read and write operations is 0 or 1, reducing the complexity of I / O operations. Furthermore, it increases the storage scale of a single bucket while ensuring efficiency, and eliminates the need to load RADOS object data from the local file system.

[0086] The metadata storage device based on a distributed key-value database provided in this application stores the metadata of objects in the form of key-value pairs in a distributed key-value cluster. It adopts a table partitioning approach, using multiple hash tables and multiple ordered lists to carry the metadata of objects within a bucket. Storing metadata through hash tables reduces the complexity of I / O operations, increasing the storage scale of a single bucket while ensuring efficiency. Storing metadata through ordered lists provides interfaces for adding, deleting, modifying, and querying object metadata. Without increasing storage costs, it achieves high-efficiency, low-space-utilization metadata retrieval applicable to multiple ordered lists.

[0087] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a metadata storage method based on a distributed key-value database, the method including:

[0088] Determine the metadata of the objects stored in the bucket, and determine the preset type and key-value pairs of the metadata;

[0089] Based on the predetermined type and key-value pairs of the metadata, the metadata is saved to the corresponding hash table and ordered list; each bucket in the distributed storage system stores metadata through at least one hash table and at least one ordered list, the hash table is used to store metadata, and the ordered list is used to store metadata retrieval information.

[0090] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the metadata storage method based on a distributed key-value database provided by the above methods, the method comprising:

[0092] Determine the metadata of the objects stored in the bucket, and determine the preset type and key-value pairs of the metadata;

[0093] Based on the predetermined type and key-value pairs of the metadata, the metadata is saved to the corresponding hash table and ordered list; each bucket in the distributed storage system stores metadata through at least one hash table and at least one ordered list, the hash table is used to store metadata, and the ordered list is used to store metadata retrieval information.

[0094] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the metadata storage method based on a distributed key-value database provided by the methods described above, the method comprising:

[0095] Determine the metadata of the objects stored in the bucket, and determine the preset type and key-value pairs of the metadata;

[0096] Based on the predetermined type and key-value pairs of the metadata, the metadata is saved to the corresponding hash table and ordered list; each bucket in the distributed storage system stores metadata through at least one hash table and at least one ordered list, the hash table is used to store metadata, and the ordered list is used to store metadata retrieval information.

[0097] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A metadata storage method based on a distributed key-value database, characterized in that, The method is applied to a server, which has a distributed object storage system installed. The method includes: Determine the metadata of the objects stored in the bucket, and determine the preset type and key-value pairs of the metadata; Based on the predetermined preset type and key-value pairs of the metadata, the metadata is saved to the corresponding hash table and ordered list, including: storing the metadata in a container group corresponding to the predetermined preset type; each preset type of metadata corresponds to one container group, each container group corresponds to at least one hash slot, and the number of hash slots corresponding to each container group is equal; determining the CRC16 value of the metadata based on the key-value pairs of the metadata, and mapping the metadata of the bucket object to the corresponding hash table using the HASH_SLOT algorithm based on the total number of hash slots and the determined CRC16 value to determine the hash slot corresponding to the metadata; mapping and storing the metadata to the hash table corresponding to the determined hash slot; each hash table corresponds to at least one hash slot; assigning a corresponding score value to the metadata, determining the ordered list corresponding to the metadata based on the assigned score value, and storing the metadata in the corresponding ordered list; the metadata items in the ordered list are sorted in order according to the score value. In the distributed storage system, each bucket stores metadata through at least one hash table and at least one ordered list. The hash table is used to store metadata, and the ordered list is used to store metadata retrieval information.

2. The metadata storage method based on a distributed key-value database according to claim 1, characterized in that, The preset types include a first type, a second type, a third type, and a fourth type. The metadata of the first type is basic metadata, the metadata of the second type is object attribute metadata, the metadata of the third type is index metadata, and the metadata of the fourth type is index order metadata.

3. The metadata storage method based on a distributed key-value database according to claim 2, characterized in that, The number of hash tables is determined based on a preset number and the total number of hash slots, wherein the number of hash tables exceeds the preset number and is a factor of the total number.

4. The metadata storage method based on a distributed key-value database according to claim 1, characterized in that, The method also includes the following steps: Determine the client's metadata retrieval request; the metadata retrieval request contains retrieval information, including the index order and the number of index elements for each of the ordered lists; Based on the retrieval information, the corresponding metadata is retrieved from each of the ordered lists; The metadata is stored in a preset map table. Based on the preset map table, the metadata stored in the preset map table is sorted sequentially. The preset map table, which has completed the storage and sorting of the metadata, is then returned to the client.

5. The metadata storage method based on a distributed key-value database according to claim 1, characterized in that, The method also includes the following steps: Determine the client's metadata retrieval request; the metadata retrieval request contains retrieval information, including baseline metadata and the number of index elements; Determine the storage location of the baseline metadata in each of the ordered lists; Based on the retrieval information and the storage location, the corresponding metadata is retrieved from each of the ordered lists; The metadata is stored in a preset map table. Based on the preset map table, the metadata stored in the preset map table is sorted sequentially. The preset map table, which has completed the storage and sorting of the metadata, is then returned to the client.

6. The metadata storage method based on a distributed key-value database according to claim 4 or 5, characterized in that, The step of storing the metadata in a preset map table, sorting the metadata stored in the preset map table according to the preset map table, and returning the preset map table with the metadata stored and sorted to the client further includes: If it is determined that the preset map table has reached its storage limit and there are unstored metadata, then based on the preset map table, the map table sorting value of the unstored metadata and the metadata at the end of the preset map table is determined. If the sort value of the map table corresponding to the last metadata exceeds the sort value of the map table corresponding to the metadata that has not been stored, delete the last metadata and store the metadata that has not been stored into the preset map table.

7. A metadata storage device based on a distributed key-value database, characterized in that, The device is used on a server, the server having a distributed object storage system installed, and the device includes: The first determining module is used to determine the metadata of the objects stored in the bucket, and to determine the preset type and key-value pairs of the metadata; A data storage module is used to save the metadata to a corresponding hash table and an ordered list based on a predetermined type and key-value pairs of the metadata. This includes: storing the metadata in a container group corresponding to the predetermined type; each type of metadata corresponds to one container group, each container group corresponds to at least one hash slot, and the number of hash slots corresponding to each container group is equal; determining the CRC16 value of the metadata based on the key-value pairs; mapping the metadata of the bucket object to the corresponding hash table using the HASH_SLOT algorithm based on the total number of hash slots and the determined CRC16 value, and determining the hash slot corresponding to the metadata; mapping and storing the metadata to the hash table corresponding to the determined hash slot; each hash table corresponds to at least one hash slot; assigning a corresponding score to the metadata; determining the ordered list corresponding to the metadata based on the assigned score, and storing the metadata in the corresponding ordered list; the metadata items in the ordered list are sorted according to the score. In the distributed storage system, each bucket stores metadata through at least one hash table and at least one ordered list. The hash table is used to store metadata, and the ordered list is used to store metadata retrieval information.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the metadata storage method based on a distributed key-value database as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the metadata storage method based on a distributed key-value database as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the metadata storage method based on a distributed key-value database as described in any one of claims 1 to 6.

Images