Object metadata storage method and device

Abstract

The present application provides an object metadata storage method and device, which are applied to the storage nodes included in the object storage system. The method includes: determining the target group to which the target object metadata to be stored in the target bucket belongs, wherein the target bucket includes N groups for storing object metadata, each group corresponding to a storage space on a different storage medium; and storing the target object metadata in the storage space corresponding to the target group. In the present application, the object metadata in the same bucket can be stored across storage media and is no longer limited by the storage capacity of a single storage medium. Therefore, the object metadata storage capacity of a single bucket can be increased.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a method and device for storing object metadata. Background Art

[0002] With the explosive growth of unstructured data, object storage has emerged. Object storage, with its flat, two-tier architecture, allows for quick search of required objects, making it ideal for storing massive amounts of unstructured data. It can be used to store massive amounts of image data, video data for medical insurance monitoring, and more.

[0003] An object is a collection of file data and its related attribute information, including object name, object data and object metadata. In order to facilitate the management of massive unstructured data (ie, objects), object data and object metadata are usually stored separately.

[0004] Currently, object metadata is commonly stored in buckets (containers for storing objects). Object metadata for a single bucket is typically stored on fixed storage media, and multiple buckets can share the same storage media. However, the storage capacity of a single storage media limits the amount of object metadata a single bucket can store. Summary of the Invention

[0005] In view of this, the present application proposes an object metadata storage method and device to increase the object metadata storage capacity of a single bucket.

[0006] To achieve the above application objectives, this application provides the following technical solutions:

[0007] In a first aspect, the present application provides an object metadata storage method, which is applied to a storage node included in an object storage system, and the method includes:

[0008] Determine a target group to which the target object metadata to be stored in the target bucket belongs, wherein the target bucket includes N groups for storing object metadata, each group corresponding to storage space on a different storage medium;

[0009] The target object metadata is stored in a storage space corresponding to the target group.

[0010] Optionally, determining the target group to which the target object metadata to be stored in the target bucket belongs includes:

[0011] Determine the target virtual node to which the target object metadata belongs;

[0012] From the recorded correspondence between the identifiers of the virtual nodes and the identifiers of the groups, the group corresponding to the target virtual node is searched as the target group.

[0013] Optionally, determining the target virtual node to which the target object metadata belongs includes:

[0014] Obtaining object information of a target object to which the target object metadata belongs, the object information including the object name of the target object and the bucket name of the target bucket;

[0015] Performing a hash operation on the object information of the target object to obtain a corresponding hash value;

[0016] The target virtual node to which the target object metadata belongs is determined according to the hash value and the total number of virtual nodes.

[0017] Optionally, if the target bucket supports multiple versions, the object information further includes the version number of the target object.

[0018] Optionally, each group is allowed to store the same maximum number of object metadata, and N is determined based on the maximum number of object metadata expected to be stored in the target bucket and the maximum number of object metadata allowed to be stored in a single group.

[0019] In a second aspect, the present application provides an object metadata storage device, which is applied to a storage node included in an object storage system, and the device includes:

[0020] a determining unit, configured to determine a target group to which the target object metadata to be stored in the target bucket belongs, wherein the target bucket includes N groups for storing object metadata, each group corresponding to a storage space on a different storage medium;

[0021] A storage unit is used to store the target object metadata in a storage space corresponding to the target group.

[0022] Optionally, the determining unit determines the target group to which the target object metadata to be stored in the target bucket belongs, including:

[0023] Determine the target virtual node to which the target object metadata belongs;

[0024] From the recorded correspondence between the identifiers of the virtual nodes and the identifiers of the groups, the group corresponding to the target virtual node is searched as the target group.

[0025] Optionally, the determining unit determines the target virtual node to which the target object metadata belongs, including:

[0026] Obtaining object information of a target object to which the target object metadata belongs, the object information including the object name of the target object and the bucket name of the target bucket;

[0027] Performing a hash operation on the object information of the target object to obtain a corresponding hash value;

[0028] The target virtual node to which the target object metadata belongs is determined according to the hash value and the total number of virtual nodes.

[0029] Optionally, if the target bucket supports multiple versions, the object information further includes the version number of the target object.

[0030] Optionally, each group is allowed to store the same maximum number of object metadata, and N is determined based on the maximum number of object metadata expected to be stored in the target bucket and the maximum number of object metadata allowed to be stored in a single group.

[0031] It can be seen from the above description that in the embodiment of the present application, the object metadata in the same bucket can be stored across storage media and is no longer limited by the storage capacity of a single storage medium. Therefore, the object metadata storage capacity of a single bucket can be increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0033] Figure 1 This is a flow chart of a method for storing object metadata shown in an embodiment of the present application;

[0034] Figure 2 This is an implementation process of step 101 shown in an embodiment of the present application;

[0035] Figure 3 This is an implementation process of step 201 shown in an embodiment of the present application;

[0036] Figure 4 This is a schematic diagram of an object metadata storage device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0037] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0038] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. The singular forms "a", "the" and "the" used in the embodiments of the present application are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0039] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the embodiments of the present application, the negotiation information may also be referred to as the second information, and similarly, the second information may also be referred to as the negotiation information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0040] In order to make the purpose, technical solutions and advantages of this application more clear, the present application is described in detail below with reference to the accompanying drawings and specific embodiments:

[0041] See also Figure 1 , is a flow chart of an object metadata storage method shown in an embodiment of the present application, which is applied to storage nodes included in an object storage system, such as a storage server.

[0042] like Figure 1 As shown, the process may include the following steps:

[0043] Step 101: Determine the target group to which the target object metadata to be stored in the target bucket belongs.

[0044] The node can parse the received write request to obtain the object to be stored (to be written) and the bucket to store the object.

[0045] Here, the object to be stored is referred to as the target object; the object metadata for the target object is referred to as the target object metadata; and the bucket in which the target object is to be stored is referred to as the target bucket. It should be understood that the names target object, target object metadata, and target bucket are simply for ease of distinction and are not intended to be limiting.

[0046] In this embodiment, a target bucket includes N groups for storing object metadata. Each group corresponds to storage space on a different storage medium. Object metadata stored in the same target bucket will ultimately be stored on different storage media. Because object metadata belonging to the same bucket can be stored across storage media, the object metadata storage capacity of a single bucket is no longer limited.

[0047] In this step, the process of determining the group to which the target object metadata belongs (the group to be written) is described below and will not be repeated here. The group to which the target object metadata belongs is referred to as the target group. It should be understood that the term "target group" is used for ease of distinction and is not intended to be limiting.

[0048] It should also be noted that the number of groups N can be determined based on the maximum number of object metadata that the target bucket is expected to store and the maximum number of object metadata that can be stored in a single group. For example, if the maximum number of object metadata that can be stored in the target bucket is 100 billion and the maximum number of object metadata that can be stored in a single group is 100 million, the number of groups is 1000.

[0049] Step 102: Store the target object metadata into the storage space corresponding to the target group.

[0050] So far, completed Figure 1 The process shown.

[0051] pass Figure 1 As can be seen from the process, in the embodiment of the present application, object metadata in the same bucket can be stored across storage media and is no longer limited by the storage capacity of a single storage medium. Therefore, the object metadata storage capacity of a single bucket can be increased.

[0052] The following describes the process of determining the target group to which the target object metadata belongs in step 101. Figure 2 , which is an implementation process of step 101 shown in an embodiment of the present application.

[0053] like Figure 2 As shown, the process may include the following steps:

[0054] Step 201: Determine the target virtual node to which the target object metadata belongs.

[0055] In the embodiment of the present application, a virtual layer is established between the bucket and the group, and the virtual layer is composed of a number of virtual nodes. The number of virtual nodes can be determined according to the scale of the storage system and the number of groups.

[0056] As an example, the number of virtual nodes may be a multiple of the number of groups. Of course, the number of virtual nodes may be determined in other ways, but generally the number of virtual nodes is much larger than the number of groups.

[0057] In this step, the virtual node to which the target object metadata belongs needs to be determined from multiple virtual nodes. Here, the virtual node to which the target object metadata belongs is referred to as the target virtual node. It should be understood that the term "target virtual node" is used for ease of distinction and is not intended to be limiting.

[0058] The specific process of determining the virtual node to which the target object metadata belongs is described below and will not be repeated here.

[0059] Step 202: Search for a group corresponding to the target virtual node from the recorded correspondence between the identifiers of the virtual nodes and the identifiers of the groups, and use it as the target group.

[0060] In this embodiment of the present application, a correspondence between virtual nodes and groups can be pre-established based on the load balancing principle. For example, if there are 20 virtual nodes, designated as virtual nodes 1 to virtual nodes 20, and 4 groups, designated as groups 1 to 4, then each group corresponds to 5 virtual nodes based on the load balancing principle, as shown in the following table.

[0061] Virtual node identifier Group identification Virtual nodes 1 to 5 Group 1 Virtual nodes 6 to 10 Group 2 Virtual nodes 11 to 15 Group 3 Virtual nodes 16 to 20 Group 4

[0062] Table 1

[0063] After determining the target virtual node to which the target object metadata belongs in step 201, the group corresponding to the target virtual node can be determined by searching the recorded correspondence between virtual nodes and groups, and the group can be used as the group to which the target object metadata belongs (target group).

[0064] So far, completed Figure 2 The process shown.

[0065] pass Figure 2 The process shown can determine the group to which the metadata of the object to be stored belongs, and then store the object metadata in the storage space corresponding to the group to which it belongs through step 102.

[0066] The following describes the process of determining the target virtual node to which the target object metadata belongs in step 201. Figure 3 , which is an implementation process of step 201 shown in an embodiment of the present application.

[0067] like Figure 3 As shown, the process may include the following steps:

[0068] Step 301: Obtain object information of a target object, where the object information includes the object name of the target object and the bucket name of the target bucket.

[0069] The bucket name is globally unique in the entire object storage system; the object name is unique within the bucket.

[0070] Step 302: Perform a hash operation on the object information of the target object to obtain a corresponding hash value.

[0071] Since bucket names and object names are unique, the hash value obtained by performing a hash operation based on the bucket name + object name is unique. In other words, a hash value that uniquely identifies the metadata of the target object in the target bucket can be obtained. Specifically, it can be expressed as:

[0072] H=hash(BucketName+ObjectName)

[0073] BucketName represents the bucket name; ObjectName represents the object name; hash represents the hash operation; and H represents the hash value.

[0074] Step 303: Determine the target virtual node to which the target object metadata belongs based on the obtained hash value and the total number of virtual nodes.

[0075] As an embodiment, the hash value obtained in step 302 can be used to calculate the remainder of the total number of virtual nodes, and the virtual node to which the target object metadata belongs can be determined based on the remainder. Specifically, it can be expressed as:

[0076] V=H%K

[0077] Where K represents the total number of virtual nodes; V is the remainder result.

[0078] As an example, when the total number of virtual nodes is 20, the remainder result is between 0 and 19, which may correspond to virtual nodes 1 to 20, respectively.

[0079] So far, completed Figure 3 The process shown.

[0080] pass Figure 3 The process shown can evenly distribute the metadata of objects in the bucket to each virtual node. Then, through the correspondence between virtual nodes and groups in step 202 (all virtual nodes are evenly distributed to each group), the metadata of objects in the bucket can be evenly distributed to each group. Each group corresponds to the storage resources of different storage media. Therefore, load balancing can be achieved within the bucket, and the storage resources of each storage medium can be fully utilized.

[0081] Here, it should be noted that if the target bucket supports multiple versions (the target bucket enables multi-version mode), the object information involved in the hash operation in step 302 also includes the version number of the target object, which can be specifically expressed as:

[0082] H=hash(BucketName+ObjectName+VersionId)

[0083] VersionId is the version number of the object.

[0084] This is because, in multi-version mode, multiple versions of the same object need to be stored and accessed. Therefore, a version number is needed to distinguish different versions of the same object within a bucket. In other words, a version of an object in a bucket can be uniquely identified by the bucket name + object name + version number.

[0085] By performing a hash operation on the bucket name + object name + version number and then taking the modulus of the number of virtual nodes, multiple versions of the same object may be assigned to different groups and then stored in different storage media. Concurrent access can be achieved when operating the multi-version object metadata of the object.

[0086] In addition, it should be noted that the embodiment of the present application can add new groups (increase the number of groups in the bucket) by adding storage media to increase the maximum amount of object metadata that the bucket can store, but the number of virtual nodes remains unchanged in this process.

[0087] This is because when reading the stored object metadata, you still need to go through Figure 3 The process shown determines the virtual node to which the object metadata belongs. If the number of virtual nodes changes, it will not be possible to Figure 3 The process shown correctly finds the stored object metadata. Therefore, the embodiment of the present application does not change the number of virtual nodes when dynamically adding groups, but needs to adjust the correspondence between virtual nodes and groups.

[0088] As an example, based on the correspondence shown in Table 1, a group is added, which is recorded as Group 5. Then the correspondence between virtual nodes 1 to 20 and each group can be adjusted as shown in Table 2.

[0089] Virtual node identifier Group identification Virtual nodes 1 to 4 Group 1 Virtual nodes 6 to 9 Group 2 Virtual nodes 11 to 14 Group 3 Virtual nodes 16 to 19 Group 4 Virtual nodes 5, 10, 15, and 20 Group 5

[0090] Table 2

[0091] As can be seen, virtual nodes 5, 10, 15, and 20 have been relocated to the newly added group (Group 5). The metadata of the objects stored in these virtual nodes needs to be migrated from the storage space corresponding to the original group to the storage space corresponding to Group 5. At this point, load balancing is re-established between the virtual nodes and the groups.

[0092] Here, it should also be noted that in this application, each group has an independent database. Therefore, the object metadata stored in each group can be accessed concurrently, eliminating the performance bottleneck of object metadata access.

[0093] The above describes the method provided in the embodiment of the present application. The following describes the device provided in the embodiment of the present application:

[0094] See also Figure 4, an object metadata storage device shown in an embodiment of the present application, is applied to a storage node included in an object storage system, and includes a determining unit 401 and a storage unit 402, wherein:

[0095] A determining unit 401 is configured to determine a target group to which target object metadata to be stored in a target bucket belongs, wherein the target bucket includes N groups for storing object metadata, each group corresponding to storage space on a different storage medium;

[0096] The storage unit 402 is configured to store the target object metadata into a storage space corresponding to the target group.

[0097] As an embodiment, the determining unit 401 determines the target group to which the target object metadata to be stored in the target bucket belongs, including:

[0098] Determine the target virtual node to which the target object metadata belongs;

[0099] From the recorded correspondence between the identifiers of the virtual nodes and the identifiers of the groups, the group corresponding to the target virtual node is searched as the target group.

[0100] As an embodiment, the determining unit 401 determines the target virtual node to which the target object metadata belongs, including:

[0101] Obtaining object information of a target object to which the target object metadata belongs, the object information including the object name of the target object and the bucket name of the target bucket;

[0102] Performing a hash operation on the object information of the target object to obtain a corresponding hash value;

[0103] The target virtual node to which the target object metadata belongs is determined according to the hash value and the total number of virtual nodes.

[0104] As an embodiment, if the target bucket supports multiple versions, the object information further includes the version number of the target object.

[0105] As an embodiment, each group is allowed to store the same maximum number of object metadata, and N is determined based on the maximum number of object metadata expected to be stored in the target bucket and the maximum number of object metadata allowed to be stored in a single group.

[0106] So far, completed Figure 4 Description of the device shown.

[0107] It can be seen from the above description that in the embodiment of the present application, the object metadata in the same bucket can be stored across storage media and is no longer limited by the storage capacity of a single storage medium. Therefore, the object metadata storage capacity of a single bucket can be increased.

[0108] The above description is only a preferred embodiment of the embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for storing object metadata, characterized in that: Applied to a storage node included in an object storage system, the method includes: Determine a target virtual node to which target object metadata to be stored in a target bucket belongs; search for a group corresponding to the target virtual node as a target group from the recorded correspondence between the identifiers of the virtual nodes and the identifiers of the groups; wherein the target bucket includes N groups for storing object metadata, each group corresponds to storage space on a different storage medium, and each group uses an independent database; a virtual layer is established between the target bucket and the groups, the virtual layer consisting of a plurality of virtual nodes, the number of the virtual nodes being greater than the number of the groups; the correspondence between the identifiers of the virtual nodes and the identifiers of the groups can be dynamically adjusted, and the number of the virtual nodes remains unchanged; The target object metadata is stored in a storage space corresponding to the target group.

2. The method according to claim 1, wherein The determining the target virtual node to which the target object metadata belongs includes: Obtaining object information of a target object to which the target object metadata belongs, the object information including the object name of the target object and the bucket name of the target bucket; Performing a hash operation on the object information of the target object to obtain a corresponding hash value; The target virtual node to which the target object metadata belongs is determined according to the hash value and the total number of virtual nodes.

3. The method according to claim 2, wherein If the target bucket supports multiple versions, the object information also includes the version number of the target object.

4. The method according to claim 1, wherein Each group is allowed to store the same maximum number of object metadata, and N is determined based on the maximum number of object metadata expected to be stored in the target bucket and the maximum number of object metadata allowed to be stored in a single group.

5. An object metadata storage device, characterized in that: Applied to a storage node included in an object storage system, the device includes: A determination unit is configured to determine a target virtual node to which target object metadata to be stored in a target bucket belongs; from the recorded correspondence between virtual node identifiers and group identifiers, search for a group corresponding to the target virtual node as a target group; wherein the target bucket includes N groups for storing object metadata, each group corresponding to storage space on a different storage medium, and each group uses an independent database; a virtual layer is established between the target bucket and the group, the virtual layer consisting of a plurality of virtual nodes, the number of the virtual nodes being greater than the number of the groups; the correspondence between the virtual node identifiers and the group identifiers can be dynamically adjusted, and the number of the virtual nodes remains unchanged; A storage unit is used to store the target object metadata in a storage space corresponding to the target group.

6. The device according to claim 5, characterized in that The determining unit determines the target virtual node to which the target object metadata belongs, including: Obtaining object information of a target object to which the target object metadata belongs, the object information including the object name of the target object and the bucket name of the target bucket; Performing a hash operation on the object information of the target object to obtain a corresponding hash value; The target virtual node to which the target object metadata belongs is determined according to the hash value and the total number of virtual nodes.

7. The device according to claim 6, characterized in that If the target bucket supports multiple versions, the object information also includes the version number of the target object.

8. The device according to claim 5, wherein Each group is allowed to store the same maximum number of object metadata, and N is determined based on the maximum number of object metadata expected to be stored in the target bucket and the maximum number of object metadata allowed to be stored in a single group.

Images