Object uploading method, system, device and storage medium of a storage system

Abstract

The present application discloses an object uploading method, system, device and storage medium for a storage system, which are applied to the field of storage technology. The method includes: setting a multi-level index directory for a bucket in the storage system, and setting a virtual bucket shard corresponding to the bucket shard for each bucket shard under each level of directory structure; determining a target bucket, and determining the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded; converting the directory structure of the object to be uploaded into a check value and mapping it into a preset numerical space to obtain a first numerical value corresponding to the check value; determining the number of the virtual bucket shard in the numerical space that is closest to the first numerical value under the directory structure of the object to be uploaded, and writing the object information of the object to be uploaded into the bucket shard corresponding to the virtual bucket shard. By applying the solution of the present application, the specification of storing objects in a single bucket is effectively improved, and the time consumption for orderly listing is ensured to be relatively low. And the balance of storage is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular, to an object uploading method, system, device and storage medium for a storage system. Background Art

[0002] Under the existing object storage framework, to ensure the reliability and performance of storage, the specification limit for storing objects in a single bucket is 100 million. The specific reason is that on an index shard of a bucket, a maximum of 100,000 object information records are stored. After exceeding this threshold, a disconnection may occur during cluster reconstruction. In the current solution, object index information is recorded in the form of object-map on the bucket shards, and the number of object-maps recorded on each bucket shard is limited to 100,000 to avoid stability and performance problems caused by exceeding the limit. And a bucket has 1024 index shards by default. If too many index shards are set, the time consumption for ordered listing will increase because the objects between the index shards are unordered.

[0003] With the continuous increase of user data, in the current solution, it is necessary to create multiple buckets for data management, access and maintenance, which increases the difficulty of development and operation and maintenance. Therefore, for the specification of storing objects in a single bucket, there is a need to store hundreds of billions of objects in a single bucket, but at the same time, it is necessary to ensure that the time consumption for ordered listing is relatively low.

[0004] In summary, how to effectively improve the specification of storing objects in a single bucket and ensure relatively low time consumption for ordered listing is a technical problem that needs to be urgently solved by those skilled in the art at present. Summary of the Invention

[0005] The purpose of the present invention is to provide an object uploading method, system, device and storage medium for a storage system, so as to effectively improve the specification of storing objects in a single bucket and ensure relatively low time consumption for ordered listing.

[0006] To solve the above technical problems, the present invention provides the following technical solutions:

[0007] An object uploading method for a storage system, comprising:

[0008] Setting a multi-level index directory for the bucket in the storage system, and setting a virtual bucket shard corresponding to each bucket shard under each level of directory structure for the bucket shard;

[0009] Determining a target bucket for storing the object information of the object to be uploaded;

[0010] Based on the object name of the object to be uploaded, determining the directory structure of the object to be uploaded under the target bucket;

[0011] Convert the directory structure of the object to be uploaded into a check value, and map the check value into a preset numerical space according to a preset mapping rule to obtain a first numerical value corresponding to the check value;

[0012] Determine the number of the virtual bucket shard closest to the first numerical value in the numerical space under the directory structure of the object to be uploaded, and use the bucket shard corresponding to the virtual bucket shard as the target bucket shard;

[0013] Write the object information of the object to be uploaded into the target bucket shard;

[0014] Wherein, multiple bucket shards are set under the directory structure of the object to be uploaded, and each bucket shard has its corresponding virtual bucket shard; the numbers of different virtual bucket shards under the directory structure of the object to be uploaded are different, and the numbers of each virtual bucket shard are mapped into the numerical space according to the mapping rule.

[0015] Preferably, setting a multi-level index directory for the bucket in the storage system and setting a virtual bucket shard corresponding to each bucket shard under each directory structure includes:

[0016] Set a first-level index directory for the bucket in the storage system; wherein, the first-level index directory includes m first-level directories and 1 metadata shard, and the metadata shard includes k bucket shards set therein;

[0017] Set a second-level index directory for the bucket in the storage system; wherein, for each of the m first-level directories, in the second-level index directory, it includes n second-level directories corresponding to the first-level directory and 1 metadata shard, and the metadata shard includes k bucket shards set therein;

[0018] Set a third-level index directory for the bucket in the storage system; wherein, for each second-level directory, in the third-level index directory, it includes p third-level directories corresponding to the second-level directory and 1 metadata shard, and the metadata shard includes k bucket shards set therein;

[0019] For any one third-level directory, the third-level directory includes k bucket shards set therein;

[0020] For each bucket shard under each directory structure, set q virtual bucket shards corresponding to the bucket shard; m, n, p, k, q are all positive integers.

[0021] Preferably, it further includes:

[0022] Receive a directory structure adjustment instruction and adjust the index directory level of the bucket.

[0023] Preferably, after determining the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded, the method further includes:

[0024] When the directory structure of the object to be uploaded does not exceed the index directory level of the bucket and the directory structure of the object to be uploaded does not exist in the index directory of the bucket, adding the directory structure of the object to be uploaded to the index directory.

[0025] Preferably, after determining the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded, the method further includes:

[0026] When the directory structure of the object to be uploaded exceeds the index directory level of the bucket, outputting a prompt message indicating that the object name of the object to be uploaded is incorrect.

[0027] Preferably, converting the directory structure of the object to be uploaded into a check value, and mapping the check value into a preset numerical space according to a preset mapping rule to obtain a first numerical value corresponding to the check value includes:

[0028] Converting the directory structure of the object to be uploaded into a hash value, and dividing the hash value by a first fixed value, and using the obtained remainder as the hash value mapped into the preset numerical space, and the obtained first numerical value corresponding to the hash value.

[0029] Preferably, the method further includes:

[0030] When the number of object information stored on any bucket shard reaches a preset first threshold, splitting the bucket shard into 2 bucket shards, and dividing the virtual bucket shard corresponding to the bucket shard before splitting into 2 parts, so that one part of the virtual bucket shards corresponds to one of the bucket shards after splitting, and the other part of the virtual bucket shards corresponds to the other bucket shard after splitting;

[0031] Wherein, when dividing the virtual bucket shard corresponding to the bucket shard before splitting into 2 parts, the difference in the number of the two parts of the virtual bucket shards does not exceed 1.

[0032] An object uploading system for a storage system, including:

[0033] A multi-level index directory construction module, configured to set a multi-level index directory for a bucket in the storage system, and set a virtual bucket shard corresponding to each bucket shard under each directory structure;

[0034] A target bucket determination module, configured to determine a target bucket for storing object information of an object to be uploaded;

[0035] An object directory structure determination module to be uploaded, which is used to determine the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded;

[0036] A numerical mapping module, which is used to convert the directory structure of the object to be uploaded into a verification value, and map the verification value into a preset numerical space according to a preset mapping rule to obtain a first numerical value corresponding to the verification value;

[0037] A target bucket shard determination module, which is used to determine the number of the virtual bucket shard closest to the first numerical value in the numerical space under the directory structure of the object to be uploaded, and use the bucket shard corresponding to the virtual bucket shard as the target bucket shard;

[0038] A writing module, which is used to write the object information of the object to be uploaded into the target bucket shard;

[0039] Among them, multiple bucket shards are set under the directory structure of the object to be uploaded, and each bucket shard has its own corresponding virtual bucket shard; the numbers of different virtual bucket shards under the directory structure of the object to be uploaded are different, and the numbers of each virtual bucket shard are mapped into the numerical space according to the mapping rule.

[0040] An object uploading device of a storage system, including:

[0041] A memory, which is used to store a computer program;

[0042] A processor, which is used to execute the computer program to implement the steps of the object uploading method of the storage system as described above.

[0043] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the object uploading method of the storage system as described above are implemented.

[0044] Applying the technical solution provided by the embodiment of the present invention, a multi-level index directory of the bucket is set for the bucket in the storage system. When an object needs to be uploaded subsequently, after determining the target bucket for storing the object information of the object to be uploaded, based on the object name of the object to be uploaded, the directory structure of the object to be uploaded under the target bucket is determined. That is to say, different objects to be uploaded with similar object names will be placed under the same or similar directory structures. In other words, in the solution of this application, based on the object name of the object to be uploaded and using the multi-level index directory of the bucket, different object information can be initially divided, which also makes the time-consuming very short if the buckets of this application are listed in an orderly manner. And, the solution of this application does not limit the number of bucket shards, that is, the specification of storing objects in a single bucket can reach the current requirement of hundreds of billions or even higher.

[0045] In addition, after determining the directory structure of the object to be uploaded under the target bucket, the present application converts the directory structure of the object to be uploaded into a check value, and maps the check value into a preset numerical space according to the preset mapping rule to obtain a first numerical value corresponding to the check value. Then, it determines the number of the virtual bucket shard closest to the first numerical value in the numerical space under the directory structure of the object to be uploaded, and uses the bucket shard corresponding to the virtual bucket shard as the target bucket shard, and writes the object information of the object to be uploaded into the target bucket shard. Such an operation is beneficial to load balancing, that is, for multiple bucket shards under the same directory structure, the objects to be uploaded placed in this directory structure can be evenly distributed to each bucket shard under this directory structure. Brief Description of the Drawings

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0047] Figure 1 It is a flowchart of an embodiment of an object uploading method for a storage system in the present invention;

[0048] Figure 2 It is a schematic structural diagram of a three-level index directory set in a specific embodiment of the present invention;

[0049] Figure 3 It is a schematic diagram of the corresponding relationship between bucket shards and virtual bucket shards in a specific embodiment of the present invention;

[0050] Figure 4 It is a schematic diagram of mapping the virtual bucket shard number and the check value into a preset numerical space in a specific embodiment of the present invention;

[0051] Figure 5 It is a schematic structural diagram of an object uploading system for a storage system in the present invention. Detailed Embodiments

[0052] The core of the present invention is to provide an object uploading method for a storage system, which effectively improves the specification of objects stored in a single bucket and ensures that the time-consuming for orderly listing is relatively low. And it ensures storage balance, that is, it is beneficial to make the storage of each bucket shard under the same directory structure balanced.

[0053] To enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0054] Please refer to Figure 1 , Figure 1 , which is the implementation flowchart of an object uploading method for a storage system in the present invention. The object uploading method of the storage system may include the following steps:

[0055] Step S101: Set a multi-level index directory for the buckets in the storage system, and set virtual bucket shards corresponding to the bucket shards for each bucket shard under each level of directory structure.

[0056] Specifically, the storage system of the present application is usually a distributed object storage system. In the storage system, one or more buckets may be set, and each bucket can implement the solution of the present application. Of course, considering that after applying the solution of the present application, a single bucket can store hundreds of billions of object information, therefore, in some cases, only a single bucket needs to be set, which is also beneficial to reducing the difficulty of development and operation and maintenance for staff.

[0057] In the solution of the present application, a multi-level index directory of the bucket needs to be set. When storing the object information of the object to be uploaded subsequently, the directory structure of the object to be uploaded is determined according to the object name of the object to be uploaded. In this way, different objects to be uploaded with similar object names will be placed under the same or similar directory structures. That is, based on the object name of the object to be uploaded, using the multi-level index directory of the bucket, the object information can be preliminarily divided, and thus if the buckets of the present application are used for orderly listing, the time consumption will be very short.

[0058] The present application also needs to set bucket shards in the multi-level index directory. The specific number of bucket shards set can be set and adjusted as needed. For example, a certain number of bucket shards are set under each directory at the lowest level. For each bucket shard under each level of directory structure, the present application also needs to set virtual bucket shards corresponding to the bucket shards. Setting virtual bucket shards can be used to achieve the balance of object information storage in subsequent steps.

[0059] When setting the multi-level index directory of the bucket, there are various specific methods. For example, in a specific embodiment of the present invention, step S101 may specifically include:

[0060] Set a first-level index directory for the buckets in the storage system; wherein, the first-level index directory includes m first-level directories and 1 metadata shard, and the metadata shard includes k bucket shards set therein;

[0061] Set a second-level index directory for the buckets in the storage system; wherein, for each of the m first-level directories, in the second-level index directory, it includes n second-level directories corresponding to this first-level directory and 1 metadata shard, and the metadata shard includes k bucket shards set therein;

[0062] Set a third-level index directory for the buckets in the storage system; wherein, for each second-level directory, in the third-level index directory, it includes p third-level directories corresponding to this second-level directory and 1 metadata shard, and the metadata shard includes k bucket shards set therein;

[0063] For any one third-level directory, this third-level directory includes k bucket shards set therein;

[0064] For each bucket shard under each level of directory structure, set q virtual bucket shards corresponding to the bucket shard; m, n, p, k, q are all positive integers.

[0065] In this implementation manner, a three-level index directory is set for the buckets. Among them, under the first-level index directory, m first-level directories and 1 metadata shard are set. m is a positive integer, and the specific value can be selected according to needs. Setting 1 metadata shard under the first-level index directory is considered that in some cases, it is allowed to directly store the object information of the object to be uploaded under the first-level index directory. Therefore, 1 metadata shard is set, and the metadata shard includes k bucket shards set therein. In Figure 2 the metadata shard shown is represented as "file", that is, a metadata shard composed of objects without a directory structure in the bucket, and it can be named bucketID+file for example.

[0066] In addition, for the first-level index directory of the bucket, it can be represented by a first-level index table. For example, Table 1 is the first-level index table in a specific case.

[0067] Table 1:

[0068] First-level directory 1 bucketID + First-level directory 1 First-level directory 2 bucketID + First-level directory 2 ... ... First-level directory m bucketID + First-level directory m File bucketID + file

[0069] On the left side of Table 1 is the key, which is the first-level directory structure reflected by the first-level index table, that is, the number of the first-level directory and 1 metadata shard. On the right side is the value, which is the name of the corresponding second-level index table. For example, Table 1 can be named bucketID+index.

[0070] For each of the m first-level directories, under the secondary index directory, n secondary directories corresponding to the first-level directory and one metadata shard are set. n is a positive integer, and the specific value can be selected according to needs.

[0071] In the secondary index directory under any one first-level directory, one metadata shard is set. This is because in some cases, it is allowed to directly store the object information of the object to be uploaded under this secondary index directory. Therefore, for each of the m first-level directories, one metadata shard corresponding to the first-level directory is set under the secondary index directory. In Figure 2 The metadata shard shown in is represented as "First-level directory 1 / File", that is, a metadata shard composed of objects without directory structure under the secondary index directory. This metadata shard can be named: bucketID + first-level directory number + file. Here, the first-level directory number refers to the number of the first-level directory corresponding to this metadata shard. For example Figure 2 The metadata shard under the secondary index directory shown in can be named: bucketID + first-level directory 1 + file.

[0072] Since there are m first-level directories, for the secondary index directory of the bucket, m secondary index tables can be used to represent it. For example, Table 2 is one of the secondary index tables.

[0073] Table 2:

[0074] First-level directory 1 / Second-level directory 1 bucketID + First-level directory 1 + Second-level directory 1 First-level directory 1 / Second-level directory 2 bucketID + First-level directory 1 + Second-level directory 2 ... ... First-level directory 1 / Second-level directory n bucketID + First-level directory 1 + Second-level directory n First-level directory 1 / File bucketID + First-level directory 1 + file

[0075] On the left side of Table 2 is the key of this secondary index table, which is the secondary directory structure reflected by this secondary index table, that is, the numbers of each secondary directory under the first-level directory 1, and one metadata shard. On the right side is the value, which is the name of the corresponding tertiary index table.

[0076] Since m first-level directories are set, and n secondary directories are set under each first-level directory, there will be m×n secondary directories. For each secondary directory, under the tertiary index directory, p tertiary directories corresponding to the secondary directory and one metadata shard are set. p is a positive integer, and the specific value can be selected according to needs.

[0077] In the tertiary index directory under any one secondary directory, one metadata shard is set. This metadata shard includes k bucket shards set. This is because in some cases, it is allowed to directly store the object information of the object to be uploaded under this tertiary index directory. Therefore, for each secondary directory, one metadata shard corresponding to the secondary directory is set under the tertiary index directory. In Figure 2The metadata shard shown is represented as "First-level directory 1 / Second-level directory 3 / File", that is, a metadata shard composed of objects without a directory structure under the three-level index directory, which can be named: bucketID + First-level directory number + Second-level directory number + file. Here, the first-level directory number and the second-level directory number refer to the numbers of the first-level directory and the second-level directory corresponding to this metadata shard. For example Figure 2 the metadata shard under the three-level index directory shown in Figure 2 can be named: bucketID + First-level directory 1 + Second-level directory 3 + file.

[0078] For the three-level index directory of a bucket, multiple three-level index tables can be used to represent it. For example, Table 3 shows one of the three-level index tables.

[0079] Table 3:

[0080]

[0081]

[0082] On the left side of Table 3 is the key of this three-level index table, which is the three-level directory structure reflected by this three-level index table, that is, each three-level directory number under First-level directory 1 / Second-level directory 3 and 1 metadata shard. The right side is the value, which can be the name of the bucket shard correspondence table under the corresponding three-level directory structure. Table 3 can be named bucketID + First-level directory 1 + Second-level directory 3.

[0083] For any one three-level directory, there are k bucket shards set in this three-level directory. For example Figure 2 Figure 2 shows k bucket shards under "First-level directory 1 / Second-level directory 3 / Third-level directory 2". k is a positive integer, and the specific value can be selected according to needs.

[0084] For each bucket shard under each level of directory structure, q virtual bucket shards corresponding to the bucket shard are set; q is a positive integer, and the specific value can be selected according to needs. Figure 2 Figure 2 marks that the bucket shard 2 among the k bucket shards under "First-level directory 1 / Second-level directory 3 / Third-level directory 2" corresponds to q virtual bucket shards.

[0085] Step S102: Determine the target bucket for storing the object information of the object to be uploaded.

[0086] Specifically, there are various ways to determine the target bucket for storing the object information of the object to be uploaded. For example, it can be determined based on the object name of the uploaded object. Another example is that it can be specified by the upload instruction and so on. Of course, in some cases, when only 1 bucket is set, this bucket can be directly used as the target bucket for storing the object information of the object to be uploaded.

[0087] Step S103: Determine the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded.

[0088] The object name of the object to be uploaded may include multiple pieces of information. In this application, based on the object name of the object to be uploaded, the directory structure of the object to be uploaded under the target bucket is determined. For example, in one scenario, if the beginning part of the object name of the object to be uploaded is bucketID + first-level directory 1, it indicates that the object information of this object to be uploaded needs to be stored in Figure 2 one of the bucket shards in "first-level directory 1 / files". Another example is that if the beginning part of the object name of the object to be uploaded is bucketID + first-level directory 1 + second-level directory 3 + third-level directory 2, it indicates that the object information of this object to be uploaded needs to be stored in Figure 2 one of the bucket shards in "first-level directory 1 / second-level directory 3 / third-level directory 2".

[0089] Step S104: Convert the directory structure of the object to be uploaded into a check value, and map the check value into a preset numerical space according to a preset mapping rule to obtain a first numerical value corresponding to the check value.

[0090] When converting the directory structure of the object to be uploaded into a check value, there are various specific algorithms. For example, considering that the hash value is a commonly used check value, therefore, the directory structure of the object to be uploaded can be converted into a hash value.

[0091] There are also various ways to map the check value into a preset numerical space. For example, a relatively convenient way is to perform division and take the remainder.

[0092] That is, in a specific embodiment of the present invention, step S104 may specifically include:

[0093] Convert the directory structure of the object to be uploaded into a hash value, and divide the hash value by a first fixed value, and use the obtained remainder as the hash value mapped into the preset numerical space, and the obtained first numerical value corresponding to the hash value.

[0094] Dividing the hash value by the first fixed value means using the first fixed value as the divisor and the hash value as the dividend, and the obtained remainder is used as the first numerical value corresponding to the hash value mapped into the preset numerical space.

[0095] For example, in one scenario, the numerical space is 0 to 2 31 -1. The first fixed value is 2 31 -1. After mapping, the obtained first numerical value corresponding to the check value falls within 0 to 2 31 -1.

[0096] Step S105: Determine the number of the virtual bucket shard in the numerical space that is closest to the first numerical value under the directory structure of the object to be uploaded, and use the bucket shard corresponding to the virtual bucket shard as the target bucket shard.

[0097] Multiple bucket shards are set under the directory structure of the object to be uploaded, and each bucket shard has its corresponding virtual bucket shard. For example, in the above example, the directory structure of the object to be uploaded is the first-level directory 1 / second-level directory 3 / third-level directory 2, then k bucket shards are set under this directory structure. Each of these k bucket shards has its corresponding virtual bucket shard. For example, k = 128, and each bucket shard corresponds to 100 virtual bucket shards respectively, that is, p is 100. That is, under the directory structure of the object to be uploaded, there are a total of 128 × 100 = 12800 virtual bucket shards.

[0098] For example Figure 3 in, bucket shard 0 corresponds to virtual bucket shards 0.0 to 0.99. Bucket shard 1 corresponds to virtual bucket shards 1.0 to 1.99. Bucket shard 127 corresponds to virtual bucket shards 127.0 to 127.99.

[0099] Moreover, the numbers of different virtual bucket shards under the directory structure of the object to be uploaded are different, and the numbers of all virtual bucket shards are mapped to the numerical space according to the mapping rule. That is to say, the numbers of the 12800 virtual bucket shards in the above example are different from each other, and the numbers of these 12800 virtual bucket shards are all mapped to the numerical space according to the mapping rule. For details, see Figure 4 that the 12800 virtual bucket shards are all mapped to the numerical space 0 to 2 31 -1.

[0100] For example, Table 4 is the corresponding table of bucket shards under a certain third-level directory structure, that is, it represents the corresponding relationship between each bucket shard and the virtual bucket shard under this directory structure.

[0101] Table 4:

[0102] Hash value (Virtual bucket shard 0#0) Bucket shard 0 Hash value (Virtual bucket shard 0#n) Bucket shard 0 Hash value (Virtual bucket shard 0#99) Bucket shard 0 Hash value (Virtual bucket shard 1#0) Bucket shard 1 Hash value (Virtual bucket shard 1#) Bucket shard 1 Hash value (Virtual bucket shard 1#99) Bucket shard 1 ... ... Hash value (Virtual bucket shard 127#0) Bucket shard 127 Hash value (Virtual bucket shard 127#n) Bucket shard 127 Hash value (Virtual bucket shard 127#99) Bucket shard 127

[0103] It can be understood that after mapping 12,800 virtual bucket shards to the numerical space, their positions are fixed. Therefore, in practical applications, for the convenience of calculation, under different directory structures, the number and numbering of the set virtual bucket shards are the same. That is, for example, for each different directory such as the first-level directory 1 / second-level directory 3 / third-level directory 1, the first-level directory 1 / second-level directory 5 / third-level directory 7, etc., k = 128 can be set as described above. Each bucket shard corresponds to 100 virtual bucket shards respectively, and the virtual bucket shards start from 0.0 and increase by 0.01 as the step size, with the numbering value of 127.99.

[0104] Figure 4 Obj1 to Obj6 in it represent the first numerical values of the 6 objects to be uploaded in the numerical space respectively. It can be seen that for Obj1, in the numerical space, the number of the virtual bucket shard closest to Obj1 is virtual shard 0.0. For Obj2, in the numerical space, the number of the virtual bucket shard closest to Obj2 is virtual shard 1.0. For Obj3, in the numerical space, the number of the virtual bucket shard closest to Obj3 is virtual shard 127.0. For Obj4, in the numerical space, the number of the virtual bucket shard closest to Obj4 is virtual shard 1.99.

[0105] Step S106: Write the object information of the object to be uploaded into the target bucket shard;

[0106] After determining the target bucket shard, the object information of the object to be uploaded can be written into the target bucket shard. For example, in the above example, the directory structure of the object to be uploaded is the first-level directory 1 / second-level directory 3 / third-level directory 2. After converting the directory structure of the object to be uploaded into a check value and mapping it to the preset numerical space to obtain the first numerical value corresponding to the check value, for example, the number of the virtual bucket shard closest to this first numerical value in the numerical space is 126.58, and the bucket shard corresponding to this virtual bucket shard is 126. Therefore, the object information of the object to be uploaded can be written into bucket shard 126 under the first-level directory 1 / second-level directory 3 / third-level directory 2.

[0107] In a specific embodiment of the present invention, it may further include:

[0108] Receive a directory structure adjustment instruction and adjust the index directory level of the bucket.

[0109] In Figure 2 the embodiment, a 3-level index directory is set. In this embodiment, directory structure adjustment is allowed, that is, after receiving a directory structure adjustment instruction, config can be adjusted to modify the index directory level of the bucket.

[0110] In a specific embodiment of the present invention, after determining the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded, it may further include:

[0111] When the directory structure of the object to be uploaded does not exceed the index directory level of the bucket and the directory structure of the object to be uploaded does not exist in the index directory of the bucket, add the directory structure of the object to be uploaded to the index directory.

[0112] For example, in a specific scenario, the index directory level of the bucket is 3 levels, and 100 first-level directories are set, that is, m is 100. For example, based on the object name of the object to be uploaded, the determined directory structure of the object to be uploaded under the target bucket is first-level directory 104 / second-level directory 3 / third-level directory 2. It can be seen that this directory structure does not exist in the index directory of the bucket, that is, the key of the first-level directory 104 does not exist in the first-level index table. However, the index directory level of this directory structure is 3 levels, which does not exceed the index directory level of the bucket, so the directory structure of the object to be uploaded can be added to the index directory. That is, adjust the corresponding first-level index table, second-level index table, and third-level index table, so as to add the directory structure of "first-level directory 104 / second-level directory 3 / third-level directory 2" to the index directory of the bucket.

[0113] It can be seen that in this embodiment, it is allowed to adjust the index directory of the bucket according to the object name of the object to be uploaded, and the flexibility is very high. In practical applications, the first-level index table, each second-level index table, and each third-level index table can all be stored in the cache.

[0114] In a specific embodiment of the present invention, after determining the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded, it may further include:

[0115] When the directory structure of the object to be uploaded exceeds the index directory level of the bucket, output a prompt message indicating that the object name of the object to be uploaded is incorrect.

[0116] Still taking the index directory level of the bucket being 3 levels and 100 first-level directories being set in the bucket as an example, for example, the determined directory structure of the object to be uploaded under the target bucket is first-level directory 104 / second-level directory 3 / third-level directory 2 / fourth-level directory 1. It can be seen that this directory structure does not exist in the index directory of the bucket, and moreover, the index directory level of this directory structure is 4 levels, which has exceeded the index directory level of the bucket. This situation is usually caused by an incorrect object name of the object to be uploaded. Therefore, a prompt message indicating that the object name of the object to be uploaded is incorrect can be output.

[0117] In a specific embodiment of the present invention, it further includes:

[0118] When the number of object information stored on any bucket shard reaches a preset first threshold, split the bucket shard into two bucket shards, and divide the virtual bucket shard corresponding to the bucket shard before splitting into two parts, so that one part of the virtual bucket shard corresponds to one of the bucket shards after splitting, and the other part of the virtual bucket shard corresponds to the other bucket shard after splitting;

[0119] Among them, when dividing the virtual bucket shard corresponding to the bucket shard before splitting into two parts, the difference in the number of virtual bucket shards in the two parts does not exceed 1.

[0120] When the number of object information stored on any bucket shard reaches a preset first threshold, it means that the number of object information stored on this bucket shard is relatively large. Continuing to store object information on this bucket shard may cause stability and performance problems. Therefore, the original bucket shard will be split to obtain two new bucket shards, and the virtual bucket shard corresponding to the bucket shard before splitting will be divided into two parts, usually into two equal parts. Of course, when the number of virtual bucket shards corresponding to the bucket shard before splitting is odd and cannot be exactly divided into two equal parts, it can be divided into two parts with a difference in number not exceeding 1.

[0121] For example, if the original bucket shard 127 corresponds to virtual bucket shards 127.0 to 127.99, after splitting, bucket shard 127 and bucket shard 128 are obtained. The new bucket shard 127 corresponds to virtual bucket shards 127.0 to 127.49, and the new bucket shard 128 corresponds to virtual bucket shards 127.50 to 127.99.

[0122] It can be seen that when splitting the bucket shard in this implementation manner, it is necessary to adjust the correspondence with the virtual bucket shard, but this process will not affect other bucket shards, so it is not easy to make mistakes. Even if mistakes are made, the impact is relatively small, that is, only a single bucket shard will be affected. In addition, such a solution is also beneficial to ensuring the normal operation of the service, that is, when a certain bucket is split, it does not affect the service based on other buckets.

[0123] Applying the technical solution provided by the embodiments of the present invention, a multi-level index directory for buckets in a storage system is set for the buckets. When an object needs to be uploaded subsequently, after determining the target bucket for storing the object information of the object to be uploaded, based on the object name of the object to be uploaded, the directory structure of the object to be uploaded under the target bucket is determined. That is to say, different objects to be uploaded with similar object names will be placed under the same or similar directory structures. In other words, in the solution of the present application, based on the object name of the object to be uploaded and using the multi-level index directory of the bucket, different object information can be preliminarily divided. That is, if the buckets of the present application are used for orderly listing, the time-consuming will be very short. Moreover, the solution of the present application does not limit the number of bucket shards, that is, the specification of storing objects in a single bucket can reach the specification of hundreds of billions or even higher required currently.

[0124] In addition, after determining the directory structure of the object to be uploaded under the target bucket, the present application will convert the directory structure of the object to be uploaded into a check value, and according to a preset mapping rule, map the check value into a preset numerical space to obtain a first numerical value corresponding to the check value. Then, the number of the virtual bucket shard closest to the first numerical value in the numerical space under the directory structure of the object to be uploaded is determined, and the bucket shard corresponding to the virtual bucket shard is used as the target bucket shard, and the object information of the object to be uploaded is written into the target bucket shard; through such an operation, it is beneficial to load balancing, that is, for multiple bucket shards under the same directory structure, the objects to be uploaded placed in this directory structure can be evenly distributed to each bucket shard under this directory structure.

[0125] Corresponding to the above method embodiments, the embodiments of the present invention also provide an object uploading system for a storage system, which can be correspondingly referred to with the above text.

[0126] See Figure 5 As shown in the figure, it is a schematic structural diagram of an object uploading system for a storage system in the present invention, including:

[0127] A multi-level index directory construction module 501, configured to set a multi-level index directory for buckets in a storage system, and set virtual bucket shards corresponding to the bucket shards for each directory structure;

[0128] A target bucket determination module 502, configured to determine a target bucket for storing the object information of the object to be uploaded;

[0129] A directory structure determination module 503 for the object to be uploaded, configured to determine the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded;

[0130] A numerical mapping module 504, configured to convert the directory structure of an object to be uploaded into a check value, and map the check value into a preset numerical space according to a preset mapping rule to obtain a first numerical value corresponding to the check value;

[0131] A target bucket shard determination module 505, configured to determine the number of the virtual bucket shard closest to the first numerical value in the numerical space under the directory structure of the object to be uploaded, and use the bucket shard corresponding to the virtual bucket shard as the target bucket shard;

[0132] A writing module 506, configured to write the object information of the object to be uploaded into the target bucket shard;

[0133] Wherein, multiple bucket shards are set under the directory structure of the object to be uploaded, and each bucket shard has its corresponding virtual bucket shard; the numbers of different virtual bucket shards under the directory structure of the object to be uploaded are different, and the numbers of each virtual bucket shard are mapped into the numerical space according to the mapping rule.

[0134] In a specific embodiment of the present invention, the multi-level index directory construction module 501 is specifically configured to:

[0135] Set a first-level index directory for the buckets in the storage system; wherein, the first-level index directory includes m first-level directories and 1 metadata shard, and the metadata shard includes k bucket shards set therein;

[0136] Set a second-level index directory for the buckets in the storage system; wherein, for each of the m first-level directories, in the second-level index directory, it includes n second-level directories corresponding to the first-level directory and 1 metadata shard, and the metadata shard includes k bucket shards set therein;

[0137] Set a third-level index directory for the buckets in the storage system; wherein, for each second-level directory, in the third-level index directory, it includes p third-level directories corresponding to the second-level directory and 1 metadata shard, and the metadata shard includes k bucket shards set therein;

[0138] For any one third-level directory, the third-level directory includes k bucket shards set therein;

[0139] For each bucket shard under each level of directory structure, set q virtual bucket shards corresponding to the bucket shard; m, n, p, k, and q are all positive integers.

[0140] In a specific embodiment of the present invention, the multi-level index directory construction module 501 is further configured to:

[0141] Receive a directory structure adjustment instruction and adjust the index directory level of the bucket.

[0142] In a specific embodiment of the present invention, after the directory structure determination module 503 of the object to be uploaded determines the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded, the multi-level index directory construction module 501 is further configured to:

[0143] When the directory structure of the object to be uploaded does not exceed the index directory level of the bucket and the directory structure of the object to be uploaded does not exist in the index directory of the bucket, add the directory structure of the object to be uploaded to the index directory.

[0144] In a specific embodiment of the present invention, after the directory structure determination module 503 of the object to be uploaded determines the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded, it further includes:

[0145] An error reporting module, configured to output a prompt message indicating that the object name of the object to be uploaded is incorrect when the directory structure of the object to be uploaded exceeds the index directory level of the bucket.

[0146] In a specific embodiment of the present invention, the numerical mapping module 504 is specifically configured to:

[0147] Convert the directory structure of the object to be uploaded into a hash value, divide the hash value by a first fixed value, and use the obtained remainder as the hash value mapped into a preset numerical space to obtain a first numerical value corresponding to the hash value.

[0148] In a specific embodiment of the present invention, it further includes a bucket classification control module, configured to:

[0149] When the number of object information stored on any bucket shard reaches a preset first threshold, split the bucket shard into 2 bucket shards, and divide the virtual bucket shard corresponding to the bucket shard before splitting into 2 parts, so that one part of the virtual bucket shard corresponds to one of the bucket shards after splitting, and the other part of the virtual bucket shard corresponds to the other bucket shard after splitting;

[0150] Wherein, when dividing the virtual bucket shard corresponding to the bucket shard before splitting into 2 parts, the difference in the number of the two parts of the virtual bucket shards does not exceed 1.

[0151] Corresponding to the above method and system embodiments, the embodiments of the present invention further provide an object uploading device for a storage system and a computer-readable storage medium, which can be correspondingly referred to with the above text.

[0152] The object uploading device of the storage system may include:

[0153] A memory, configured to store a computer program;

[0154] A processor for executing a computer program to implement the steps of the object uploading method of the storage system in any of the above embodiments.

[0155] A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the steps of the object uploading method of the storage system in any of the above embodiments are implemented. The computer-readable storage medium mentioned here includes random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium well known in the technical field.

[0156] It should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0157] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0158] Specific examples are used in this article to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the technical solution and its core idea of the present invention. It should be pointed out that for those of ordinary skill in the art of this technology, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. An object uploading method for a storage system, characterized in that, Including: Setting a multi-level index directory for a bucket in a storage system, and setting a virtual bucket shard corresponding to each bucket shard under each level of directory structure for the bucket; Determining a target bucket for storing object information of an object to be uploaded; Based on the object name of the object to be uploaded, determining the directory structure of the object to be uploaded under the target bucket; Converting the directory structure of the object to be uploaded into a check value, and mapping the check value into a preset numerical space according to a preset mapping rule to obtain a first numerical value corresponding to the check value; Determining the number of the virtual bucket shard closest to the first numerical value in the numerical space under the directory structure of the object to be uploaded, and using the bucket shard corresponding to the virtual bucket shard as the target bucket shard; Writing the object information of the object to be uploaded into the target bucket shard; Wherein, multiple bucket shards are provided under the directory structure of the object to be uploaded, and each bucket shard has its corresponding virtual bucket shard; the numbers of different virtual bucket shards under the directory structure of the object to be uploaded are different, and the numbers of all virtual bucket shards are mapped into the numerical space according to the mapping rule.

2. The object uploading method of the storage system according to claim 1, characterized in that The setting a multi-level index directory for a bucket in a storage system, and setting a virtual bucket shard corresponding to each bucket shard under each level of directory structure for the bucket includes: Setting a first-level index directory for a bucket in a storage system; wherein, the first-level index directory includes m first-level directories and 1 metadata shard, and the metadata shard includes k bucket shards set therein; Setting a second-level index directory for a bucket in a storage system; wherein, for each of the m first-level directories, in the second-level index directory, it includes n second-level directories corresponding to the first-level directory and 1 metadata shard, and the metadata shard includes k bucket shards set therein; Setting a third-level index directory for a bucket in a storage system; wherein, for each second-level directory, in the third-level index directory, it includes p third-level directories corresponding to the second-level directory and 1 metadata shard, and the metadata shard includes k bucket shards set therein; For any one third-level directory, the third-level directory includes k bucket shards set therein; For each bucket shard under each level of directory structure, setting q virtual bucket shards corresponding to the bucket shard; m, n, p, k, q are all positive integers.

3. The object uploading method of the storage system according to claim 2, wherein Further including: Receiving a directory structure adjustment instruction and adjusting the index directory level of the bucket.

4. The object uploading method of the storage system according to claim 1, characterized in that, After determining the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded, it further includes: When the directory structure of the object to be uploaded does not exceed the index directory level of the bucket and the directory structure of the object to be uploaded does not exist in the index directory of the bucket, adding the directory structure of the object to be uploaded to the index directory.

5. The object uploading method of the storage system according to claim 4, wherein After determining the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded, it further includes: When the directory structure of the object to be uploaded exceeds the index directory level of the bucket, a prompt message indicating an error in the object name of the object to be uploaded is output.

6. The object uploading method of the storage system according to claim 1, wherein Converting the directory structure of the object to be uploaded into a check value, and mapping the check value into a preset numerical space according to a preset mapping rule to obtain a first numerical value corresponding to the check value, includes: Converting the directory structure of the object to be uploaded into a hash value, dividing the hash value by a first fixed value, and using the obtained remainder as the hash value mapped into the preset numerical space, and obtaining the first numerical value corresponding to the hash value.

7. The object uploading method of the storage system according to any one of claims 1 to 6, characterized in that, It also includes: When the number of object information stored on any bucket shard reaches a preset first threshold, splitting the bucket shard into 2 bucket shards, and dividing the virtual bucket shard corresponding to the bucket shard before splitting into 2 parts, so that one part of the virtual bucket shards corresponds to one of the bucket shards after splitting, and the other part of the virtual bucket shards corresponds to the other bucket shard after splitting; Among them, when dividing the virtual bucket shard corresponding to the bucket shard before splitting into 2 parts, the difference in the number of the two parts of the virtual bucket shards does not exceed 1.

8. An object uploading system for a storage system, characterized in that It includes: A multi-level index directory construction module, configured to set a multi-level index directory for the bucket in the storage system, and set a virtual bucket shard corresponding to each bucket shard under each level of directory structure; A target bucket determination module, configured to determine a target bucket for storing the object information of the object to be uploaded; An object to be uploaded directory structure determination module, configured to determine the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded; A numerical value mapping module, configured to convert the directory structure of the object to be uploaded into a check value, and map the check value into a preset numerical space according to a preset mapping rule to obtain a first numerical value corresponding to the check value; A target bucket shard determination module, configured to determine the number of the virtual bucket shard closest to the first numerical value in the numerical space under the directory structure of the object to be uploaded, and use the bucket shard corresponding to the virtual bucket shard as the target bucket shard; A writing module, configured to write the object information of the object to be uploaded into the target bucket shard; Among them, multiple bucket shards are set under the directory structure of the object to be uploaded, and each bucket shard has its own corresponding virtual bucket shard; the numbers of different virtual bucket shards under the directory structure of the object to be uploaded are different, and the numbers of each virtual bucket shard are mapped into the numerical space according to the mapping rule.

9. An object uploading device of a storage system, characterized in that, It includes: A memory, configured to store a computer program; A processor, configured to execute the computer program to implement the steps of the object uploading method of the storage system according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the steps of the object uploading method of the storage system according to any one of claims 1 to 7 are implemented.

Images