Index data storage method, system, device and storage medium

Abstract

The present application discloses an index data storage method, which is applied to an object storage system including multiple shards. The index data storage method includes: receiving an uploaded object; determining a shard corresponding to the object; and storing the index data of the object in the determined shard. The solution of the present application will not consume the disk resources of the node where the OSD is located in the traditional solution, which is conducive to ensuring the stability of the object storage system. The present application also discloses an index data storage system, device and storage medium, which have corresponding technical effects.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to an index data storage method, system, device and storage medium. Background Art

[0002] Object storage is a massive cloud storage for objects and files. Object storage does not have a multi-level file structure like a file system, but only has a concept of "bucket", that is, storage space, or bucket. "Bucket" is used to store objects, that is, a container for storing objects, which is a very flat storage method. Currently, Ceph is commonly used as the backend distributed object storage.

[0003] Ceph is a unified, distributed storage system with excellent performance, high reliability, and high scalability. When users upload objects to object storage buckets, they need to save some metadata of the objects to the bucket index for easy search. In the current solution, the bucket index data is stored in the RocksDB database of the node where the OSD (Object Storage Device) is located in the form of omap (object-map). Therefore, when the number of objects stored in a single bucket reaches a certain number, performing operations such as bucket list or Ceph silent checker deep-scrub will greatly consume the disk resources of the node where the OSD is located, causing OSD abnormalities. In addition, if the underlying deep-scrub time is too long, it will also cause slow requests in the cluster, resulting in performance jitter and affecting the stability of the storage system.

[0004] In summary, how to effectively store index data and ensure storage reliability is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide an index data storage method, system, device and storage medium to effectively store index data and ensure storage reliability.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] An index data storage method is applied to an object storage system including multiple shards, the index data storage method comprising:

[0008] Receive the uploaded object;

[0009] Determining a shard corresponding to the object;

[0010] The index data of the object is stored in the determined shard.

[0011] Preferably, the determining the shard corresponding to the object comprises:

[0012] Calculate the hash value of the object name of the object;

[0013] Add the hash value and 2 n -1 performs an AND operation, and uses the slice whose slice number is consistent with the value obtained by the AND operation as the slice corresponding to the object;

[0014] Where n is a positive integer, and 2 n =m, where m represents the current number of shards in the object storage system.

[0015] Preferably, it also includes:

[0016] When the amount of index data stored in any shard in the object storage system reaches a preset quantity threshold, the shard capacity of the object storage system is expanded to increase the number of shards in the object storage system.

[0017] Preferably, performing shard expansion of the object storage system includes:

[0018] Double the number of shards in the object storage system, and migrate the index data in each shard before the expansion according to a preset migration rule;

[0019] The migration rules include:

[0020] For any index data stored in any shard before the expansion, determine whether the p+1th bit of the hash value of the object name of the object corresponding to the index data is 0;

[0021] If yes, keep the index data in the shard where the index data currently resides;

[0022] If not, migrate the index data to the target shard corresponding to the index data, and the number of the target shard corresponding to the index data = the shard number where the index data currently resides + k;

[0023] The number of shards in the object storage system before expansion is k, and k=2 p After the shard expansion, the number of shards in the object storage system is 2k.

[0024] Preferably, the migration of index data in each shard before expansion is performed in a loop traversal manner, and when migrating index data in any shard, a read-write lock is added to the shard.

[0025] Preferably, it also includes:

[0026] When a write bucket index operation is received during the process of expanding the shard, a shard corresponding to the object is determined according to the object specified by the write bucket index operation;

[0027] Determine whether the determined shard is the source shard before expansion;

[0028] If it is not the source shard, the shard is determined to be the target shard after expansion, and a write operation is performed on the target shard;

[0029] If it is a source shard, determine whether the source shard has a read-write lock;

[0030] If not, performing a write operation on the source shard;

[0031] If so, a write operation is performed on the cache object, and after the expansion process is completed, a write operation is performed on the source shard based on the cache object.

[0032] Preferably, it also includes:

[0033] When a read bucket index operation is received during the process of expanding the shard, a shard corresponding to the object is determined according to the object specified by the write bucket index operation;

[0034] Determine whether the determined shard is the source shard before expansion;

[0035] If it is not the source shard, the shard determined is the target shard after expansion, and it is determined whether there is index data to be read in the target shard;

[0036] If the target shard contains the index data to be read, a read operation is performed;

[0037] If the target shard does not contain the index data to be read, determine whether the source shard corresponding to the target shard is locked with a read-write lock;

[0038] If the source shard corresponding to the target shard is not locked with a read-write lock, an error message is returned;

[0039] If the source shard corresponding to the target shard is locked with a read-write lock, determine whether there is index data to be read in the source shard corresponding to the target shard. If so, perform a read operation; otherwise, return an error prompt.

[0040] If it is a source shard, determine whether there is index data to be read in at least one place in the source shard and the cache object; if so, perform a read operation, otherwise return an error prompt.

[0041] An object storage system, the object storage system comprising a plurality of shards, the object storage system comprising:

[0042] An object receiving module, used for receiving uploaded objects;

[0043] A shard determination module, used to determine the shard corresponding to the object;

[0044] A saving module is used to save the index data of the object in the determined slice.

[0045] An object storage device, comprising:

[0046] Memory for storing computer programs;

[0047] A processor is used to execute the computer program to implement the steps of the index data storage method as described in any one of the above items.

[0048] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the index data storage method as described in any one of the above are implemented.

[0049] By applying the technical solution provided in the embodiment of the present invention, multiple shards are set up in the object storage system. After receiving the uploaded object, the present application does not save the index data in the OSD node as in the traditional solution, but determines the shard corresponding to the object, thereby saving the index data of the object in the determined shard. In other words, the present application uses multiple shards to save the index data, that is, saves it in a distributed manner. Therefore, the situation in which the disk resources of the node where the OSD is located are greatly consumed in the traditional solution will not occur, which is beneficial to ensuring the stability of the object storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 It is a flowchart of an implementation method of an index data storage method in the present invention;

[0052] Figure 2 It is an implementation flow chart of receiving a write bucket index operation during the process of shard expansion in a specific implementation manner of the present invention;

[0053] Figure 3 It is an implementation flow chart of receiving a bucket index reading operation during the process of shard expansion in a specific implementation manner of the present invention;

[0054] Figure 4 It is a structural schematic diagram of an index data storage system in the present invention. DETAILED DESCRIPTION

[0055] The core of the present invention is to provide an index data storage method, which will not consume the disk resources of the node where the OSD is located greatly in the traditional solution, and is conducive to ensuring the stability of the object storage system.

[0056] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Please refer to Figure 1 , Figure 1 This is a flowchart of an implementation method of an index data storage method in the present invention. The index data storage method is applied to an object storage system including multiple shards. The index data storage method includes the following steps:

[0058] Step S101: receiving an uploaded object.

[0059] Specifically, when a user uploads an object to a bucket in an object storage system, relevant metadata needs to be recorded, including object name, object size, modification time, ETAG, and other information. This information is collectively referred to as index data.

[0060] Step S102: Determine the fragment corresponding to the object.

[0061] Since the solution of the present application uses multiple shards to store index data, for any object received, it is necessary to determine the shard corresponding to the object, that is, the shard used to store the index data of the object. Of course, the specific sharding rules corresponding to the object can be set and adjusted according to actual needs, for example, the shard can be determined by taking the modulus operation of the hash value of the object name.

[0062] Step S103: Save the index data of the object in the determined shard.

[0063] After determining the shard corresponding to the object, the index data of the object can be saved in the determined shard. In practical applications, the index data can be stored in the omap of the shard object in the form of key-value. In addition, in some cases, each shard object can be stored as multiple copies, for example, if it is stored as 3 copies, the corresponding index data is also stored in 3 copies, thereby ensuring the reliability of metadata.

[0064] In a specific implementation of the present invention, step S102 may specifically include:

[0065] Calculate the hash value of the object name of the object;

[0066] Add the hash value and 2 n -1 performs an AND operation, and takes the slice whose slice number is consistent with the value obtained by the AND operation as the slice corresponding to the object;

[0067] Where n is a positive integer, and 2 n =m, where m represents the current number of shards in the object storage system.

[0068] In this implementation, the hash value of the object name is calculated. For example, in one case, n=4, that is, the current number of shards in the object storage system is m=16, and the calculated hash value of the object name is 1111 1111111111 1111 0011 1011 1101 0110. The hash value is compared with 2 n -1 is ANDed with 0000 00000000 0000 0000 0000 0000 1111, and the result is 0110. That is to say, in this example, the shard number obtained after the AND operation is 6, and the shard corresponding to the object is determined to be the sixth shard, that is, the index data of the object will be saved in the sixth shard later.

[0069] And it can be seen that the hash value of the object name and 2 n -1 performs an AND operation, which is equivalent to taking the last n digits of the hash value of the object name.

[0070] In this implementation, the number of shards m in the object storage system is set to 2 to the power of n, and the shard corresponding to the object is determined by an AND operation, which greatly improves the sharding efficiency compared to performing a modulo operation on the hash value of the object name.

[0071] In a specific embodiment of the present invention, it may also include:

[0072] When the amount of index data stored in any shard in the object storage system reaches a preset quantity threshold, the shard capacity of the object storage system is expanded to increase the number of shards in the object storage system.

[0073] In the solution of the present application, multiple shards are used to store index data, that is, distributed storage, so the situation of greatly consuming the disk resources of the node where the OSD is located in the traditional solution will not occur, which is beneficial to ensuring the stability of the object storage system.

[0074] As the number of objects uploaded increases, it may happen that the index data stored in each shard is relatively large. In this regard, the solution of the present application can also expand the capacity of the shards so that the index data in each shard will not be too much. Specifically, when it is detected that the amount of index data stored in any shard in the object storage system reaches a preset threshold, the shard capacity of the object storage system can be expanded.

[0075] There are many specific ways to expand the shard capacity, as long as the number of shards in the object storage system after the shard capacity expansion exceeds that before the expansion, such as adding a specified number of shards.

[0076] In a specific implementation of the present invention, performing shard expansion of an object storage system may specifically include:

[0077] Double the number of shards in the object storage system and migrate the index data in each shard before the expansion according to the preset migration rules;

[0078] The migration rules include:

[0079] For any index data stored in any shard before the expansion, determine whether the p+1th bit of the hash value of the object name of the object corresponding to the index data is 0;

[0080] If yes, keep the index data in the shard where the index data currently resides;

[0081] If not, migrate the index data to the target shard corresponding to the index data, and the number of the target shard corresponding to the index data = the shard number where the index data currently resides + k;

[0082] The number of shards in the object storage system before expansion is k, and k = 2 p After the shard expansion, the number of shards in the object storage system is 2k.

[0083] This implementation is conducive to the rapid migration of index data. Specifically, in this implementation, before sharding, the number of shards k in the object storage system is set to 2 to the power of p, and after the sharding is expanded, the number of shards doubles, so after the sharding is expanded, the number of shards in the object storage system is 2k.

[0084] All shards before expansion are source shards. Each source shard has a corresponding target shard. The corresponding relationship can be expressed as: the number of the source shard + k = the number of the target shard corresponding to the source shard.

[0085] After the shard expansion is performed in this way, for any index data stored in any shard, when the index data is migrated, there is no need to determine the storage location of the index data through a mapping operation, that is, there is no need to determine the shard to which the index data needs to be migrated through the operation of step S102, but directly determine it based on the value of the p+1th bit of the hash value of the object name of the object corresponding to the index data, so that this implementation method can greatly shorten the time consumed for shard expansion. And because some index data does not need to be migrated, the migration time of the index data can also be reduced.

[0086] For example, in the aforementioned implementation, the hash value of an object name is 1111 11111 1111 11110011 1011 1101 0110. In the aforementioned implementation, the number of shards in the object storage system before expansion is 16=2. 4 , the index data of the object is stored in the 6th shard. After the shard expansion, the number of shards increases from 16 to 32 = 2 5 . Since the first to eighth bits of the index data are: 1101 0110, that is, the value of the fifth bit is 1, the index data will be migrated to the target shard corresponding to the index data. In this example, the number of the target shard is 22 = the shard number 6 + 16 where the index data is currently located. It can be understood that for each index data stored in the 6th shard, the above method can be used to determine whether the corresponding index data is retained in the 6th shard or needs to be migrated to the 22nd shard.

[0087] In addition, it should be noted that in actual applications, when migrating any index data in any shard, it is possible to determine whether the p+1th bit of the hash value of the object name is 0 according to the above implementation method, thereby deciding whether the index data needs to be migrated. Moreover, when it is determined that migration is required, it is possible to further determine whether the index data already exists in the target shard. This is because in some cases, during the migration of the shard's index data, the user performs a write operation on the target shard. Therefore, when it is determined that the index data already exists in the target shard, the index data on the source shard can be directly discarded.

[0088] In a specific implementation of the present invention, the migration of index data in each shard before expansion is performed in a loop traversal manner, and when migrating index data in any shard, a read-write lock is added to the shard.

[0089] Each shard needs to migrate index data, and in this implementation, the migration of index data in each shard before expansion is performed in a loop, that is, the index data in each shard is migrated in sequence. In addition, when migrating index data in any shard, only the shard is added with a read-write lock, that is, the shard is only readable but not writable, and other shards do not need to be added with a read-write lock. After the index data migration process of any shard is completed, the read-write lock of the shard can be released. This method can effectively reduce the impact of shard expansion on user business, because if the user's read and write operations do not involve the shard that is currently undergoing index data migration, they will not be affected. If they involve the shard that is currently undergoing index data migration, only the write operation will be affected.

[0090] Further, in a specific embodiment of the present invention, please refer to Figure 2 , is a flowchart of receiving a write bucket index operation during the process of shard expansion in a specific implementation manner, which may include:

[0091] Step S201: when a write bucket index operation is received during the process of expanding the shard, a shard corresponding to the object is determined according to the object specified by the write bucket index operation;

[0092] Step S202: Determine whether the determined shard is the source shard before expansion;

[0093] If it is not the source shard, execute step S203: determine that the shard is the target shard after expansion, and perform a write operation on the target shard;

[0094] If it is a source shard, execute step S204: determine whether a read-write lock is added to the source shard;

[0095] If not, execute step S205: write operation to the source shard;

[0096] If yes, execute step S206: perform a write operation on the cache object, and perform a write operation on the source shard based on the cache object after the expansion process is completed.

[0097] As described above, the solution of the present application has little impact on user services during the process of shard expansion. This implementation method specifically describes the situation where a write bucket index operation is received during the process of shard expansion, which will not affect user services at all.

[0098] Specifically, when a write bucket index operation is received, the write bucket index operation specifies the write object, so that the shard corresponding to the object can be determined, that is, the shard storing the index data of the object can be determined. For example, in the above example, the hash value of the object name is calculated and then combined with 2 n -1 and operation, the shard corresponding to the object can be determined, and it can be understood that at this time 2 n Should = the number of shards in the object storage system after expansion, 2k. For example, if the number of shards before expansion is 16, the number of shards after expansion is 32, which is to perform an AND operation on the hash value of the object name and 31 to determine the shard corresponding to the object being written.

[0099] After determining the shard corresponding to the object, it can be determined whether the determined shard is the source shard before the expansion. The source shard in this application refers to each shard before this expansion, and correspondingly, each shard added after this expansion is the target shard. For example, if the number of shards before the expansion is 16, then these 16 shards are all source shards, and the number of shards after the expansion is 32, then the 17th shard to the 32nd shard are all target shards for this expansion. Of course, if another expansion is carried out later, and the number of shards becomes 64, then during this expansion, the 32 shards before the expansion are all source shards, and the 32 shards added after the expansion are all target shards.

[0100] You can judge whether the determined shard is the source shard before expansion according to the shard number. If it is not the source shard, it means it is the target shard. The target shard will not have a read-write lock, so you can directly write to the target shard. It can be seen that it will not affect user business.

[0101] If it is a source shard, it is necessary to further determine whether the source shard has a read-write lock. If not, it means that the source shard can also be written directly without affecting user services. If a read-write lock is added, in order not to affect user services, this implementation method can first write to the cache object. The cache object can be a log object, for example. Of course, other forms of cache objects can be selected in other implementation methods, which can temporarily record the index data to be written. By writing to the cache object, after the expansion process is completed, the source shard can be written based on the cache object, that is, each index data recorded in the cache object is written to the corresponding source shard.

[0102] In a specific embodiment of the present invention, please refer to Figure 3 , is a flowchart of receiving a bucket index read operation during shard expansion in a specific implementation manner, which may include:

[0103] Step S301: when a read bucket index operation is received during the process of expanding the shard, a shard corresponding to the object is determined according to the object specified by the write bucket index operation;

[0104] Step S302: Determine whether the determined shard is the source shard before expansion;

[0105] If it is not the source shard, execute step S303: determine that the shard is the target shard after expansion, and determine whether there is index data to be read in the target shard;

[0106] If the target shard contains index data to be read, then step S304 is executed: a read operation is performed;

[0107] If the index data to be read does not exist in the target shard, step S305 is executed: determining whether the source shard corresponding to the target shard is locked with a read-write lock;

[0108] If the source shard corresponding to the target shard is not locked with a read-write lock, execute step S306: return an error prompt;

[0109] If the source shard corresponding to the target shard is locked with a read-write lock, then execute step S307: determine whether there is index data to be read in the source shard corresponding to the target shard; if so, execute step S304: perform a read operation; otherwise, execute step S306: return an error prompt;

[0110] If it is a source shard, execute step S308: determine whether there is index data to be read in at least one place in the source shard and the cache object; if so, execute step S304: perform a read operation, otherwise execute step S306: return an error prompt.

[0111] This implementation specifically describes the situation where a bucket index read operation is received during the process of shard expansion, which will not affect user services at all.

[0112] During the process of expanding the shard, when a read bucket index operation is received, the shard corresponding to the object can be determined according to the object specified by the write bucket index operation, and then it is determined whether the determined shard is the source shard before the expansion.

[0113] If the determined shard is not the source shard, it means that the determined shard is the target shard after expansion, and it can be determined whether there is index data to be read in the target shard. If there is index data to be read in the target shard, the read operation can be performed directly, that is, the index data in the target shard is read. If there is no index data to be read in the target shard, it means that the index data may be in the source shard and has not yet been migrated to the target shard. Therefore, it can be determined whether the source shard corresponding to the target shard has a read-write lock. If no read-write lock is added, it means that the index data is not in the source shard either. It may be that the current bucket index reading operation itself is wrong or the index data is lost, so an error prompt can be returned. If the source shard has a read-write lock, it can be further determined whether there is index data to be read in the source shard. If so, a read operation is performed, that is, the index data is read from the source shard and fed back to the user. On the contrary, if the source shard has a read-write lock but there is no index data to be read, an error prompt can be returned.

[0114] If the determined shard is the source shard, it means that the index data should be stored in the source shard or cache object. Therefore, it is determined whether there is at least one place in the source shard and the cache object where the index data to be read exists. If so, a read operation is performed, that is, if there is index data to be read in the cache object, the index data in the cache object is read and fed back to the user. If there is index data to be read in the source shard, the index data in the source shard is read and fed back to the user. If there is no such data, it means that there may be an error in the bucket index read operation itself or the index data is lost, so an error prompt can be returned.

[0115] By applying the technical solution provided in the embodiment of the present invention, multiple shards are set up in the object storage system. After receiving the uploaded object, the present application does not save the index data in the OSD node as in the traditional solution, but determines the shard corresponding to the object, thereby saving the index data of the object in the determined shard. In other words, the present application uses multiple shards to save the index data, that is, saves it in a distributed manner. Therefore, the situation in which the disk resources of the node where the OSD is located are greatly consumed in the traditional solution will not occur, which is beneficial to ensuring the stability of the object storage system.

[0116] Corresponding to the above method embodiment, the embodiment of the present invention further provides an object storage system, which can be referred to in correspondence with the above.

[0117] The object storage system consists of multiple shards, see Figure 4 , the object storage system includes:

[0118] The object receiving module 401 is used to receive the uploaded object;

[0119] A fragment determination module 402, configured to determine fragments corresponding to an object;

[0120] The saving module 403 is used to save the index data of the object in the determined shards.

[0121] In a specific implementation of the present invention, the shard determination module 402 is specifically configured to:

[0122] Calculate the hash value of the object name of the object;

[0123] Add the hash value and 2 n -1 performs an AND operation, and takes the slice whose slice number is consistent with the value obtained by the AND operation as the slice corresponding to the object;

[0124] Where n is a positive integer, and 2 n =m, where m represents the current number of shards in the object storage system.

[0125] In a specific embodiment of the present invention, a shard expansion module is also included, which is used to:

[0126] When the amount of index data stored in any shard in the object storage system reaches a preset quantity threshold, the shard capacity of the object storage system is expanded to increase the number of shards in the object storage system.

[0127] In a specific embodiment of the present invention, the shard expansion of the object storage system of the shard expansion module specifically includes:

[0128] Double the number of shards in the object storage system and migrate the index data in each shard before the expansion according to the preset migration rules;

[0129] The migration rules include:

[0130] For any index data stored in any shard before the expansion, determine whether the p+1th bit of the hash value of the object name of the object corresponding to the index data is 0;

[0131] If yes, keep the index data in the shard where the index data currently resides;

[0132] If not, migrate the index data to the target shard corresponding to the index data, and the number of the target shard corresponding to the index data = the shard number where the index data currently resides + k;

[0133] The number of shards in the object storage system before expansion is k, and k = 2 p After the shard expansion, the number of shards in the object storage system is 2k.

[0134] In a specific implementation of the present invention, the shard expansion module migrates the index data in each shard before expansion in a loop traversal manner, and when migrating the index data in any shard, a read-write lock is added to the shard.

[0135] In a specific implementation of the present invention, a write bucket index execution module is also included, which is used to:

[0136] When a write bucket index operation is received during the process of shard expansion, the shard corresponding to the object is determined according to the object specified by the write bucket index operation;

[0137] Determine whether the determined shard is the source shard before expansion;

[0138] If it is not the source shard, the shard determined is the target shard after expansion, and the write operation is performed on the target shard;

[0139] If it is a source shard, determine whether the source shard has a read-write lock;

[0140] If not, write to the source shard;

[0141] If so, a write operation is performed on the cache object, and after the expansion process is completed, a write operation is performed on the source shard based on the cache object.

[0142] In a specific implementation of the present invention, a bucket index execution module is also included, which is used to:

[0143] When a read bucket index operation is received during the process of shard expansion, the shard corresponding to the object is determined according to the object specified by the write bucket index operation;

[0144] Determine whether the determined shard is the source shard before expansion;

[0145] If it is not the source shard, the shard determined is the target shard after expansion, and it is determined whether there is index data to be read in the target shard;

[0146] If the target shard contains the index data to be read, the read operation is performed;

[0147] If the index data to be read does not exist in the target shard, determine whether the source shard corresponding to the target shard has a read-write lock;

[0148] If the source shard corresponding to the target shard is not locked with a read-write lock, an error message is returned;

[0149] If the source shard corresponding to the target shard is locked with a read-write lock, determine whether the source shard corresponding to the target shard has the index data to be read. If so, perform a read operation; otherwise, return an error prompt.

[0150] If it is a source shard, determine whether the index data to be read exists in at least one place in the source shard and the cache object; if so, perform a read operation, otherwise return an error prompt.

[0151] Corresponding to the above method and system embodiments, the embodiments of the present invention further provide an object storage device and a computer-readable storage medium, which can be referred to in correspondence with the above.

[0152] The object storage device includes:

[0153] Memory for storing computer programs;

[0154] A processor is used to execute a computer program to implement the steps of the index data storage method in any of the above embodiments.

[0155] The computer readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the index data storage method in any of the above embodiments are implemented. The computer readable storage medium mentioned here includes random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the technical field.

[0156] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0157] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0158] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the technical solution and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, the present invention can also be improved and modified without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for storing index data, characterized in that: Applied to an object storage system including multiple shards, the index data storage method includes: Receive the uploaded object; Determining a shard corresponding to the object; Storing the index data of the object in the determined shard; Also includes: When the amount of index data stored in any shard in the object storage system reaches a preset amount threshold, performing shard expansion of the object storage system to increase the number of shards in the object storage system; Performing shard expansion of the object storage system includes: Double the number of shards in the object storage system, and migrate the index data in each shard before the expansion according to a preset migration rule; The migration rules include: For any index data stored in any shard before the expansion, determine whether the p+1th bit of the hash value of the object name of the object corresponding to the index data is 0; If yes, keep the index data in the shard where the index data currently resides; If not, migrate the index data to the target shard corresponding to the index data, and the number of the target shard corresponding to the index data = the shard number where the index data currently resides + k; The number of shards in the object storage system before expansion is k, and k=2 p After the shard expansion, the number of shards in the object storage system is 2k.

2. The index data storage method according to claim 1, characterized in that: The determining of the shards corresponding to the object includes: Calculate the hash value of the object name of the object; Add the hash value and 2 n -1 performs an AND operation, and uses the slice whose slice number is consistent with the value obtained by the AND operation as the slice corresponding to the object; Where n is a positive integer, and 2 n =m, where m represents the current number of shards in the object storage system.

3. The index data storage method according to claim 2, characterized in that: The index data in each shard before the expansion is migrated in a circular traversal manner, and when migrating the index data in any shard, a read-write lock is added to the shard.

4. The index data storage method according to claim 3, characterized in that: Also includes: When a write bucket index operation is received during the process of expanding the shard, a shard corresponding to the object is determined according to the object specified by the write bucket index operation; Determine whether the determined shard is the source shard before expansion; If it is not the source shard, the shard is determined to be the target shard after expansion, and a write operation is performed on the target shard; If it is a source shard, determine whether the source shard has a read-write lock; If not, performing a write operation on the source shard; If so, a write operation is performed on the cache object, and after the expansion process is completed, a write operation is performed on the source shard based on the cache object.

5. The index data storage method according to claim 3, characterized in that: Also includes: When a bucket index read operation is received during the process of expanding the shard, a shard corresponding to the object is determined according to the object specified by the bucket index read operation; Determine whether the determined shard is the source shard before expansion; If it is not the source shard, the shard determined is the target shard after expansion, and it is determined whether there is index data to be read in the target shard; If the target shard contains the index data to be read, a read operation is performed; If the target shard does not contain the index data to be read, determine whether the source shard corresponding to the target shard is locked with a read-write lock; If the source shard corresponding to the target shard is not locked with a read-write lock, an error message is returned; If the source shard corresponding to the target shard is locked with a read-write lock, determine whether there is index data to be read in the source shard corresponding to the target shard. If so, perform a read operation; otherwise, return an error prompt. If it is a source shard, determine whether there is index data to be read in at least one place in the source shard and the cache object; If yes, perform the read operation, otherwise return an error prompt.

6. An object storage system, characterized in that: The object storage system includes a plurality of shards, and the object storage system includes: An object receiving module, used for receiving uploaded objects; A shard determination module, used to determine the shard corresponding to the object; A saving module, used for saving the index data of the object in the determined shard; It also includes a shard expansion module for: When the amount of index data stored in any shard in the object storage system reaches a preset amount threshold, performing shard expansion of the object storage system to increase the number of shards in the object storage system; Performing shard expansion of the object storage system includes: Double the number of shards in the object storage system, and migrate the index data in each shard before the expansion according to a preset migration rule; The migration rules include: For any index data stored in any shard before the expansion, determine whether the p+1th bit of the hash value of the object name of the object corresponding to the index data is 0; If yes, keep the index data in the shard where the index data currently resides; If not, migrate the index data to the target shard corresponding to the index data, and the number of the target shard corresponding to the index data = the shard number where the index data currently resides + k; The number of shards in the object storage system before expansion is k, and k=2 p After the shard expansion, the number of shards in the object storage system is 2k.

7. An object storage device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the index data storage method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the index data storage method according to any one of claims 1 to 5 are implemented.

Images