Object storage method and device, electronic equipment and storage medium

Abstract

The application discloses an object storage method and device, electronic equipment and storage medium. The method comprises the following steps: receiving a first request; the first request is used for requesting to store a first object into a first container; the first request carries at least the first object; determining at least two sub-containers associated with the first container; determining a first sub-container corresponding to the first object by using a consistent hash algorithm and the determined at least two sub-containers; and storing the first object into the first sub-container. The method provided by the application determines the sub-containers associated with the container for the container storage request of the object, and determines the sub-containers associated with the object by using the consistent hash algorithm, so that the object is actually stored into the corresponding sub-container. The object is stored by using the sub-container instead of the container, and the processing efficiency of the container can be improved when processing the traversal request for the object.

Description

Technical Field

[0001] This application relates to the field of data storage, and more particularly to an object storage method, apparatus, electronic device, and storage medium. Background Technology

[0002] As distributed network storage technology matures, more and more upper-layer applications are choosing object storage as their underlying storage medium; among them, Ceph object storage features high performance, high availability, and high scalability. However, when users store a certain number of objects in a single container, i.e., when the container stores a large amount of data, the container encounters a performance bottleneck when users request to traverse objects, leading to low processing efficiency. Summary of the Invention

[0003] To address the related technical problems, embodiments of this application provide an object storage method, apparatus, electronic device, and storage medium.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides an object storage method, including:

[0006] Receive a first request; the first request is used to request that a first object be stored in a first container; the first request carries at least the first object;

[0007] Identify at least two sub-containers associated with the first container;

[0008] Using a consistent hashing algorithm and at least two determined sub-containers, determine the first sub-container corresponding to the first object;

[0009] Store the first object in the first sub-container.

[0010] In the above method, the first request also carries the name of the first object; determining the first sub-container corresponding to the first object using a consistent hashing algorithm and at least two determined sub-containers includes:

[0011] The hash value corresponding to the name is calculated using the consistent hashing algorithm;

[0012] The first sub-container is determined using the hash value corresponding to the name and the hash rings corresponding to the at least two sub-containers; the hash rings are obtained using a consistent hashing algorithm.

[0013] The above method further includes:

[0014] The preset number of sub-containers is determined using the configuration parameters of the first container;

[0015] Based on the preset quantity, determine the number of sub-containers on the hash ring;

[0016] The hash ring is determined based on the number of sub-containers on the hash ring, the sub-container identifier in the configuration parameters of the first container, and the consistent hashing algorithm.

[0017] In the above method, determining the number of sub-containers on the hash ring based on the preset quantity includes:

[0018] When the preset quantity is lower than the first threshold, a first parameter is determined; the first parameter represents the minimum multiple by which the result obtained by multiplying the preset quantity is greater than or equal to the first threshold.

[0019] The number of sub-containers on the hash ring is determined using the first parameter.

[0020] In the above method, determining the number of sub-containers on the hash ring based on the preset quantity includes:

[0021] When the preset quantity is greater than or equal to the first threshold, the preset quantity is determined as the number of sub-containers on the hash ring.

[0022] In the above method, determining the first sub-container using the hash value corresponding to the name and the hash rings corresponding to the at least two sub-containers includes:

[0023] The position of the first object on the hash ring is determined using the hash value corresponding to the name;

[0024] The first sub-container is determined based on the position of the first object on the hash ring and the position of each sub-container on the hash ring.

[0025] In the above method, storing the first object in the first sub-container includes:

[0026] The number of shards for the first sub-container is determined using the configuration parameters of the first container.

[0027] Based on the number of shards, the first sub-container is sharded;

[0028] The shard corresponding to the first object is determined using the consistent hashing algorithm and the first sub-container after sharding;

[0029] Store the first object in the shard corresponding to the first object.

[0030] This application embodiment also provides an object storage device, including:

[0031] A receiving unit is configured to receive a first request; the first request is configured to request that a first object be stored in a first container; the first request carries at least the first object.

[0032] A first processing unit is configured to determine at least two sub-containers associated with the first container;

[0033] The second processing unit is used to determine the first sub-container corresponding to the first object using a consistent hashing algorithm and at least two determined sub-containers;

[0034] A storage unit is used to store the first object into the first sub-container.

[0035] This application also provides an electronic device, including: a processor and a memory for storing a computer program capable of running on the processor.

[0036] When the processor runs the computer program, the computer program, when executed by the processor, implements the steps of any of the above-described object storage methods.

[0037] This application also provides a storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of any of the object storage methods described above.

[0038] The object storage method, apparatus, electronic device, and storage medium provided in this application embodiment, upon receiving a first request, determines at least two sub-containers associated with a first container; the first request is used to request the storage of a first object into the first container; the first request carries at least the first object; using a consistent hashing algorithm and the determined at least two sub-containers, the first sub-container corresponding to the first object is determined, and the first object is stored in the first sub-container. The technical solution provided in this application embodiment, upon receiving a container storage request for an object, determines at least two sub-containers associated with the container (which can be considered a virtual container), and then uses a consistent hashing algorithm to determine the sub-container corresponding to the object, so as to facilitate the storage of the object in the corresponding sub-container. By storing the object in at least two sub-containers associated with the container, it is possible to traverse each sub-container in parallel when processing user object traversal requests, avoiding the problems of long traversal time and low efficiency caused by storing the object in only one container, thereby improving the performance and efficiency of the container during traversal. Attached Figure Description

[0039] Figure 1 This is a system architecture diagram of an object storage system in related technologies;

[0040] Figure 2 This is a schematic diagram of an object storage method in an embodiment of this application;

[0041] Figure 3 This is a schematic diagram of the structure of the sub-container determined by the hash ring in an embodiment of this application;

[0042] Figure 4 This is a system architecture diagram of object storage in an application embodiment of this application;

[0043] Figure 5 This is a schematic diagram of the object storage method in an application embodiment of this application;

[0044] Figure 6 This is a schematic diagram of an object storage device structure in an embodiment of this application;

[0045] Figure 7 This is a schematic diagram of an electronic device structure in an embodiment of this application. Detailed Implementation

[0046] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0047] Before providing a further detailed description of this application, the nouns and terms used in the embodiments of this application will be explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows.

[0048] (1) Load balancing: Load balancing is a computer technology whose main function is to distribute a large number of jobs reasonably across multiple operating units for execution.

[0049] Driven by specific business needs, users upload a large number of objects to a single container for storage. Theoretically, each container has an infinite capacity, meaning it can store a number of objects. However, in related technologies, when the number of objects stored in a container reaches a certain level (e.g., millions), the time spent processing client traversal requests (i.e., traversing the index information of all objects stored in the container) increases, leading to performance bottlenecks such as request waiting. This also causes other requests to be blocked and unable to be processed in a timely manner.

[0050] To solve the above problems, such as Figure 1As shown, this paper proposes to improve container traversal efficiency by increasing the number of container shards. Typically, the index information of objects within a single container is stored in the container's corresponding storage pool (default.rgw.buckets.index), and by default, all objects are stored in a single region of this pool; the storage region corresponds to the number of container shards. In this approach, the object storage gateway (RGW, RADOS Gateway) can read configuration information, set up multiple shards within the container, and use a hash algorithm to determine the correspondence between object storage locations and shards, thus distributing object index information across different regions of the storage pool. When processing traversal requests using this method, parallel traversal of the index information corresponding to different regions can improve traversal efficiency and enhance container processing performance.

[0051] However, the above approach has two main drawbacks. First, since the shard count needs to be set when the container is created before object storage, a large shard count coupled with a small number of objects will result in excessive invalid traversals when processing traversal requests, wasting time and resources. Conversely, a small shard count coupled with a large number of objects will lead to a large number of objects on a single shard, thus affecting traversal efficiency. In other words, while setting the number of shards can improve traversal efficiency, the shard count must be set during container creation and cannot dynamically change based on storage requests, potentially wasting storage resources or impacting container traversal performance. Second, the theoretical performance of a single container during traversal requests still depends on the hardware; that is, the container's processing speed (IOPS, Input / Output Operations Per Second) is related to the container's hardware. Therefore, setting the number of shards can only improve container traversal performance to a certain extent.

[0052] In conclusion, storing objects in a single container does not provide a reasonable solution to the container performance bottleneck that occurs when handling traversal requests.

[0053] Based on this, in various embodiments of this application, the concept of a virtual container is introduced. For an object's container storage request, the container at the request's entry point is actually a logical concept. When an object is stored in this container, the underlying virtual container has multiple configurable actual sub-containers. Therefore, when an object's container storage request is received, at least two sub-containers associated with the container are determined, and a consistent hashing algorithm is used to determine the sub-container associated with the object, thereby actually storing the object in the corresponding sub-container. By using sub-containers instead of containers to store objects, when processing traversal requests, the processing efficiency of the container can be improved by performing parallel traversal on each of the at least two sub-containers associated with the container, avoiding the problem of low processing efficiency caused by the performance bottleneck of the container, and improving the performance of the container.

[0054] This application provides an object storage method, such as... Figure 2 As shown, the method includes the following steps:

[0055] Step 201: Receive the first request;

[0056] Wherein, the first request is used to request that the first object be stored in the first container; the first request carries at least the first object;

[0057] Step 202: Identify at least two sub-containers associated with the first container;

[0058] Step 203: Using the consistent hashing algorithm and at least two determined sub-containers, determine the first sub-container corresponding to the first object;

[0059] Step 204: Store the first object in the first sub-container.

[0060] The object storage method can be applied to electronic devices, specifically RGW; the first request can be a request sent by a user through a client.

[0061] In practical applications, the RGW can be viewed as a Hypertext Transfer Protocol (HTTP) server.

[0062] In practical applications, after the electronic device receives the first request through the application programming interface (API), it needs to parse the first request in order to perform operations such as user authentication, permission verification, and quota checks. That is to say, the first request may also include user authentication information, access control permission information, and quota information. The API can be an interface of Amazon S3 or an interface of OpenStack Swift.

[0063] In practical applications, the electronic device can create a corresponding first container and at least two associated sub-containers for the user based on the creation request and configuration information, and generate attribute information corresponding to the first container. The creation request is used to request the creation of the first container; the creation request carries at least the user name and container identifier. Furthermore, when the electronic device creates at least two sub-containers associated with the first container based on the configuration information, a mapping relationship is established between the first container and the at least two sub-containers, i.e., a mapping relationship is established between the first container and the sub-container identifiers. This allows the electronic device to store objects in the sub-containers associated with the first container when processing object storage requests.

[0064] The configuration information includes at least sub-container quantity configuration information and sub-container shard configuration information; the attribute information includes at least container identifier (e.g., container ID), storage pool identifier (e.g., marker), user name (e.g., owner), quota, etc.; wherein, the container attribute information represents various attribute information of the container; the sub-container quantity configuration information is used to determine the preset number of sub-containers corresponding to the container; the sub-container shard configuration information is used to determine the preset number of shards for the sub-container; the container identifier can identify the container; the storage pool identifier is used to determine the storage pool for storing object index information; the user name represents the user's identity information; the quota represents the container's quota parameters, such as the maximum number of objects that can be stored, the capacity of the stored objects, etc.

[0065] In practical applications, the configuration information for the number of sub-containers and the configuration information for sub-container fragmentation can be set as needed.

[0066] In practical applications, after receiving the first request, the electronic device can determine the corresponding first container and at least two associated sub-containers based on the username carried in the first request, and then store the first object.

[0067] In practical applications, when creating a container, the electronic device also configures a corresponding storage pool for each container to store the object's index data, metadata, and other information. For example, the storage pool `data_pool` can be configured to store the metadata information of the objects in the container; and the storage pool `index_pool` can be configured to store the index data information of the objects in the container.

[0068] In practical applications, after receiving the first request, the electronic device can determine at least two associated sub-containers by reading the sub-container number configuration information of the first container.

[0069] Here, the first container can be understood as a virtual container, that is, a logical concept, and each virtual container is associated with at least two actual sub-containers. In other words, a request to store the first object in the first container actually means storing the first object in the first sub-container associated with the virtual container.

[0070] In practical applications, for load balancing purposes, consistent hashing can be used to determine the first sub-container to achieve even storage of objects. This is because in traditional hash tables, adding or deleting a sub-container requires remapping all objects to determine the new sub-container. This leads to a large amount of data migration and increases the operational cost of dynamically adjusting the number of sub-containers. Consistent hashing, on the other hand, uses a modulo method to form a virtual ring, or hash ring, around the entire hash value space, with values ​​ranging from 0 to 2. 32 , total 2 32 The hash ring is composed of -1 integers, arranged in a clockwise direction. Then, based on the positions of each sub-container and object on the hash ring, the sub-container for storing objects can be determined, achieving even distribution of objects across sub-containers. During this process, if a sub-container changes and can no longer store objects, only K / n objects (i.e., the average number of objects stored in each sub-container) need to be remapped to determine the new sub-container, without affecting the objects stored in other sub-containers; where K represents the total number of objects stored in all sub-containers; and n represents the number of sub-containers. This reduces the time and resource costs of adjusting the number of sub-containers.

[0071] In step 204, in practical application, when the electronic device stores the first object into the first sub-container, it will actually store the index information of the first object into the storage pool corresponding to the first sub-container, such as a storage pool named default.rgw.buckets.index.

[0072] Since the first container is associated with at least two sub-containers, and each sub-container corresponds to a storage pool (i.e., a one-to-one correspondence exists between sub-containers and storage pools), when processing client traversal requests, the electronic device can directly and in parallel traverse the index information in the storage pools corresponding to each sub-container. This improves the efficiency of container traversal. Furthermore, using sub-containers instead of the container to store objects increases the maximum theoretical number of objects the container can store. Specifically, the maximum theoretical number of objects the container can store can be understood as the maximum number of objects the container can store without encountering a performance bottleneck when the electronic device processes client traversal requests. When the container and sub-containers are on the same hardware, and at least two sub-containers are used instead of the container to store objects, the maximum number of objects that the sub-containers can store can increase accordingly because the electronic device can perform parallel traversal of each sub-container, thus increasing the maximum theoretical number of objects that can be stored.

[0073] In one embodiment, the first request further carries the name of the first object; determining the first sub-container corresponding to the first object using a consistent hashing algorithm and at least two determined sub-containers includes:

[0074] The hash value corresponding to the name is calculated using the consistent hashing algorithm;

[0075] The first sub-container is determined using the hash value corresponding to the name and the hash rings corresponding to the at least two sub-containers; the hash rings are obtained using a consistent hashing algorithm.

[0076] Here, when a user sends the first request through the client, the first request also carries the name of the first object (i.e., the key), so that the corresponding hash value can be calculated based on the name. Specifically, the hash function in the consistent hashing algorithm can be used to calculate the hash value corresponding to the first object.

[0077] In practical applications, the consistent hashing algorithm can generate a virtual hash ring, with values ​​ranging from 0 to 2 on the ring. 32 The hash values ​​are composed of [a number of hash values], arranged in a clockwise direction.

[0078] For each sub-container of the first container, the electronic device configures a corresponding sub-container identifier (e.g., sub-container ID). By inputting the identifier of each sub-container into a hash function, the corresponding hash value can be obtained. Based on the hash values ​​of the sub-containers and the hash rings, at least two hash rings corresponding to the sub-containers can be obtained.

[0079] In practical applications, when the electronic device creates the first container, it configures the number of sub-containers for that first container. This sub-container number configuration information can at least be used to determine the preset number and identifiers of the sub-containers associated with the first container. Therefore, the actual number of sub-containers can be determined by reading the sub-container number configuration information, and thus the first object can be stored.

[0080] Based on this, in one embodiment, the method may further include:

[0081] The preset number of sub-containers is determined using the configuration parameters of the first container;

[0082] Based on the preset quantity, determine the number of sub-containers on the hash ring;

[0083] The hash ring is determined based on the number of sub-containers on the hash ring, the sub-container identifier in the configuration parameters of the first container, and the consistent hashing algorithm.

[0084] In practical applications, the electronic device can determine the preset number of sub-containers by reading the configuration parameters of the first container (which can be child_bucket_num), i.e., the sub-container quantity configuration information, and thus determine the number of sub-containers on the hash ring.

[0085] Here, the electronic device can also determine the identifier of each sub-container in a preset number of sub-containers by configuring parameters, so as to determine its position on the hash ring.

[0086] In practical applications, if the number of sub-containers is initially small, the distribution of objects across these sub-containers may become uneven. This is understandable because the consistent hashing algorithm determines sub-containers by using the hash values ​​of the objects and the corresponding hash rings of the sub-containers. When the number of sub-containers is small, such as two sub-containers, their positions on the hash ring may be unevenly distributed, resulting in a large number of objects being concentrated in the first sub-container and a small number of objects in the second sub-container, causing data skew.

[0087] To address the aforementioned issues, when storing objects, a preset number of sub-containers associated with the first container can be compared with a threshold, and the number of sub-containers on the hash ring can be determined based on the comparison result, so as to achieve uniform storage of objects.

[0088] Based on this, in one embodiment, determining the number of sub-containers on the hash ring based on the preset quantity includes:

[0089] When the preset quantity is lower than the first threshold, a first parameter is determined; the first parameter represents the minimum multiple by which the result obtained by multiplying the preset quantity is greater than or equal to the first threshold.

[0090] The number of sub-containers on the hash ring is determined using the first parameter.

[0091] Here, the sub-containers on the hash ring can be understood as the virtual nodes corresponding to the sub-containers.

[0092] Here, when the preset number of sub-containers is lower than the first threshold, the objects can be stored evenly by configuring a certain number of virtual nodes for each sub-container; wherein, the first threshold can be set manually as needed, for example, set to 8.

[0093] For example, when the preset number of sub-containers is 3 and the number of the first threshold is 8, it can be determined that the minimum multiple (i.e., the first parameter) for the result obtained by multiplying the preset number by a factor is greater than or equal to the first threshold is 3. It can be understood that the first parameter can be calculated by the following formula: number of sub-containers = preset number * first parameter. That is to say, three virtual nodes need to be configured for each sub-container, thereby determining that the number of sub-containers on the hash ring is 9.

[0094] In one embodiment, determining the number of sub-containers on the hash ring based on the preset quantity includes:

[0095] When the preset quantity is greater than or equal to the first threshold, the preset quantity is determined as the number of sub-containers on the hash ring.

[0096] For example, such as Figure 3 As shown, the process of determining the number of sub-containers on a hash ring may include the following steps:

[0097] Step 301: Read the preset number of sub-containers;

[0098] In practical applications, the preset number of sub-containers can be determined by the configuration parameters of the first container, and then step 302 can be executed.

[0099] Step 302: Determine whether the preset quantity is less than the first threshold;

[0100] Here, when the preset number of sub-containers is greater than or equal to the first threshold, step 305 is executed; if the preset number of sub-containers is less than the first threshold, step 303 is executed.

[0101] Step 303: Determine if the preset number of child containers is greater than 1;

[0102] When the preset quantity is greater than 1, proceed to step 304; when the preset quantity is less than or equal to 1, proceed to step 306.

[0103] Step 304: Determine the first parameter;

[0104] The first parameter represents the minimum multiple at which the result of multiplying the preset quantity by a factor is greater than or equal to the first threshold.

[0105] Step 305: Determine the number of sub-containers on the hash ring;

[0106] Here, the first parameter can be used to determine the number of sub-containers on the hash ring; specifically, the number = preset number * first parameter.

[0107] Step 306: Determine that the configuration parameters are invalid;

[0108] In practical applications, if the preset quantity is less than or equal to 1, the preset quantity is considered an invalid parameter.

[0109] Here, after determining that the configuration parameter being read is invalid, step 307 is then executed.

[0110] Step 307: End the current processing flow.

[0111] In practical applications, after obtaining the hash value corresponding to the name and the hash rings corresponding to at least two sub-containers, the position of the first object on the hash ring is determined to identify the first sub-container.

[0112] Based on this, in one embodiment, determining the first sub-container using the hash value corresponding to the name and the hash rings corresponding to the at least two sub-containers includes:

[0113] The position of the first object on the hash ring is determined using the hash value corresponding to the name;

[0114] The first sub-container is determined based on the position of the first object on the hash ring and the position of each sub-container on the hash ring.

[0115] In practical applications, when the preset number of sub-containers is greater than or equal to the first threshold, the number of sub-containers on the hash ring is equal to the preset number. In this case, the electronic device can obtain the identifier corresponding to each of the preset number of sub-containers through the configuration information of the first container. Then, it obtains the hash value corresponding to each sub-container through the hash function in the consistent hashing algorithm (exemplarily, the hash function can be understood as taking the modulo of the identifier corresponding to the sub-container), thereby determining the position of each sub-container on the hash ring.

[0116] Here, after determining the position of the first object on the hash ring, it can be agreed that the first sub-container encountered in a clockwise direction from the position is the corresponding first sub-container.

[0117] When the preset number of sub-containers is less than a first threshold, corresponding virtual nodes are configured for the preset number of sub-containers to achieve uniform storage of objects. In this case, after the electronic device reads the configuration information to obtain the identifiers of the preset number of sub-containers, it adds a corresponding value, such as ID001, ID002, and ID003, to each sub-container identifier as the ID of the virtual node corresponding to the sub-container. Then, based on the ID of the virtual node, the position of each virtual node on the hash ring is determined. At this point, after determining the position of the first object on the hash ring, the virtual node closest to the position of the first object on the hash ring can be found clockwise from the position, and the sub-container corresponding to the found virtual node is used as the first container.

[0118] For example, such as Figure 4 As shown, for objects 1, 2, 3, and 4, their positions on the hash ring can be determined by calculating their hash values ​​using a hash function. Similarly, for sub-containers 1, 2, 3, and 4, their positions on the hash ring can be determined by calculating their hash values ​​using a hash function. Furthermore, based on the principles of consistent hashing, it can be determined that object 1 is stored in sub-container 1, object 2 in sub-container 4, object 3 in sub-container 2, and object 4 in sub-container 3, respectively.

[0119] In practical applications, when the electronic device creates a sub-container for each sub-container of the first container, it can set a corresponding number of shards for each sub-container and store the first object based on the shards. In this way, for each sub-container, the efficiency of traversal can be further improved by traversing different shards in parallel, thereby improving the processing performance of the container.

[0120] Based on this, in one embodiment, storing the first object in the first sub-container includes:

[0121] The number of shards for the first sub-container is determined using the configuration parameters of the first container.

[0122] Based on the number of shards, the first sub-container is sharded;

[0123] The shard corresponding to the first object is determined using the consistent hashing algorithm and the first sub-container after sharding;

[0124] Store the first object in the shard corresponding to the first object.

[0125] In practical applications, the configuration information of the first container may include the sharding configuration information of each sub-container. The electronic device may determine the number of shards and the corresponding shard identifier (such as shard ID) of the sub-container based on the sharding configuration information. The sharding configuration information can be manually set according to requirements.

[0126] In practical applications, after determining the first sub-container, a virtual hash ring is first generated using the consistent hashing algorithm, with values ​​ranging from 0 to 2 on the ring. 32 The hash values ​​are composed of [a set of hash values] arranged clockwise. Then, the hash value corresponding to the name of the first object and the hash value corresponding to each shard identifier of the first sub-container are determined using the hash function in the consistent hash algorithm. Next, based on the hash value corresponding to the name, the position of the first object on the hash ring is determined. Simultaneously, based on the hash value corresponding to each shard, the position of each shard on the hash ring is determined. Finally, the first shard found clockwise from the position of the first object on the hash ring is used as the shard for storing the first object, and the first object is stored in the found shard.

[0127] The object storage method provided in the embodiments of this application, such as Figure 5 As shown, after receiving a first request, at least two sub-containers associated with the first container are determined; the first request is used to request the storage of a first object in the first container; the first request carries at least the first object; using a consistent hashing algorithm and the determined at least two sub-containers, the first sub-container corresponding to the first object is determined, and the first object is stored in the first sub-container. The technical solution provided in this application, for an object storage request of a container, determines at least two sub-containers associated with the container and uses a consistent hashing algorithm to determine the sub-container corresponding to the object, so as to facilitate the storage of the object in the corresponding sub-container. By storing the object in at least two sub-containers associated with the container, it is possible to achieve parallel traversal of each sub-container when processing user traversal requests, avoiding the problems of long traversal time and low efficiency caused by storing the object in only one container, thus improving the traversal performance and efficiency of the container.

[0128] In addition, in this embodiment of the application, the preset number of sub-containers can be obtained through the configuration parameters of the first container. At the same time, by comparing the preset number of sub-containers with a threshold, the object is stored by setting virtual nodes. This not only enables the objects to be stored evenly in the sub-containers, but also avoids the waste of storage resources by supporting the dynamic addition or reduction of sub-containers.

[0129] To implement the solutions of the embodiments of this application, the embodiments of this application also provide an object storage device, disposed on an electronic device, such as... Figure 6 As shown, the device includes:

[0130] Receiving unit 601 is configured to receive a first request; the first request is configured to request that a first object be stored in a first container; the first request carries at least the first object;

[0131] The first processing unit 602 is configured to determine at least two sub-containers associated with the first container;

[0132] The second processing unit 603 is used to determine the first sub-container corresponding to the first object using a consistent hashing algorithm and at least two determined sub-containers;

[0133] Storage unit 604 is used to store the first object into the first sub-container.

[0134] In practical applications, the object storage device can be deployed on the corresponding node of RGW.

[0135] In one embodiment, the first request also carries the name of the first object; the second processing unit 603 is configured to:

[0136] The hash value corresponding to the name is calculated using the consistent hashing algorithm;

[0137] The first sub-container is determined using the hash value corresponding to the name and the hash rings corresponding to the at least two sub-containers; the hash rings are obtained using a consistent hashing algorithm.

[0138] In one embodiment, the second processing unit 603 is further configured to:

[0139] The preset number of sub-containers is determined using the configuration parameters of the first container;

[0140] Based on the preset quantity, determine the number of sub-containers on the hash ring;

[0141] The hash ring is determined based on the number of sub-containers on the hash ring, the sub-container identifier in the configuration parameters of the first container, and the consistent hashing algorithm.

[0142] In one embodiment, the second processing unit 603 is configured to:

[0143] When the preset quantity is lower than the first threshold, a first parameter is determined; the first parameter represents the minimum multiple by which the result obtained by multiplying the preset quantity is greater than or equal to the first threshold.

[0144] The number of sub-containers on the hash ring is determined using the first parameter.

[0145] In one embodiment, the second processing unit 603 is configured to:

[0146] When the preset quantity is greater than or equal to the first threshold, the preset quantity is determined as the number of sub-containers on the hash ring.

[0147] In one embodiment, the second processing unit 603 is configured to:

[0148] The position of the first object on the hash ring is determined using the hash value corresponding to the name;

[0149] The first sub-container is determined based on the position of the first object on the hash ring and the position of each sub-container on the hash ring.

[0150] In one embodiment, the storage unit 604 is used for:

[0151] The number of shards for the first sub-container is determined using the configuration parameters of the first container.

[0152] Based on the number of shards, the first sub-container is sharded;

[0153] The shard corresponding to the first object is determined using the consistent hashing algorithm and the first sub-container after sharding;

[0154] Store the first object in the shard corresponding to the first object.

[0155] In practical applications, the receiving unit 601 can be implemented by a processor in the object storage device; the first processing unit 602, the second processing unit 603 and the storage unit 604 can be implemented by a processor in the object storage device.

[0156] It should be noted that the object storage device provided in the above embodiments is only illustrated by the division of the above-described program units when performing object storage. In practical applications, the above processing can be assigned to different program units as needed, that is, the internal structure of the device can be divided into different program units to complete all or part of the processing described above. In addition, the object storage device and the object storage method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0157] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiments of this application, the embodiments of this application also provide an electronic device, such as... Figure 7 As shown, the electronic device 700 includes:

[0158] The communication interface 701 enables interaction with other devices;

[0159] The processor 702 is connected to the communication interface 701 to enable interaction with other devices and to execute the methods provided by one or more technical solutions of the above-mentioned electronic device when running a computer program;

[0160] The computer program is stored in memory 703.

[0161] Specifically, the communication interface 701 is used to receive a first request; the first request is used to request that a first object be stored in a first container; the first request carries at least the first object;

[0162] The processor 702 is configured to determine at least two sub-containers associated with the first container;

[0163] Using a consistent hashing algorithm and at least two determined sub-containers, determine the first sub-container corresponding to the first object;

[0164] Store the first object in the first sub-container.

[0165] In one embodiment, the first request also carries the name of the first object; the processor 702 is configured to:

[0166] The hash value corresponding to the name is calculated using the consistent hashing algorithm;

[0167] The first sub-container is determined using the hash value corresponding to the name and the hash rings corresponding to the at least two sub-containers; the hash rings are obtained using a consistent hashing algorithm.

[0168] In one embodiment, the processor 702 is further configured to:

[0169] The preset number of sub-containers is determined using the configuration parameters of the first container;

[0170] Based on the preset quantity, determine the number of sub-containers on the hash ring;

[0171] The hash ring is determined based on the number of sub-containers on the hash ring, the sub-container identifier in the configuration parameters of the first container, and the consistent hashing algorithm.

[0172] In one embodiment, the processor 702 is configured to:

[0173] When the preset quantity is lower than the first threshold, a first parameter is determined; the first parameter represents the minimum multiple by which the result obtained by multiplying the preset quantity is greater than or equal to the first threshold.

[0174] The number of sub-containers on the hash ring is determined using the first parameter.

[0175] In one embodiment, the processor 702 is configured to:

[0176] When the preset quantity is greater than or equal to the first threshold, the preset quantity is determined as the number of sub-containers on the hash ring.

[0177] In one embodiment, the processor 702 is configured to:

[0178] The position of the first object on the hash ring is determined using the hash value corresponding to the name;

[0179] The first sub-container is determined based on the position of the first object on the hash ring and the position of each sub-container on the hash ring.

[0180] In one embodiment, the processor 702 is configured to:

[0181] The number of shards for the first sub-container is determined using the configuration parameters of the first container.

[0182] Based on the number of shards, the first sub-container is sharded;

[0183] The shard corresponding to the first object is determined using the consistent hashing algorithm and the first sub-container after sharding;

[0184] Store the first object in the shard corresponding to the first object.

[0185] It should be noted that the specific processing procedure of processor 702 can be understood by referring to the above method.

[0186] Of course, in practical applications, the various components in an electronic device are coupled together through a bus system 704. It can be understood that the bus system 704 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 7 The general designated all buses as Bus System 704.

[0187] The memory 703 in this embodiment is used to store various types of data to support the operation of the electronic device 700. Examples of such data include any computer program used to operate on the electronic device 700.

[0188] The methods disclosed in the embodiments of this application can be applied to the processor 702, or implemented by the processor 702. The processor 702 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 702 or by instructions in the form of software. The processor 702 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 702 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 703. The processor 702 reads the information in the memory 703 and combines its hardware to complete the steps of the aforementioned method.

[0189] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0190] It should be noted that the specific processing procedure of the electronic device 700 has been described in detail above and will not be repeated here.

[0191] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 703 storing a computer program, which can be executed by the processor 702 of the electronic device 700 to complete the steps of the aforementioned object storage method. The computer-readable storage medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk storage device or a magnetic tape storage device.

[0192] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0193] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0194] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An object storage method, characterized in that, include: Receive the first request; The first request is used to request that the first object be stored in the first container; The first request carries at least the first object; Identify at least two sub-containers associated with the first container; Using a consistent hashing algorithm and at least two determined sub-containers, determine the first sub-container corresponding to the first object; The first object is stored in the first sub-container; wherein... The first request also carries the name of the first object; determining the first sub-container corresponding to the first object using a consistent hashing algorithm and at least two determined sub-containers includes: The hash value corresponding to the name is calculated using the consistent hashing algorithm; The first sub-container is determined using the hash value corresponding to the name and the hash rings corresponding to the at least two sub-containers; the hash rings are obtained using a consistent hashing algorithm. The method further includes: The preset number of sub-containers is determined using the configuration parameters of the first container; Based on the preset quantity, determine the number of sub-containers on the hash ring; The hash ring is determined based on the number of sub-containers on the hash ring, the sub-container identifier in the configuration parameters of the first container, and the consistent hashing algorithm.

2. The method according to claim 1, characterized in that, Determining the number of sub-containers on the hash ring based on the preset quantity includes: When the preset quantity is lower than the first threshold, a first parameter is determined; the first parameter represents the minimum multiple by which the result obtained by multiplying the preset quantity is greater than or equal to the first threshold. The number of sub-containers on the hash ring is determined using the first parameter.

3. The method according to claim 1, characterized in that, Determining the number of sub-containers on the hash ring based on the preset quantity includes: When the preset quantity is greater than or equal to the first threshold, the preset quantity is determined as the number of sub-containers on the hash ring.

4. The method according to claim 1, characterized in that, The step of determining the first sub-container using the hash value corresponding to the name and the hash rings corresponding to the at least two sub-containers includes: The position of the first object on the hash ring is determined using the hash value corresponding to the name; The first sub-container is determined based on the position of the first object on the hash ring and the position of each sub-container on the hash ring.

5. The method according to any one of claims 1 to 4, characterized in that, The step of storing the first object into the first sub-container includes: The number of shards for the first sub-container is determined using the configuration parameters of the first container. Based on the number of shards, the first sub-container is sharded; The shard corresponding to the first object is determined using the consistent hashing algorithm and the first sub-container after sharding; Store the first object in the shard corresponding to the first object.

6. An object storage device, characterized in that, include: The receiving unit is used to receive the first request; The first request is used to request that the first object be stored in the first container; The first request carries at least the first object; A first processing unit is configured to determine at least two sub-containers associated with the first container; The second processing unit is used to determine the first sub-container corresponding to the first object using a consistent hashing algorithm and at least two determined sub-containers; A storage unit is used to store the first object in the first sub-container; wherein the first request also carries the name of the first object; The second processing unit is configured to calculate the hash value corresponding to the name using a consistent hashing algorithm; and determine the first sub-container using the hash value corresponding to the name and the hash rings corresponding to the at least two sub-containers; wherein the hash rings are obtained using a consistent hashing algorithm. The second processing unit is further configured to: determine a preset number of sub-containers using the configuration parameters of the first container; determine the number of sub-containers on the hash ring based on the preset number; and determine the hash ring based on the number of sub-containers on the hash ring, the sub-container identifier in the configuration parameters of the first container, and the consistent hashing algorithm.

7. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the object storage method according to any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the object storage method according to any one of claims 1 to 5.

Images