Distributed object storage method, apparatus, device, and medium

Abstract

Embodiments of the present disclosure relate to a distributed object storage method, apparatus, device and medium, wherein the method comprises: splitting a target object to be stored into a plurality of key-value pair data; determining a target partition corresponding to each of the plurality of key-value pair data from a plurality of partitions of a preset distributed object storage system; determining a data storage node currently holding the target partition based on a preset partition view to obtain a data storage node corresponding to each of the plurality of key-value pair data; wherein the partition view is used to indicate meta information of the plurality of partitions, and the meta information at least contains information of the data storage node holding the partition; the data storage node is a storage node of the distributed object storage system; and caching the plurality of key-value pair data into the corresponding data storage node. Embodiments of the present disclosure can effectively alleviate the local overload problem of the object storage service cluster.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a distributed object storage method, apparatus, device, and medium. Background Technology

[0002] Object storage services can store and access various unstructured data files, including text, images, audio, and video, anytime via the network. Typically provided to users as cloud services, they offer advantages such as high scalability, low cost, and high durability, effectively meeting massive data storage needs. However, existing object storage services suffer from cluster overload issues, where the load on some nodes exceeds their processing capacity, impacting service availability. Therefore, improvements to object storage services are urgently needed. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a distributed object storage method, apparatus, device, and medium.

[0004] This disclosure provides a distributed object storage method, the method comprising: splitting a target object to be stored into multiple key-value pairs; determining a target partition corresponding to each of the multiple key-value pairs from multiple partitions of a preset distributed object storage system; determining the data storage node currently holding the target partition based on a preset partition view, thereby obtaining the data storage node corresponding to each of the multiple key-value pairs; wherein the partition view is used to indicate the metadata of the multiple partitions, the metadata including at least information about the data storage node holding the partition; the data storage node is a storage node of the distributed object storage system; and caching the multiple key-value pairs in their respective corresponding data storage nodes.

[0005] Optionally, determining the target partition corresponding to each of the multiple key-value pairs from multiple partitions of a preset distributed object storage system includes: performing hash processing on each of the multiple key-value pairs to obtain a hash value corresponding to each of the multiple key-value pairs; and determining the target partition corresponding to each of the multiple key-value pairs from multiple partitions of the preset distributed object storage system based on the hash values ​​corresponding to each of the multiple key-value pairs.

[0006] Optionally, the plurality of partitions includes a first partition, which is a partition with replica data, and the first partition corresponds to a plurality of data storage nodes, wherein the plurality of identical replica data corresponding to the first partition are distributed on different data storage nodes.

[0007] Optionally, the plurality of data storage nodes corresponding to the first partition include a master node and a slave node, wherein the master node is used to write data to the first partition, the slave node is used to copy the data written to the first partition by the master node, and either the master node or the slave node is used to read data from the first partition.

[0008] Optionally, the multiple data storage nodes corresponding to the first partition are determined in the following way: based on a uniform allocation strategy, multiple data storage nodes corresponding to each first partition in the multiple partitions are determined from the currently available data storage node set, wherein the uniform allocation strategy is used to indicate that the difference in the number of partitions held by different data storage nodes is within a preset range.

[0009] Optionally, the method further includes: if the currently available set of data storage nodes changes, reallocating multiple data storage nodes corresponding to the first partition based on the changed set of currently available data storage nodes and the uniform allocation strategy.

[0010] Optionally, the target object to be stored does not belong to the first-level hot data. The first-level hot data is data whose current popularity is higher than a preset popularity threshold. The first-level hot data is used to store in the client cache.

[0011] Optionally, before performing the step of splitting the target object to be stored into multiple key-value pairs, the method further includes: upon receiving the object to be stored, determining whether the object to be stored is long-tail data; if so, determining that the object to be stored is the target object to be stored; wherein the target object to be stored is used to store in the corresponding data storage node on the server; if not, caching the object to be stored in the memory of the server.

[0012] This disclosure also provides a distributed object storage device, comprising: an object splitting module for splitting a target object to be stored into multiple key-value pairs; a partition determination module for determining the target partition corresponding to each of the multiple key-value pairs from multiple partitions of a preset distributed object storage system; a node determination module for determining the data storage node currently holding the target partition based on a preset partition view, thereby obtaining the data storage node corresponding to each of the multiple key-value pairs; wherein the partition view is used to indicate the metadata of the multiple partitions, and the metadata includes at least information about the data storage node holding the partition; the data storage node is a storage node of the distributed object storage system; and a data caching module for caching the multiple key-value pairs to their respective data storage nodes.

[0013] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the distributed object storage method provided in this disclosure.

[0014] This disclosure also provides a computer-readable storage medium storing a computer program for executing the distributed object storage method provided in this disclosure.

[0015] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the distributed object storage method provided in this disclosure.

[0016] The technical solution provided in this disclosure can split the target object to be stored into multiple key-value pairs, determine the target partition corresponding to each key-value pair from multiple partitions of a preset distributed object storage system, and then determine the data storage node currently holding the target partition based on a preset partition view, thus obtaining the data storage node corresponding to each key-value pair. The multiple key-value pairs can then be cached in their respective data storage nodes. Through this method, the target object can be split into key-value pairs, and based on the target partitions corresponding to the split key-value pairs, the multiple key-value pairs can be distributed and cached in their respective data storage nodes, achieving the purpose of distributed node load and effectively alleviating the problem of local overload in the object storage service cluster.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram of the overall cluster architecture of an object storage service provided in this embodiment of the disclosure;

[0021] Figure 2 A flowchart illustrating a distributed object storage method provided in an embodiment of this disclosure;

[0022] Figure 3 This is a schematic diagram illustrating the copying of partitioned data according to an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of object storage provided in an embodiment of this disclosure;

[0024] Figure 5 This is a schematic diagram of a partition reallocation provided in an embodiment of the present disclosure;

[0025] Figure 6 This is a schematic diagram of a multi-level storage architecture provided in an embodiment of the present disclosure;

[0026] Figure 7 This is a schematic diagram illustrating an application of distributed object storage provided in an embodiment of this disclosure;

[0027] Figure 8 This is a schematic diagram of the structure of a distributed object storage device provided in an embodiment of the present disclosure;

[0028] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0029] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0030] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0031] To facilitate understanding of object storage services, this disclosure first provides the following embodiments: Figure 1 The diagram illustrates the overall cluster architecture of an object storage service. Users can initiate requests such as read / write requests to the access layer of the cluster via domain name or SDK (Software Development Kit). For ease of understanding, the access layer, storage layer, and index layer are described below:

[0032] The access layer can parse based on the object storage protocol, thereby retrieving the data corresponding to the read and write requests in the storage layer and the index layer. Specifically, based on the read and write requests, the access layer can determine the corresponding object metadata in the index layer and the object data in the storage layer, thereby performing read IO operations and read IO (Input / Output) operations, and then return the operation results to the upper layer according to the protocol requirements.

[0033] Storage layer: can contain multiple data storage nodes for storing object data, and can specifically use a distributed pooled storage system as the underlying storage engine.

[0034] Index layer: It can contain multiple partition nodes to store object metadata, such as object name, object size, content type, data distribution on the storage engine, etc. In existing technologies, it is mostly implemented using dictionary-ordered technology.

[0035] Based on the above architecture, a cluster local overload problem can easily occur, meaning that the load on some nodes in the cluster exceeds their processing capacity, thereby affecting service availability. For example, the following scenarios can all lead to this problem:

[0036] (1) Hot object scenario: A large number of users request to download the same object, and the data IO pressure is concentrated on a single partition of the index layer and a few data disks of the storage layer, causing the corresponding data nodes of the index layer or storage layer to be overloaded.

[0037] (2) High QPS (Queries Per Second) download scenario: Users repeatedly read a batch of data at a high QPS, and the data IO pressure is concentrated on several nodes in the index layer or storage layer, causing local node CPU (Central Processing Unit) overload.

[0038] (3) High-throughput download scenario: Users concurrently load a batch of large object data, and the data bandwidth pressure is concentrated on several nodes of the storage layer, causing the disk bandwidth or network bandwidth of local nodes to reach the upper limit.

[0039] (4) Download acceleration scenario: In acceleration scenarios, it is generally desirable to have a shorter data IO link. On the other hand, due to cost considerations, the underlying index and storage may use slow storage devices.

[0040] To address the cluster overload issue that occurs in one or more of the above scenarios, this disclosure provides a distributed object storage method, apparatus, device, and medium, which are well-suited for the overall cluster architecture of the aforementioned object storage services. For example, the functional layers of the architecture, such as the access layer, can be improved based on the distributed object storage method provided in this disclosure. For ease of understanding, a detailed explanation follows.

[0041] Figure 2 This is a flowchart illustrating a distributed object storage method provided in an embodiment of this disclosure. The method can be executed by a distributed object storage device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device, such as a server. Figure 2 As shown, the method mainly includes the following steps S202 to S208:

[0042] Step S202 involves splitting the target object to be stored into multiple key-value pairs. These key-value pairs can also be referred to as KV (key-value) sharding.

[0043] This disclosure does not limit the target object, which may specifically include object metadata and object data. In some embodiments, the target object to be stored can be any received object to be stored; in other embodiments, the target object to be stored can be an object that has been determined to be cached in a partition of a data storage node after multi-level filtering. This is not a limitation. This disclosure does not limit the method of splitting the target object. For example, the target object can be converted into key-value pairs through a specific slicing method. Exemplarily, one or more of the following implementations can be used for the target object: field splitting, size splitting, logical splitting, and time series splitting. These are all examples and should not be considered as limitations. By splitting the target object, regardless of its size, it can be broken down into smaller key-value pairs, which facilitates subsequent processing.

[0044] Step S204: Determine the target partitions corresponding to each key-value pair of data from multiple partitions of the preset distributed object storage system. In practical applications, the partitions mentioned in this embodiment can also be called logical partitions corresponding to user data. The number of partitions can be preset when creating the distributed object storage system (also referred to as a storage cluster), such as N partitions, and this number can be kept unchanged thereafter. In practical applications, partitioning can be based on the scale of the distributed object storage system. Specifically, the storage space of the distributed object storage system can be pre-divided into N logical partitions for storing user data, which is to obtain the aforementioned multiple partitions of the preset distributed object storage system. After obtaining multiple key-value pairs, this embodiment can determine the target partition corresponding to each key-value pair based on a partitioning strategy. To ensure that the multiple key-value pairs of the target object can be well distributed across multiple partitions, the partitioning strategy can be a hash partitioning strategy. Specifically, in some implementations, the multiple key-value pairs can be hashed separately to obtain their respective hash values. Based on the hash values ​​of the multiple key-value pairs, the target partitions corresponding to each key-value pair are determined from multiple partitions of a pre-defined distributed object storage system. The hash algorithm used in the hash processing can be flexibly set. This embodiment can use hashing for partitioning, storing each key-value pair in a randomized manner. The amount of metadata required to be recorded in each partition will be correspondingly less. In addition, this method can also effectively ensure load balancing.

[0045] Step S206: Based on a preset partition view, determine the data storage node currently holding the target partition, obtaining the data storage nodes corresponding to each of the multiple key-value pairs. The partition view indicates the metadata of multiple partitions, which includes at least information about the data storage node holding the partition. The data storage node is a storage node in a distributed object storage system. The data storage node information includes identification information, as well as basic node information such as location and capacity. By recording the data storage node holding the partition in the metadata, the data storage node holding the partition can be accurately located. In other words, the correspondence between each partition and the data storage node can be accurately determined, thus quickly identifying the data storage node holding the target partition based on the metadata, given the known target partition. Furthermore, the partition's metadata may also include information such as the partition's location and allowed capacity, which is not limited here. In practical applications, a single data storage node can hold multiple partitions simultaneously, and the target partitions corresponding to multiple key-value pairs can be held by different data storage nodes.

[0046] Step S208: Cache multiple key-value pairs of data into their respective data storage nodes.

[0047] The above method enables key-value pair splitting of the target object. Based on the target partitions corresponding to the multiple key-value pairs obtained from the splitting, the multiple key-value pairs are distributed and cached in the corresponding data storage nodes, achieving the purpose of node load distribution and thus effectively alleviating the problem of local overload in the object storage service cluster.

[0048] In some implementations, multiple partitions include a first partition, which is a partition with replicated data, and the first partition corresponds to multiple data storage nodes. Multiple identical replicates of the first partition are distributed across different data storage nodes. In practical applications, each of the aforementioned multiple partitions can be the first partition, meaning each partition can be a partition with replicated data. It is understood that the data in the first partition itself is held by the corresponding data storage node, and the multiple replicates of the first partition's data are also held by different data storage nodes; therefore, it can be considered that the first partition corresponds to multiple data storage nodes.

[0049] In practical applications, the data corresponding to a partition can be copied to multiple storage devices. This data redundancy addresses data reliability and availability issues, ensuring that even if a single storage device fails, data loss or inaccessibility will not occur. Furthermore, it distributes I / O access pressure across multiple nodes. For example, such as... Figure 3 The diagram illustrates a method for copying partitioned data, which can copy partitioned data multiple times to obtain multiple copies of the data. Figure 3 The example of creating two additional copies is merely a simplified illustration and should not be considered a limitation. In practical applications, the multiple data storage nodes corresponding to the first partition include a master node and slave nodes. The master node is used to write data to the first partition, and the slave nodes are used to replicate the data written by the master node. Additionally, both the master and slave nodes are used to read data from the first partition. In other words, all write requests are processed by the master node, and then the data written by the master node is replicated to the slave nodes synchronously or asynchronously. This processing mode is called the master-slave topology, and it effectively ensures data consistency among the replicas.

[0050] For easier understanding, you can also refer to, for example Figure 4 The diagram shown illustrates an object storage method. Figure 4 The diagram illustrates that the target object to be stored is divided into multiple key-value pairs, and these key-value pairs are mapped to different partitions. Figure 4 The diagram briefly illustrates three partitions, each capable of holding multiple key-value pairs. Figure 4 As can be seen, multiple key-value pairs can be distributed and mapped to multiple partitions. The specific mapping method depends on the specific algorithm, such as hash partitioning, and is not limited here. Furthermore, Figure 4 The diagram also illustrates that partitioned data can be placed on different data storage nodes as multiple mirror copies. This method can distribute node load and alleviate the problem of localized cluster overload.

[0051] To further ensure load balancing and evenly distribute partitions within the cluster across data storage nodes, in some specific implementations, the multiple data storage nodes corresponding to the first partition are determined as follows: Based on a uniform distribution strategy, multiple data storage nodes corresponding to each first partition are determined from the currently available data storage node set. The uniform distribution strategy indicates that the difference in the number of partitions held by different data storage nodes is within a preset range. This preset range can be flexibly set, for example, 0 to M. For any two data storage nodes, if the difference in the number of partitions they hold is within this range, then these two data storage nodes can be considered to have evenly distributed partitions. The currently available data storage node set can be obtained through a node view, which can be used to indicate information about the currently available data storage nodes. In practical applications, based on the number of partitions indicated by the partition view and the replication strategy of each partition, each partition and its corresponding replica can be evenly distributed across the data storage nodes, generating a partition list. The partition list can be used to indicate the partition ID and information about the data storage nodes holding each partition (such as access address, master / slave information, etc.), without limitation.

[0052] Furthermore, the method provided in this disclosure embodiment also includes: when the currently available set of data storage nodes changes, i.e., when data storage nodes become offline or go offline, the multiple data storage nodes corresponding to the first partition can be reallocated based on the changed set of currently available data storage nodes and a uniform allocation strategy. For ease of understanding, taking the case where data storage nodes go offline as an example, please refer to... Figure 5 The diagram illustrates a partition redistribution scheme. The left image shows available nodes A, B, and C. Taking four partitions, each corresponding to two data storage nodes, as an example, the data and replicas for partition 1 are held by nodes A and B respectively. Similarly, partition 2 corresponds to nodes C and A, partition 3 to nodes B and C, and partition 4 to nodes A and B. In other words, node A simultaneously holds data for partitions 1, 2, and 4; node B holds data for partitions 1, 3, and 4; and node C holds data for partitions 2, 3, and 4. This allocation method can achieve good load balancing. After node C goes offline, partition redistribution is performed based on a uniform allocation strategy. (See [link to relevant documentation]). Figure 5In the diagram on the right, since partition 1 and partition 4 both correspond to nodes A and B, they are unaffected. Partition 2, which originally corresponded to nodes C and A, is reassigned to nodes B and A. Partition 3, which originally corresponded to nodes B and C, is reassigned to nodes B and A. In other words, after the reassignment, nodes A and B both hold data from partitions 1, 2, 3, and 4, thus achieving load balancing.

[0053] Considering the limitations of existing single-disk caching methods, which still pose a risk of stability degradation under high single-disk load during periods of high-volume request traffic, and to rationally allocate cache resources and improve high-concurrency, low-latency single-machine read / write capabilities, this embodiment of the disclosure can adopt a multi-level caching strategy. First-level hot data (also known as header hot data) is directly cached on the client side. That is, the target object to be stored in this embodiment of the disclosure does not belong to first-level hot data; first-level hot data is data whose current popularity exceeds a preset popularity threshold, and it is stored in the client-side cache. For example, an algorithm such as LRU (Least Recently Used) can be used to prioritize the eviction of the least recently used data, thereby caching first-level hot data on the client side. A small amount of space (e.g., 1%) is used to cache recent first-level hot data on the client side to facilitate rapid response to user requests for this data, greatly reducing latency for such data and alleviating server pressure. As mentioned above, primary hot data can be cached on the client side, while non-primary hot data can be stored on the server side. Specifically, before performing the step of splitting the target object to be stored into multiple key-value pairs, the method provided in this disclosure embodiment further includes the following steps a to c:

[0054] Step a: Upon receiving the object to be stored, determine whether the object is long-tail data. If yes, proceed to step b; otherwise, proceed to step c. It is understood that most of the stored data in object storage services exhibits a long-tail distribution, consisting of a large volume of data with low access frequency. This type of data is long-tail data. Methods for identifying long-tail data can be found in relevant technologies and will not be elaborated upon here. For example, a Bloom filter can be used to determine whether the object to be stored is long-tail data.

[0055] Step b involves determining the object to be stored as the target object; whereby the target object is stored in the data storage node corresponding to the server. If the object to be stored is long-tail data, the aforementioned method provided in this disclosure can be used to distribute the multiple key-value pairs of the object across multiple data storage nodes, i.e., storage via the external storage corresponding to the server. In other words, object storage is performed via the disk corresponding to the server, and data can be written using methods such as Append Only, meaning that each write operation appends to the end of existing data on the disk, rather than updating or deleting at random positions. This helps reduce random read / write operations on the disk and improves write efficiency. Furthermore, a FIFO (First In First Out) algorithm can be used to evict the earliest written data; in other words, when space needs to be freed, deletion starts from the earliest written data. In practical applications, the aforementioned target object to be stored can be cached on the server using the server's partitioning engine.

[0056] Step c: Cache the objects to be stored in the server's memory. If the object to be stored is not long-tail data, it indicates relatively hot data, and therefore can be preferentially stored in the server's memory. Admission control can be performed using algorithms such as TinyLFU (Tiny Least Frequently Used) to determine whether the data can enter the memory cache. When new data needs to be stored, if memory is full, less frequently used data may be evicted according to the TinyLFU algorithm. Furthermore, a forced admission policy can be set to allow some objects to be stored to be written to the server's memory even if they are not approved by TinyLFU.

[0057] For easier understanding, please refer to Figure 6 The diagram illustrates a multi-level storage architecture where frequently accessed data is prioritized for caching on the client side, relatively frequently accessed data is cached in the server's memory, and subsequent data is stored in the server's disk cache. In practical applications, server-side memory caching can be implemented using a server-side memory engine, leveraging server-side memory to provide faster access speeds. Server-side disk caching can be implemented using a server-side partitioning engine, utilizing corresponding data storage nodes on the server for partitioned storage. Data not cached in client or server memory can be cached using the server's disk. Furthermore... Figure 6 The diagram also illustrates the data source, which can be persistently stored. When querying the stored data, it's possible to first query the data cached on the client (first-level query), then query the data stored on the server (second-level query), and finally query the source data.

[0058] Furthermore, for ease of understanding, embodiments of this disclosure also provide, as follows: Figure 7 This diagram illustrates an application of distributed object storage. Users can implement object storage through either Access Layer 1 or Access Layer 2. For example, Access Layer 1 can be an application programming interface (API) for object storage, specifically corresponding to SDK traffic. Access Layer 2 can be a routing server, specifically corresponding to domain name traffic. The SDKs illustrated in Access Layer 1 and Access Layer 2 can be SDKs providing caching services; that is, the access layer needs to access the caching service based on this SDK. In other words, the access layer performs read and write caching through the SDK providing caching services. The target object to be stored can be converted into key-value pairs through a specific slicing method and written to the caching service. The caching service corresponds to two types of servers: one type corresponds to a metadata management node, and the other type corresponds to a data storage node. The metadata management node is responsible for cluster data partitioning, partition scheduling, and replica master election, etc. Data access can be achieved by querying the partition distribution information through the metadata management node. Figure 7 The arrowed circles illustrating multiple metadata management nodes briefly indicate that synchronization and leader election are possible between these nodes. Data storage nodes can provide semantic interfaces such as key-value pairs and can read and write the partitioned data they hold. Figure 7 The diagram briefly illustrates how the access layer communicates with the partition API (Application Programming Interface) through a caching service SDK, thereby enabling data read and write operations via data storage nodes. Partition data can have multiple replicas, and correspondingly, the multiple data storage nodes for each partition can be classified as master and slave nodes. Data storage nodes and metadata management nodes can interact to achieve data read and write operations. Subsequently, during data retrieval, the metadata management node can determine which data storage node the partition corresponding to the key-value pair data is located on, thus ensuring accurate access to that node.

[0059] In summary, the distributed object storage method provided in this disclosure has the following advantages:

[0060] (1) By splitting an object into multiple key-value pairs, these pairs can be distributed across multiple pre-defined partitions and held by multiple data storage nodes. In other words, the object can be split and stored on multiple data storage nodes, effectively solving the single-machine storage space problem. In addition, distributing the load across multiple nodes not only helps to achieve load balancing but also effectively improves service throughput.

[0061] (2) Data redundancy is achieved through data replication, synchronizing partitioned replica data to multiple data storage nodes. This provides a certain degree of disaster recovery capability for cached data and allows for reading replicas located on different data storage nodes based on demand in practical applications. This solves the problem of requests being concentrated on a single node in hotspot scenarios and increases the upper limit of read capacity in the caching system. At the same time, data replication effectively ensures data consistency among replicas.

[0062] (3) Automatic data scheduling can be achieved. In other words, when data storage nodes go online or offline, partitions can be automatically redistributed according to node load. For example, partitions can be automatically scheduled to data storage nodes with lower loads without manual intervention. This better realizes the elastic scaling capability of the cache system and effectively ensures load balancing.

[0063] (4) A multi-layer caching strategy is proposed, that is, by combining memory caching and disk caching, high concurrency and low latency read and write capabilities can be provided, which can effectively enhance the high concurrency and low latency single-machine read and write capabilities.

[0064] In summary, the above-described methods provided in this disclosure can significantly alleviate local overload problems in the cluster, such as mitigating hotspot traffic issues in object storage services, reducing the probability of accidents caused by sudden surges in hotspot traffic, improving the stability of the object storage system, accelerating the response to user hotspot requests, and effectively reducing the average latency of read requests to the access cache.

[0065] Corresponding to the aforementioned distributed object storage method, this disclosure further provides a distributed object storage device. Figure 8 This is a schematic diagram of a distributed object storage device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device, such as... Figure 8 As shown, the distributed object storage device includes:

[0066] The object splitting module 802 is used to split the target object to be stored into multiple key-value pairs of data.

[0067] The partition determination module 804 is used to determine the target partition corresponding to each of the multiple key-value pair data from multiple partitions of the preset distributed object storage system;

[0068] The node determination module 806 is used to determine the data storage node currently holding the target partition based on a preset partition view, and obtain the data storage node corresponding to each of the multiple key-value pairs; wherein, the partition view is used to indicate the metadata of multiple partitions, and the metadata includes at least the information of the data storage node holding the partition; the data storage node is the storage node of the distributed object storage system.

[0069] The data caching module 808 is used to cache multiple key-value pairs of data to the target partition in their respective data storage nodes.

[0070] The above-mentioned device can perform key-value pair splitting on the target object and distribute the resulting key-value pair data in the corresponding partitions to achieve the purpose of node load distribution, thereby effectively alleviating the problem of local overload in the object storage service cluster.

[0071] In some implementations, the partition determination module 804 is specifically used to: perform hash processing on the plurality of key-value pairs of data respectively to obtain the hash value corresponding to each of the plurality of key-value pairs of data; and determine the target partition corresponding to each of the plurality of key-value pairs of data from a plurality of partitions of a preset distributed object storage system based on the hash value corresponding to each of the plurality of key-value pairs of data.

[0072] In some implementations, the plurality of partitions includes a first partition, which is a partition with duplicate data, and the first partition corresponds to a plurality of data storage nodes, wherein the plurality of identical duplicate data corresponding to the first partition are distributed on different data storage nodes.

[0073] In some implementations, the plurality of data storage nodes corresponding to the first partition include a master node and a slave node, wherein the master node is used to write data to the first partition, the slave node is used to copy the data written to the first partition by the master node, and either the master node or the slave node is used to read data from the first partition.

[0074] In some implementations, the multiple data storage nodes corresponding to the first partition are determined as follows: based on a uniform allocation strategy, multiple data storage nodes corresponding to each first partition in the multiple partitions are determined from the currently available set of data storage nodes, wherein the uniform allocation strategy is used to indicate that the difference in the number of partitions held by different data storage nodes is within a preset range.

[0075] In some embodiments, the apparatus further includes a reallocation module for reallocating multiple data storage nodes corresponding to the first partition based on the changed set of currently available data storage nodes and the uniform allocation strategy, in the event of a change in the set of currently available data storage nodes.

[0076] In some implementations, the target object to be stored is not part of the first-level hot data. The first-level hot data is data whose current popularity is higher than a preset popularity threshold. The first-level hot data is used to store in the client cache.

[0077] In some embodiments, the device further includes a hierarchical caching module, which, upon receiving an object to be stored, determines whether the object to be stored is long-tail data; if so, determines that the object to be stored is a target object to be stored; wherein the target object to be stored is stored in the data storage node corresponding to the server; if not, caches the object to be stored in the memory of the server.

[0078] The distributed object storage device provided in this disclosure can execute the distributed object storage method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0079] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0080] This disclosure provides an electronic device, which includes: a storage device storing a computer program thereon; and a processing device for executing the computer program in the storage device to implement the steps of any method of this disclosure.

[0081] The following is for reference. Figure 9 This document illustrates a structural schematic diagram of an electronic device 900 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0082] like Figure 9 As shown, electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 902 or a program loaded from storage device 908 into random access memory (RAM) 903. RAM 903 also stores various programs and data required for the operation of electronic device 900. Processing device 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0083] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 9 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0084] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0085] In addition to the methods and devices described above, embodiments of this disclosure can also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the methods provided in the embodiments of this disclosure. The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user computing device, partially on a user device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0086] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the distributed object storage method provided in embodiments of this disclosure.

[0087] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0088] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the distributed object storage method of this disclosure.

[0089] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0090] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0091] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0092] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0093] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0094] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A distributed object storage method, characterized in that, include: Split the target object to be stored into multiple key-value pairs; The target partition corresponding to each of the multiple key-value pair data is determined from multiple partitions of a pre-defined distributed object storage system. Each partition includes a first partition containing multiple replicas of the data, and these replicas are distributed across multiple data storage nodes corresponding to the first partition. The multiple data storage nodes corresponding to the first partition are determined based on a uniform allocation strategy according to the currently available data storage node set indicated by the node view. The uniform allocation strategy is used to indicate that the difference in the number of partitions held by different data storage nodes is within a preset range. Based on a preset partition view, the data storage node currently holding the target partition is determined, thereby obtaining the data storage node corresponding to each of the multiple key-value pairs; wherein, the partition view is used to indicate the metadata of the multiple partitions, and the metadata includes at least information about the data storage node holding the partition; the data storage node is a storage node of the distributed object storage system; The multiple key-value pairs are cached in their respective data storage nodes.

2. The method according to claim 1, characterized in that, The step of determining the target partition corresponding to each of the multiple key-value pair data from multiple partitions of a preset distributed object storage system includes: The multiple key-value pairs are hashed separately to obtain the hash values ​​corresponding to each of the multiple key-value pairs. Based on the hash values ​​corresponding to the multiple key-value pairs, the target partitions corresponding to the multiple key-value pairs are determined from multiple partitions of a preset distributed object storage system.

3. The method according to claim 1, characterized in that, The first partition has multiple data storage nodes, including a master node and a slave node. The master node is used to write data to the first partition, the slave node is used to copy the data written to the first partition by the master node, and either the master node or the slave node is used to read data from the first partition.

4. The method according to claim 1, characterized in that, The method further includes: If the set of currently available data storage nodes changes, the multiple data storage nodes corresponding to the first partition are reallocated based on the changed set of currently available data storage nodes and the uniform allocation strategy.

5. The method according to claim 1, characterized in that, The target object to be stored does not belong to the first-level hot data. The first-level hot data is data whose current popularity is higher than a preset popularity threshold. The first-level hot data is used to store in the client cache.

6. The method according to claim 5, characterized in that, Before performing the step of splitting the target object to be stored into multiple key-value pairs, the method further includes: Upon receiving an object to be stored, determine whether the object to be stored is long-tail data; If so, the object to be stored is determined to be the target object to be stored; wherein, the target object to be stored is used to store in the data storage node corresponding to the server. If not, the object to be stored is cached in the server's memory.

7. A distributed object storage device, characterized in that, include: The object splitting module is used to split the target object to be stored into multiple key-value pairs; A partition determination module is used to determine the target partition corresponding to each of the multiple key-value pair data from multiple partitions of a preset distributed object storage system. The multiple partitions include a first partition with multiple replicas of data, and the multiple replicas of data are distributed on multiple data storage nodes corresponding to the first partition. The multiple data storage nodes corresponding to the first partition are determined based on a uniform allocation strategy and according to the currently available set of data storage nodes indicated by the node view. The uniform allocation strategy is used to indicate that the difference in the number of partitions held by different data storage nodes is within a preset range. The node determination module is used to determine the data storage node currently holding the target partition based on a preset partition view, thereby obtaining the data storage node corresponding to each of the multiple key-value pairs; wherein, the partition view is used to indicate the metadata of the multiple partitions, and the metadata includes at least information about the data storage node holding the partition; the data storage node is a storage node of the distributed object storage system; The data caching module is used to cache the multiple key-value pairs of data to their respective data storage nodes.

8. An electronic device, characterized in that, The electronic device includes: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the distributed object storage method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is used to execute the distributed object storage method according to any one of claims 1-6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the distributed object storage method according to any one of claims 1-6.

Images