Method and system for quickly traversing object storage bucket

Abstract

The invention discloses a method and system for quickly traversing an object storage bucket, and the method comprises the steps: a client sends a break point obtaining request to a server; the client receives the segmentation points corresponding to the logic fragments in the storage bucket fed back by the server; the client creates a plurality of threads according to the plurality of segmentation points, wherein each thread corresponds to at least two segmentation points; the client side traverses the data objects stored in the storage buckets in parallel based on the corresponding segmentation points through multiple threads, and traversal results of the storage buckets are obtained; the client side sends a traversal result to the server side, the traversal result comprises the mapping relation between the logic fragment in the storage bucket and the data interval to which the data object belongs, and the traversal result is used for querying the data object in the storage bucket.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a method and system for quickly traversing object storage buckets. Background Art

[0002] With the development of the Internet and cloud computing technology, the storage and processing of massive data has become increasingly important. Object storage has become one of the main ways to store unstructured data (such as pictures, videos, documents, etc.) due to its high scalability, low cost and easy management.

[0003] Bucket is a logical unit in the object storage system. The object storage system organizes and manages objects (i.e., unstructured data such as pictures, videos, and documents) stored in the system based on buckets.

[0004] However, with the rapid increase in the amount of data in the bucket, how to efficiently traverse and retrieve the data in the bucket has become a problem that needs to be solved urgently.

[0005] Traditional traversal methods usually use a single-threaded sequential traversal method, which is inefficient when facing large-scale data sets and cannot fully utilize the computing power of modern multi-core processors. In addition, as the storage scale grows, the performance bottleneck of a single server also limits the improvement of traversal speed. Summary of the invention

[0006] To this end, this application discloses the following technical solutions:

[0007] A first aspect of the present application provides a method for quickly traversing an object storage bucket, comprising:

[0008] The client sends a split point acquisition request to the server;

[0009] The client receives the split points corresponding to the logical shards in the storage bucket fed back by the server;

[0010] The client creates a plurality of threads according to the plurality of segmentation points, each of the threads corresponding to at least two segmentation points;

[0011] The client traverses the data objects stored in the bucket based on the corresponding split points in parallel through the multiple threads to obtain a traversal result of the bucket;

[0012] The client sends the traversal result to the server, the traversal result including a mapping relationship between a logical shard in the storage bucket and a data interval to which the data object belongs, and the traversal result is used to query the data object in the storage bucket.

[0013] Optionally, each of the threads corresponds to at least one starting segmentation point and at least one ending segmentation point;

[0014] The client traverses the data objects stored in the bucket based on the corresponding split points in parallel through the multiple threads to obtain the traversal result of the bucket, including:

[0015] For each of the threads running in parallel, the client traverses the data objects stored in the bucket between the starting segmentation point and the ending segmentation point through the thread to obtain the sub-traversal result of the thread, and the sub-traversal results of multiple threads constitute the traversal result of the bucket.

[0016] A second aspect of the present application provides a method for quickly traversing an object storage bucket, comprising:

[0017] The server responds to the client's split point acquisition request and determines the split points corresponding to each logical shard in the bucket;

[0018] The server sends the plurality of segmentation points to the client, so that the client traverses the data objects stored in the storage bucket in parallel based on the plurality of segmentation points;

[0019] The server receives a traversal result of the storage bucket fed back by the client, wherein the traversal result includes a mapping relationship between a logical shard in the storage bucket and a data interval to which the data object belongs, and the traversal result is used to query a data object in the storage bucket.

[0020] Optionally, also include:

[0021] The server responds to the query list request of the client and determines the target data interval to which the target data belongs, wherein the target data includes the data object to be queried by the query list request;

[0022] The server determines the target logical shard corresponding to the target data interval according to the traversal result;

[0023] The server determines the target physical server where the leader node corresponding to the target logical slice is located according to the server information of the target logical slice;

[0024] The server performs a query operation on the target physical server according to the query start position and the query data volume carried by the query list request to obtain the target data;

[0025] The server feeds back the target data to the client.

[0026] Optionally, also include:

[0027] The server detects the data volume of the data object corresponding to each of the logical slices;

[0028] When it is detected that the data volume of the data object corresponding to any of the logical shards is greater than the data volume threshold, the server divides the logical shard into two new logical shards according to the intermediate key of the logical shard.

[0029] Optionally, each of the logical shards corresponds to a leader node and at least one subordinate node, and the leader node and the subordinate node are both used to store data objects of the corresponding logical shard;

[0030] The method further comprises:

[0031] The server detects node status parameters, logical shard distribution parameters and network delay parameters of the nodes corresponding to each of the logical shards;

[0032] The server adjusts the data objects stored in the nodes corresponding to each of the logical slices according to the node status parameters, logical slice distribution parameters and network delay parameters of the nodes corresponding to each of the logical slices.

[0033] Optionally, also include:

[0034] The server synchronizes the data objects stored in each of the leader nodes to the slave nodes belonging to the same logical shard in real time;

[0035] When the server detects a failure of any of the leader nodes, it selects a slave node belonging to the same logical slice as a new leader node.

[0036] Optionally, also include:

[0037] The server detects system operation indicators of the system to which the storage bucket belongs in real time, and the system operation indicators include at least one of node status parameters, logical shard distribution parameters, network delay parameters, CPU utilization and memory utilization.

[0038] Optionally, also include:

[0039] When any of the system operation indicators meets the corresponding alarm condition, the server outputs alarm information.

[0040] A third aspect of the present application provides a system for quickly traversing object storage buckets, including a client and a server;

[0041] The client is used to:

[0042] Send a split point acquisition request to the server;

[0043] Receive the split points corresponding to the logical shards in the storage bucket fed back by the server;

[0044] Creating a plurality of threads according to the plurality of segmentation points, each of the threads corresponding to at least two of the segmentation points;

[0045] The data objects stored in the storage bucket are traversed in parallel by the multiple threads based on the corresponding split points to obtain a traversal result of the storage bucket;

[0046] Sending the traversal result to the server, the traversal result including a mapping relationship between the logical shards in the storage bucket and the data interval to which the data object belongs, the traversal result being used to query the data object in the storage bucket;

[0047] The server is used to:

[0048] Respond to the client's split point acquisition request and determine the split points corresponding to each logical shard in the bucket;

[0049] The server sends the plurality of segmentation points to the client, so that the client traverses the data objects stored in the storage bucket in parallel based on the plurality of segmentation points;

[0050] The server receives a traversal result of the storage bucket fed back by the client, wherein the traversal result includes a mapping relationship between a logical shard in the storage bucket and a data interval to which the data object belongs, and the traversal result is used to query a data object in the storage bucket.

[0051] The beneficial effects of this solution are:

[0052] When it is necessary to traverse the data objects stored in the bucket, the client first obtains the split points corresponding to the logical shards in the bucket, creates multiple threads based on the split points, and uses multiple threads to traverse the data objects corresponding to each logical shard in parallel, thereby significantly improving the traversal efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0054] Figure 1 is a flow chart of a method for quickly traversing object storage buckets provided by an embodiment of the present application;

[0055] Figure 2 is a flowchart of another method for quickly traversing object storage buckets provided by an embodiment of the present application;

[0056] Figure 3 This is a schematic diagram of interaction between a client and a server provided in an embodiment of the present application;

[0057] Figure 4 is a flow chart of a data query process provided by an embodiment of the present application;

[0058] Figure 5 It is a structural diagram of a system for quickly traversing object storage buckets provided by an embodiment of the present application;

[0059] Figure 6 It is a structural diagram of a client provided in an embodiment of the present application;

[0060] Figure 7 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] This application embodiment provides a method for quickly traversing object storage buckets. This method can be applied to the client. Figure 1 , the method may include the following steps.

[0063] S101, the client sends a split point acquisition request to the server.

[0064] When the client needs to traverse the data in the bucket, it can send a split point acquisition request to the server to obtain multiple split points corresponding to the bucket fed back by the server. These split points can effectively define the boundaries of different traversal tasks, thereby supporting the client to perform parallel traversal operations.

[0065] When there are multiple buckets, the split point acquisition request sent by the client may carry the bucket identifier of the bucket to be traversed, so that the server feeds back the split point of the corresponding bucket according to the identifier.

[0066] S102, the client receives the split points corresponding to each logical shard in the storage bucket fed back by the server.

[0067] The storage bucket referred to in this embodiment can be regarded as a data set consisting of a large number of data objects. These data objects can be divided into multiple logical slices according to corresponding keys, each logical slice corresponds to multiple data objects, and a logical slice can include the keys of the corresponding multiple data objects.

[0068] The storage bucket can be specifically stored in a computer system, which can be called a storage bucket system. The storage bucket system can specifically include multiple physical servers, each physical server is deployed with one or more nodes, each node occupies a certain storage space of the physical server, and the data objects contained in the storage bucket can be specifically stored in the storage space of these nodes.

[0069] The client and server used to implement the method of this embodiment can be a traversal and query system specifically used to traverse and query data objects stored in a bucket. The traversal and query system can be integrated into the bucket system or can be independent of the bucket system and communicated with the bucket system.

[0070] The above data objects include but are not limited to video objects, picture objects, audio objects, document objects and other unstructured data objects.

[0071] A storage bucket may include multiple logical shards, each of which may include a starting position and an ending position. The starting position is used to indicate the storage address of the first data object corresponding to the logical shard in the server, and the ending position is used to indicate the storage address of the last data object corresponding to the logical shard in the server. The data objects corresponding to each logical shard may be stored continuously in the storage module of the server, that is, all data objects corresponding to a logical shard are stored between the starting position and the ending position corresponding to the logical shard.

[0072] After receiving the split point acquisition request, the server can query the starting position of each logical shard in the bucket that the client needs to traverse and the ending position of the last logical shard based on the current state of the bucket and the specific request parameters of the client, and then send the multiple starting positions and the ending position of the last logical shard obtained from the query to the client as the multiple split points corresponding to the bucket.

[0073] Optionally, the server can store the queried locations in the server's memory for a period of time. If other clients send split point acquisition requests for the bucket during this period, the server can directly send these locations stored in the memory as split points to the client, thereby optimizing response time and reducing resource consumption, and avoiding repeated calculation of split points.

[0074] When sending the segmentation points, the server can assemble these segmentation points according to the communication protocol pre-agreed between the server and the client, obtain one or more data packets containing these segmentation points, and then feed the data packets back to the client. The client parses these data packets based on the communication protocol to obtain the segmentation points therein.

[0075] S103: The client creates multiple threads according to the multiple segmentation points, and each thread corresponds to at least two segmentation points.

[0076] For any thread, the thread can correspond to two adjacent split points or multiple consecutive split points. Among the multiple split points corresponding to a thread, the split point with the smallest address can be called the starting split point corresponding to the thread, and the split point with the largest address can be called the ending split point corresponding to the thread. The storage space between the starting split point and the ending split point is equivalent to the traversal range corresponding to the thread. The traversal ranges of multiple threads do not overlap.

[0077] In step S103, the client can directly create threads corresponding to each two adjacent segmentation points. Alternatively, the client can also create threads according to the size of the storage space between the segmentation points, so that the traversal range of each thread is as close as possible, thereby improving the traversal efficiency.

[0078] S104, the client traverses the data objects stored in the bucket based on the corresponding split points in parallel through multiple threads to obtain the traversal result of the bucket.

[0079] As mentioned above, each thread corresponds to at least one starting segmentation point and at least one ending segmentation point.

[0080] In step S104, for each thread running in parallel, the client can traverse the data objects stored in the bucket between the start segmentation point and the end segmentation point through the thread to obtain the sub-traversal result of the thread, and the sub-traversal results of multiple threads constitute the traversal result of the bucket.

[0081] Specifically, for each thread, when the client runs the thread, it can use a request similar to the Amazon S3 protocol (or other similar protocols) for traversing data objects (such as ListObjects request) to query the data objects in the traversal range corresponding to the thread one by one, and obtain the object list of the traversal range corresponding to the thread, which is the sub-traversal result corresponding to the thread. The object list can record the key corresponding to each data object in the traversal range and the logical shard corresponding to the data object.

[0082] When traversing through ListObjects request, the starting split point corresponding to the thread can be passed in as the marker parameter of the request, and the ending split point can be passed in as the ending parameter. At the same time, the number of entries for each query can also be set. Based on the marker parameter and the number of entries, the ListObjects request can start from the position indicated by the marker parameter, query the data objects corresponding to the number of entries one by one, and then return the keys and logical shards of the traversed multiple data objects, as well as the storage address of the last data object on the server. Then the client passes the storage address of the last server as the marker parameter again, so that the ListObjects request continues to query the data objects corresponding to the number of entries starting from the new marker parameter, and the cycle continues until the ListObjects request queries the address corresponding to the ending parameter. When the address corresponding to the ending parameter is queried, the traversal process of the thread ends. At this time, the object list consisting of the keys of all data objects and the corresponding logical shards obtained from the query is the sub-traversal result obtained after the thread traverses.

[0083] After obtaining the sub-traversal results of all threads, the client can integrate multiple sub-traversal results into an object list. The integrated object list containing all sub-traversal results is the traversal result of the bucket.

[0084] S105, the client sends a traversal result to the server, the traversal result includes a mapping relationship between the logical shards in the bucket and the data interval to which the data object belongs, and the traversal result is used to query the data object in the bucket.

[0085] The data interval to which a data object belongs refers to the data interval to which the key of the data object belongs.

[0086] As mentioned above, after the server obtains the traversal results of the bucket, it can determine which keys each logical shard corresponds to based on the correspondence between the keys and logical shards in the traversal results, thereby determining the data interval of the keys corresponding to each logical shard. When it is necessary to query any one or more data objects, the server can quickly determine which logical shards these data objects correspond to based on the data interval to which the keys of these data objects belong and the traversal results, and then quickly query the corresponding data objects according to the corresponding logical shards.

[0087] The beneficial effects of this solution are:

[0088] When it is necessary to traverse the data objects stored in the bucket, the client first obtains the split points corresponding to the logical shards in the bucket, creates multiple threads based on the split points, and uses multiple threads to traverse the data objects corresponding to each logical shard in parallel, thereby significantly improving the traversal efficiency.

[0089] The present application also provides a method for quickly traversing object storage buckets. This method can be applied to the server. Figure 2 , the method may include the following steps.

[0090] S201, the server responds to the split point acquisition request of the client and determines the split points corresponding to each logical shard in the bucket.

[0091] The implementation of step S201 may refer to the aforementioned step S102 and will not be described in detail.

[0092] S202: The server sends multiple split points to the client, so that the client traverses the data objects stored in the bucket in parallel based on the multiple split points.

[0093] The manner in which the client traverses the storage buckets may refer to steps S103 and S104.

[0094] S203, the server receives the traversal result of the bucket fed back by the client, the traversal result includes the mapping relationship between the logical shards in the bucket and the data interval to which the data object belongs, and the traversal result is used to query the data object in the bucket.

[0095] When there are multiple clients and multiple buckets, the server may send the split point corresponding to each bucket to a corresponding client, so that the server may receive the traversal results of multiple buckets fed back by multiple clients in step S203.

[0096] The beneficial effects of this embodiment are consistent with those of the above-mentioned embodiments and will not be described in detail.

[0097] To facilitate understanding of the interaction process between the client and the server in the above method of this embodiment, please refer to Figure 3 , is a schematic diagram of an interaction process between a client and a server provided in this embodiment, and the interaction process may include the following steps.

[0098] S301, the client sends a split point acquisition request to the server.

[0099] S302, the server responds to the split point acquisition request of the client and determines the split points corresponding to each logical shard in the bucket.

[0100] S303: The server sends multiple segmentation points to the client.

[0101] Step S303 may be equivalent to step S102 or S202 of the aforementioned embodiment.

[0102] S304: The client creates multiple threads according to the multiple segmentation points.

[0103] S305 , the client traverses the data objects stored in the bucket based on the corresponding split points in parallel through multiple threads to obtain the traversal result of the bucket.

[0104] S306: The client sends the traversal result to the server.

[0105] Step S306 may be equivalent to step S105 or S203 of the aforementioned embodiment.

[0106] In some optional embodiments, the server can press Figure 4 The method shown queries the corresponding data object based on the obtained traversal results.

[0107] S401, the server responds to the query list request of the client and determines the target data interval to which the target data belongs, where the target data includes the data object to be queried by the query list request.

[0108] The query list request may carry marker information, a query starting position, and a query data volume. The marker information may include the key of the data object to be queried. The query starting position is used to indicate which address of the server to start the query from. The query data volume represents how many data objects the client needs to query starting from the query starting position.

[0109] After receiving the query list request, the server may first parse the tag information in the request to determine, based on the tag information, which data interval the key corresponding to the data object to be queried belongs to. The data interval is the target data interval.

[0110] S402, the server determines the target logical shard corresponding to the target data interval according to the traversal result.

[0111] As described above, the server can determine the data interval of the key corresponding to each logical shard based on the correspondence between the key and the logical shard in the traversal result. Therefore, in S402, the server can determine the target logical shard corresponding to the target data interval based on the correspondence, that is, determine which logical shard the data object of the data interval is mapped to.

[0112] S403, the server determines the target physical server where the leader node corresponding to the target logical slice is located according to the server information of the target logical slice.

[0113] Each logical slice has server information corresponding to it, and the server information can at least indicate the physical server where the leader node managing the logical slice is located. In S403, the server can read the server information of the target logical slice, thereby determining the target physical server where the leader node corresponding to the target logical slice is located.

[0114] S404, the server performs a query operation on the target physical server according to the query starting position and the query data volume carried in the query list request to obtain the target data.

[0115] In step S404, the server may use the query start position and query data volume specified in the query list request to perform a query operation on the physical server where the leader node is located to obtain target data that meets the query list request.

[0116] S405: The server feeds back the target data to the client.

[0117] In step S405, the server may first collect the target data obtained by the query, format the target data to obtain data that meets the format requirements of the client, and then encapsulate the formatted data into one or more response data packets according to a preset communication protocol, and then send the response data packets to the client. The client can parse the response data packets to obtain the target data therein.

[0118] In this embodiment, the server can create logical shards through the index metadata logical sharding module. Each logical shard can be called a "shard", and each logical shard corresponds to a continuous key interval. The data objects corresponding to each key in the key interval are the data objects corresponding to the logical shard.

[0119] Each logical shard corresponds to a leader node and multiple slave nodes, where the leader node is used to process client requests, for example, to output corresponding target data based on the query list request of the client in the above embodiment. The slave nodes corresponding to the logical shards can maintain data synchronization with the leader node through a data consistency protocol (such as the Raft protocol) to ensure data consistency between the slave nodes and the leader node.

[0120] In the initial state, the index metadata logical sharding module can divide the entire key space into one or more logical shards.

[0121] The metadata of each logical shard (including information such as identification, start key, end key, etc.) can be stored in the unified scheduler module on the server side. The metadata of the logical shard also includes information about the leader node and slave nodes, as well as the status of each node.

[0122] The index metadata logical sharding module can also split a logical shard into two logical shards in the following manner when the data volume of the data object corresponding to any logical shard is too large, thereby reducing the data volume corresponding to a single logical shard.

[0123] Optionally, also include:

[0124] The server detects the data volume of each logical shard corresponding to the data object;

[0125] When it is detected that the data volume of the data object corresponding to any logical shard is greater than the data volume threshold, the server divides the logical shard into two new logical shards according to the intermediate key of the logical shard.

[0126] The module manages data shards through automatic splitting and redistribution strategies, ensuring that data is evenly distributed as it grows, avoiding hot spots, and maintaining data consistency through the Raft protocol.

[0127] The above steps can be performed by the index metadata logical sharding module (Shard Management) and the unified scheduler module (Placement Driver, PD) on the server side.

[0128] PD is responsible for monitoring the status of all logical shards, including the data volume. When PD detects that the data volume corresponding to any logical shard is greater than a specific threshold (for example, greater than 96MB), PD outputs an instruction to split the logical shard to the index metadata logical shard module to trigger the splitting process of the logical shard.

[0129] After receiving the above instructions, the index metadata logical sharding module can drive the logical shards with data volume greater than the threshold to split based on the data consistency protocol (such as the Raft protocol). Specifically, the index metadata logical sharding module can sort all the keys corresponding to the logical shard in lexicographic order, and then select the middle key of the key interval corresponding to the logical shard as the split point, split all the keys corresponding to the logical shard into two parts, and retain one part of them, and assign the other part to a new logical shard, thereby splitting the logical shard into two logical shards.

[0130] Splitting and redistributing logical shards in the above manner allows the bucket system to scale seamlessly as data grows, ensuring that data is evenly distributed and avoiding hot spots.

[0131] Optionally, each logical shard corresponds to a leader node and at least one slave node, and both the leader node and the slave node are used to store data objects of the corresponding logical shard;

[0132] In the method of this embodiment, the server can also adjust the data objects stored in each node in the following manner, for example, transferring a data object from the current node to another node:

[0133] The server detects the node status parameters, logical shard distribution parameters and network delay parameters of the node corresponding to each logical shard;

[0134] The server adjusts the data objects stored in the nodes corresponding to each logical shard according to the node status parameters, logical shard distribution parameters and network delay parameters of the nodes corresponding to each logical shard.

[0135] The above steps can be executed by a unified scheduler module (Placement Driver, PD) on the server side.

[0136] In this embodiment, PD can collect node status parameters, logical shard distribution parameters, and network delay parameters of each node in the bucket system in real time, and adjust the data objects stored in each node according to the collected parameters.

[0137] The node status parameters may include whether the corresponding node is faulty, the size of the free storage space of the node, and other parameters.

[0138] The logical shard distribution parameter may indicate in which nodes the data objects corresponding to a logical shard are stored.

[0139] The network delay parameter corresponding to a node can represent the transmission delay when the node transmits data to the client.

[0140] The server can adjust the data objects stored in the nodes corresponding to each logical shard as follows:

[0141] For each logical shard, first determine in which nodes the data objects corresponding to the logical shard are stored based on the logical shard distribution parameters, and then determine whether the node status parameters and network delay parameters of these nodes meet the preset target conditions. If at least one of the node status parameters and network delay parameters of a node meets the target condition, the data object corresponding to the logical shard stored in the node can be deleted, and these data objects can be transferred to another node whose node status parameters and network delay parameters do not meet the target conditions.

[0142] The target conditions may include that a network delay parameter is greater than or equal to a specific delay threshold, free storage space is less than or equal to a specific storage threshold, and a node is in a fault state.

[0143] Optionally, PD can also dynamically adjust the configuration of the cluster based on the collected node status parameters, logical shard distribution parameters, and network delay parameters to adapt to changes in different workloads, and can be used to predict future capacity requirements and plan resources in advance. For example, when it is determined based on the above parameters that a node stores a large amount of data, more storage resources and computing resources can be allocated to the node to meet the needs of storing and querying data.

[0144] Furthermore, the unified scheduler module can also be used to maintain the topology information of the cluster and ensure that data is evenly distributed throughout the cluster. Specifically, PD can detect whether the amount of data corresponding to each logical shard is greater than a certain threshold, and when it is detected that the amount of data corresponding to a logical shard is greater than the threshold, the index metadata logical shard module is triggered to split the logical shard into two logical shards.

[0145] PD can also be used to manage the process of nodes joining and leaving the cluster to ensure that data consistency and availability are not affected. Specifically, when PD detects that any new node has joined the bucket system, PD can determine the node as a slave node of a certain logical shard, and synchronize the data objects stored by the leader node of the logical shard to the slave node. When PD detects that any node is about to leave the bucket system, PD can first detect whether the data objects stored in the node have backups on other nodes. If there are backups, the node can be removed from the bucket system. If there are no backups, the node's data objects will be transferred to other nodes first, and then the node will be removed from the bucket system.

[0146] Optionally, PD can also provide the API interface required by the client to query data objects, provide an API interface for the client to monitor the status of the bucket system (including node status, logical shard distribution, data volume statistics), and provide an API interface for managing nodes joining and exiting the cluster.

[0147] In some optional embodiments, the server may also include a data replica and consistency assurance module, through which the server may implement a data consistency protocol (such as the Raft protocol) to ensure data consistency in the bucket system, that is, to ensure that data object query services are available in the event of network partitions or node failures.

[0148] Specifically, through the data copy and consistency guarantee module, the server can perform the following steps:

[0149] The server synchronizes the data objects stored in each leader node to the slave nodes belonging to the same logical shard in real time;

[0150] When the server detects the failure of any leader node, it selects a slave node belonging to the same logical shard as the new leader node.

[0151] The above steps can be performed by the data copy and consistency assurance module of the server.

[0152] As mentioned above, a logical shard can correspond to a leader node and multiple slave nodes, which synchronize data in real time through the Raft protocol. At the same time, each slave node can periodically send heartbeat messages to the leader node. If the leader node can regularly receive the heartbeat message of a slave node, it can be determined that the slave node is not in a faulty state. If the leader node cannot regularly receive the heartbeat message of a slave node, it can be determined that the slave node is in a faulty state. When it is determined that a slave node is in a faulty state, the leader node can report the message that the slave node is in a faulty state to the data replica and consistency assurance module, so that the data replica and consistency assurance module selects a new slave node to replace the faulty slave node, thereby ensuring that the data object corresponding to a logical shard is always stored in multiple non-faulty slave nodes.

[0153] On the other hand, the data copy and consistency assurance module can detect in real time whether each leader node is faulty. When it is detected that any leader node is in a faulty state, the data copy and consistency assurance module can trigger the election mechanism for the logical shard to which the leader node belongs. At this time, each slave node belonging to the logical shard can output voting information, and the voting information can be used to designate any slave node as the new leader node. The data copy and consistency assurance module can integrate the voting information of each slave node in the corresponding logical shard, and finally determine a target slave node, and determine the target slave node as the new leader node corresponding to the logical shard, and continue to provide data query services through this new leader node.

[0154] Through the above mechanism, this embodiment can promptly determine a new leader node when a leader node fails to ensure service continuity, and can shorten service interruption time as much as possible when any node fails through real-time data replication and status synchronization between nodes.

[0155] Optionally, the method of this embodiment may further include the following steps:

[0156] The server detects the system operation indicators of the system to which the storage bucket belongs in real time. The system operation indicators include at least one of node status parameters, logical shard distribution parameters, network delay parameters, CPU usage, and memory usage.

[0157] The above steps may be performed by the monitoring and alarm module of the server. Specifically, the monitoring and alarm module may monitor the above one or more system operation indicators of the system to which the bucket belongs (ie, the aforementioned bucket system) in real time through the Prometheus monitoring system or other similar monitoring systems.

[0158] Furthermore, the server can also:

[0159] When any system operation indicator meets the corresponding alarm condition, the alarm information is output.

[0160] The above steps can also be executed by the monitoring and alarm module of the server.

[0161] The monitoring and alarm module can be pre-configured with alarm conditions corresponding to various system operation indicators, and different system operation indicators can correspond to different alarm conditions. Alarm conditions can be flexibly configured according to business needs without limitation. Each alarm condition can include one or more alarm thresholds. If the system operation indicator is greater than the alarm threshold contained in the corresponding alarm condition, it can be determined that the indicator meets the alarm condition. If the system operation indicator is less than or equal to the alarm threshold contained in the corresponding alarm condition, it can be determined that the indicator does not meet the alarm condition.

[0162] Exemplarily, the alarm condition corresponding to the CPU usage may include a CPU usage threshold of 80%. If the current CPU usage is greater than 80%, it can be determined that the CPU usage meets the alarm condition. If the CPU usage is less than or equal to 80%, it can be determined that the CPU usage does not meet the alarm condition.

[0163] The beneficial effects of this embodiment are:

[0164] First, through the parallel traversal mechanism, the client can simultaneously process object subsets defined by multiple split points, significantly improving the traversal speed and shortening the business processing time.

[0165] Second, the server uses a caching mechanism to cache the obtained segmentation points, which reduces the repeated calculations on the server, reduces the computing cost and response time, and improves the overall performance and resource utilization of the system.

[0166] Third, by managing logical shards through automatic splitting and redistribution strategies, we ensure that data is evenly distributed as it grows, avoid hot spots, and enhance the scalability and load balancing capabilities of the system.

[0167] Fourth, a data consistency protocol is used to ensure data consistency and availability of each node in each logical shard, which keeps services available even in the event of network partitions or node failures, improving system reliability and stability.

[0168] Fifth, the unified scheduler module monitors the status of nodes corresponding to all logical shards and is responsible for data shard splitting decisions, simplifying system management and maintenance.

[0169] Sixth, through the automatic fault detection and recovery mechanism, node failures can be detected in a short time and services can be restored quickly, improving the availability and robustness of the system.

[0170] Seventh, it provides a wealth of monitoring indicators and alarm mechanisms, which can monitor the health of the cluster in real time, detect potential problems in a timely manner, and ensure the stable operation of the system.

[0171] In summary, the present invention provides an efficient, scalable and reliable method and system for quickly traversing object storage buckets, which significantly improves traversal efficiency, optimizes resource utilization, and enhances the scalability, consistency and availability of the system.

[0172] The present application also provides a system for quickly traversing object storage buckets. Figure 5 , the system may include a client 501 and a server 502;

[0173] Client 501 is used for:

[0174] Send a split point acquisition request to the server;

[0175] Receive the split points corresponding to each logical shard in the bucket fed back by the server;

[0176] Create multiple threads according to multiple split points, each thread corresponds to at least two split points;

[0177] The data objects stored in the bucket are traversed in parallel based on the corresponding split points by multiple threads to obtain the traversal results of the bucket;

[0178] Send the traversal result to the server. The traversal result includes the mapping relationship between the logical shards in the bucket and the data interval to which the data object belongs. The traversal result is used to query the data object in the bucket.

[0179] Server 502 is used for:

[0180] Respond to the client's split point acquisition request and determine the split points corresponding to each logical shard in the bucket;

[0181] The server sends multiple split points to the client, so that the client can traverse the data objects stored in the bucket in parallel based on the multiple split points;

[0182] The server receives the bucket traversal result fed back by the client. The traversal result includes the mapping relationship between the logical shards in the bucket and the data interval to which the data object belongs. The traversal result is used to query the data object in the bucket.

[0183] See also Figure 6 , the client can include:

[0184] The communication unit 601 is used to send a segmentation point acquisition request to the server;

[0185] The communication unit 601 is further used to receive the split points corresponding to each logical shard in the storage bucket fed back by the server;

[0186] A creating unit 602, configured to create multiple threads according to multiple segmentation points, each thread corresponding to at least two segmentation points;

[0187] A traversal unit 603 is used to traverse the data objects stored in the bucket based on the corresponding split points in parallel through multiple threads to obtain a traversal result of the bucket;

[0188] The communication unit 601 is also used to send a traversal result to the server. The traversal result includes a mapping relationship between a logical partition in a storage bucket and a data interval to which a data object belongs. The traversal result is used to query a data object in the storage bucket.

[0189] Optionally, each thread corresponds to at least one starting segmentation point and at least one ending segmentation point;

[0190] The traversal unit 603 traverses the data objects stored in the bucket based on the corresponding segmentation points in parallel through multiple threads, and when the traversal result of the bucket is obtained, it can be used to:

[0191] For each thread running in parallel, the client traverses the data objects stored in the bucket between the start segmentation point and the end segmentation point through the thread to obtain the sub-traversal result of the thread. The sub-traversal results of multiple threads constitute the traversal result of the bucket.

[0192] See also Figure 7 , the server can include:

[0193] The splitting module 701 is used to respond to the splitting point acquisition request of the client and determine the splitting points corresponding to each logical shard in the bucket;

[0194] The communication module 702 is used to send multiple segmentation points to the client, so that the client can traverse the data objects stored in the bucket in parallel based on the multiple segmentation points;

[0195] The communication module 702 is also used to receive the traversal result of the bucket fed back by the client, the traversal result includes the mapping relationship between the logical shards in the bucket and the data interval to which the data object belongs, and the traversal result is used to query the data object in the bucket.

[0196] Optionally, the server further includes a query module 703, which is used to:

[0197] Responding to a query list request from a client, determining a target data interval to which target data belongs, the target data including a data object to be queried by the query list request;

[0198] Determine the target logical shard corresponding to the target data interval according to the traversal result;

[0199] Determine the target physical server where the leader node corresponding to the target logical slice is located according to the server information of the target logical slice;

[0200] Perform a query operation on the target physical server according to the query starting position and the query data volume carried by the query list request to obtain the target data;

[0201] The communication module 702 is also used to feed back the target data to the client.

[0202] Optionally, the server further includes an index metadata logical sharding module 704, which is used to:

[0203] The server detects the data volume of each logical shard corresponding to the data object;

[0204] When it is detected that the data volume of the data object corresponding to any logical shard is greater than the data volume threshold, the server divides the logical shard into two new logical shards according to the intermediate key of the logical shard.

[0205] Optionally, each logical shard corresponds to a leader node and at least one slave node, and both the leader node and the slave node are used to store data objects of the corresponding logical shard;

[0206] The server also includes a unified scheduler module 705, which is used to:

[0207] Detect node status parameters, logical shard distribution parameters, and network delay parameters of the nodes corresponding to each logical shard;

[0208] According to the node status parameters, logical shard distribution parameters and network delay parameters of the nodes corresponding to each logical shard, the data objects stored in the nodes corresponding to each logical shard are adjusted.

[0209] Optionally, the server also includes a data copy and consistency guarantee module 706, which is used to:

[0210] Synchronize the data objects stored in each leader node to the slave nodes belonging to the same logical shard in real time;

[0211] When a failure of any leader node is detected, a slave node belonging to the same logical shard is selected as the new leader node.

[0212] Optionally, the server also includes a monitoring and alarm module 707, which is used to:

[0213] The server detects the system operation indicators of the system to which the storage bucket belongs in real time. The system operation indicators include at least one of node status parameters, logical shard distribution parameters, network delay parameters, CPU usage, and memory usage.

[0214] Optionally, the monitoring and alarm module 707 is also used for:

[0215] When any system operation indicator meets the corresponding alarm condition, the server outputs an alarm message.

[0216] The specific working principles of the client and server of the system for quickly traversing object storage buckets provided in this embodiment can be found in the relevant steps of the method for quickly traversing object storage buckets in the aforementioned embodiment, and will not be repeated here.

[0217] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0218] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0219] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0220] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0221] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for quickly traversing an object storage bucket, characterized in that: include: The client sends a split point acquisition request to the server; The client receives the split points corresponding to the respective logical shards in the storage bucket fed back by the server; The client creates a plurality of threads according to the plurality of segmentation points, each of the threads corresponding to at least two segmentation points; The client traverses the data objects stored in the bucket based on the corresponding split points in parallel through the multiple threads to obtain a traversal result of the bucket; The client sends the traversal result to the server, the traversal result including a mapping relationship between a logical shard in the storage bucket and a data interval to which the data object belongs, and the traversal result is used to query the data object in the storage bucket.

2. The method according to claim 1, characterized in that Each of the threads corresponds to at least one starting segmentation point and at least one ending segmentation point; The client traverses the data objects stored in the bucket based on the corresponding split points in parallel through the multiple threads to obtain the traversal result of the bucket, including: For each of the threads running in parallel, the client traverses the data objects stored in the bucket between the starting segmentation point and the ending segmentation point through the thread to obtain the sub-traversal result of the thread, and the sub-traversal results of multiple threads constitute the traversal result of the bucket.

3. A method for quickly traversing object storage buckets, characterized in that: include: The server responds to the client's split point acquisition request and determines the split points corresponding to each logical shard in the bucket; The server sends the plurality of segmentation points to the client, so that the client traverses the data objects stored in the storage bucket in parallel based on the plurality of segmentation points; The server receives a traversal result of the storage bucket fed back by the client, wherein the traversal result includes a mapping relationship between a logical shard in the storage bucket and a data interval to which the data object belongs, and the traversal result is used to query a data object in the storage bucket.

4. The method according to claim 3, characterized in that: Also includes: The server responds to the query list request of the client and determines the target data interval to which the target data belongs, wherein the target data includes the data object to be queried by the query list request; The server determines the target logical shard corresponding to the target data interval according to the traversal result; The server determines the target physical server where the leader node corresponding to the target logical slice is located according to the server information of the target logical slice; The server performs a query operation on the target physical server according to the query start position and the query data volume carried by the query list request to obtain the target data; The server feeds back the target data to the client.

5. The method according to claim 3, characterized in that: Also includes: The server detects the data volume of the data object corresponding to each of the logical slices; When it is detected that the data volume of the data object corresponding to any of the logical shards is greater than the data volume threshold, the server divides the logical shard into two new logical shards according to the intermediate key of the logical shard.

6. The method according to claim 3, characterized in that Each of the logical shards corresponds to a leader node and at least one slave node, and the leader node and the slave node are used to store data objects of the corresponding logical shard; The method further comprises: The server detects node status parameters, logical shard distribution parameters and network delay parameters of the nodes corresponding to each of the logical shards; The server adjusts the data objects stored in the nodes corresponding to each of the logical slices according to the node status parameters, logical slice distribution parameters and network delay parameters of the nodes corresponding to each of the logical slices.

7. The method according to claim 6, characterized in that Also includes: The server synchronizes the data objects stored in each of the leader nodes to the slave nodes belonging to the same logical shard in real time; When the server detects a failure of any of the leader nodes, it selects a slave node belonging to the same logical slice as a new leader node.

8. The method according to claim 3, characterized in that Also includes: The server detects system operation indicators of the system to which the storage bucket belongs in real time, and the system operation indicators include at least one of node status parameters, logical shard distribution parameters, network delay parameters, CPU utilization and memory utilization.

9. The method according to claim 8, characterized in that Also includes: When any of the system operation indicators meets the corresponding alarm condition, the server outputs alarm information.

10. A system for quickly traversing object storage buckets, characterized in that: Including client and server; The client is used to: Send a split point acquisition request to the server; Receive the split points corresponding to the logical shards in the storage bucket fed back by the server; Creating a plurality of threads according to the plurality of segmentation points, each of the threads corresponding to at least two of the segmentation points; The multiple threads traverse the data objects stored in the bucket based on the corresponding split points in parallel to obtain a traversal result of the bucket; Sending the traversal result to the server, the traversal result including a mapping relationship between the logical shards in the storage bucket and the data interval to which the data object belongs, the traversal result being used to query the data object in the storage bucket; The server is used to: Respond to the client's split point acquisition request and determine the split points corresponding to each logical shard in the bucket; The server sends the plurality of segmentation points to the client, so that the client traverses the data objects stored in the storage bucket in parallel based on the plurality of segmentation points; The server receives a traversal result of the storage bucket fed back by the client, wherein the traversal result includes a mapping relationship between a logical shard in the storage bucket and a data interval to which the data object belongs, and the traversal result is used to query a data object in the storage bucket.

Images