Data scheduling method and device of distributed storage system and storage medium

Abstract

The invention discloses a data scheduling method and device of a distributed storage system and a storage medium, and relates to the field of cloud computing. The method comprises the following steps: acquiring a cluster identifier set from a configuration center; a Hash mapping ring is generated according to the cluster identifier set, and the Hash mapping ring is of a logic closed-loop data structure obtained through a space mapping mode based on the Hash value of identifier information in the cluster identifier set; an access request for the target file is received, and the access request comprises a target file identifier; performing mapping query on a Hash mapping ring based on the target file identifier, and determining a target storage cluster; and routing the access request to a target storage cluster, and then executing a data scheduling operation on data associated with the access request in the target storage cluster. According to the method and the device, the technical problem of poor storage expansion flexibility caused by the fact that each node needs to be equivalently expanded along with the increase of the data volume in the existing distributed storage technology is solved.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and more specifically, to a data scheduling method, apparatus, and storage medium for a distributed storage system. Background Technology

[0002] Current distributed storage technologies face the challenge of scalability in node storage capacity when implementing large-scale data storage. Traditional distributed storage systems, especially those built on Internet Protocol File Systems (IPLs), while achieving decentralized data storage and efficient data integrity verification, suffer from inherent data distribution mechanisms that require all participating nodes to store the same dataset or a large copy of it. This design is acceptable when data volume is small or growing slowly; however, with a rapid increase in data volume, each node must correspondingly expand its storage space, leading to poor storage scalability.

[0003] Figure 1 A schematic diagram illustrating the limitations of the prior art is shown, such as... Figure 1 As shown, all file system nodes (e.g., file system-1, file system-2, file system-3) are peers. Each file system node synchronizes its stored data to other nodes through its internal data synchronization module. While each storage node on a distributed cloud storage server has peer-to-peer data, as the amount of stored data increases, each node needs to expand its storage space accordingly, making elastic scaling impossible.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a data scheduling method, apparatus, and storage medium for a distributed storage system, which at least solves the technical problem in existing distributed storage technologies where each node needs to be expanded equally as the amount of data increases, resulting in poor storage expansion flexibility.

[0006] According to one aspect of the embodiments of this application, a data scheduling method for a distributed storage system is provided, comprising: obtaining a cluster identifier set from a configuration center, wherein the cluster identifier set includes identifier information of N storage clusters, where N is an integer greater than 1; generating a hash mapping ring based on the cluster identifier set, wherein the hash mapping ring is a logical closed-loop data structure obtained by spatial mapping based on the hash values ​​of the identifier information in the cluster identifier set; receiving an access request for a target file, wherein the access request includes a target file identifier; performing a mapping query on the hash mapping ring based on the target file identifier to determine a target storage cluster; routing the access request to the target storage cluster, and then performing a data scheduling operation on the data associated with the access request within the target storage cluster, wherein the data scope of the data scheduling operation is limited to the set of storage nodes managed by the target storage cluster.

[0007] Optionally, determining the target storage cluster by performing a mapping query on the hash mapping ring based on the target file identifier includes: performing a hash operation on the target file identifier to obtain a first mapping value; performing a matching operation between the first mapping value and a position on the hash mapping ring, wherein the matching operation includes determining a target logical position on the hash mapping ring, and the hash value corresponding to the target logical position is the first value greater than or equal to the first mapping value in a preset ring traversal direction; and determining the storage cluster associated with the target logical position as the target storage cluster.

[0008] Optionally, the access request is routed to the target storage cluster for processing, including: if the access request is a file storage request for a target file, the target file and the file storage request are routed to the target storage cluster, wherein the target storage cluster performs the following operations after receiving the target file: dividing the target file into multiple data blocks and calculating the hash value of each data block; then generating a Merkle tree structure based on the hash values ​​of the multiple data blocks, wherein the root hash of the Merkle tree structure serves as the content identifier of the target file; and synchronously storing the multiple data blocks and the Merkle tree structure among multiple peer nodes within the target storage cluster.

[0009] Optionally, a Merkle tree structure is generated based on the hash values ​​of multiple data blocks, including: using the hash value of each data block as a leaf node hash value of the Merkle tree structure, and then combining all the obtained leaf node hash values ​​into a data set; performing multiple combined hash operations on the data set according to a preset number of node aggregations until the node hash values ​​of each level corresponding to the target file are obtained, wherein the top-level node hash value corresponding to the target file is the root node hash value of the target file; and generating a Merkle tree structure based on the node hash values ​​of each level corresponding to the target file.

[0010] Optionally, multiple data blocks and Merkle tree structures are synchronously stored among multiple peer nodes within the target storage cluster, including: receiving multiple data blocks and Merkle tree structures through the master node in the target storage cluster and storing them locally; and synchronizing the storage operation status of the target file to at least one slave node in the target storage cluster through the master node based on a conflict-free replication data type protocol, wherein the slave node stores multiple data blocks and Merkle tree structures locally according to the received operation status.

[0011] Optionally, the access request is routed to the target storage cluster for processing, including: if the access request is a file read request about the target file, the file read request is routed to the target storage cluster so that the target storage cluster performs the following operations: obtains the corresponding content identifier based on the target file identifier; obtains multiple data blocks constituting the target file from multiple peer nodes within the target storage cluster based on the Merkle tree structure corresponding to the content identifier, and then restores the target file by combining the multiple data blocks.

[0012] Optionally, the data scheduling method of the distributed storage system further includes: monitoring changes in storage clusters; adding new storage clusters to the distributed storage system and updating the cluster identifier set based on the changes; obtaining the updated cluster identifier set from the configuration center; and updating the hash mapping ring based on the updated cluster identifier set.

[0013] Optionally, obtaining a cluster identifier set from the configuration center includes: accessing a distributed key-value database, wherein the distributed key-value database is used to represent a database system that organizes data into key-value pairs and distributes storage and management across multiple nodes; reading cluster registration entries organized with a preset naming prefix, wherein registration entries of multiple nodes belonging to the same storage cluster share the same naming prefix; the registration entry is used to represent a structured data record including at least one of the following: node network address, status identifier, and cluster identifier; the preset naming prefix is ​​used to represent a string used to divide and classify data entries belonging to different logical groups in the key space of the distributed key-value database.

[0014] According to another aspect of the embodiments of this application, a data scheduling apparatus for a distributed storage system is also provided, comprising: an acquisition unit, configured to acquire a cluster identifier set from a configuration center, wherein the cluster identifier set includes identifier information of N storage clusters, where N is an integer greater than 1; a first processing unit, configured to generate a hash mapping ring based on the cluster identifier set, wherein the hash mapping ring is a logical closed-loop data structure obtained by spatial mapping based on the hash values ​​of the identifier information in the cluster identifier set; a second processing unit, configured to receive an access request for a target file, wherein the access request includes a target file identifier; a query unit, configured to perform a mapping query on the hash mapping ring based on the target file identifier to determine the target storage cluster; and a third processing unit, configured to route the access request to the target storage cluster, and then perform a data scheduling operation on the data associated with the access request within the target storage cluster, wherein the data scope of the data scheduling operation is limited to the set of storage nodes managed by the target storage cluster.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the data scheduling method of the distributed storage system described above.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the data scheduling method of the distributed storage system described above.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the data scheduling method of the distributed storage system described above.

[0018] In this application, the data scheduling method of the distributed storage system first obtains a cluster identifier set from the configuration center, wherein the cluster identifier set includes the identifier information of N storage clusters, where N is an integer greater than 1; generates a hash mapping ring based on the cluster identifier set, wherein the hash mapping ring is a logical closed-loop data structure obtained by spatial mapping based on the hash values ​​of the identifier information in the cluster identifier set; receives an access request for a target file, wherein the access request includes a target file identifier; performs a mapping query on the hash mapping ring based on the target file identifier to determine the target storage cluster; routes the access request to the target storage cluster, and then performs a data scheduling operation on the data associated with the access request within the target storage cluster, wherein the data scope of the data scheduling operation is limited to the set of storage nodes managed by the target storage cluster.

[0019] In this embodiment, a data scheduling method for a distributed storage system is employed. By obtaining the identifier set of the storage cluster from the configuration center and then generating a hash mapping ring, file access requests can be intelligently allocated to the most suitable storage cluster, achieving efficient data scheduling. This method not only helps reduce data replication and storage resource waste across all network nodes but also improves data access speed and the dynamic expansion capability of storage nodes by limiting operations to specific clusters. This solves the technical problem in existing distributed storage technologies where each node needs to be expanded equally as data volume increases, resulting in poor storage expansion flexibility. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a diagram illustrating the limitations of existing technology;

[0022] Figure 2 This is a flowchart of an optional data scheduling method for a distributed storage system according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of an optional hash ring according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of an optional decentralized storage network according to an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of an optional access control technology according to an embodiment of this application;

[0026] Figure 6 This is a schematic diagram of an optional proximity access technology according to an embodiment of this application;

[0027] Figure 7 This is a schematic diagram of an optional Merkle storage structure according to an embodiment of this application;

[0028] Figure 8 This is an optional file content identifier tree structure diagram according to an embodiment of this application;

[0029] Figure 9 This is a schematic diagram of an optional file deduplication technique according to an embodiment of this application;

[0030] Figure 10This is a schematic diagram of an optional multi-source transmission technology according to an embodiment of this application;

[0031] Figure 11 This is a schematic diagram of an optional hash mapping ring adding a node according to an embodiment of this application;

[0032] Figure 12 This is an optional network deployment diagram according to an embodiment of this application;

[0033] Figure 13 This is a schematic diagram of the main flow of an optional data scheduling method for a distributed storage system according to an embodiment of this application;

[0034] Figure 14 This is a schematic diagram of a data scheduling device for an optional distributed storage system according to an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] According to an embodiment of this application, an embodiment of a data scheduling method for a distributed storage system is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] According to the embodiments of this application, a data scheduling system of a distributed storage system (hereinafter referred to as the system) can be used as the execution subject of the data scheduling method of the distributed storage system in the embodiments of this application. The data scheduling system of the distributed storage system can be a software system or an embedded system combining software and hardware. Of course, the execution subject of the method in the embodiments of this application can also be other forms of execution subject, such as devices or equipment. Those skilled in the art should know that this application does not particularly limit the specific form of the execution subject.

[0039] Figure 2 This is a flowchart of a data scheduling method for a distributed storage system according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0040] Step S201: Obtain the cluster identifier set from the configuration center. The cluster identifier set includes the identifier information of N storage clusters, where N is an integer greater than 1.

[0041] Optionally, the configuration center is a centralized service that stores the configuration information of all nodes or components in the system for distributed system coordination and service discovery. The cluster identifier set refers to the set of identifier information for all storage clusters in the system. Here, N storage clusters represent the number of independent clusters participating in storage within the system, reflecting the system's multi-cluster mode.

[0042] Optionally, the system can proactively or periodically retrieve a set of cluster identifiers from the configuration center during startup or operation. For example, when a cluster node corresponding to a file system starts up, it can establish a communication connection with the configuration center to obtain the identifier information of all storage clusters in the current system. This information may include metadata such as cluster name, status, and storage node address. Through centralized management by the configuration center, the system can reduce inconsistencies in cluster information and enhance system stability and maintainability.

[0043] Step S202: Generate a hash mapping ring based on the cluster identifier set, wherein the hash mapping ring is a logical closed-loop data structure obtained by spatial mapping based on the hash values ​​of the identifier information in the cluster identifier set.

[0044] Alternatively, a hash mapping ring is a logical structure used for data location and load balancing in distributed systems. The system performs hash operations on cluster identification information, mapping the identifier to a virtual ring, thereby selecting which cluster data and requests should be assigned to.

[0045] Optionally, the system performs a hash operation on each identifier based on the acquired set of cluster identifiers to generate a virtual ring mapping. The construction of the hash mapping ring generally needs to consider the ring size and partitioning strategy, which helps to improve the uniformity of data distribution and query efficiency. For example, the hash ring can be set to a range of 0 to (2^32-1), and virtual node technology can be used to increase the partition density on the ring, thereby improving load balancing.

[0046] Optionally, after generating the hash mapping ring, the system can quickly locate data requests, reduce full network scanning, and lower data access latency. Simultaneously, through the dynamic characteristics of the hash ring, the system does not need to redistribute existing data on a large scale during expansion; it only needs to add new cluster identifiers to the ring, simplifying the management and expansion of storage resources.

[0047] Step S203: Receive an access request for the target file, wherein the access request includes the target file identifier.

[0048] Optionally, the target file identifier is information that can identify the target file, which can be a filename, a file identifier, or a hash value generated by a certain algorithm.

[0049] Optionally, when a business application or external system sends a file access request, the system receives the request and extracts the target file identifier, preparing for further scheduling and querying. Through the file identifier, the system can track and process requests for specific files, improving the accuracy and efficiency of data access.

[0050] Step S204: Perform a mapping query on the hash mapping ring based on the target file identifier to determine the target storage cluster.

[0051] Optionally, mapping query refers to the process of mapping the target file identifier onto a hash ring through a hash operation, in order to select the storage cluster responsible for the file.

[0052] Optionally, the system hashes the target file identifier and then searches for the point on the hash ring that is closest to this hash value. The cluster corresponding to this hash value is the target storage cluster. For example, if the hash value of the target file identifier falls at a certain position on the hash ring, the system will find the cluster identifier point closest to this value along the direction of the ring, thereby selecting the target storage cluster.

[0053] Optionally, using a query mechanism helps to allocate each file access request to an appropriate storage cluster, reducing hotspot clusters. At the same time, due to the characteristics of hash rings, data distribution can be smoothly adjusted even when the network topology changes, maintaining the efficient operation of the system.

[0054] Step S205: The access request is routed to the target storage cluster, and then a data scheduling operation is performed on the data associated with the access request within the target storage cluster. The data scope of the data scheduling operation is limited to the set of storage nodes managed by the target storage cluster.

[0055] Optionally, data scheduling operations refer to the actual operations performed on the requested file within the storage cluster, such as data processing tasks like reading, writing, updating, or deleting.

[0056] Optionally, once a target storage cluster is selected, the system directs file access requests to that cluster. Storage nodes within the cluster will perform appropriate data scheduling operations based on the request type. For example, in the case of a read request, the storage node will locate and read file fragments, assemble them into a complete file, and return it to the requester. By limiting operations to the target cluster, the system reduces cross-cluster data transfer and processing, improving file access speed and overall system responsiveness.

[0057] In one optional embodiment, determining the target storage cluster by performing a mapping query on a hash mapping ring based on the target file identifier includes: performing a hash operation on the target file identifier to obtain a first mapping value; performing a matching operation between the first mapping value and a position on the hash mapping ring, wherein the matching operation includes determining a target logical position on the hash mapping ring, the hash value corresponding to the target logical position being the first value greater than or equal to the first mapping value in a preset ring traversal direction; and determining the storage cluster associated with the target logical position as the target storage cluster.

[0058] Optionally, the target file identifier can be the file name, identifier, or a file identifier generated through a hash operation. A hash operation is an algorithm that transforms an input of arbitrary length into a fixed-length output, often called a hash value or hash code, used for fast file lookup and data integrity verification.

[0059] Optionally, when the system receives an access request containing a target file identifier, it performs a hash algorithm on this identifier to obtain a fixed-length first mapping value. This first mapping value will be used as the basis for finding the target location on the hash mapping ring.

[0060] Optionally, the hash mapping ring is a virtual closed-loop data structure. By hashing the cluster identifier and distributing the results across the ring, a logical node distribution graph is formed. The default ring traversal direction is usually clockwise, used to select which cluster a file is mapped to.

[0061] Optionally, the system compares the obtained first mapping value with the hash mapping ring, starting from the first mapping value, and searches for the position on the ring where the first hash value is greater than or equal to the first mapping value according to the preset ring traversal direction (such as clockwise), which is the target logical position.

[0062] Optionally, if the target logical position is found, the system will retrieve the cluster name or identifier corresponding to this position, and use the corresponding cluster as the target storage cluster. Subsequently, the access request will be routed to this cluster to perform subsequent data scheduling operations.

[0063] Figure 3 It is a schematic diagram of an optional hash ring according to an embodiment of the present application. The system connects the hash value space end to end to form a ring (usually from 0 to (2^32 - 1)). During initialization, the hash value of the incoming file system cluster name is calculated and mapped to the ring. When obtaining, the data key (the requested object name) is also hashed to the ring, and the nearest node in the clockwise direction is found as the归属. For example, the cluster names are File System - 01 (i.e., Node 1), File System - 02 (i.e., Node 2), File System - 03 (i.e., Node 3), and the requested object name is: abc.txt. The calculated values after hashing are shown in Table 1:

[0064] Table 1

[0065]

[0066] The system queries the nearest node in the clockwise direction, then File System - 01 < abc.txt < File System - 02 < File System - 03, so the finally hit node is File System - 02. As Figure 3 shown, the found node after mapping the object name is Node 2.

[0067] In an optional embodiment, routing the access request to the target storage cluster for processing operations includes: in the case where the access request is a file storage request for a target file, routing the target file and the file storage request to the target storage cluster. The operations performed by the target storage cluster after receiving the target file include: splitting the target file into multiple data blocks, calculating the hash value of each data block; then generating a Merkle tree structure based on the hash values of the multiple data blocks, where the root hash of the Merkle tree structure is used as the content identifier of the target file; synchronously storing the multiple data blocks and the Merkle tree structure among multiple peer nodes within the target storage cluster.

[0068] Optionally, the file storage request comes from a client or a business system and is a request to store a new file or update an existing file in a distributed storage system.

[0069] Optionally, when the system receives a file storage request, it first locates the target storage cluster using a hash ring based on the target file identifier in the request. Then, it sends the target file and its associated storage request information (such as file attributes, storage policies, etc.) to the target storage cluster, waiting for the cluster to complete the storage operation. This helps the file storage request to be allocated to a more suitable storage cluster, reducing data transmission across the entire network.

[0070] Optionally, after receiving a file storage request, the target storage cluster first divides the target file into multiple data blocks according to a pre-defined size (e.g., 256KB). Then, it performs a hash operation on each data block to generate a corresponding hash value, which is used for subsequent file integrity verification and data location.

[0071] Alternatively, a Merkle tree is a binary tree structure where each non-leaf node is a hash of the hash values ​​of its child nodes. Therefore, the root node hash value can be considered a unique content identifier for the entire file. The target storage cluster constructs the Merkle tree using the hash values ​​of data blocks. Starting from the bottom-level data block hash values, the calculation is performed by merging and calculating at each level until the root hash value of the tree is generated, which is the content identifier of the target file.

[0072] Optionally, the Merkle tree structure can simplify the file verification process, requiring only a comparison of the root hash value to determine if the file has been tampered with, without needing to download and compare the entire file. Simultaneously, the root hash value serves as an identifier for the target file, facilitating file location and management across different storage nodes.

[0073] Optionally, peer nodes within the target storage cluster will store each data block and the Merkle tree structure through a data synchronization mechanism. Nodes can periodically exchange state information, which helps all nodes hold the same copy of the data blocks and maintain the consistency of the Merkle tree structure.

[0074] Figure 4This is a schematic diagram of an optional decentralized storage network according to an embodiment of this application. The system first divides the file into blocks, for example, dividing a 768kb file into three data blocks of 256kb each. Subsequently, the system generates n (n is an integer greater than 1) storage nodes. For each storage node, a node file block object is constructed. The node file block object contains three key parts: encrypted data (Data), link, and content hash (Cid). Among them, the encrypted data is the data block that has been securely processed, the link is used to maintain the logical relationship between nodes, and the content hash is obtained by performing a hash operation on the content of the data block and is used for data integrity and consistency verification. For example, the encrypted data (Data) included in the node file block object of storage node 1 can be denoted as [By80BX], and the encrypted data (Data) included in the node file block object of storage node 2 can be denoted as [Ht15EB].

[0075] Figure 5 This is a schematic diagram of an optional access control technology according to an embodiment of this application. The distributed storage system integrates a security token service, supports the use of secure access protocols, and generates temporary security token service credentials. Users can access data with restricted (read / write permissions) within the credential's validity period, providing data access authorization management services, reducing malicious data access, and improving data access security. Figure 5 As shown, the access steps are as follows: First, the user authenticates themselves on the application client. After successful login, the application server initiates a request to obtain a security token service credential to the blockchain storage node of the distributed storage system. The blockchain storage node generates the security token service credential and securely returns it to the application server. The application server then forwards the security token service credential to the application client, completing the credential transfer process. Finally, the application client uses the received security token service credential to interact with the distributed storage system, uploading or downloading resources.

[0076] Figure 6 A schematic diagram of an optional proximity access technology according to an embodiment of this application is shown. Distributed storage provides two proximity access strategies: Internet Protocol (IP) segment-based and peer-to-peer-based. Storage nodes can calculate the communication distance between different nodes, select the optimal scheduling link, and improve file upload and download speeds. Figure 6 As shown, when a user in region A requests a file stored in region B, if there is a neighboring node in region A that is related to the requested data block, the system will first attempt to retrieve the data from the neighboring node. If there is no relevant node in region A, or to further improve the data retrieval speed, the system will use a point-to-point strategy to directly retrieve data fragments from the node in region B with the best network conditions, thereby achieving fast and stable file transfer without being limited by geographical location.

[0077] In one optional embodiment, a Merkle tree structure is generated based on the hash values ​​of multiple data blocks, including: using the hash value of each data block as a leaf node hash value of the Merkle tree structure, and then combining all the obtained leaf node hash values ​​into a data set; performing multiple combined hash operations on the data set according to a preset number of node aggregations until the node hash values ​​of each level corresponding to the target file are obtained, wherein the top-level node hash value corresponding to the target file is the root node hash value of the target file; and generating a Merkle tree structure based on the node hash values ​​of each level corresponding to the target file.

[0078] Optionally, after the target file is divided into multiple data blocks, the system will perform a hash operation on each data block to generate leaf node hash values. Subsequently, all leaf node hash values ​​will be collected to form a data set, which serves as the basic elements for constructing the Merkle tree.

[0079] Optionally, the system groups the leaf node hash values ​​in the dataset according to a preset number of aggregations, and performs a combined hash operation on the hash values ​​of each group to generate higher-level intermediate node hash values. This process is repeated multiple times until the top-level root node hash value is generated. Through combined hash operations, the system gradually constructs the hierarchical structure of a Merkle tree, realizing hierarchical management and verification of target file data blocks, and improving the efficiency and flexibility of file integrity verification.

[0080] Optionally, after the hash values ​​of all levels of nodes have been calculated, the system uses the calculated hash values ​​as the basis to construct the complete structure of the Merkle tree from bottom to top. The root node hash value serves as a unique identifier for the target file and is stored in a hash table, while the hash values ​​of other nodes in the tree are used for data block management and verification.

[0081] Figure 7 A schematic diagram of an optional Merkel storage structure according to an embodiment of this application is shown. Distributed storage uses a Merkel directed acyclic graph structure to store data, such as... Figure 7 As shown, a raw file storage structure consists of a root node, a set of intermediate nodes, and a set of leaf nodes. Each leaf node contains a data fragment, intermediate nodes are hash values ​​of their child node data, and the root node is the hash value of all intermediate node data. Distributed storage uses the root node as the unique identifier of the file, and by verifying the hash value, it can detect whether any data block has been modified, reducing the possibility of malicious data tampering. For example, Figure 7As shown, the top hash node is the root node; hash node 0 and hash node 1 are two intermediate nodes; hash node 0_0 and hash node 0_1 are child nodes of hash node 0, and hash node 1_0 and hash node 1_1 are child nodes of hash node 1. Among them, hash node 0_0, hash node 0_1, hash node 1_0 and hash node 1_1 are four leaf nodes, corresponding to data shards L1, L2, L3 and L4 respectively.

[0082] Figure 8 An optional file content identifier tree structure diagram according to an embodiment of this application is shown. For example... Figure 8 As shown, the file system reads a file in 256KB (maximum) chunks, then uses a hash algorithm to generate content identifiers (such as the content identifier of child node 1) based on the chunk content, forming a Merkle tree of the file. Each level of the tree can contain up to 174 child nodes. When there are more than 174 nodes, a new parent node is created, and leaf nodes are reattached to it. For example, node 1 contains 174 child nodes, and node 2 contains the 175th child node. Node 1 and node 2 are both children of the root node, which represents the root content identifier (including the content identifiers of all child nodes under the root node, such as the content identifiers of node 1 and node 2).

[0083] In one optional embodiment, multiple data blocks and Merkle tree structures are synchronously stored among multiple peer nodes within the target storage cluster, including: receiving multiple data blocks and Merkle tree structures locally through a master node in the target storage cluster; and synchronizing the storage operation status of the target file to at least one slave node in the target storage cluster based on a conflict-free replication data type protocol, wherein the slave node stores multiple data blocks and Merkle tree structures locally according to the received operation status.

[0084] Optionally, the master node is responsible for receiving external file storage requests and data blocks, while also storing the Merkle tree structure. When a target file and its corresponding Merkle tree structure are routed to the target storage cluster via an access request, the master node first receives the information, stores the data blocks in its local storage medium, and saves the Merkle tree structure for subsequent data verification and management.

[0085] Optionally, conflict-free replication data types allow simultaneous read and write operations on multiple nodes without concern for data conflicts, automatically resolving conflicts via timestamps or version numbers. After completing local storage of the data block, the master node transmits the storage operation status (including operation timestamps, version numbers, etc.) to at least one slave node via the network, according to the conflict-free replication data type protocol. Based on the received status information, the slave node determines whether the data block already exists locally. If it does not exist or the version is too old, it obtains and stores the data block and Merkle tree structure from the master node or other nodes in the network, thereby helping to maintain data consistency within the cluster.

[0086] Figure 9 A schematic diagram of an optional file deduplication technique according to an embodiment of this application is shown. After data files are divided into blocks, a Merkle directed acyclic graph is used to store the block data. Each file has a unique content identifier and records the corresponding block index. The smaller the file differences, the higher the probability that the hash values ​​are the same after file division, and the more file indexes point to the same block. For identical data shards, distributed storage only retains a certain amount of redundancy for fault recovery. The more files stored in the cluster, the higher the file content redundancy, and the more obvious the effect of compressing storage space, thereby achieving file deduplication, saving storage space, and reducing the storage cost of large files. Figure 9 As shown, the original files (including files A1, A2, and A3) are first divided into chunks. Then, a unique content identifier is generated using a hash algorithm, and an index relationship is established between each content identifier and its corresponding original file (e.g., the index relationship between content identifier 1 and file A1, content identifier 2 and file A2, and content identifier 3 and file A3). This mapping between files and data chunks is recorded. When the system receives a new file upload request, it checks whether the content identifier generated after the file chunking already exists in the system. If it does, it means that two files have duplicate parts in this chunk. In this case, the system no longer stores this data chunk, but only adds a reference to the chunk in the content identifier index.

[0087] In one optional embodiment, routing the access request to the target storage cluster for processing includes: if the access request is a file read request about a target file, routing the file read request to the target storage cluster so that the target storage cluster performs the following operations: obtaining the corresponding content identifier based on the target file identifier; obtaining multiple data blocks constituting the target file from multiple peer nodes within the target storage cluster based on the Merkle tree structure corresponding to the content identifier, and then restoring the target file by combining the multiple data blocks.

[0088] Optionally, when the system detects that the access request is a file read request, it will determine the target storage cluster based on the target file identifier using a hash ring and forward the read request to that cluster. By accurately locating the target storage cluster, the system can respond to read requests quickly, reducing the time spent searching the entire network and improving data access efficiency.

[0089] Optionally, upon receiving a file read request, the target storage cluster first searches the hash table for the corresponding content identifier (i.e., the root node hash value of the Merkle tree) based on the target file identifier. If the content identifier of the target file is obtained, the target storage cluster will parse the file's block structure according to the Merkle tree structure, and then select the optimal node from multiple peer nodes within the cluster (usually based on the node's current load and data redundancy) to obtain the data blocks constituting the target file.

[0090] Optionally, after the target storage cluster has collected the necessary data blocks, it will merge and reassemble the data blocks according to the original file's block order and size to restore the complete file data. Finally, the file data will be returned to the party that initiated the read request. The system utilizes file restoration operations to help clients receive file data consistent with the original storage, thus meeting the business requirements for data integrity and consistency.

[0091] Figure 10 A schematic diagram of an optional multi-source transmission technology according to an embodiment of this application is shown. Distributed storage allows different parts of a file to be retrieved simultaneously from multiple nodes, thereby accelerating file transfer speed. Multiple parallel connections can improve the efficiency and reliability of file transfer. When a user issues a file download request, the system locates the storage node set based on the file's content identifier and selects the optimal multi-node for parallel data transmission based on the node's current state and location information. Afterward, the client or proxy server receives data blocks in parallel from the selected nodes, and upon completion of reception, reassembles the data blocks based on the content identifier metadata to restore the original file. Figure 10 As shown, file fragment 1 is stored on storage node 1, file fragment 2 is stored on storage node 2, and file fragment 3 is stored on storage node 3. After receiving the data, the data blocks are reassembled to restore the file for users to download.

[0092] Optionally, to improve the consistency of data block state across multiple storage nodes, the system employs a rumor propagation protocol. This is a decentralized protocol that uses message propagation between random nodes to eventually achieve state consistency across all nodes. Whenever a new node joins or node data changes, the rumor propagation protocol helps to quickly update the storage location information of file blocks, ensuring the up-to-date state of data distribution among nodes.

[0093] Optionally, the file system cluster uses conflict-free replicated data types to achieve state consistency among distributed nodes, which helps to efficiently manage data (such as Pin / Unpin operations) in a decentralized environment. The steps are as follows:

[0094] 1. Users initiate Pin / Unpin operations to any node.

[0095] 2. Nodes locally update the status of conflict-free replicated data types (record operation timestamps and logical clocks).

[0096] 3. The status is asynchronously propagated to other nodes via the rumor propagation protocol.

[0097] Among them, conflict resolution means that if two nodes simultaneously pin and unpin the same content identifier, the final state of the conflict-free replicated data type will be determined based on the timestamp or version number; the synchronization mechanism is that nodes periodically broadcast their own status or push incremental updates through the file system message publishing mechanism channel.

[0098] In an optional embodiment, the data scheduling method of the distributed storage system further includes: monitoring change information of the storage cluster; adding the newly added storage cluster to the distributed storage system and updating the cluster identifier set according to the change information; obtaining the updated cluster identifier set from the configuration center; and updating the hash mapping ring according to the updated cluster identifier set.

[0099] Optionally, if a new storage cluster is detected joining, the system will automatically trigger a cluster registration process. First, the newly joined storage cluster will report its existence to the configuration center, submitting its cluster name, node list, and other metadata. Subsequently, the configuration center updates the storage cluster's identifier set, adding the new cluster's information to form an updated cluster identifier set. This helps the system recognize cluster changes and prepare to adapt to the new cluster layout.

[0100] Optionally, after changes occur in the storage cluster, all routing nodes and distributed cloud storage server nodes in the system will obtain the latest cluster identifier set from the configuration center. The system achieves this operation by subscribing to change notifications from the configuration center; any update to the cluster identifier set can be pushed to the relevant nodes, helping to keep abreast of the latest cluster status.

[0101] Optionally, finally, the system reconstructs its hash map ring based on the updated cluster identifier set. Updating the hash map ring involves recalculating the hash values ​​of all nodes and then constructing a coherent ring structure based on the recalculated hash values. The hash value of the data object will be used to locate the target node on the ring.

[0102] Figure 11This diagram illustrates an optional hash mapping ring for adding nodes according to an embodiment of this application. When the throughput of nodes within the same cluster is relatively low, more peer nodes (those reporting keys to the distributed key-value storage system with the same key name prefix, such as " / storage prefix / cluster name / node name") can be added to the file system cluster to obtain higher throughput. When the storage system's storage space is insufficient, storage resources can be further increased by adding file system clusters. Figure 11 As shown, after adding cluster nodes, node 4 was added to the hash ring, but the original mapping data may not need to be migrated, reducing the amount of data migration required due to expansion. Figure 11 For explanations of other nodes and the clockwise search method, please refer to [link / reference]. Figure 3 The relevant explanation.

[0103] In one optional embodiment, obtaining a cluster identifier set from a configuration center includes: accessing a distributed key-value database, wherein the distributed key-value database is used to characterize a database system that organizes data into key-value pairs and distributes storage and management across multiple nodes; reading cluster registration entries organized with a preset naming prefix, wherein registration entries of multiple nodes belonging to the same storage cluster share the same naming prefix; the registration entries are used to characterize structured data records including at least one of node network address, status identifier, and cluster identifier; the preset naming prefix is ​​used to characterize strings that divide and classify data entries belonging to different logical groups in the key space of the distributed key-value database.

[0104] Optionally, a distributed key-value database stores data as key-value pairs across multiple nodes, supporting high parallelism and elastic scaling, making it suitable for storing and managing configuration information in distributed systems. The system sends a request to the configuration center, which responds to the distributed key-value database running within the configuration center. The configuration center acts as the storage and management center for the storage cluster's metadata. By accessing the distributed key-value database, the system can obtain the latest metadata of the storage cluster, including the node's network address, status identifier, and cluster identifier, providing a data source for subsequent cluster identifier set extraction and retrieval operations.

[0105] Optionally, a default naming prefix is ​​a string identifier used in the keyspace of a distributed key-value database to identify and categorize data entries belonging to the same logical group (such as the same storage cluster). For example, " / storage prefix / cluster name / " can be used as a default naming prefix for registering and retrieving node information of different storage clusters on the distributed key-value database.

[0106] Optionally, the system uses the API of the key-value database to read keys that begin with a preset naming prefix, thereby obtaining the associated cluster registration entries. These registration entries contain key information about each node in the storage cluster. The system then parses the cluster registration entries read with the preset naming prefix, extracting the cluster identifier for each entry and compiling these identifiers into a set. By extracting the set of cluster identifiers, the system can distinguish which nodes belong to the same storage cluster, thus helping to direct access requests to nodes in the target storage cluster.

[0107] Figure 12 An optional network deployment diagram according to an embodiment of this application is shown. For example... Figure 12 As shown, firstly, internal users access internal business services via Hypertext Transfer Protocol (HTTP), triggering calls to the business server. Upon receiving a user request, the internal server invokes the file storage service, sending instructions to the distributed cloud storage system. The distributed cloud storage system can use F5 devices for rate limiting and forwarding, and accesses distributed cloud storage service clusters located in Campus A and Campus B respectively via HTTP. These two campuses represent different geographical regions or data centers, providing file storage services through their own independent yet coordinated mechanisms. Each campus deploys a management component, a distributed key-value store, to handle tasks such as user-level credential management, node management, and communication between servers.

[0108] In Zone A, the file system cluster command-line tool interacts with the file system cluster service in Zone A across multiple blockchain PaaS clusters. The "file system cluster service" mentioned here is essentially a collection of storage nodes; for example, "File System Cluster - Service 1" might contain "File System Storage Node 1". Services communicate using a conflict-free replication data type protocol. The same operational model applies to Zone B, where the file system cluster command-line tool interacts with the file system cluster service within the zone, and inter-service communication also relies on the conflict-free replication data type protocol. Furthermore, a hash ring is introduced in the deployment diagram, dividing the file system storage nodes into several groups. Nodes within each group have identical data, and data synchronization within a group is achieved through a rumor propagation protocol. The data between groups constitutes a complete data set, rather than each node storing the same data, thus enabling horizontal scaling of storage nodes.

[0109] Figure 13 The main flowchart of a data scheduling system for an optional distributed storage system according to an embodiment of this application is shown. Figure 13As shown, user requests are first initiated through a software development kit (SDK), whether for storing or retrieving files. The SSD forwards the request to a distributed cloud storage server node, which calculates the target file system cluster node based on a preset hash ring algorithm. This node is determined by matching the hash value of the file object name with the hash value of the cluster node name. Subsequently, data is transmitted to the selected file system cluster node, which uses a conflict-free replication data type algorithm and a rumor propagation protocol to achieve data state synchronization and consistency. Furthermore, the system supports dynamic expansion. Newly added file system cluster nodes are automatically registered to the distributed key-value storage system center. After the distributed cloud storage server node detects an update to the node list, it recalculates the hash ring, allowing the new node to participate in the distributed storage and transmission of data, further enhancing the system's throughput and elastic scalability. The main process includes:

[0110] Distributed key-value storage system registry center: Stores file system cluster node information. When a file system cluster node starts up, it registers its node information with the distributed key-value storage system. When a file system cluster node is destroyed, the distributed key-value storage system information is automatically invalidated.

[0111] Distributed cloud storage server node: Obtains file system cluster node information stored in the distributed key-value storage system, generates a hash ring based on the node name, and when a transaction request is received, obtains the file system cluster node on the corresponding hash ring based on the hash result of the requested object name, and transmits the transaction data.

[0112] File system cluster nodes: Request the corresponding file system node, and then synchronize the transaction operation to the file system cluster nodes in the same cluster through a conflict-free replication data type protocol, so that different file system nodes can achieve final data consistency.

[0113] File system node data storage: The transaction data is divided into blocks of 256k (maximum block size), and then a content identifier is generated based on the block content using a hash algorithm. A Merkle tree is generated and the data is saved locally. Finally, the interface returns the content identifier of the root node of the Merkle tree.

[0114] File system node data query: During the query, the root node content identifier is first found based on the mapping relationship between the file name and the root node content identifier. Then, the block content (a list of child node content identifiers or a list of leaf nodes, with leaf nodes containing file data blocks) is retrieved based on the root content identifier. Finally, all the file data block contents are returned one by one to reassemble a complete file.

[0115] Figure 14This is a schematic diagram of a data scheduling device for an optional distributed storage system according to an embodiment of this application. According to another aspect of an embodiment of this application, a data scheduling device for a distributed storage system is also provided, comprising: an acquisition unit 1401, a first processing unit 1402, a second processing unit 1403, a query unit 1404, and a third processing unit 1405.

[0116] The system includes the following components: an acquisition unit 1401, which acquires a cluster identifier set from a configuration center, wherein the cluster identifier set includes identifier information for N storage clusters, where N is an integer greater than 1; a first processing unit 1402, which generates a hash mapping ring based on the cluster identifier set, wherein the hash mapping ring is a logical closed-loop data structure obtained by spatial mapping based on the hash values ​​of the identifier information in the cluster identifier set; a second processing unit 1403, which receives an access request for a target file, wherein the access request includes a target file identifier; a query unit 1404, which performs a mapping query on the hash mapping ring based on the target file identifier to determine the target storage cluster; and a third processing unit 1405, which routes the access request to the target storage cluster and then performs a data scheduling operation on the data associated with the access request within the target storage cluster, wherein the data scope of the data scheduling operation is limited to the set of storage nodes managed by the target storage cluster.

[0117] Optionally, the query unit 1404 includes: a first mapping value subunit, used to perform a hash operation on the target file identifier to obtain a first mapping value; a matching subunit, used to perform a matching operation between the first mapping value and a position on the hash mapping ring, wherein the matching operation includes determining a target logical position on the hash mapping ring, and the hash value corresponding to the target logical position is the first value greater than or equal to the first mapping value in a preset ring traversal direction; and a determining subunit, used to determine the storage cluster associated with the target logical position as the target storage cluster.

[0118] Optionally, the third processing unit 1405 includes: a Merkle tree generation subunit, used to route the target file and the file storage request to the target storage cluster when the access request is a file storage request for the target file, wherein the target storage cluster performs the following operations after receiving the target file: dividing the target file into multiple data blocks and calculating the hash value of each data block; then generating a Merkle tree structure based on the hash values ​​of the multiple data blocks, wherein the root hash of the Merkle tree structure serves as the content identifier of the target file; and synchronously storing the multiple data blocks and the Merkle tree structure among multiple peer nodes within the target storage cluster.

[0119] Optionally, the Merkle tree subunit is generated, comprising: a data set module, used to take the hash value of each data block as a leaf node hash value of the Merkle tree structure, and then combine all the obtained leaf node hash values ​​into a data set; a hash operation module, used to perform multiple combined hash operations on the data set according to a preset number of node aggregations, until the node hash values ​​of each level corresponding to the target file are obtained, wherein the top-level node hash value corresponding to the target file is the root node hash value of the target file; and a generation module, used to generate a Merkle tree structure according to the node hash values ​​of each level corresponding to the target file.

[0120] Optionally, generating the Merkle tree subunit further includes: a local storage module, used to receive multiple data blocks and the Merkle tree structure through the master node in the target storage cluster and store them locally; and a synchronization module, used to synchronize the storage operation status of the target file to at least one slave node in the target storage cluster through the master node based on a conflict-free replication data type protocol, wherein the slave node stores multiple data blocks and the Merkle tree structure locally according to the received operation status.

[0121] Optionally, the third processing unit 1405 further includes: when the access request is a file read request about the target file, routing the file read request to the target storage cluster so that the target storage cluster performs the following operations: obtaining the corresponding content identifier based on the target file identifier; obtaining multiple data blocks constituting the target file from multiple peer nodes within the target storage cluster based on the Merkle tree structure corresponding to the content identifier, and then restoring the target file by combining the multiple data blocks.

[0122] Optionally, the data scheduling device of the distributed storage system further includes: a monitoring unit for monitoring change information of the storage cluster; an update unit for adding the new storage cluster to the distributed storage system and updating the cluster identifier set according to the change information; a cluster identifier set acquisition unit for acquiring the updated cluster identifier set from the configuration center; and a hash mapping ring update unit for updating the hash mapping ring according to the updated cluster identifier set.

[0123] Optionally, the acquisition unit 1401 includes: an access subunit for accessing a distributed key-value database, wherein the distributed key-value database represents a database system that organizes data into key-value pairs and distributes storage and management across multiple nodes; and a read subunit for reading cluster registration entries organized with a preset naming prefix, wherein registration entries of multiple nodes belonging to the same storage cluster share the same naming prefix; the registration entry represents a structured data record including at least one of node network address, status identifier, and cluster identifier; and the preset naming prefix represents a string used to divide and classify data entries belonging to different logical groups in the key space of the distributed key-value database.

[0124] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the data scheduling method of the distributed storage system described above.

[0125] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the data scheduling method of the distributed storage system described above.

[0126] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the data scheduling method of the distributed storage system described above.

[0127] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0128] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0133] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data scheduling method for a distributed storage system, characterized in that, include: Obtain a cluster identifier set from the configuration center, wherein the cluster identifier set includes identifier information of N storage clusters, where N is an integer greater than 1; A hash mapping ring is generated based on the cluster identifier set, wherein the hash mapping ring is a logical closed-loop data structure obtained by spatial mapping based on the hash values ​​of the identifier information in the cluster identifier set. Receive an access request for a target file, wherein the access request includes a target file identifier; Based on the target file identifier, a mapping query is performed on the hash mapping ring to determine the target storage cluster; The access request is routed to the target storage cluster, and then a data scheduling operation is performed on the data associated with the access request within the target storage cluster, wherein the data scope of the data scheduling operation is limited to the set of storage nodes managed by the target storage cluster.

2. The data scheduling method for a distributed storage system according to claim 1, characterized in that, Based on the target file identifier, a mapping query is performed on the hash mapping ring to determine the target storage cluster, including: Perform a hash operation on the target file identifier to obtain a first mapping value; The first mapping value is matched with the position on the hash mapping ring. The matching operation includes determining a target logical position on the hash mapping ring. The hash value corresponding to the target logical position is the first value that is greater than or equal to the first mapping value in a preset ring traversal direction. The storage cluster associated with the target logical location is determined as the target storage cluster.

3. The data scheduling method for a distributed storage system according to claim 1, characterized in that, The access request is routed to the target storage cluster for processing, including: When the access request is a file storage request for the target file, the target file and the file storage request are routed to the target storage cluster. The target storage cluster, upon receiving the target file, performs the following operations: dividing the target file into multiple data blocks and calculating the hash value of each data block; then generating a Merkle tree structure based on the hash values ​​of the multiple data blocks, wherein the root hash of the Merkle tree structure serves as the content identifier of the target file; and synchronously storing the multiple data blocks and the Merkle tree structure among multiple peer nodes within the target storage cluster.

4. The data scheduling method for a distributed storage system according to claim 3, characterized in that, Based on the hash values ​​of the multiple data blocks, a Merkle tree structure is generated, including: The hash value of each data block is used as the hash value of a leaf node in the Merkle tree structure, and then all the obtained leaf node hash values ​​are combined into a data set; Based on the preset number of node aggregations, the data set is subjected to multiple combined hash operations until the node hash values ​​of each level corresponding to the target file are obtained, wherein the top-level node hash value corresponding to the target file is the root node hash value of the target file. The Merkle tree structure is generated based on the node hash values ​​of each level corresponding to the target file.

5. The data scheduling method for a distributed storage system according to claim 3, characterized in that, Synchronously storing the multiple data blocks and the Merkle tree structure among multiple peer nodes within the target storage cluster includes: The master node in the target storage cluster receives the multiple data blocks and the Merkle tree structure and performs local storage. The master node synchronizes the storage operation status of the target file to at least one slave node in the target storage cluster based on the conflict-free replication data type protocol. The slave node stores the multiple data blocks and the Merkle tree structure locally according to the received operation status.

6. The data scheduling method for a distributed storage system according to claim 1, characterized in that, The access request is routed to the target storage cluster for processing, including: If the access request is a file read request for the target file, the file read request is routed to the target storage cluster so that the target storage cluster performs the following operations: Obtain the corresponding content identifier based on the target file identifier; based on the Merkle tree structure corresponding to the content identifier, obtain multiple data blocks constituting the target file from multiple peer nodes within the target storage cluster, and then combine the multiple data blocks to restore the target file.

7. The data scheduling method for a distributed storage system according to claim 1, characterized in that, The data scheduling method of the distributed storage system further includes: Monitor changes to the storage cluster; Based on the change information, the newly added storage cluster is added to the distributed storage system and the cluster identifier set is updated; Obtain the updated cluster identifier set from the configuration center; The hash mapping ring is updated based on the updated set of cluster identifiers.

8. The data scheduling method for a distributed storage system according to claim 1, characterized in that, Retrieve the cluster identifier set from the configuration center, including: Access a distributed key-value database, wherein the distributed key-value database is used to characterize a database system that organizes data into key-value pairs and distributes and manages them across multiple nodes; Read cluster registration entries organized with a preset naming prefix, wherein registration entries of multiple nodes belonging to the same storage cluster share the same naming prefix; the registration entries are used to represent structured data records including at least one of node network address, status identifier and cluster identifier; the preset naming prefix is ​​used to represent strings that divide and classify data entries belonging to different logical groups in the key space of the distributed key-value database.

9. A data scheduling device for a distributed storage system, characterized in that, include: The acquisition unit is used to acquire a cluster identifier set from the configuration center, wherein the cluster identifier set includes the identifier information of N storage clusters, where N is an integer greater than 1; The first processing unit is configured to generate a hash mapping ring based on the cluster identifier set, wherein the hash mapping ring is a logical closed-loop data structure obtained by spatial mapping based on the hash values ​​of the identifier information in the cluster identifier set. The second processing unit is configured to receive an access request for a target file, wherein the access request includes a target file identifier; The query unit is used to perform a mapping query on the hash mapping ring based on the target file identifier to determine the target storage cluster; The third processing unit is configured to route the access request to the target storage cluster, and then perform a data scheduling operation on the data associated with the access request within the target storage cluster, wherein the data scope of the data scheduling operation is limited to the set of storage nodes managed by the target storage cluster.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device in which the computer-readable storage medium is located performs the data scheduling method of the distributed storage system according to any one of claims 1 to 8.

11. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the data scheduling method of the distributed storage system according to any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the data scheduling method of the distributed storage system according to any one of claims 1 to 8.

Images