A method and system for distributed data storage based on erasure codes

Abstract

The embodiment of the present application provides a method and system for distributed data storage based on erasure codes, the method comprising: obtaining n blocks of data blocks and check blocks to be stored, and obtaining a dynamic parameter γ, wherein γ is a number greater than zero and less than 1, and the result of γ·n is an integer; selecting γ·n nodes as the first type of storage nodes according to the node identifier of each node in the distributed system; selecting (1‑γ)·n nodes as the second type of storage nodes from at least part of the remaining nodes included in the distributed system except the first type of storage nodes according to the node performance; using each node in the first type of storage node and the second type of storage node to store a check block or a data block respectively. The embodiment of the present application provides an improved content hash value-based addressing method, and a site selection method combined with comprehensive indicator evaluation. After the data is stored for the first time, the storage location is dynamically adjusted with load balancing as the goal to improve the read and write efficiency and performance of the system.

Description

Technical Field

[0001] The present application relates to the field of distributed system data storage. Specifically, embodiments of the present application relate to a method and system for distributed data storage based on erasure codes. Background Art

[0002] With the advent of the big data era, traditional centralized storage systems are no longer able to meet the storage and processing needs of massive amounts of data, giving rise to distributed storage systems. Compared to centralized storage systems, distributed storage systems offer high reliability, security, and scalability. By distributing and storing complete data across independent nodes, they prevent data loss caused by single points of failure and improve data reliability. Furthermore, distributed storage systems employ load balancing strategies, eliminating the performance bottlenecks caused by central nodes in centralized storage systems and improving read and write efficiency.

[0003] To improve the storage efficiency of distributed storage systems and ensure data consistency, related technologies introduce data fault tolerance. Existing fault tolerance technologies include multi-copy fault tolerance and erasure code fault tolerance. Multi-copy fault tolerance involves storing multiple copies of data on multiple nodes, ensuring that when a node fails, fault tolerance can be achieved by reading data copies from other nodes. Erasure code fault tolerance involves sharding the complete data into multiple data blocks, calculating parity blocks using an erasure code algorithm, and storing the data blocks and parity blocks on different nodes according to specific rules, providing data redundancy without increasing excessive storage overhead. Because erasure codes can enhance fault tolerance while reducing storage overhead compared to multi-copy technology, they have been widely used in the distributed storage field.

[0004] Erasure codes, such as RS(n,k), first split the original data into k pieces, and use the coding matrix to obtain the corresponding (nk) check blocks by performing linear combination calculations on the data blocks. According to different location strategies, the data blocks and check blocks are stored on n different nodes. The additional storage overhead is When no more than nk nodes fail, the original data can be restored using the remaining normal nodes.

[0005] While using erasure coding to improve the reliability of distributed storage systems also presents some challenges. When writing data, data blocks must be encoded and calculated to generate parity blocks. These blocks and parity blocks must then be stored on different nodes. This not only incurs computational overhead but also increases the network overhead associated with node selection. When updating data, all relevant data blocks and parity blocks must be read before the old data is updated, increasing write latency. When reading data, frequently accessed nodes become hot data, leading to hotspots and degrading system performance.

[0006] Existing data block storage location selection strategies are generally based on location addressing. When reading data, the client must send a request to a central node, which then finds the target data at the corresponding storage location. This addressing method makes the reading process heavily dependent on the central node and a specific network topology. It is not fully decentralized and is prone to single points of failure. Summary of the Invention

[0007] The purpose of the embodiments of the present application is to provide a method and system for distributed data storage based on erasure codes. The embodiments of the present application provide an improved content hash value-based addressing method, combined with a dynamic addressing method based on comprehensive indicator evaluation. After the data is initially stored, the embodiments of the present application dynamically adjust the storage location with load balancing as the goal, thereby improving the read and write efficiency and performance of the system. At the same time, the storage and update methods provided by the present invention implement decentralized node storage, and each node maintains an independent index table, which helps prevent system operation abnormalities caused by single point failures.

[0008] In a first aspect, an embodiment of the present application provides a method for distributed data storage based on erasure codes, the method comprising: obtaining n data blocks and check blocks to be stored, and obtaining a dynamic parameter γ, wherein n is an integer greater than 1, γ is a number greater than zero and less than 1, and the result of γ·n is an integer; selecting γ·n nodes as first-type storage nodes according to the node identifier of each node in the distributed system; selecting (1-γ)·n nodes as second-type storage nodes from at least part of the remaining nodes included in the distributed system except the first-type storage nodes according to the node performance; and using each of the first-type storage nodes and the second-type storage nodes to store a check block or a data block respectively.

[0009] Some embodiments of the present application adopt a strategy that combines node identification and node performance selection to select storage nodes when storing n data blocks for the first time, thereby improving the speed of subsequent reading, updating and processing of the stored data, thereby improving the access efficiency of the distributed storage system.

[0010] In some embodiments, the selecting of γ·n nodes as the first type of storage nodes based on the node identifier of each node in the distributed system includes: sorting each node in the distributed system on a ring according to the common prefix length of the node name; and searching for γ·n nodes with a common prefix on the ring as the first type of storage nodes based on the file identifier of the data to be stored.

[0011] Some embodiments of the present application arrange corresponding nodes according to node names and then select storage nodes for data blocks and check blocks to be stored according to the arrangement results, which can improve the subsequent access and reading speed of these data.

[0012] In some embodiments, the searching for γ·n nodes with a common prefix on the ring as the first type of storage node based on the file identifier of the file to be stored includes: if the node on the ring does not have a common prefix with the file identifier, randomly selecting a direction to continue matching nodes until the node with the first γ·n maximum prefix length is found to obtain the first type of storage node; if the node on the ring has a common prefix length with the file identifier, continuing matching nodes in a clockwise direction until the node with the first γ·n maximum prefix length is found to obtain the first type of storage node.

[0013] Some embodiments of the present application determine the distribution direction of the first type of storage nodes on the ring based on the name of the file to be stored, which can improve the subsequent reading and processing speed of these stored data.

[0014] In some embodiments, the method of selecting (1-γ)·n nodes as the second type of storage nodes from at least a portion of the remaining nodes excluding the first type of storage nodes in the distributed system according to node performance includes: dividing the remaining nodes excluding the first type of storage nodes in the distributed system into high-performance nodes, medium-performance nodes and low-performance nodes according to performance; and selecting (1-γ)·n nodes from the high-performance nodes as the second type of storage nodes.

[0015] Some embodiments of the present application also select storage nodes for part of the data in the n data blocks and check blocks stored for the first time based on the dynamic coefficient and node performance, thereby improving the storage speed of these data blocks and check blocks.

[0016] In some embodiments, the method further includes: calculating hash values of the data block and the check block; storing the hash values of the corresponding data block or check block in an index table on each node in the first type of storage node and the second type of storage node; wherein the hash value is used to locate the data block or check block when updating or repairing data.

[0017] In some embodiments of the present application, it is also necessary to store hash values of the stored data blocks and check blocks in each storage node to improve the speed of subsequent reading and locating these data blocks.

[0018] In some embodiments, the method further includes: in response to the operation of updating the data block, dynamically adjusting the node used to store the data to be updated according to a comprehensive evaluation index, wherein the comprehensive evaluation index is related to the node performance difference and the data block heat attribute, the node performance difference is related to the load index and usage index of the corresponding node, and the data block heat index is related to the priority index, access count index and importance index of the data to be updated; updating the description information of each node after dynamic adjustment, wherein the description information includes: a file identifier, a data block or check block hash value, an identifier for distinguishing between data blocks and check blocks, a node identifier or the storage location of the data block and the check block, the data block or check block hash value is used to identify the data block or check block stored by the corresponding node, the identifier for distinguishing between data blocks or check blocks is used to indicate whether the data stored by the corresponding node belongs to a data block or a check block, the node identifier is a node domain name or a set unique node name, and the storage location of the data block or the check block is used to record the specific location of the data stored on the corresponding node.

[0019] Some embodiments of the present application also use comprehensive evaluation indicators to dynamically adjust the nodes storing data blocks when updating data blocks, and use the node description information to dynamically adjust the corresponding information of the data stored in the adjusted node. In this way, the storage node can be reselected for the corresponding data block when the data block is updated, and the adjustment status of the data stored on the relevant node can be obtained in real time according to the description information, thereby improving the efficiency of the distributed storage system in subsequent storage, reading and updating of related data.

[0020] In some embodiments, the method further includes: storing an identifier for distinguishing whether a stored block belongs to a data block or a check block in an index table of each node in the first type of storage node and the second type of storage node; upon receiving a file read request from a client, adopting the following strategy to read the required data block and / or check block according to the read request attribute and the identifier for distinguishing between data blocks and check blocks: if it is confirmed that none of the data blocks to be read have failed, reading the corresponding data block through the identifier; if it is confirmed that the data block to be read has failed, reading the data block and the check block at the same time through the identifier, and restoring the failed data block based on the read data; if it is confirmed that the data needs to be checked for integrity, reading all data blocks and check blocks used for integrity checking through the identifier; If it is confirmed that no integrity check is required for the data, and when no fault occurs in the data blocks to be read, multiple data blocks are read through the identifier, and the multiple data blocks are connected into target data and then output to the client; wherein, the reading of all data blocks and check blocks for integrity check through the identifier includes: upon receiving a file read request from the client, finding the node with the largest prefix length as the read file name in the ring, and searching counterclockwise for the next (1-γ)·n nodes, and reading the data block or check block required for recovery in each node according to the hash value included in the index table; sending a query request to the index table of each node in the corresponding high-performance list, locking the query range according to the file unique identifier, and reading the required data block and check block in the queried node position.

[0021] Some embodiments of the present application determine whether to read only the data block or to read both the check block and the data block at the same time by storing an identifier for distinguishing whether the stored block belongs to a data block or a check block and the purpose of this reading, thereby improving the efficiency of data reading and increasing the speed of data reading.

[0022] In a second aspect, some embodiments of the present application provide a distributed data storage device based on erasure codes, the device comprising: a data acquisition module, configured to acquire n data blocks and check blocks to be stored, and acquire a dynamic parameter γ, wherein n is an integer greater than 1, γ is a number greater than zero and less than 1, and the result of γ·n is an integer; a first-class storage node selection module, configured to select γ·n nodes as first-class storage nodes based on the node identifiers of each node in the distributed system; a second-class storage node selection module, configured to select (1-γ)·n nodes as second-class storage nodes from at least part of the remaining nodes included in the distributed system except the first-class storage nodes based on node performance; a block data storage module, configured to use each of the first-class storage nodes and the second-class storage nodes to store a check block or a data block respectively.

[0023] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any one of the embodiments included in the first aspect.

[0024] In a fourth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the embodiments of the first aspect can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A storage architecture diagram of a distributed system provided in an embodiment of the present application;

[0027] Figure 2 One of the flowcharts of the method for distributed data storage based on erasure coding provided in an embodiment of the present application;

[0028] Figure 3 Flowchart 2 of the method for distributed data storage based on erasure coding provided in an embodiment of the present application;

[0029] Figure 4 This is a block diagram of a distributed data storage device based on erasure coding provided in an embodiment of the present application;

[0030] Figure 5 The second block diagram of the distributed data storage device based on erasure coding provided in an embodiment of the present application;

[0031] Figure 6 The third block diagram of the distributed data storage device based on erasure coding provided in an embodiment of the present application;

[0032] Figure 7 This is a fourth block diagram of a distributed data storage device based on erasure codes provided in an embodiment of the present application;

[0033] Figure 8 A schematic diagram of a method for obtaining a comprehensive evaluation value provided in an embodiment of the present application;

[0034] Figure 9The fifth block diagram of the distributed data storage device based on erasure coding provided in an embodiment of the present application;

[0035] Figure 10 A schematic diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0037] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0038] As described in the background section, distributed storage systems store data in a distributed manner across multiple independent devices. Traditional network storage systems mostly use centralized storage servers to store all data. These servers become a bottleneck for system performance and a focal point for reliability and security, failing to meet the needs of large-scale storage applications. Distributed network storage systems employ a scalable system architecture, utilizing multiple storage servers to share the storage load and location servers to locate stored information. This improves system reliability, availability, and access efficiency to a certain extent. However, as the number of servers increases, a critical issue arises: how to load-balance the data storage node locations to improve system read and write efficiency and performance, while also preventing system operational anomalies caused by single point failures.

[0039] Traditional data centers or storage clusters usually adopt a multi-copy data redundancy strategy, which has high access performance, but also has the disadvantages of high storage redundancy and storage cost. To solve the above problems, more and more commercial storage systems are introducing erasure code (including traditional erasure code, storage regeneration code / network coding) data protection strategies to achieve the same reliability with less data redundancy. The erasure code algorithm encodes multiple data blocks and generates multiple check blocks, where data blocks and check blocks are collectively referred to as coding blocks. The embodiments of the present application are based on the erasure code data storage strategy.

[0040] As described in the background technology section, the data block storage location selection strategy in the existing erasure code distributed data storage system is generally based on location addressing. When reading data, the client needs to send a request to the central node and then find the target data from the corresponding storage location. This addressing method makes the reading process heavily dependent on the central node and the specific network topology. It is not completely decentralized and is prone to single point failure problems. In the data storage process based on address addressing, there is randomness between data and storage nodes. It is difficult to predict the distribution of hot data and cold data, which can easily lead to hot spot problems and unbalanced system load. When considering the storage node of the data block, it is also necessary to start from the data and the node itself, fully consider the node indicators and the data indicators, and balance the load between the nodes.

[0041] At least in order to solve these technical problems, the embodiments of the present application provide a method for distributed data storage based on erasure codes. When selecting storage nodes, the method needs to determine the use of a digital identification selection method (i.e., a method for selecting the first type of storage node) and a performance selection method (i.e., a method for selecting the second type of storage node) to allocate storage nodes for the data blocks and check blocks to be stored in combination with the dynamic parameter γ, wherein the method of selecting the storage location for the data block or check block according to the unique identifier of the node is a digital identification selection method, and the method of selecting the storage location for the data block and check block according to the performance of the node is recorded as a performance selection method. Some embodiments of the present application also complete data reading based on the content hash value addressing method, and dynamically adjust the storage nodes of the data block or check block in combination with the comprehensive indicator evaluation method. That is, in some embodiments of the present application, after the data is stored for the first time (the first storage is based on the selected first type of storage node and the second type of storage node), the storage location is dynamically adjusted with load balancing as the goal to improve the read and write efficiency and performance of the system. At the same time, the storage and update methods provided in some embodiments of the present application realize decentralized node storage, and an index table is maintained separately in each node. The columns in the index table include: file identifier, data block or check block hash value, identifier of the data block or check block, node identifier, data block or check block storage location. The hash value in the index table can improve data query efficiency and help prevent system operation abnormalities caused by single point failures.

[0042] Please see Figure 1 , Figure 1 The distributed storage system provided in some embodiments of the present application includes a data center, which includes multiple clusters, for example, Figure 1 There are multiple clusters (1, 2, 3, 4, etc.), and each cluster includes multiple racks, for example, Figure 1There are multiple racks (001, 002, 003, 004, 005), and each rack is provided with multiple nodes, which can be selected as storage nodes, for example, Figure 1 It includes multiple storage nodes (100, 101, 102, 103, 104, 105).

[0043] It is understandable that the distributed data storage method based on erasure code provided in this application can be applied to Figure 1 The distributed storage system.

[0044] Figure 1 The example data center datacenter.example.com in the distributed storage system is shown in FIG. , which includes five different clusters, each of which is named, for example, cluster1.datacenter.example.com, cluster2.datacenter.example.com,

[0045] cluster3.datacenter.example.com, cluster4.datacenter.example.com,

[0046] cluster5.datacenter.example.com.

[0047] Take cluster 1 (cluster1.datacenter.example.com) as an example, which includes 5 racks (001, 002, 003, 004, 005), each named, for example:

[0048] rack1.cluster1.datacenter.example.com,

[0049] rack2.cluster1.datacenter.example.com,

[0050] rack3.cluster1.datacenter.example.com,

[0051] rack4.cluster1.datacenter.example.com, rack5.cluster1.datacenter.example.com.

[0052] Taking rack 001 as an example, it includes three storage nodes (100, 101, 102). The storage nodes can be further represented as follows:

[0053] node1.rack1.cluster1.datacenter.example.com, node2.rack1.cluster1.datacenter.example.com, node3.rack1.cluster1.datacenter.example.com.

[0054] Here is an example:

[0055] During the data write phase, for the file exampleDoc to be stored from cluster1.datacenter.example.com, its file identifier is first recorded as com.example.datacenter.cluster1.exampleDoc. Based on the digital identification selection method, γ·n storage nodes with the minimum number of routes are selected as the first-category storage nodes (assuming that these first-category storage nodes are all located in cluster 1) to store the encoded γ·n data blocks and check blocks. Secondly, the performance of all nodes in the storage system is calculated and divided into three categories: low, medium, and high performance. Then, based on the performance selection method, (1-γ)·n nodes in clusters other than cluster 1 in the high-performance category are selected as the second-category storage nodes to store the remaining (1-γ)·n encoded data blocks and check blocks.

[0056] In the subsequent update phase, the load index, usage index, data priority index, access index and importance index of the storage system node are comprehensively considered to calculate the difference between the node load score and the load mean (D 100 ,D 101 ,D 102 ,D 103 ,D 104 ,D 105 ,…), and readjust the node storage position accordingly.

[0057] like Figure 2 As shown, an embodiment of the present application provides a method for distributed data storage based on erasure codes, the method comprising:

[0058] S110 , obtaining n data blocks and check blocks to be stored, and obtaining a dynamic parameter γ, where n is an integer greater than 1, γ is a number greater than zero and less than 1, and the result of γ·n is an integer.

[0059] S120 : Select γ·n nodes as first-type storage nodes according to the node identifier of each node in the distributed system.

[0060] S130 : Select (1-γ)·n nodes as second-category storage nodes from at least a portion of the remaining nodes of the distributed system except the first-category storage nodes according to node performance.

[0061] S140: Use each of the first-type storage nodes and the second-type storage nodes to store a check block or a data block respectively.

[0062] The following is an example of how to implement the above steps.

[0063] In some embodiments of the present application, the process of selecting γ·n nodes as the first type of storage nodes based on the node identifier of each node in the distributed system as described in S120 exemplarily includes: sorting each node in the distributed system on a ring according to the common prefix length of the node name; and searching for γ·n nodes with a common prefix on the ring as the first type of storage nodes based on the file identifier of the data to be stored.

[0064] It should be noted that the node name, for example, if a node is located at the dataLocation position in the domain name www.example.com, is flipped to the com-example-dataLocation identification string. The com identification is stored on ring r0, the example identification is stored on ring r1, and the dataLocation identification is stored on r2. Therefore, minimum routing locality is provided for nodes with the same prefix name. For storage files in the same domain, the node with the smallest number of routes is selected from all possible nodes as the storage node to achieve local load balancing and improve data read and write rates.

[0065] In some embodiments of the present application, the file identifier includes its subdomain name, host name, and file name. For example, for a file to be stored named storageDataHtml, which is located in the domain name myhost.dataLocation.example.com, the file identifier is recorded as com.example.dataLocation.myhost.storageDataHtml.

[0066] The structure of the ring in some embodiments of the present application is as follows: the ring includes multiple layers of sub-rings (r0, r1, r2, ... rz), and the sub-rings from r0 to rz store subdomain names, host names, and file names respectively. Each sub-ring uses a bidirectional linked list data structure to meet the requirements of clockwise and counterclockwise queries; the strings in each sub-ring are arranged clockwise based on the number of identical prefix characters. Pointers are used to connect the upper and lower sub-rings to indicate the next string that the stored prefix in the upper sub-ring may have. For example, for the prefixes exam1, exam2, and exam3, which have suffixes aaa, bbb, aab, and ccc, respectively, the ring is constructed as follows:

[0067] r0 stores (exam1, exam2, exam3) in clockwise order; r1 includes three subrings: (aaa, aab, bbb, ccc), (aaa, aab, bbb, ccc), and (aaa, aab, bbb, ccc), which correspond to the three prefix strings in r0 respectively. The strings in each subring are sorted according to the same number of prefix characters.

[0068] For example, in some embodiments of the present application, S120 includes a process of searching for γ·n nodes with a common prefix on the ring as the first type of storage nodes based on the file identifier of the file to be stored, which exemplarily includes: if the node on the ring does not have a common prefix with the file identifier, then randomly selecting a direction to continue matching nodes until the node with the first γ·n maximum prefix length is found to obtain the first type of storage node; if the node on the ring has a common prefix length with the file identifier, then continuing to match nodes in a clockwise direction until the node with the first γ·n maximum prefix length is found to obtain the first type of storage node.

[0069] In some embodiments of the present application, the process of selecting (1-γ)·n nodes as the second type of storage nodes from at least part of the remaining nodes of the distributed system except the first type of storage nodes according to node performance in S130 exemplarily includes: dividing the remaining nodes of the distributed system except the first type of storage nodes into high-performance nodes, medium-performance nodes and low-performance nodes according to performance; and selecting (1-γ)·n nodes from the high-performance nodes as the second type of storage nodes.

[0070] In some embodiments of the present application, each node maintains a separate index table. The columns in the index table include: file identifier, data block or check block hash value, identifier for marking data blocks or check blocks (or identifiers used to distinguish data blocks from check blocks), node identifier, and data block or check block storage location. The information stored in these columns (i.e., descriptions of the corresponding nodes) can improve data query efficiency in distributed storage systems and help prevent system operational anomalies caused by single points of failure. The file identifier is the unique identifier of the file stored on the node, i.e., the node unique identifier plus the concatenation of the file name. The data block or check block hash value is used to identify which data block or check block of the file is stored on the node, primarily used to locate the data block or check block when updating or repairing data. The identifier for the marker block or check block indicates whether the file stored at the location is the original data block or the check block, explicitly distinguishing between data blocks and check blocks. This facilitates development and testing by coders and provides methods for directly reading data blocks and recovering data using check block decoding during data updates or repairs. The node identifier uses the node's domain name or a uniquely configured node name as a unique identifier to identify a node in the storage network. The data block or check block storage location is the specific storage location of the data block or check block of the storage file, including the file path or memory address or starting offset, which is used to locate the specific storage location of the data block or check block in the node.

[0071] In some embodiments of the present application, the method also includes: calculating hash values of the data block and the check block; storing the hash value of the corresponding data block or check block in an index table on each node in the first type of storage node and the second type of storage node; wherein the hash value is used to locate the data block or check block when updating or repairing data.

[0072] In some embodiments of the present application, the method further includes: in response to an operation of updating a data block (i.e., a data block stored on a node of a distributed storage system), dynamically adjusting the node used to store the data to be updated (i.e., the data block to be updated) according to a comprehensive evaluation index, wherein the comprehensive evaluation index is related to the node performance difference and the data block heat attribute, the node performance difference is related to the load index and usage index of the corresponding node, and the data block heat index is related to the priority index, access count index and importance index of the data to be updated; updating the description information of each node in the dynamically adjusted node, and these description information exemplarily include: a file identifier, a data block or a check block hash value, an identifier for distinguishing between a data block and a check block, a node identifier or the storage location of the data block and the check block; wherein the file identifier is a unique identifier of the file stored at the node; the data block or check block hash value: used to identify Identify which data block or check block of the file is stored in the node, that is, the hash value of the data block or check block is used to identify the data block or check block stored in the corresponding node; the identifier for distinguishing between the data block or check block is used to indicate whether the stored block belongs to the data block or the check block, and is used to explicitly indicate the difference between the data block and the check block, that is, the identifier for distinguishing between the data block or the check block is used to indicate whether the data stored in the corresponding node belongs to the data block or the check block; the node identifier uses the node's domain name or the set unique node name as the identifier that uniquely represents the corresponding node in the storage network, that is, the node identifier is the node domain name or the set unique node name; the storage location of the data block or the check block is used to record the specific storage location of the data block or check block of the stored file, including the file path or memory address or the starting offset, that is, the storage location of the data block or the check block is used to record the specific location of the data stored on the corresponding node.

[0073] In some embodiments of the present application, the method further includes: storing an identifier for distinguishing whether the stored block belongs to a data block or a check block in the index table of each node in the first type of storage node and the second type of storage node; upon receiving a file read request from the client, adopting the following strategy to read the required data block and / or check block according to the read request attribute and the identifier for distinguishing between data blocks and check blocks: if it is confirmed that none of the data blocks to be read have failed, then only the data block is read through the identifier, that is, if it is confirmed that none of the data blocks to be read have failed, then the corresponding data block is read through the identifier; if it is confirmed that the data block to be read has failed, then k data blocks and check blocks are read simultaneously through the identifier, and the data is restored using the data decoding module, that is, if it is confirmed that the data block to be read has failed, then the data block and check block are read simultaneously through the identifier, and the faulty data block is restored based on the read data; if it is confirmed that the read data needs to be checked for integrity, then n data blocks and check blocks are read through the identifier, that is, if it is confirmed that the data needs to be checked for integrity , then all data blocks and check blocks for integrity verification are read through the identifier; if it is confirmed that the read data does not need to be integrity verified, and when none of the data blocks to be read has any faults, only the data blocks are read through the identifier, the data blocks are connected into target data, and then output to the client, that is, if it is confirmed that the data does not need to be integrity verified, and when none of the data blocks to be read has any faults, multiple data blocks are read through the identifier, the multiple data blocks are connected into target data, and then output to the client; wherein, if it is necessary to read n data blocks and check blocks, the method further includes: upon receiving a file read request from the client, finding a node with a maximum prefix length with the read file name in the ring, and searching for the next (1-γ)·n nodes counterclockwise, and reading the data blocks or check blocks required to restore the file in each node according to the hash value included in the index table; sending a query request to the index table of each node in the corresponding high-performance list, locking the query range according to the file unique identifier, and reading the required data blocks and check blocks in the queried node position.

[0074] The following combination Figure 3 The distributed data storage method based on erasure codes provided by some embodiments of the present application is exemplified.

[0075] The embodiment of the present application provides a distributed data storage method based on erasure codes. When storing data for the first time, the specific storage location is selected based on the CPU performance and network performance of the node, as well as the node's unique identifier. The method also includes using a dynamic parameter γ to select the allocation ratio of the storage location selection method based on the node's unique identifier (digital identifier selection method) and the storage location selection method based on the node's performance (performance selection method). In the subsequent update phase, the node's load index, utilization index, and data priority index, access count index, and importance index are comprehensively considered to propose a load-balanced node selection strategy (i.e., adjusting the storage node of the data block to be updated based on the comprehensive evaluation index) to improve the read and write performance and update efficiency of the distributed storage system, reduce network overhead, and prevent hot spots.

[0076] The present invention provides a method for distributed data storage of erasure codes. Figure 3 As shown, the method includes:

[0077] The first step is to obtain the complete input data and perform sharding processing to obtain data blocks of uniform size.

[0078] In the data writing phase: In the distributed storage system, the data block size is set to s, the number of data blocks is k, the number of check blocks is m, the sum of the number of check blocks and data blocks is n, the erasure code is selected as RS(n,k) and the finite field is selected as GF(2 w ).

[0079] In the second step, the coding matrix is used to calculate the data block to obtain the check block and synthesize multiple stripes.

[0080] Generate and save the encoding matrix to calculate the check block when writing the data block. Slice the data block according to the preset block size s and number, obtaining k sharded data blocks. If the complete data size N cannot be evenly divided into M groups, each with k data blocks of size s, the last data block that does not meet the size s is padded with zeros to ensure normal encoding.

[0081] Using the coding matrix, linear combination calculations are performed on the data blocks that meet the requirements to obtain the corresponding (nk) check blocks. M groups of data (each group contains k data blocks and m check blocks) are stored in different nodes according to the location selection rules.

[0082] The third step is to calculate the content hash value of the data block and the check block.

[0083] In the fourth step, if the data is written for the first time, the first write location selection strategy is adopted to select a storage node from two methods: a digital identification selection method and a performance selection method.

[0084] Selecting a storage location based on a digital identification selection method includes: (1) setting a dynamic parameter γ, which can be used to determine the number of storage blocks allocated for the digital identification selection method and the performance selection method; (2) selecting a storage location based on a node's unique identifier, denoted as the digital identification selection method; and (3) selecting a storage location method based on the node's performance, denoted as the performance selection method. For example, in some embodiments of the present application, the dynamic parameter γ is used to determine whether to use the digital identification selection method to store γ·n blocks or to use the performance selection method to store (1-γ)·n blocks.

[0085] For example, in some embodiments of the present application, the digital identification selection method includes:

[0086] On the ring, nodes are sorted according to the length of the common prefix with other node names, so that the common prefix length between adjacent nodes in the clockwise direction is as large as possible.

[0087] b. Search for a node on the ring with a common prefix based on the file identifier. If the current node on the ring does not share a common prefix with the file identifier, randomly select a direction to continue matching nodes. If the current node and the file identifier share a common prefix length, continue matching nodes in a clockwise direction until a node with the largest prefix length (γ·n) is found. A data block or check block is stored on each node.

[0088] For example, in some embodiments of the present application, the performance selection method includes: evaluating the CPU performance and network performance of the node.

[0089] Furthermore, in some embodiments of the present application, evaluating a node's CPU performance and network performance includes evaluating the CPU performance and network performance of a node that has not yet stored data when data is first written. CPU performance includes the main frequency, number of cores, and number of threads, and network performance includes network throughput and network latency. CPU performance is one of the important factors affecting node performance, and the CPU performance score is calculated as follows:

[0090] CPU=CPU 主频 *CPU 核心 *CPU 线程

[0091] Among them, CPU represents the CPU performance score of the node. 主频 Indicates the oscillation speed of the digital pulse signal in the node's CPU, which affects the CPU processing speed to a certain extent; CPU 核心 Indicates the number of CPU cores of the node. The more CPU cores, the stronger the processing capability. 线程 Indicates the number of CPU threads on a node, which reflects the CPU's parallel processing capability.

[0092] Network performance is also an important indicator for evaluating node performance. The network performance score is calculated as follows:

[0093]

[0094] Among them, mem represents the network performance score of the node, MEM 吞吐量 Indicates the network throughput of the node, which can reflect the data transmission capacity of the node; MEM 延迟 Indicates the network latency of a node, reflecting the speed and efficiency of network data transmission.

[0095] The weight parameters of the node's CPU performance index and network performance are set to 0.5 and 0.5 respectively. For node n, the weighted calculation formula is:

[0096] w n =0.5*cpu+0.5*mem

[0097] Some embodiments of the present application group nodes by performance level based on the calculated node weights and store the corresponding nodes in low-performance, medium-performance, and high-performance lists, respectively. Combining the file identifier and the node unique identifier, embodiments of the present application select a node storage location in the high-performance list.

[0098] The node storage location is selected in the high-performance list. The node selection range is a random (1-γ)·n element in the high-performance node, and the γ·n nodes selected by the digital identification selection method are excluded. It can be expressed by the following formula:

[0099]

[0100] Among them, Deque high represents the high-performance list, and B represents the γ·n nodes selected in the digital identification selection method.

[0101] The above process is as follows Figure 1 The distributed architecture shown is described as follows:

[0102] The distributed system consists of H data centers, then S={S1,S2,…S H}, each data center contains multiple clusters. The total number of nodes in the data center is J, then the summary point NODE = {node1, node2, ...node J For each summary point NODE on the rack, calculate the node weighted score weight = {w1,w2,…w J}, store the nodes into three weighted queues: low, medium, and high according to their scores:

[0103] Dequelow ={node1, node3, ...node J-1}

[0104] Deque medium ={node2, node4, ...node J-2}

[0105] Deque high ={node5, node6, ...node J}

[0106] Among them, Deque low Represents a low-performance list, Deque medium Represents a medium-performance list, Deque high Represents a high-performance list.

[0107] It should be noted that the digital identification selection method (i.e., the method for selecting the first type of storage node) in some embodiments of the present application takes locality into consideration and selects the node with the smallest number of routes from all possible nodes as the storage node, but does not consider the performance indicators of the node. In addition, the fault tolerance of the distributed system may be greatly reduced due to the storage of data blocks or check blocks in nodes under one domain name, leading to hot spot problems. Some embodiments of the present application also write data directly into the Deque high The technical solution of using the second type of storage nodes to store the remaining data blocks or check blocks in the random nodes in the list and excluding the nodes selected in the digital identification selection method not only reduces the data reading and writing delay to a certain extent, but also evenly distributes the data on the nodes, ensuring the load balancing and fault tolerance of the distributed system.

[0108] Step 5: If the data is not written for the first time, a comprehensive evaluation method is used to dynamically adjust the storage node.

[0109] Furthermore, the storing of M groups of data (each group containing k data blocks and m check blocks) in different nodes according to the site selection rules includes: evaluating the CPU performance and network performance of the node, and calculating content hash values of the data blocks and check blocks in the M groups of data based on their contents; comprehensively evaluating the load index, utilization index, and priority index, access count index, and importance index of the node, and dynamically optimizing the storage location selection.

[0110] Furthermore, in some embodiments of the present application, the subsequent dynamic adjustment of the storage node position based on the comprehensive evaluation index includes: comprehensively evaluating the node's load index, utilization index, and data priority index, access count index, and importance index, calculating the performance difference between nodes and the heat level of the data block, so as to optimize the load balance in the system, dynamically adjust the node storage, and ensure that the reading and writing frequency of the data is relatively balanced.

[0111] In some embodiments of the present application, the dynamic adjustment of node storage uses a comprehensive evaluation method to obtain the current load status of the node. That is, in the process of calculating the comprehensive score, an algorithm such as the entropy weight method, the TOPSIS method, the grey correlation analysis method, the fuzzy comprehensive evaluation method, etc. can be selected to obtain the comprehensive load score RE. Then, dynamic adjustment is performed based on the load of different nodes to balance their load.

[0112] The process of obtaining the comprehensive load score includes:

[0113] For the node load index (A 11 ), utilization index (A 12 ) and data priority indicators (A 21 ), visit count indicator (A 22 ) and importance index (A 23 ), which can be divided into two parts: nodes and data, among which the load index (A 11 ), utilization index (A 12 ), visit count indicator (A 22 ) is an objective indicator, and the priority indicator (A 21 ), importance index (A 23 ) is a subjective value indicator. Set the weight vector to: W A =[0.25,0.25,0.16,0.18,0.16].

[0114] In some embodiments of the present application, for subjective indicators: for priority indicators (A 21 ), the assignment range is [0, 100], of which the high priority assignment range is (50, 100], the medium priority assignment range is (20, 50], and the low priority assignment range is [0, 20]; for the importance index (A 23 ), the value range is [0, 100], of which the core data value range is (50, 100], the important data value range is (20, 50], and the general data value range is [0, 20]. For objective indicators: For load indicators (A 11 ), utilization index (A 12 ), visit count indicator (A 22) The normalization formula is used to scale the original values to the range [0, 100] to facilitate subsequent comprehensive evaluation.

[0115] For each storage node, the evaluation grade membership function is used to calculate the evaluation grade membership corresponding to the five indicators. For each node, the membership of each indicator is calculated for each evaluation grade to form a single indicator evaluation matrix. For the five indicators, a comprehensive indicator evaluation matrix R = [R 1,1 ,R 1,2 ,R 2,1 ,R 2,2 ,R 2,3 The evaluation matrix E includes four elements: excellent (e1), good (e2), general (e3), and poor (e4). The excellent grade is assigned 90 points, the good grade is assigned 80 points, the general grade is assigned 70 points, and the poor grade is assigned 60 points. The comprehensive evaluation matrix B is calculated based on the weight vector W and the single indicator evaluation matrix R. The formula is as follows:

[0116] B=W×R=(b1,b2,b3,b4,b5)

[0117] Here, "X" indicates the corresponding operation of W and R, not the multiplication of two matrices. ik It represents the comprehensive membership of the i-th node at the k-th evaluation level, as follows:

[0118]

[0119] Among them, r ijk It represents the probability that the i-th node belongs to the k-th evaluation level on the j-th indicator.

[0120] The node load score RE is calculated based on the evaluation matrix B and the evaluation level matrix E i , where RE i represents the load score of the i-th node:

[0121]

[0122] The process of dynamically adjusting the load to balance the load includes:

[0123] For nodes a, b, c, ..., d, e, the load scores (REa, REb, REc, ..., REd, REe) are obtained according to the comprehensive evaluation method above, and the average load of the nodes is calculated as follows:

[0124]

[0125] make It represents the difference between the load score of node n and the average load of the node. If D = 0, it means that the load of the node is relatively stable and meets the requirements of data storage. If D > 0, it means that the load score of the node is high and the storage data needs to be adjusted so that the D value approaches 0. Use the function f(RE a ,RE b ,...,RE e ) indicates that load balancing among nodes is achieved by re-adjusting the node storage locations.

[0126] During the dynamic adjustment process, it is necessary to update the node load average in real time. and the load difference D to determine which node to adjust the data storage location.

[0127] Step 6: Update the file identifier, data block or check block hash value, identifier of the data block and check block, node unique identifier and data block or check block storage location in the node.

[0128] Furthermore, the storage of data according to hash values described in some embodiments of the present application includes: when a data block is created or modified, calculating the content hash value of the data block as a unique identifier, writing the data block content, and updating the file identifier, data block or check block hash value, identifier marking the data block or check block, node unique identifier, and data block or check block storage location in its index table.

[0129] Furthermore, in some embodiments of the present application, the process of calculating the content hash value of a data block includes:

[0130] Initialize eight buffers {h0,h1,h2,…h7} and fill the data block D with zero values block ′ and check block V block ' is divided into Z blocks, and R rounds of transformation are performed on each block to update the buffer value.

[0131] Concatenate the buffer values after the last round of transformation to form the final hash value

[0132] H(D block )=h 0,z ||h 1,Z ||…h 7,Z || and H(V block )=h 0,Z ||h 1,Z ||…h 7,z ||.

[0133] The data writing in some embodiments of the present application involves initial writing and subsequent update operations. If it is the initial writing, the verification of new and old data is not involved. If it is a subsequent update operation, the new and old data need to be verified in order to obtain the incremental part and perform incremental updates on the data block.

[0134] In some embodiments of the present application, when data is written, if it is the first time to write, the storage node is directly selected according to the digital identification selection method and the performance selection method, and the file identifier, data block or check block hash value, identifier of the data block or check block, node unique identifier, and data block or check block storage location in the node are updated.

[0135] In some embodiments of the present application, when writing data, if it is a subsequent update operation, it is necessary to query the storage node information through the index table, query the storage data block and check block in the corresponding node, use the updated data block to replace the old data block, use the coding matrix to calculate the check block of the new data block, and replace the old check block to achieve incremental data update.

[0136] In some embodiments of the present application, during the data reading phase:

[0137] According to the encoding matrix saved in the data writing stage, the decoding matrix is generated by using the matrix inverse operation to decode the data blocks and check blocks in the node and restore them to the original data.

[0138] After receiving a file read request from the client, the node with the largest prefix length of the read file name is found in the ring, and the next (1-γ)·n nodes are searched counterclockwise. In each node, the data block or check block required to recover the file is read according to the index table; a query request is sent to the index table of each node in the high-performance list, the query range is locked according to the file unique identifier, and the required data block and check block are read from the queried node position to improve reading efficiency and ensure data consistency and integrity.

[0139] During the reading phase, the hash value of the required data is directly matched in the index table of each node, rather than querying the data node location information through the central node. Data is read at the data node, which simplifies the query steps, improves the query hit rate, and prevents single point failure problems in the central node.

[0140] Please refer to Figure 4 , Figure 4 The distributed data storage device based on erasure code provided by the embodiment of the present application is shown. It should be understood that the device is similar to the above Figure 2The method embodiment corresponds to the method embodiment and can execute each step involved in the above method embodiment. The specific functions of the device can be found in the description above. To avoid repetition, the detailed description is omitted here. The device includes at least one software functional module that can be stored in the memory in the form of software or firmware or fixed in the operating system of the device. The distributed data storage device based on erasure code includes: a data acquisition module 110, a first-type storage node selection module 120, a second-type storage node selection module 130, and a block data storage module 140.

[0141] The data acquisition module is configured to acquire n data blocks and check blocks to be stored, and acquire a dynamic parameter γ, wherein n is an integer greater than 1, γ is a number greater than zero and less than 1, and the result of γ·n is an integer.

[0142] The first type storage node selection module is configured to select γ·n nodes as the first type storage nodes according to the node identifiers of the nodes in the distributed system.

[0143] The second type storage node selection module is configured to select (1-γ)·n nodes as second type storage nodes from at least part of the remaining nodes included in the distributed system except the first type storage nodes according to node performance.

[0144] The block data storage module is configured to use each of the first-type storage nodes and the second-type storage nodes to store a check block or a data block respectively.

[0145] like Figure 5 As shown, in some embodiments of the present application, the device further includes: a data writing module and a data reading module.

[0146] The data writing module is mainly responsible for distributing and storing the data sent by the client to multiple nodes. The data writing module includes: Figure 6 The following are the components of the system: a write request receiving module, a data slicing module, a data encoding module, a storage location selection module, a storage location dynamic adjustment module, and a data update module. The write request receiving module is used to receive data write operations sent by the client, first store the data in a buffer to avoid blocking subsequent write operations, and provide complete data for subsequent data slicing.

[0147] The data slicing module is configured to slice the complete data N received by the request receiving module into stripes of size s, with each stripe containing k data blocks. If the complete data N cannot be evenly divided into M groups, each containing k data blocks of size s, the last data block that does not meet the size s is padded with zeros to facilitate proper encoding. The data slicing module is connected to the data encoding module and outputs the sliced stripes to the data encoding module.

[0148] The data encoding module includes: using the encoding matrix generated by the set erasure code RS (n, k), calculating the data block to obtain the check block, and each stripe includes k data blocks and nk check blocks.

[0149] The storage location selection module includes: when performing the first data writing, evaluating the CPU performance index and network performance index of the node, and dividing the node into three levels: low performance, medium performance and high performance.

[0150] The storage location selection module also includes: each node maintains a separate index table, which includes a file identifier, a data block or check block hash value, an identifier marking the data block or check block, a node unique identifier, and a data block or check block storage location.

[0151] When selecting a storage location, the dynamic parameter γ is first determined. Its function is to store γ·n blocks (data blocks or check blocks) in the digital identification selection method and (1-γ)·n blocks (data blocks or check blocks) in the performance selection method.

[0152] Secondly, in the digital identification selection method (used to select the first type of storage nodes), the nodes with the first γ·n maximum prefix lengths are found on the ring, and a data block or a check block is stored on each node.

[0153] Next, in the performance selection method (for selecting the second type of storage nodes), (1-γ)·n nodes are randomly selected from the high-performance node list, and a data block or a check block is stored in each node.

[0154] After selecting a node and storing a data block or a check block in the above-mentioned digital identification selection method and performance selection method, the node index table is updated, including the file identifier, the data block or check block hash value, the identifier of the data block or check block, the node unique identifier, and the data block or check block storage location.

[0155] like Figure 7 or Figure 8As shown, the contents of the storage location dynamic adjustment module described in some embodiments of the present application include: a node index evaluation module, a data index evaluation module, and a comprehensive evaluation module. The node index evaluation module is used to evaluate the load index and usage index of different nodes, weighted to obtain the comprehensive score of the node, and filter out nodes that meet different needs; the data index evaluation module is used to evaluate the comprehensive index of the data, including data priority, data access times and data importance index; the comprehensive evaluation module obtains a comprehensive load-balancing node selection strategy based on the node index evaluation module and the data index evaluation module, and dynamically adjusts the existing storage data. Figure 8 As shown, the load indicators include: CPU load, disk load, memory load, network load, obtain the load indicator scores of these loads, and determine a score based on the weight coefficient; the utilization index exemplarily includes: CPU utilization, disk utilization, memory utilization, network bandwidth utilization, obtain the utilization indicator scores of these utilizations, and then obtain an evaluation value based on the corresponding weight coefficient; the data indicator evaluation module obtains the data indicator score through data priority, data access times and data importance, and then obtains an evaluation value based on the data indicator score and a weight coefficient value; finally, a total evaluation value is obtained through the comprehensive evaluation module, which is used to consider node allocation during dynamic switching. In other words, the process of obtaining the comprehensive score of each node when determining the dynamic adjustment of storage nodes in some embodiments of the present application includes: determining the indicator system, calculating the module scores, determining the weight coefficients and calculating the comprehensive score.

[0156] The relevant indicators are listed in Tables 1, 2 and 3 respectively.

[0157] Table 1 shows the detailed description of the node load indicators

[0158]

[0159] Table 2 shows the detailed description of the node load indicators

[0160]

[0161] Table 3 shows the detailed description of the node load indicators

[0162]

[0163]

[0164] In some embodiments of the present application, when the data block and the check block are written for the first time, the storage location dynamic adjustment module is not involved. The data update module includes: a module for obtaining the data block and the check block content from the data storage module located by the index query module, and incrementally updating them to the new data block content and the new check block content.

[0165] In some embodiments of the present application, the data reading module is mainly responsible for receiving the read request of the client, first searching the required data storage location from the index linked list, then reading the required data from the storage location, and selecting whether to perform integrity check on the data in the optional configuration, and finally sending the data to the client. Figure 9 As shown) exemplarily includes a read request receiving module, a data query module, a data decoding module, an integrity check module and a data sending module.

[0166] The read request receiving module of the embodiment of the present application is used to obtain the data read operation sent by the client. The data query module is used to obtain the data block and check block of the target data from the node. Optionally, if no data blocks have any faults, only the target data block is read; if some data blocks have any faults, both the data block and the check block are read simultaneously, and the data is restored using the data decoding module. The data decoding module is used to restore the data block to the original data using the decoding matrix. The integrity check module is used to optionally, if an integrity check is selected, then when no data blocks have any faults, the target data block and check block are still read, and the integrity of the data block is verified using the check block. If the verification passes, the target data is output to the client normally. If the verification fails, a data corruption warning is output. The integrity check module is used to optionally, if no integrity check is selected, then when no data blocks have any faults, only the data blocks are read, and the data blocks are concatenated into the target data and output to the client without performing a data integrity check. The optional integrity check module allows the data reading module to adapt to different scenarios. When data integrity check is selected, the reliability of the data is improved; when data integrity check is not selected, the data reading steps are simplified and the reading efficiency is improved. The data sending module is used to output the target data requested by the client to the client.

[0167] Some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method described in any one of the embodiments of the method for distributed data storage based on erasure codes can be implemented.

[0168] like Figure 10 Some embodiments of the present application provide an electronic device 700, comprising a memory 710, a processor 720, and a computer program stored on the memory 710 and executable on the processor 720, wherein the processor 720 can implement a method described in any one of the embodiments of the above-mentioned method for distributed data storage based on erasure codes when reading and executing the program through a bus 730.

[0169] Processor 720 can process digital signals and can include various computing architectures, such as a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements a combination of multiple instruction sets. In some examples, processor 720 can be a microprocessor.

[0170] The memory 710 may be used to store instructions executed by the processor 720 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 720 of the embodiment of the present disclosure may be used to execute the instructions in the memory 710 to implement Figure 2 The memory 710 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory or other memory known to those skilled in the art.

[0171] Some embodiments of the present application introduce an index table in the node, including fields such as file identifier, data block or check block hash value, identifier of the data block or check block (i.e., identifier used to distinguish whether the block belongs to a data block or a check block), node unique identifier, data block or check block storage location, etc., which serve to identify the relationship between the file, data block or check block, and storage node. Among them, the identifier of the data block or check block can not only improve the maintainability and reliability of the system, but also provide convenience in the data update and recovery process. When the stored data needs to be updated, the identifier can quickly locate the original data block, so that only these blocks are updated without involving the check block. This can reduce unnecessary computing and storage overhead. The site selection strategy for the first write in some embodiments of the present application, the strategy for selecting the storage node location when the data is first written, includes three parts: digital identifier selection method, performance selection method, and dynamic parameter selection. Not only does it use the hierarchical information of the node unique identifier to optimize path selection and improve communication efficiency, but it also significantly improves the overall processing speed and responsiveness of the system by assigning more data processing tasks to high-performance nodes. Some embodiments of the present application provide an optimized data update method, that is, when data is not written for the first time, it is necessary to use a comprehensive indicator evaluation method to comprehensively evaluate the node load indicator, utilization indicator, and data priority indicator, access count indicator, and importance indicator to provide a strategy for dynamically adjusting storage nodes. The embodiments of the present application provide a distributed data storage system based on erasure codes, including the first write site selection strategy and optimized data update method provided above, and integrated into the data write module and data read module.

[0172] It's easy to understand that some embodiments of the present application improve hit rates by introducing data block and check block identifiers, for example, by using an index table to store relevant attribute information for data blocks and check blocks. Data block and check block identifiers are added to the index table values to distinguish whether the storage node corresponding to the index table entry stores a data block or a check block. Because the index table in the embodiments of the present application provides the ability to quickly locate the storage location of a data block, required data can be found more efficiently, and the logical implementation is relatively simple. During the data read phase, the system provides the ability to directly read data blocks and recover faulty data based on check blocks, adapting to different scenarios and improving read efficiency. Some embodiments of the present application also provide a first-write location selection strategy, specifically designing an optimized first-write data location selection strategy that improves the correlation between data and nodes and reduces randomness. Compared to current storage location selection methods, this approach takes into account node performance and divides the node range into three levels: low-performance, medium-performance, and high-performance. Storing first-write data in medium-performance nodes fully utilizes network performance and facilitates subsequent dynamic adjustment of storage locations based on the hotness or coldness of the data. Furthermore, using a content hash-based location selection method avoids the problem of high reliance on centralized nodes, improving read and write efficiency. Some embodiments of the present application also provide a comprehensive index evaluation method, that is, a data update method is provided. When the data is not written for the first time, the node load index, usage index and data priority index, access count index and importance index are comprehensively evaluated according to the comprehensive index evaluation method, the balance of the storage location selection strategy is optimized, the storage nodes are dynamically adjusted, and the efficiency of the system is improved. This method avoids the problem that the storage location cannot be changed once the data is stored in the node, and can be adjusted in time according to the node load and hot data to adapt to the ever-changing data reading and writing process. Some embodiments of the present application also design a storage and update system, specifically designing a distributed data storage system based on erasure code, combining the first write site selection strategy and the update strategy of the comprehensive index evaluation method, significantly improving the load balancing of the system's storage and update process, and making full use of the system's bandwidth. The system can avoid the randomness of data storage node selection, select the storage location that best suits the data according to actual conditions, avoid single point failure problems, and provide an optimized data storage and update strategy.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0174] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0175] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0176] The foregoing is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.

[0177] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0178] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

Claims

1. A method for distributed data storage based on erasure codes, characterized in that: The method comprises: Obtain n data blocks and check blocks to be stored, and obtain a dynamic parameter γ, where n is an integer greater than 1 and γ is a number greater than zero and less than 1. The result of is an integer; Select according to the node identification of each node in the distributed system Nodes are used as first-class storage nodes; Select at least part of the remaining nodes of the distributed system except the first type of storage nodes according to the node performance Nodes serve as the second type of storage nodes; Using each of the first type of storage nodes and the second type of storage nodes to store a check block or a data block respectively; in, The node identification of each node in the distributed system is selected Nodes are used as first-class storage nodes, including: Sort each node in the distributed system on a ring according to the common prefix length of the node name; Search the ring for a file with a common prefix based on the file identifier of the data to be stored. nodes as the first type of storage nodes; The method searches for a file with a common prefix on the ring according to the file identifier of the file to be stored. The nodes as the first type of storage nodes include: If the nodes on the ring do not have a common prefix with the file identifier, a random direction is selected to continue matching nodes until a preceding prefix is found. The node with the largest prefix length obtains the first type of storage node; If the node on the ring has a common prefix length with the file identifier, continue matching nodes in a clockwise direction until the prefix is found. The node with the largest prefix length obtains the first type of storage node; The method of selecting at least part of the remaining nodes of the distributed system except the first type of storage nodes according to the node performance Nodes as the second type of storage nodes, including: Dividing the remaining nodes of the distributed system except the first type of storage nodes into high-performance nodes, medium-performance nodes and low-performance nodes according to performance; Select from the high performance nodes nodes as the second type of storage nodes; The method further comprises: Calculating hash values of the data block and the check block; Storing a hash value of a corresponding data block or a check block in an index table on each of the first-type storage nodes and the second-type storage nodes; The hash value is used to locate the data block or check block when updating or repairing data; The method further comprises: The index table of each node in the first type of storage node and the second type of storage node stores an identifier for distinguishing whether a stored block belongs to a data block or a check block; After receiving a file read request from a client, the following strategy is adopted to read the required data blocks and / or check blocks according to the read request attributes and the identifier used to distinguish between data blocks and check blocks: If it is confirmed that none of the data blocks to be read has any fault, then the corresponding data block is read using the identifier; If it is confirmed that the data block to be read is faulty, the data block and the check block are read simultaneously using the identifier, and the faulty data block is restored based on the read data; If it is confirmed that the data needs to be checked for integrity, all data blocks and check blocks used for integrity checking are read through the identifier; If it is determined that the data does not need to be checked for integrity, and when no failure occurs in any of the data blocks to be read, multiple data blocks are read using the identifier, and the multiple data blocks are concatenated into target data and then output to the client; The method of reading all data blocks and check blocks for integrity verification by the identifier includes: after receiving a file read request from the client, finding the node with the largest prefix length with the read file name in the ring, and searching for the next node in a counterclockwise direction. nodes, and read the data blocks or check blocks required for recovery in each node according to the hash values included in the index table; send a query request to the index table of each node in the corresponding high-performance list, lock the query range according to the unique identifier of the file, and read the required data blocks and check blocks in the queried node position.

2. The method according to claim 1, wherein The method further comprises: In response to a data update operation, dynamically adjusting nodes used to store the data to be updated according to a comprehensive evaluation index, wherein the comprehensive evaluation index is related to node performance differences and data block heat attributes, the node performance differences are related to load indicators and usage indicators of corresponding nodes, and the data block heat indicators are related to priority indicators, access count indicators, and importance indicators of the data to be updated; Update the description information of each node after dynamic adjustment, wherein the description information includes: a file identifier, a data block or a check block hash value, an identifier for distinguishing between a data block and a check block, a node identifier or the storage location of the data block and the check block, the data block or check block hash value is used to identify the data block or check block stored by the corresponding node, the identifier for distinguishing between a data block or a check block is used to indicate whether the data stored by the corresponding node belongs to a data block or a check block, the node identifier is a node domain name or a set unique node name, and the storage location of the data block or the check block is used to record the specific location of the data stored on the corresponding node.

3. A device for distributed data storage based on erasure coding, configured to execute the method according to claim 1, the device comprising: The data acquisition module is configured to acquire n data blocks and check blocks to be stored, and acquire a dynamic parameter γ, where γ is a number greater than zero and less than 1. The result of is an integer; The first type of storage node selection module is configured to select nodes based on the node identifiers of each node in the distributed system. Nodes are used as first-class storage nodes; The second type of storage node selection module is configured to select at least part of the remaining nodes of the distributed system except the first type of storage nodes according to the node performance. Nodes serve as the second type of storage nodes; The block data storage module is configured to use each of the first-type storage nodes and the second-type storage nodes to store a check block or a data block respectively.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 2 can be implemented.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method described in any one of claims 1 to 2 can be implemented.

Images