Distributed data storage method and system based on erasure codes

Abstract

The embodiment of the invention provides a distributed data storage method and system based on erasure codes, and the method comprises the steps: obtaining n data blocks and check blocks to be stored, and obtaining a dynamic parameter gamma, where gamma is a number greater than zero and less than 1, and the result of gamma * n is an integer; according to the node identifier of each node in the distributed system, selecting gamma * n nodes as a first type of storage nodes; according to node performance, selecting (1-gamma). N nodes from at least part of remaining nodes, except the first type of storage nodes, included in the distributed system as second type of storage nodes; and each node in the first type of storage nodes and the second type of storage nodes is adopted to store a verification block or a data block. The embodiment of the invention provides an improved addressing method based on a content hash value and combined with comprehensive index evaluation. And after the data is stored for the first time, the storage position is dynamically adjusted by taking load balancing as a target, so that the read-write efficiency and performance of the system are improved.

Description

Technical Field

[0001] The present application relates to the field of distributed system data storage. Specifically, embodiments of the present application relate to a method and system for distributed data storage based on erasure codes. Background Art

[0002] With the advent of the big data era, traditional centralized storage systems can no longer meet the storage and processing needs of massive data, and distributed storage systems have emerged. Compared with centralized storage systems, distributed storage systems have high reliability, security, and scalability. By storing complete data in independent nodes, data loss caused by single point failures can be prevented, thereby improving data reliability. At the same time, distributed storage systems have load balancing strategies and are no longer limited by the performance bottleneck caused by the central node of centralized storage systems, which can improve read and write efficiency.

[0003] In order to improve the storage efficiency of distributed storage systems and ensure data consistency, related technologies will introduce data fault tolerance. Existing fault tolerance technologies include multi-copy fault tolerance technology and erasure code fault tolerance technology. Multi-copy fault tolerance is to store multiple copies of data on multiple nodes to ensure that when a node fails, fault tolerance can be achieved by reading data copies from other nodes. Erasure code fault tolerance technology is to split the complete data into multiple data blocks, calculate the check blocks using the erasure code algorithm, store the data blocks and check blocks on different nodes according to certain rules, and provide data redundancy without increasing too much storage burden. Compared with multi-copy technology, erasure code can enhance fault tolerance while reducing storage overhead, so it has been widely used in the field of distributed storage.

[0004] Erasure codes, such as RS(n,k), first divide the original data into k pieces, use the coding matrix, and calculate the linear combination of the data blocks to obtain the corresponding (nk) check blocks. According to different location strategies, the data blocks and check blocks are stored on n different nodes. The additional storage overhead is When no more than nk nodes fail, the original data can be restored using the remaining normal nodes.

[0005] While using erasure coding technology to improve the reliability of distributed storage systems, some problems will also arise. When writing data, it is necessary to encode and calculate the data block to obtain the check block, and store the data block and the check block on different nodes, which not only generates computing overhead, but also increases the network overhead generated when selecting nodes. During the data update process, it is necessary to read all relevant data blocks and check blocks, and then update the old data, which increases the write delay. When reading data, frequently accessed nodes will become hot data, causing hot spots and reducing system performance.

[0006] The existing data block storage location selection strategy is generally based on location addressing. When reading data, the client needs to send a request to the central node and then find the target data from the corresponding storage location. This addressing method makes the reading process heavily dependent on the central node and a specific network topology. It is not completely decentralized and is prone to single point failure problems. Summary of the invention

[0007] The purpose of the embodiments of the present application is to provide a method and system for distributed data storage based on erasure codes. The embodiments of the present application provide an improved content hash value-based addressing method and a dynamic addressing method combined with comprehensive index evaluation. After the data is stored for the first time, the embodiments of the present application dynamically adjust the storage location with load balancing as the goal to improve the read and write efficiency and performance of the system. At the same time, the storage and update method provided by the present invention realizes decentralized node storage, and an index table is maintained separately in each node, which is conducive to preventing system operation abnormalities caused by single point failures.

[0008] In a first aspect, an embodiment of the present application provides a method for distributed data storage based on erasure codes, the method comprising: obtaining a total of n data blocks and check blocks to be stored, and obtaining a dynamic parameter γ, wherein n is an integer greater than 1, γ is a number greater than zero and less than 1, and the result of γ·n is an integer; selecting γ·n nodes as first-type storage nodes according to the node identifier of each node in the distributed system; selecting (1-γ)·n nodes as second-type storage nodes from at least part of the remaining nodes included in the distributed system except the first-type storage nodes according to the node performance; and using each of the first-type storage nodes and the second-type storage nodes to store a check block or a data block, respectively.

[0009] Some embodiments of the present application adopt a strategy that combines node identification and node performance selection to select storage nodes when storing n data blocks for the first time, thereby increasing the speed of subsequent reading, updating and processing of the stored data, thereby improving the access efficiency of the distributed storage system.

[0010] In some embodiments, the selecting of γ·n nodes as the first type of storage nodes based on the node identifier of each node in the distributed system includes: sorting each node in the distributed system on a ring according to the common prefix length of the node name; and searching for γ·n nodes with a common prefix on the ring as the first type of storage nodes based on the file identifier of the data to be stored.

[0011] Some embodiments of the present application arrange corresponding nodes according to node names and then select storage nodes for data blocks and check blocks to be stored according to the arrangement results, which can improve the subsequent access and reading speed of these data.

[0012] In some embodiments, searching for γ·n nodes with a common prefix on the ring as the first type of storage nodes according to the file identifier of the file to be stored includes: if the node on the ring has no common prefix with the file identifier, randomly selecting a direction to continue matching nodes until the node with the first γ·n maximum prefix length is found to obtain the first type of storage node; if the node on the ring has a common prefix length with the file identifier, continuing to match nodes in a clockwise direction until the node with the first γ·n maximum prefix length is found to obtain the first type of storage node.

[0013] Some embodiments of the present application determine the distribution direction of the first type of storage nodes on the ring according to the name of the file to be stored, which can improve the subsequent reading and processing speed of these stored data.

[0014] In some embodiments, the method of selecting (1-γ)·n nodes as the second type of storage nodes from at least a portion of the remaining nodes excluding the first type of storage nodes in the distributed system according to node performance includes: dividing the remaining nodes excluding the first type of storage nodes in the distributed system into high-performance nodes, medium-performance nodes and low-performance nodes according to performance; and selecting (1-γ)·n nodes from the high-performance nodes as the second type of storage nodes.

[0015] Some embodiments of the present application also select storage nodes for part of the data in the n data blocks and check blocks stored for the first time based on the dynamic coefficient and the node performance, so as to improve the storage speed of these data blocks and check blocks.

[0016] In some embodiments, the method also includes: calculating hash values ​​of the data block and the check block; storing the hash value of the corresponding data block or check block in an index table on each node in the first type of storage node and the second type of storage node; wherein the hash value is used to locate the data block or check block when updating or repairing data.

[0017] In some embodiments of the present application, it is also necessary to store hash values ​​of the stored data blocks and check blocks in each storage node to improve the speed of subsequent reading and locating these data blocks.

[0018] In some embodiments, the method further includes: in response to an operation of updating a data block, dynamically adjusting the nodes used to store the data to be updated according to a comprehensive evaluation index, wherein the comprehensive evaluation index is related to the node performance difference and the data block heat attribute, the node performance difference is related to the load index and the usage index of the corresponding node, and the data block heat index is related to the priority index, the number of accesses index and the importance index of the data to be updated; updating the description information of each node after dynamic adjustment, wherein the description information includes: a file identifier, a data block or a check block hash value, an identifier for distinguishing between a data block and a check block, a node identifier or a storage location of the data block and the check block, the data block or the check block hash value is used to identify the data block or the check block stored by the corresponding node, the identifier for distinguishing between the data block or the check block is used to indicate whether the data stored by the corresponding node belongs to a data block or a check block, the node identifier is a node domain name or a set unique node name, and the storage location of the data block or the check block is used to record the specific location of the data stored on the corresponding node.

[0019] Some embodiments of the present application also use comprehensive evaluation indicators to dynamically adjust the nodes storing data blocks when updating data blocks, and use the description information of the nodes to dynamically adjust the corresponding information of the data stored in the adjusted nodes. In this way, the storage nodes can be reselected for the corresponding data blocks when the data blocks are updated, and the adjustment status of the data stored on the relevant nodes can be instantly obtained according to the description information, thereby improving the efficiency of the distributed storage system in subsequent storage, reading and updating of related data.

[0020] In some embodiments, the method further includes: storing in the index table of each node in the first type of storage node and the second type of storage node an identifier for distinguishing whether the stored block belongs to a data block or a check block; upon receiving a file read request from a client, adopting the following strategy to read the required data block and / or check block according to the read request attribute and the identifier for distinguishing between the data block and the check block: if it is confirmed that none of the data blocks to be read have failed, reading the corresponding data block through the identifier; if it is confirmed that the data block to be read has failed, reading the data block and the check block at the same time through the identifier, and restoring the failed data block based on the read data; if it is confirmed that the data needs to be checked for integrity, reading all the data blocks and check blocks used for integrity checking through the identifier; If it is confirmed that there is no need to perform integrity check on the data, and when no failure occurs in the data blocks to be read, multiple data blocks are read through the identifier, and the multiple data blocks are connected into target data and then output to the client; wherein, the reading of all data blocks and check blocks for integrity check through the identifier includes: upon receiving a file read request from the client, finding a node with a maximum prefix length with the read file name in the ring, and searching counterclockwise for the next (1-γ)·n nodes, and reading the data blocks or check blocks required for recovery in each node according to the hash values ​​included in the index table; sending a query request to the index table of each node in the corresponding high-performance list, locking the query range according to the file unique identifier, and reading the required data blocks and check blocks in the queried node position.

[0021] Some embodiments of the present application determine whether to read only the data block or to read both the check block and the data block at the same time by storing an identifier for distinguishing whether the stored block belongs to a data block or a check block and the purpose of this reading, thereby improving the efficiency of data reading and the speed of data reading.

[0022] In a second aspect, some embodiments of the present application provide a distributed data storage device based on erasure codes, the device comprising: a data acquisition module, configured to acquire a total of n data blocks and check blocks to be stored, and acquire a dynamic parameter γ, wherein n is an integer greater than 1, γ is a number greater than zero and less than 1, and the result of γ·n is an integer; a first-category storage node selection module, configured to select γ·n nodes as first-category storage nodes according to the node identifiers of each node in the distributed system; a second-category storage node selection module, configured to select (1-γ)·n nodes as second-category storage nodes from at least part of the remaining nodes included in the distributed system except the first-category storage nodes according to the node performance; a block data storage module, configured to use each of the first-category storage nodes and the second-category storage nodes to store a check block or a data block respectively.

[0023] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any one of the embodiments included in the first aspect.

[0024] In a fourth aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the embodiments included in the first aspect can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A storage architecture diagram of a distributed system provided in an embodiment of the present application;

[0027] Figure 2 One of the flowcharts of the method for distributed data storage based on erasure coding provided in an embodiment of the present application;

[0028] Figure 3 Flow chart 2 of the method for distributed data storage based on erasure coding provided in an embodiment of the present application;

[0029] Figure 4 One of the component block diagrams of the distributed data storage device based on erasure code provided in an embodiment of the present application;

[0030] Figure 5 The second block diagram of the distributed data storage device based on erasure coding provided in the embodiment of the present application;

[0031] Figure 6 The third block diagram of the distributed data storage device based on erasure coding provided in the embodiment of the present application;

[0032] Figure 7 The fourth block diagram of the distributed data storage device based on erasure coding provided in the embodiment of the present application;

[0033] Figure 8 A schematic diagram of a method for obtaining a comprehensive evaluation value provided in an embodiment of the present application;

[0034] Fig. 9A fifth block diagram of a distributed data storage device based on erasure codes provided in an embodiment of the present application;

[0035] Fig.10 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0037] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0038] As described in the background technology section, the distributed storage system stores data in multiple independent devices. Most traditional network storage systems use centralized storage servers to store all data. The storage server becomes the bottleneck of system performance and the focus of reliability and security, which cannot meet the needs of large-scale storage applications. The distributed network storage system adopts an expandable system structure, uses multiple storage servers to share the storage load, and uses location servers to locate storage information. It improves the reliability, availability and access efficiency of the system to a certain extent. However, with the increase in the number of servers, a very important problem has emerged, that is, how to load-balance the selection of data storage node locations to improve the read and write efficiency and performance of the system, and to prevent system operation abnormalities caused by single point failures.

[0039] Traditional data centers or storage clusters usually adopt a multi-copy data redundancy strategy, which has high access performance, but also has the disadvantages of high storage redundancy and storage cost. To solve the above problems, more and more commercial storage systems have introduced erasure code (including traditional erasure code, storage regeneration code / network coding) data protection strategies to achieve equal reliability through less data redundancy. The erasure code algorithm encodes multiple data blocks and generates multiple check blocks, where data blocks and check blocks are collectively referred to as coding blocks. The embodiments of the present application are based on the erasure code data storage strategy.

[0040] As described in the background technology section, the data block storage location selection strategy in the existing erasure code distributed data storage system is generally based on location addressing. When reading data, the client needs to send a request to the central node and then find the target data from the corresponding storage location. This addressing method makes the reading process heavily dependent on the central node and a specific network topology. It is not completely decentralized and is prone to single point failure problems. In the data storage process based on address addressing, there is randomness between data and storage nodes, and it is difficult to predict the distribution of hot data and cold data, which can easily lead to hot spot problems and unbalanced system load. When considering the storage nodes of data blocks, it is also necessary to start from the data and the nodes themselves, fully consider the node indicators and the indicators of the data, and balance the load between the nodes.

[0041] At least to solve these technical problems, the embodiments of the present application provide a method for distributed data storage based on erasure codes. When selecting storage nodes, the method needs to combine the dynamic parameter γ to determine the use of a digital identification selection method (i.e., a method for selecting the first type of storage node) and a performance selection method (i.e., a method for selecting the second type of storage node) to allocate storage nodes for the data blocks and check blocks to be stored, wherein the method for selecting the storage location for the data block or check block according to the unique node identifier is a digital identification selection method, and the method for selecting the storage location for the data block and check block according to the performance of the node is recorded as a performance selection method. Some embodiments of the present application also complete data reading based on the content hash value addressing method, and dynamically adjust the storage nodes of the data block or check block in combination with the comprehensive indicator evaluation method. That is, in some embodiments of the present application, after the data is stored for the first time (the first storage is based on the selected first type of storage node and the second type of storage node), the storage location is dynamically adjusted with load balancing as the goal to improve the read and write efficiency and performance of the system. At the same time, the storage and update methods provided in some embodiments of the present application realize decentralized node storage, and an index table is maintained separately in each node. The columns in the index table include: file identifier, data block or check block hash value, identifier of the data block or check block, node identifier, data block or check block storage location. The hash value in the index table can improve data query efficiency and help prevent system operation abnormalities caused by single point failures.

[0042] Please see Figure 1 , Figure 1 A distributed storage system provided in some embodiments of the present application includes a data center, which includes multiple clusters, for example, Figure 1 There are multiple clusters (1, 2, 3, 4, etc.), and each cluster includes multiple racks, for example, Figure 1There are multiple racks (001, 002, 003, 004, 005), and each rack is provided with multiple nodes, which can be selected as storage nodes, for example, Figure 1 Includes multiple storage nodes (100, 101, 102, 103, 104, 105).

[0043] It is understandable that the distributed data storage method based on erasure coding provided in this application can be applied to Figure 1 The distributed storage system.

[0044] Figure 1 The example data center datacenter.example.com in the distributed storage system is shown in FIG. 1 . The data center includes five different clusters, each of which is named, for example, cluster1.datacenter.example.com, cluster2.datacenter.example.com,

[0045] cluster3.datacenter.example.com, cluster4.datacenter.example.com,

[0046] cluster5.datacenter.example.com.

[0047] Take cluster 1 (cluster1.datacenter.example.com) as an example, which includes 5 racks (001, 002, 003, 004, 005), each rack is named, for example:

[0048] rack1.cluster1.datacenter.example.com,

[0049] rack2.cluster1.datacenter.example.com,

[0050] rack3.cluster1.datacenter.example.com,

[0051] rack4.cluster1.datacenter.example.com, rack5.cluster1.datacenter.example.com.

[0052] Take rack 001 as an example, which includes three storage nodes (100, 101, 102). The storage nodes can be further represented as:

[0053] node1.rack1.cluster1.datacenter.example.com, node2.rack1.cluster1.datacenter.example.com, node3.rack1.cluster1.datacenter.example.com.

[0054] Here is an example:

[0055] In the data writing phase, for the file exampleDoc to be stored from cluster1.datacenter.example.com, first record its file identifier as com.example.datacenter.cluster1.exampleDoc. According to the digital identification selection method, select γ·n storage nodes with the minimum number of routes as the first-class storage nodes (assuming that the first-class storage nodes are all located in cluster 1) to store the encoded γ·n data blocks and check blocks; secondly, calculate the performance of all nodes in the storage system and divide the nodes into three categories: low, medium and high performance; then, according to the performance selection method, select (1-γ)·n nodes in other clusters except cluster 1 from the nodes in the high-performance category as the second-class storage nodes to store the remaining (1-γ)·n data blocks and check blocks after encoding.

[0056] In the subsequent update phase, the load index, usage index, data priority index, access count index, and importance index of the storage system node are comprehensively considered to calculate the difference between the node load score and the load mean (D 100 ,D 101 ,D 102 ,D 103 ,D 104 ,D 105 ,…), and use this to readjust the node storage position.

[0057] like Figure 2 As shown, an embodiment of the present application provides a method for distributed data storage based on erasure codes, the method comprising:

[0058] S110, obtaining n data blocks and check blocks to be stored, and obtaining a dynamic parameter γ, wherein n is an integer greater than 1, γ is a number greater than zero and less than 1, and the result of γ·n is an integer.

[0059] S120 , selecting γ·n nodes as first-type storage nodes according to the node identifier of each node in the distributed system.

[0060] S130: Select (1-γ)·n nodes as second-type storage nodes from at least a portion of the remaining nodes excluding the first-type storage nodes included in the distributed system according to node performance.

[0061] S140: Use each of the first-type storage nodes and the second-type storage nodes to store a check block or a data block respectively.

[0062] The implementation process of the above steps is described below by way of example.

[0063] In some embodiments of the present application, the process of selecting γ·n nodes as the first type of storage nodes according to the node identifier of each node in the distributed system described in S120 exemplarily includes: sorting each node in the distributed system on a ring according to the common prefix length of the node name; and searching for γ·n nodes with a common prefix on the ring as the first type of storage nodes according to the file identifier of the data to be stored.

[0064] It should be noted that the node name, for example: if a node is located at the dataLocation position in the domain name www.example.com, it will be flipped to the com-example-dataLocation identification string. The com identification is stored on ring r0, the example identification is stored on ring r1, and the dataLocation identification is stored on r2. Therefore, minimum routing locality is provided for nodes with the same prefix name. For storage files in the same domain, the node with the smallest number of routes is selected from all possible nodes as the storage node to achieve local load balancing and improve data reading and writing rates.

[0065] The file identifier of some embodiments of the present application includes its subdomain name, host name and file name. For example, for a file to be stored storageDataHtml, which is located in the domain name myhost.dataLocation.example.com, the file identifier is recorded as com.example.dataLocation.myhost.storageDataHtml.

[0066] The structure of the ring in some embodiments of the present application is as follows: the ring includes multiple layers of sub-rings (r0, r1, r2, ... rz), and the sub-rings from r0 to rz store subdomain names, host names, and file names respectively. Each sub-ring uses a bidirectional linked list data structure to achieve the requirement of clockwise and counterclockwise query; the strings in each sub-ring are arranged clockwise with the number of the same prefix characters as the standard; the upper sub-ring and the lower sub-ring are connected by pointers to indicate the next string that the stored prefix in the upper sub-ring may have. For example, for the prefixes exam1, exam2, and exam3, which have the suffixes aaa, bbb, aab, and ccc respectively, the ring is constructed as follows:

[0067] r0 stores (exam1, exam2, exam3) in clockwise order; r1 includes three subrings: (aaa, aab, bbb, ccc), (aaa, aab, bbb, ccc), and (aaa, aab, bbb, ccc), which correspond to the three prefix strings in r0 respectively, and the strings in each subring are sorted according to the same number of prefix characters.

[0068] For example, in some embodiments of the present application, S120 includes a process of searching for γ·n nodes with a common prefix on the ring as the first type of storage nodes according to the file identifier of the file to be stored, which exemplarily includes: if the node on the ring has no common prefix with the file identifier, randomly selecting a direction to continue matching nodes until the node with the first γ·n maximum prefix length is found to obtain the first type of storage node; if the node on the ring has a common prefix length with the file identifier, continuing to match nodes in a clockwise direction until the node with the first γ·n maximum prefix length is found to obtain the first type of storage node.

[0069] In some embodiments of the present application, the process of selecting (1-γ)·n nodes as the second type of storage nodes from at least a part of the remaining nodes other than the first type of storage nodes included in the distributed system according to node performance in S130 exemplarily includes: dividing the remaining nodes other than the first type of storage nodes included in the distributed system into high-performance nodes, medium-performance nodes and low-performance nodes according to performance; and selecting (1-γ)·n nodes from the high-performance nodes as the second type of storage nodes.

[0070] In some embodiments of the present application, an index table is maintained separately in each node. The columns in the index table include: file identifier, data block or check block hash value, identifier of the data block or check block (or identifier for distinguishing data block and check block), node identifier, data block or check block storage location. The information stored in these columns (i.e., description information of the corresponding node) can improve the data query efficiency of the distributed storage system and help prevent system operation abnormalities caused by single point failures. Among them, the file identifier is the unique identifier of the file stored in the node, that is, the node unique identifier plus the connection name of the file name. Data block or check block hash value: used to identify which data block or check block of the file is stored in the node, mainly used to locate the data block or check block when updating or repairing data. The identifier of the marker block or check block is used to indicate whether the file stored at the location is the original data block or the check block, which is used to explicitly indicate the difference between the data block and the check block, facilitates the development and testing of coders, and provides a method for directly reading the data block and a method for decoding and recovering data using the check block when updating or repairing data. The node identifier uses the domain name of the node or the unique node name set as the identifier that uniquely represents a node in the storage network. The data block or check block storage location is the specific storage location of the data block or check block of the storage file, including the file path or memory address or starting offset, which is used to locate the specific storage location of the data block or check block in the node.

[0071] In some embodiments of the present application, the method also includes: calculating hash values ​​of the data block and the check block; storing the hash value of the corresponding data block or check block in an index table on each node in the first type of storage node and the second type of storage node; wherein the hash value is used to locate the data block or check block when updating or repairing data.

[0072] In some embodiments of the present application, the method further includes: in response to an operation of updating a data block (i.e., a data block stored on a node of a distributed storage system), dynamically adjusting a node for storing data to be updated (i.e., a data block to be updated) according to a comprehensive evaluation index, wherein the comprehensive evaluation index is related to a node performance difference and a data block heat attribute, the node performance difference is related to a load index and a usage index of a corresponding node, and the data block heat index is related to a priority index, a number of accesses index, and an importance index of the data to be updated; updating the description information of each node in the dynamically adjusted node, and the description information exemplarily includes: a file identifier, a data block or a check block hash value, an identifier for distinguishing between a data block and a check block, a node identifier, or a storage location of the data block and the check block; wherein the file identifier is a unique identifier for a file stored at the node; the data block or check block hash value: used to identify Identify which data block or check block of the file is stored in the node, that is, the hash value of the data block or check block is used to identify the data block or check block stored in the corresponding node; the identifier used to distinguish the data block or check block is used to indicate whether the stored block belongs to the data block or the check block, and is used to explicitly indicate the difference between the data block and the check block, that is, the identifier used to distinguish the data block or the check block is used to indicate whether the data stored in the corresponding node belongs to the data block or the check block; the node identifier uses the domain name of the node or the unique node name set as the identifier that uniquely represents the corresponding node in the storage network, that is, the node identifier is the node domain name or the unique node name set; the storage location of the data block or the check block is used to record the specific storage location of the data block or the check block of the stored file, including the file path or the memory address or the starting offset, that is, the storage location of the data block or the check block is used to record the specific location of the data stored on the corresponding node.

[0073] In some embodiments of the present application, the method also includes: storing in the index table of each node in the first type of storage node and the second type of storage node an identifier for distinguishing whether the stored block belongs to a data block or a check block; after receiving a file read request from the client, adopting the following strategy to read the required data block and / or check block according to the read request attribute and the identifier for distinguishing between data blocks and check blocks: if it is confirmed that none of the data blocks to be read has failed, then only the data block is read through the identifier, that is, if it is confirmed that none of the data blocks to be read has failed, then the corresponding data block is read through the identifier; if it is confirmed that the data block to be read has failed, then k data blocks and check blocks are read simultaneously through the identifier, and the data is restored using the data decoding module, that is, if it is confirmed that the data block to be read has failed, then the data block and check block are read simultaneously through the identifier, and the faulty data block is restored according to the read data; if it is confirmed that the read data needs to be checked for integrity, then n data blocks and check blocks are read through the identifier, that is, if it is confirmed that the data needs to be checked for integrity , then all data blocks and check blocks for integrity verification are read through the identifier; if it is confirmed that the read data does not need to be checked for integrity, and when none of the data blocks to be read has failed, only the data blocks are read through the identifier, and the data blocks are connected to form target data and then output to the client, that is, if it is confirmed that the data does not need to be checked for integrity, and when none of the data blocks to be read has failed, multiple data blocks are read through the identifier, and the multiple data blocks are connected to form target data and then output to the client; wherein, if it is necessary to read n data blocks and check blocks, the method further includes: after receiving a file read request from the client, finding a node with the maximum prefix length with the read file name in the ring, and counterclockwise searching for the next (1-γ)·n nodes, and reading the data blocks or check blocks required to restore the file in each node according to the hash value included in the index table; sending a query request to the index table of each node in the corresponding high-performance list, locking the query range according to the file unique identifier, and reading the required data blocks and check blocks in the queried node position.

[0074] Combine the following Figure 3 The method of distributed data storage based on erasure codes provided by some embodiments of the present application is exemplified.

[0075] The embodiment of the present application provides a distributed data storage method based on erasure codes. When storing data for the first time, the specific storage location is selected according to the CPU performance and network performance of the node, and according to the unique identifier of the node. It also includes using a dynamic parameter γ to select the allocation rate of the storage location selection method according to the unique identifier of the node (digital identifier selection method) and the storage location selection method according to the performance of the node (performance selection method). In the subsequent update stage, the node load index, usage index, and data priority index, access count index, and importance index are comprehensively considered, and a load-balanced node selection strategy (i.e., adjusting the storage node of the data block to be updated according to the comprehensive evaluation index) is proposed to improve the read and write performance and update efficiency of the distributed storage system, reduce network overhead, and prevent hot spot problems.

[0076] The present application provides a distributed data storage method for erasure codes. Figure 3 As shown, the method includes:

[0077] The first step is to obtain the complete input data and perform sharding processing to obtain data blocks of uniform size.

[0078] In the data writing phase: In the distributed storage system, set the data block size to s, the number of data blocks to k, the number of check blocks to m, the sum of the number of check blocks and data blocks to n, the erasure code to RS(n,k) and the finite field to GF(2 w ).

[0079] In the second step, the coding matrix is ​​used to calculate the data block to obtain the check block and synthesize multiple stripes.

[0080] Generate and save the encoding matrix so that the check block can be calculated when writing the data block. Slice the data block according to the preset size s and number to obtain k slicing data blocks. If the complete data size N cannot be evenly divided into M groups, each group has k data blocks of size s, then the last data block that does not meet the size s is padded with zero values ​​to enable normal encoding operations.

[0081] Using the coding matrix, linear combination calculations are performed on the data blocks that meet the requirements to obtain the corresponding (nk) check blocks, and M groups of data (each group contains k data blocks and m check blocks) are stored in different nodes according to the location selection rules.

[0082] The third step is to calculate the content hash values ​​of the data block and the check block.

[0083] In the fourth step, if the data is written for the first time, the first write site selection strategy is adopted to select a storage node from two methods: a digital identification selection method and a performance selection method.

[0084] Selecting a storage location according to a digital identification selection method includes: (1) setting a dynamic parameter γ, by which the number of storage blocks allocated for the digital identification selection method and the performance selection method can be determined; (2) selecting a storage location according to a node unique identifier, denoted as a digital identification selection method; (3) selecting a storage location method according to the performance of the node, denoted as a performance selection method. For example, in some embodiments of the present application, it is determined according to the dynamic parameter γ that the digital identification selection method is used to store γ·n blocks, and the performance selection method is used to store (1-γ)·n blocks.

[0085] For example, in some embodiments of the present application, the digital identification selection method includes:

[0086] a. On the ring, nodes are sorted according to the common prefix lengths with other node names, so that the common prefix lengths between adjacent nodes in the clockwise direction are as large as possible.

[0087] b. Search for nodes with a common prefix on the ring according to the file identifier. If the current node on the ring does not have a common prefix with the file identifier, randomly select a direction to continue matching nodes; if the current node and the file identifier have a common prefix length, continue matching nodes in a clockwise direction until a node with the maximum prefix length of the first γ·n is found, and a data block or a check block is stored on each node.

[0088] For example, in some embodiments of the present application, the performance selection method includes: evaluating the CPU performance and network performance of the node.

[0089] Further, in some embodiments of the present application, evaluating the CPU performance and network performance of a node includes: when data is first written, for a node that has not yet stored data, evaluating its CPU performance index and network performance index; wherein CPU performance includes main frequency, number of cores, number of threads, and network performance includes network throughput and network latency. CPU performance is one of the important factors affecting node performance, and the CPU performance score calculation formula is as follows:

[0090] CPU=CPU 主频 *CPU 核心 *CPU 线程

[0091] Among them, CPU represents the CPU performance score of the node. 主频 Indicates the oscillation speed of the digital pulse signal in the node's CPU, which affects the CPU processing speed to a certain extent; CPU 核心 Indicates the number of CPU cores of the node. The more CPU cores there are, the stronger the processing power is. 线程 Indicates the number of CPU threads of the node, which can reflect the CPU parallel processing capability.

[0092] Network performance is also an important indicator for evaluating node performance. The network performance score is calculated as follows:

[0093]

[0094] Among them, mem represents the network performance score of the node, MEM 吞吐量 Indicates the network throughput of the node, which can reflect the data transmission capacity of the node; MEM 延迟 Indicates the network latency of a node, reflecting the speed and efficiency of network data transmission.

[0095] The weight parameters of the node's CPU performance index and network performance are set to 0.5 and 0.5 respectively. For node n, the weighted calculation formula is:

[0096] w n =0.5*cpu+0.5*mem

[0097] Some embodiments of the present application group nodes by performance level according to the calculated node weights, and store the corresponding nodes in low performance, medium performance, and high performance lists, respectively. Combining the file identifier and the node unique identifier, embodiments of the present application select a node storage location in the high performance list.

[0098] The node storage location is selected in the high-performance list, and the node selection range is random (1-γ)·n elements in the high-performance node, and the γ·n nodes selected in the digital identification selection method are excluded. It can be expressed by the following formula:

[0099]

[0100] Among them, Deque high represents the high-performance list, and B represents the γ·n nodes selected in the digital identification selection method.

[0101] The above process is as follows Figure 1 The distributed architecture shown is described below:

[0102] The distributed system consists of H data centers, then S = {S1, S2, ... S H}, each data center contains multiple clusters. The total number of nodes in the data center is J, then the summary point NODE = {node1, node2, ...node J For each summary point NODE on the rack, calculate the node weighted score weight = {w1,w2,…w J}, according to the score, the nodes are stored in three weight queues: low, medium and high:

[0103] Dequelow ={node1, node3, ...node J-1}

[0104] Deque medium ={node2, node4, ...node J-2}

[0105] Deque high ={node5, node6, ...node J}

[0106] Among them, Deque low Represents a low-performance list, Deque medium Represents a list of properties, Deque high Represents a high-performance list.

[0107] It should be noted that the digital identification selection method (i.e., the method for selecting the first type of storage node) in some embodiments of the present application takes locality into consideration and selects the node with the smallest number of routes from all possible nodes as the storage node, but does not consider the performance indicators of the node, and may greatly reduce the fault tolerance of the distributed system because the data blocks or check blocks are stored in the nodes under one domain name, leading to hot spot problems. Some embodiments of the present application also write data directly into the Deque high The technical solution of using the second type of storage nodes to store the remaining data blocks or check blocks in the random nodes in the list and excluding the nodes selected in the digital identification selection method not only reduces the data reading and writing delay to a certain extent, but also makes the data evenly distributed on the nodes, thereby ensuring the load balancing and fault tolerance of the distributed system.

[0108] Step 5: If the data is not written for the first time, a comprehensive evaluation method is used to dynamically adjust the storage node.

[0109] Furthermore, the storing of M groups of data (each group including k data blocks and m check blocks) in different nodes according to the site selection rules includes: evaluating the CPU performance and network performance of the node, and calculating the content hash values ​​of the data blocks and check blocks in the M groups of data according to their contents; and comprehensively evaluating the load index, utilization index, and priority index, access count index, and importance index of the node, and dynamically optimizing the storage location selection.

[0110] Furthermore, in some embodiments of the present application, the subsequent dynamic adjustment of the storage node position based on the comprehensive evaluation index includes: comprehensively evaluating the node's load index, utilization index, and data priority index, access count index, and importance index, and calculating the performance difference between nodes and the heat level of data blocks, so as to optimize the load balance in the system, dynamically adjust node storage, and ensure that the reading and writing frequency of data is relatively balanced.

[0111] In some embodiments of the present application, the dynamic adjustment node storage adopts a comprehensive evaluation method to obtain the current load state of the node, that is, in the process of calculating the comprehensive score, algorithms such as entropy weight method, TOPSIS method, grey correlation analysis method, fuzzy comprehensive evaluation method, etc. can be selected to obtain the load comprehensive score RE. Then, according to the load of different nodes, dynamic adjustment is performed to make their load tend to be balanced.

[0112] The process of obtaining the comprehensive load score includes:

[0113] For the node load index (A 11 ), utilization index (A 12 ) and data priority indicators (A 21 ), visit count indicator (A 22 ) and importance index (A 23 ), which can be divided into two parts: nodes and data. The load index (A 11 ), utilization index (A 12 ), visit count indicator (A 22 ) is an objective indicator, and the priority indicator (A 21 ), importance index (A 23 ) is a subjective value assignment indicator. Set the weight vector to: W A =[0.25,0.25,0.16,0.18,0.16].

[0114] In some embodiments of the present application, for subjective indicators: for priority indicators (A 21 ), the value range is [0, 100], of which the high priority value range is (50, 100], the medium priority value range is (20, 50], and the low priority value range is [0, 20]; for the importance index (A 23 ), the value range is [0, 100], of which the value range of core data is (50, 100], the value range of important data is (20, 50], and the value range of general data is [0, 20]. For objective indicators: For load indicators (A 11 ), utilization index (A 12 ), visit count indicator (A 22) The normalization formula is used to scale the original values ​​to the range [0, 100] to facilitate subsequent comprehensive evaluation.

[0115] For each storage node, the evaluation level membership function is used to calculate the evaluation level membership corresponding to the five indicators. For each node, the membership of each evaluation level is calculated on each indicator to form a single indicator evaluation matrix. For the five indicators, a comprehensive indicator evaluation matrix R = [R 1,1 ,R 1,2 ,R 2,1 ,R 2,2 ,R 2,3 ]. The evaluation grade matrix E includes four elements: excellent (e1), good (e2), general (e3), and poor (e4). Among them, the excellent grade is assigned 90 points, the good grade is assigned 80 points, the general grade is assigned 70 points, and the poor grade is assigned 60 points. The comprehensive evaluation matrix B is calculated based on the weight vector W and the single indicator evaluation matrix R. The formula is as follows:

[0116] B=W×R=(b1,b2,b3,b4,b5)

[0117] Here, "X" indicates that W and R perform corresponding operations, not the multiplication of two matrices. ik It represents the comprehensive membership of the i-th node at the k-th evaluation level, as follows:

[0118]

[0119] Among them, r ijk It represents the probability that the i-th node belongs to the k-th evaluation level on the j-th indicator.

[0120] The node load score RE is calculated based on the evaluation matrix B and the evaluation level matrix E i , where RE i Represents the load score of the i-th node:

[0121]

[0122] The process of dynamically adjusting the load to balance the load includes:

[0123] For nodes a, b, c, ..., d, e, the respective load scores (REa, REb, REc, ..., REd, REe) are obtained according to the above comprehensive evaluation method, and the node load average is calculated:

[0124]

[0125] make It represents the difference between the load score of node n and the average load of the node. If D = 0, it means that the load of the node is relatively stable and meets the requirements of data storage. If D>0, it means that the load score of the node is high and the storage data needs to be adjusted to make the D value approach 0. Use the function f(RE a ,RE b ,...,RE e ) indicates that load balancing among nodes is achieved by re-adjusting the node storage locations.

[0126] During the dynamic adjustment process, the node load average needs to be updated in real time. and the load difference D to determine which node to adjust the data storage location.

[0127] The sixth step is to update the file identifier, data block or check block hash value, identifier of the data block and check block, node unique identifier and data block or check block storage location in the node.

[0128] Furthermore, the storage of data according to hash values ​​as described in some embodiments of the present application includes: when a data block is created or modified, calculating the content hash value of the data block as a unique identifier, writing the data block content, and updating the file identifier, data block or check block hash value, identifier marking the data block or check block, node unique identifier, and data block or check block storage location in its index table.

[0129] Furthermore, in some embodiments of the present application, the process of calculating the content hash value of a data block includes:

[0130] Initialize eight buffers {h0,h1,h2,…h7} and fill the data block D with zero values. block ′ and check block V block ′ is divided into Z blocks, and R rounds of transformation are performed on each block to update the buffer value.

[0131] Concatenate the buffer values ​​after the last round of transformation to form the final hash value

[0132] H(D block )=h 0,z ||h 1,Z ||…h 7,Z || and H(V block )=h 0,Z ||h 1,Z ||…h 7,z ||.

[0133] The data writing of some embodiments of the present application involves initial writing and subsequent update operations. If it is the initial writing, no verification of new and old data is involved. If it is a subsequent update operation, the new and old data need to be verified in order to obtain the incremental part and perform incremental updates on the data block.

[0134] In some embodiments of the present application, when data is written, if it is the first time to write, the storage node is directly selected according to the digital identification selection method and the performance selection method, and the file identifier, data block or check block hash value, identifier of the data block or check block, node unique identifier, and data block or check block storage location in the node are updated.

[0135] In some embodiments of the present application, when data is written, if it is a subsequent update operation, it is necessary to query the storage node information through the index table, query the storage data block and the check block in the corresponding node, use the updated data block to replace the old data block, use the coding matrix to calculate the check block of the new data block, and replace the old check block to achieve incremental data update.

[0136] In some embodiments of the present application, during the data reading phase:

[0137] According to the coding matrix saved in the data writing stage, a decoding matrix is ​​generated by using matrix inverse operation to decode the data blocks and check blocks in the node and restore them to the original data.

[0138] After receiving the client's request to read a file, find the node with the largest prefix length of the read file name in the ring, and search for the next (1-γ)·n nodes counterclockwise. In each node, read the data block or check block required to restore the file according to the index table; send a query request to the index table of each node in the high-performance list, lock the query range according to the file's unique identifier, and read the required data block and check block in the queried node position to improve reading efficiency and ensure data consistency and integrity.

[0139] During the reading phase, the hash value of the required data is directly matched in the index table of each node, rather than querying the data node location information through the central node. Data is read at the data node, which simplifies the query steps, improves the query hit rate, and prevents single point failure problems in the central node.

[0140] Please refer to Figure 4 , Figure 4 The distributed data storage device based on erasure code provided in the embodiment of the present application is shown. It should be understood that the device is similar to the above Figure 2Corresponding to the method embodiment, each step involved in the above method embodiment can be executed. The specific functions of the device can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in the memory in the form of software or firmware or solidified in the operating system of the device. The distributed data storage device based on erasure code includes: a data acquisition module 110, a first type of storage node selection module 120, a second type of storage node selection module 130 and a block data storage module 140.

[0141] The data acquisition module is configured to acquire n data blocks and check blocks to be stored, and acquire a dynamic parameter γ, wherein n is an integer greater than 1, γ is a number greater than zero and less than 1, and the result of γ·n is an integer.

[0142] The first type storage node selection module is configured to select γ·n nodes as the first type storage nodes according to the node identifiers of the nodes in the distributed system.

[0143] The second type storage node selection module is configured to select (1-γ)·n nodes as second type storage nodes from at least part of the remaining nodes included in the distributed system except the first type storage nodes according to node performance.

[0144] The block data storage module is configured to use each of the first-type storage nodes and the second-type storage nodes to store a check block or a data block respectively.

[0145] like Figure 5 As shown, in some embodiments of the present application, the device further includes: a data writing module and a data reading module.

[0146] The data writing module is mainly responsible for distributing and storing the data sent by the client to multiple nodes. The data writing module includes: Figure 6 As shown: write request receiving module, data slicing module, data encoding module, storage location selection module, storage location dynamic adjustment module, data update module. The write request receiving module includes: for obtaining the data write operation sent by the client, first storing the data in the buffer to avoid blocking the subsequent write operation execution, and providing complete data for subsequent data slicing.

[0147] The data slicing module includes: being used to slice the complete data N obtained by the request receiving module, the sliced ​​data size is set to s, and each stripe includes k data blocks. If the complete data size N cannot be evenly divided into M groups, each group has k data blocks of size s, then the last data block that does not meet the size s is padded with zero values ​​so as to perform the encoding operation normally. The data slicing module is connected to the data encoding module, and outputs the sliced ​​stripes to the data encoding module.

[0148] The data encoding module includes: using the encoding matrix generated by the set erasure code RS (n, k), calculating the data block to obtain the check block, each stripe includes k data blocks and nk check blocks.

[0149] The storage location selection module includes: when writing data for the first time, evaluating the CPU performance index and network performance index of the node, and dividing the node into three levels: low performance, medium performance and high performance.

[0150] The storage location selection module also includes: each node maintains a separate index table, which includes a file identifier, a data block or a check block hash value, an identifier marking a data block or a check block, a node unique identifier, and a data block or a check block storage location.

[0151] When selecting a storage location, the dynamic parameter γ is first determined. Its function is to store γ·n blocks (data blocks or check blocks) for the digital identification selection method and (1-γ)·n blocks (data blocks or check blocks) for the performance selection method.

[0152] Secondly, in the digital identification selection method (used to select the first type of storage nodes), the nodes with the first γ·n maximum prefix lengths are found on the ring, and a data block or a check block is stored on each node.

[0153] Next, in the performance selection method (for selecting the second type of storage nodes), (1-γ)·n nodes are randomly selected from the high-performance node list, and a data block or a check block is stored in each node.

[0154] After selecting a node and storing a data block or a check block in the above-mentioned digital identification selection method and performance selection method, the node index table is updated, including the file identifier, the data block or check block hash value, the identifier of the data block or check block, the node unique identifier, and the data block or check block storage location.

[0155] like Figure 7 or Figure 8As shown, the contents of the storage location dynamic adjustment module described in some embodiments of the present application include: a node indicator evaluation module, a data indicator evaluation module, and a comprehensive evaluation module. The node indicator evaluation module is used to evaluate the load indicators and usage indicators of different nodes, and to obtain a comprehensive score of the node by weight, and to screen out nodes that meet different needs; the data indicator evaluation module is used to evaluate the comprehensive indicators of the data, including data priority, data access times, and data importance indicators; the comprehensive evaluation module obtains a comprehensive load-balancing node selection strategy based on the node indicator evaluation module and the data indicator evaluation module, and dynamically adjusts the existing storage data. Figure 8 As shown, the load index includes: CPU load, disk load, memory load, network load, obtain the load index scores of these loads, and determine a score according to the weight coefficient; the utilization index exemplarily includes: CPU utilization, disk utilization, memory utilization, network bandwidth utilization, obtain the utilization index scores of these utilizations, and then obtain an evaluation value according to the corresponding weight coefficient; the data index evaluation module obtains the data index score through data priority, data access times and data importance, and then obtains an evaluation value according to the data index score and a weight coefficient value; finally, a total evaluation value is obtained through the comprehensive evaluation module, which is used to consider node allocation during dynamic switching. That is to say, the process of obtaining the comprehensive score of each node when determining the dynamic adjustment of storage nodes in some embodiments of the present application includes: determining the index system, calculating the module scores, determining the weight coefficients and calculating the comprehensive scores.

[0156] The following Tables 1, 2 and 3 list the relevant indicators respectively.

[0157] Table 1 shows the detailed description of the node load indicators

[0158]

[0159] Table 2 shows the detailed description of the node load indicators

[0160]

[0161] Table 3 shows the detailed description of the node load indicators

[0162]

[0163]

[0164] In some embodiments of the present application, when the data block and the check block are written for the first time, the storage location dynamic adjustment module is not involved. The data update module content includes: for obtaining the data block and the check block content from the data storage module located by the index query module, and incrementally updating them to new data block content and new check block content.

[0165] In some embodiments of the present application, the data reading module is mainly responsible for receiving the read request of the client, preferentially searching the required data storage location from the index linked list, then reading the required data from the storage location, and selecting whether to perform integrity check on the data in the optional configuration, and finally sending the data to the client. Fig. 9 exemplarily includes a read request receiving module, a data query module, a data decoding module, an integrity checking module and a data sending module.

[0166] The read request receiving module of the embodiment of the present application is used to obtain the data read operation sent by the client. The data query module is used to obtain the data block and the check block of the target data from the node. Optionally, if no data block fails, only the target data block is read; if some data blocks fail, the data block and the check block are read at the same time, and the data is restored by the data decoding module. The data decoding module is used to restore the data block to the original data using the decoding matrix. The integrity check module: Optionally, if the integrity check is selected, when no data block fails, the target data block and the check block are still read, and the integrity of the data block is verified by the check block. If the verification is passed, the target data is output to the client normally. If the verification is not passed, a data corruption warning is output. The integrity check module: Optionally, if the integrity check is not selected, when no data block fails, only the data block is read, and the data blocks are connected into the target data and then output to the client without performing the data integrity check. The optional integrity check module enables the data reading module to adapt to different scenarios. When data integrity check is selected, the reliability of the data is improved; when data check is not selected, the data reading steps are simplified and the reading efficiency is improved. The data sending module is used to output the target data requested by the client to the client.

[0167] Some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method described in any one of the embodiments of the method for distributed data storage based on erasure codes can be implemented.

[0168] like Fig.10 Some embodiments of the present application provide an electronic device 700, including a memory 710, a processor 720, and a computer program stored in the memory 710 and executable on the processor 720, wherein the processor 720 can implement the method described in any one of the embodiments of the above-mentioned method for distributed data storage based on erasure codes when reading and executing the program through a bus 730.

[0169] Processor 720 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 720 can be a microprocessor.

[0170] The memory 710 may be used to store instructions executed by the processor 720 or data related to the execution of instructions. These instructions and / or data may include code to implement some or all functions of one or more modules described in the embodiments of the present application. The processor 720 of the present embodiment may be used to execute the instructions in the memory 710 to implement Figure 2 The memory 710 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory or other memory known to those skilled in the art.

[0171] Some embodiments of the present application introduce an index table in the node, including a file identifier, a data block or a check block hash value, an identifier for marking a data block or a check block (i.e., an identifier for distinguishing whether a block belongs to a data block or a check block), a node unique identifier, a data block or a check block storage location, and other fields, which are used to identify the relationship between the file, the data block or the check block, and the storage node. Among them, the identifier of the marking data block or the check block can not only improve the maintainability and reliability of the system, but also provide convenience in the data update and recovery process. When the stored data needs to be updated, the identifier can quickly locate the original data block, so that only these blocks are updated without involving the check block. This can reduce unnecessary calculation and storage overhead. The site selection strategy for the first write of some embodiments of the present application, the strategy for selecting the storage node location when the data is first written, includes three parts: a digital identifier selection method, a performance selection method, and a dynamic parameter selection method. Not only the hierarchical information of the node unique identifier is used to optimize the path selection and improve the communication efficiency, but also by assigning more data processing tasks to high-performance nodes, the overall processing speed and responsiveness of the system are significantly improved. Some embodiments of the present application provide an optimized data update method, that is, when the data is not written for the first time, it is necessary to use a comprehensive indicator evaluation method to comprehensively evaluate the node load indicator, usage rate indicator and data priority indicator, access number indicator, and importance indicator to provide a strategy for dynamically adjusting storage nodes. An embodiment of the present application provides a distributed data storage system based on erasure codes, including the site selection strategy for the first write and the optimized data update method provided above, and integrated into the data writing module and the data reading module.

[0172] It is not difficult to understand that some embodiments of the present application improve the hit rate by introducing data block and check block identifiers, for example, using an index table to store relevant attribute information of data blocks and check blocks. Add data block and check block identifiers to the value of the index table to distinguish whether the storage node corresponding to the index table item stores a data block or a check block. Since the index table of the embodiment of the present application provides the ability to quickly locate the storage location of the data block, the required data can be found more efficiently, and the logic implementation is relatively simple. In the data reading stage, the function of directly reading the data block and the function of recovering the fault data according to the check block are provided to adapt to different scenarios and improve the reading efficiency. Some embodiments of the present application also provide a site selection strategy for the first write, and specifically design an optimized site selection strategy for the first write data, which improves the correlation between data and nodes and reduces randomness. Compared with the current method of selecting the storage location, the performance of the node is taken into account, and the node range of three levels of low performance, medium performance and high performance is divided. The data written for the first time is stored in the node with medium performance, which can make full use of the network performance and is conducive to the subsequent dynamic adjustment of the storage location according to the hot and cold degree of the data. At the same time, the site selection is carried out in a content hash-based manner, which avoids the problem of high dependence on centralized nodes and improves the read and write efficiency. Some embodiments of the present application also provide a comprehensive index evaluation method, that is, a data update method is provided. When the data is not written for the first time, the node load index, usage index and data priority index, access number index, and importance index are comprehensively evaluated according to the comprehensive index evaluation method, the balance of the storage location selection strategy is optimized, and the storage node is dynamically adjusted to improve the efficiency of the system. This method avoids the problem that the storage location cannot be changed once the data is stored in the node, and can be adjusted in time according to the load situation of the node and the hot data to adapt to the ever-changing data reading and writing process. Some embodiments of the present application also design a storage and update system, and specifically design a distributed data storage system based on erasure code, which combines the first write site selection strategy with the update strategy of the comprehensive index evaluation method, significantly improves the load balancing of the system's storage and update process, and makes full use of the system's bandwidth. The system can avoid the randomness of data storage node selection, select the storage location that best suits the data according to the actual situation, avoid the single point failure problem, and provide an optimized data storage and update strategy.

[0173] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0174] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0175] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0176] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0177] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0178] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for distributed data storage based on erasure coding, characterized in that: The method comprises: Obtain n data blocks and check blocks to be stored, and obtain a dynamic parameter γ, where n is an integer greater than 1, γ is a number greater than zero and less than 1, and the result of γ·n is an integer; Select γ·n nodes as the first type of storage nodes according to the node identifiers of each node in the distributed system; Selecting (1-γ)·n nodes as second-type storage nodes from at least a portion of the remaining nodes of the distributed system except the first-type storage nodes according to node performance; Each of the first-type storage nodes and the second-type storage nodes is used to store a check block or a data block respectively.

2. The method according to claim 1, characterized in that The step of selecting γ·n nodes as first-type storage nodes according to the node identifier of each node in the distributed system includes: Sorting each node in the distributed system on a ring according to the common prefix length of the node name; According to the file identifier of the data to be stored, γ·n nodes with a common prefix are searched on the ring as the first type of storage nodes.

3. The method according to claim 2, characterized in that The searching, according to the file identifier of the file to be stored, for γ·n nodes having a common prefix on the ring as the first type of storage nodes comprises: If the node on the ring has no common prefix with the file identifier, a direction is randomly selected to continue matching nodes until the node with the maximum prefix length of the first γ·n is found to obtain the first type of storage node; If the node on the ring has a common prefix length with the file identifier, the nodes are matched continuously in a clockwise direction until the node with the first γ·n largest prefix length is found to obtain the first type of storage node.

4. The method according to any one of claims 1 to 3, characterized in that The selecting (1-γ)·n nodes as second-type storage nodes from at least a portion of the remaining nodes included in the distributed system except the first-type storage nodes according to the node performance includes: Dividing the remaining nodes of the distributed system except the first type of storage nodes into high-performance nodes, medium-performance nodes and low-performance nodes according to performance; (1-γ)·n nodes are selected from the high-performance nodes as the second-type storage nodes.

5. The method according to claim 4, characterized in that The method further comprises: Calculating hash values ​​of the data block and the check block; Storing a hash value of a corresponding data block or a check block in an index table on each of the first-type storage nodes and the second-type storage nodes; The hash value is used to locate the data block or the check block when updating or repairing the data.

6. The method according to claim 1, characterized in that The method further comprises: In response to the operation of updating data, dynamically adjusting the nodes for storing the data to be updated according to the comprehensive evaluation index, wherein the comprehensive evaluation index is related to the node performance difference and the data block heat attribute, the node performance difference is related to the load index and the usage index of the corresponding node, and the data block heat index is related to the priority index, the access number index and the importance index of the data to be updated; Update the description information of each node after dynamic adjustment, wherein the description information includes: a file identifier, a data block or a check block hash value, an identifier for distinguishing between a data block and a check block, a node identifier or a storage location of the data block and the check block, the data block or the check block hash value is used to identify the data block or the check block stored by the corresponding node, the identifier for distinguishing between the data block or the check block is used to indicate whether the data stored by the corresponding node belongs to a data block or a check block, the node identifier is a node domain name or a set unique node name, and the storage location of the data block or the check block is used to record the specific location of the data stored on the corresponding node.

7. The method according to claim 5, characterized in that The method further comprises: The index table of each node in the first type of storage node and the second type of storage node stores an identifier for distinguishing whether the stored block belongs to a data block or a check block; After receiving the file reading request from the client, the following strategy is adopted to read the required data blocks and / or check blocks according to the read request attributes and the identifier used to distinguish the data blocks and check blocks: If it is confirmed that none of the data blocks to be read has any fault, then the corresponding data block is read through the identifier; If it is confirmed that the data block to be read is faulty, the data block and the check block are read simultaneously by the identifier, and the faulty data block is restored according to the read data; If it is confirmed that the data needs to be checked for integrity, all data blocks and check blocks used for the integrity check are read through the identifier; If it is confirmed that the data does not need to be checked for integrity, and when no failure occurs in the data blocks to be read, multiple data blocks are read through the identifier, and the multiple data blocks are connected into target data and then output to the client; Among them, the reading of all data blocks and check blocks for integrity verification through the identifier includes: after receiving a file reading request from the client, finding a node with a maximum prefix length with the read file name in the ring, and counterclockwise searching for the next (1-γ)·n nodes, and reading the data blocks or check blocks required for recovery in each node according to the hash value included in the index table; sending a query request to the index table of each node in the corresponding high-performance list, locking the query range according to the file unique identifier, and reading the required data blocks and check blocks in the queried node position.

8. A distributed data storage device based on erasure codes, characterized in that: The device comprises: A data acquisition module is configured to acquire n data blocks and check blocks to be stored, and acquire a dynamic parameter γ, wherein γ is a number greater than zero and less than 1, and a result of γ·n is an integer; A first-type storage node selection module is configured to select γ·n nodes as first-type storage nodes according to node identifiers of each node in the distributed system; A second-type storage node selection module is configured to select (1-γ)·n nodes as second-type storage nodes from at least part of the remaining nodes included in the distributed system except the first-type storage nodes according to node performance; The block data storage module is configured to use each of the first-type storage nodes and the second-type storage nodes to store a check block or a data block respectively.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 7 can be implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method described in any one of claims 1 to 7 can be implemented.

Images