Data storage method combining erasure code and consistent hash algorithm, and terminal device

Abstract

The invention discloses an erasure code and consistency hash algorithm combined data storage method and a terminal device, and relates to the technical field of data storage. The method comprises the following steps of: dividing a file into a plurality of file blocks, generating redundant blocks according to an erasure code algorithm, and marking the file blocks and the redundant blocks; and determining an initial storage node according to the hash value of the file, and storing the labeled data block in the initial node and continuous nodes behind the initial node. When the nodes are changed, a data migration strategy is used to ensure that the fault-tolerant capability of the system is not changed: when the nodes are increased, if the storage node sequence of the file is not continuous any more, the tail data block of the file is migrated to a new node; when the node is deleted, the data block in the node is migrated to the next node after the corresponding file storage node sequence; and modifying the data block corresponding to the file by adopting incremental updating. According to the method, the fault-tolerant capability and the storage efficiency of the consistent Hash algorithm are improved, the expandability of the system is ensured, and the method has small data updating and node addition and deletion maintenance bandwidth.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a data storage method and terminal device combining an erasure code with a consistent hash algorithm. Background Art

[0002] A distributed system is a system in which multiple computer nodes communicate and coordinate actions through message passing. It is applicable to all computer systems that can be networked. The original intention of creating this system is to allow ordinary users to use ordinary computers to complete large-scale storage and computing tasks that a single computer cannot handle. These computers may be widely distributed in space and may have different performances, but their common goal is to use more computers to process more data. From an architectural perspective, distributed file systems are mainly divided into two types: centralized and decentralized.

[0003] In decentralized storage systems, consistent hashing is a commonly used data placement strategy that works by constructing the entire hash value space into a continuous virtual ring, which not only maintains the high scalability of the system, but also reduces the amount of data migration when nodes change. In the consistent hashing algorithm, there are two main entities: data items (or requests) that need to be stored and server nodes, which are located in a virtual ring structure (i.e., a hash ring). Server nodes can be regarded as tags to which one or more requests can be mapped. Unlike a fixed number of servers, a hash ring has an unlimited number of points, allowing server nodes to be located anywhere on the ring. Through the hash function, the server's IP address or other identifier can be converted into a hash value and its position can be determined on the hash ring.

[0004] In a distributed storage environment, data may be lost or become inaccessible due to unexpected situations such as hard disk failure and server crash, which is unacceptable to users. In order to ensure the security and accessibility of data, distributed storage systems must implement some strategies to avoid the risk of data loss. A basic and effective method is to replicate data, that is, to implement a multi-copy strategy. It automatically replicates data and creates multiple copies and distributes them to different nodes. As long as not all copies become unavailable at the same time, the security and accessibility of the data can be maintained. Although the consistent hashing algorithm combined with the multi-copy storage mechanism has strong scalability and fault tolerance, and improves the availability of data, it also affects the performance of the storage system. The specific limitations are as follows:

[0005] 1. The consistent hashing algorithm based on the replica fault tolerance strategy has low storage efficiency. If a file corresponds to N replica files, the storage efficiency of the multi-copy storage mechanism is The storage system must provide (N+1) times the cost to tolerate N failures. Therefore, the consistent hashing algorithm based on the replication fault tolerance strategy requires a lot of additional space, which leads to additional storage costs.

[0006] 2. The consistent hashing algorithm based on the replica fault tolerance strategy has a large amount of data migration. In the actual storage environment, the addition and deletion of nodes are very common. In the consistent hashing algorithm, when the system nodes change, part of the data will be migrated. Since there are copies of files on multiple nodes, the consistent hashing algorithm combined with the multi-copy storage mechanism will result in a large amount of data migration when adding and deleting nodes, which will cause additional burden on the system.

[0007] Compared with the multi-copy strategy, under the same data redundancy, erasure coding can achieve better fault tolerance performance, so it shows great potential in distributed systems. The data placement strategy currently used in distributed file systems is mainly designed based on multi-copy storage. Although there are also solutions that combine erasure coding and consistent hashing algorithms in the existing technology, these solutions all have defects to varying degrees. The details are as follows:

[0008] 1. Existing solutions use the method of grouping and numbering data streams to meet the MDS (Maximum Distance Separable) property of RS codes (Reed-Solomon codes), and propose a multi-version hash ring, that is, when a new node is added, a new hash ring is built to deal with the situation when a new node is added. This solution lacks a response strategy when deleting a node, a method for updating data content, etc. In addition, in actual systems, frequently building new hash rings will require the system to maintain more linked lists, making the system complex and redundant;

[0009] 2. Existing solutions have proposed grouping nodes and converting the mapping of data blocks to nodes into the mapping process of stripes to node groups to reduce the amount of data migration during node changes. However, this solution needs to compare the node group to be added during node changes with all original node groups to find the pairing with the smallest difference. This process requires multiple traversals and is inefficient.

[0010] Therefore, there is an urgent need for an effective method to address the above-mentioned shortcomings of combining replica or erasure code fault tolerance strategies with consistent hashing algorithms. Summary of the invention

[0011] In order to overcome the technical problems of additional storage space and storage cost brought by the existing replica fault tolerance strategy, as well as the huge data migration burden, the present invention provides a data storage method and terminal device combining erasure code and consistent hash algorithm. The many technical effects that can be produced by the preferred technical solution among the many technical solutions provided by the present invention are described in detail below. To achieve the above purpose, the present invention provides the following technical solutions:

[0012] The present invention provides a new data storage method combining erasure code and consistent hash algorithm, comprising the following steps:

[0013] The received file is divided into multiple file blocks, and corresponding redundant blocks are generated according to the multiple file blocks according to the erasure code algorithm; each of the file blocks and the redundant block is respectively labeled according to the labeling format of "name of the file + serial number + tail block or non-tail block" to obtain labeled data blocks; the starting storage node is determined on the hash ring according to the hash value of the file, and the labeled data blocks are stored in the starting storage node and subsequent consecutive storage nodes in sequence according to the set direction to obtain the storage node sequence and data block sequence corresponding to the file.

[0014] In some embodiments, the data storage method combining erasure coding with consistent hashing algorithm further includes adding a new storage node on the hash ring, and when the migration condition is met, migrating the data blocks of the data block sequence corresponding to the file, including the following steps:

[0015] Migrate the last data block in the data block sequence corresponding to the file to the new storage node to obtain a new storage node sequence corresponding to the file; when the new storage node is not the tail node of the new storage node sequence corresponding to the file, modify the tail identifier of the data block migrated to the new storage node to a non-tail block, and modify the tail identifier of the data block on the tail node of the new storage node sequence to a tail block.

[0016] In some embodiments, the data storage method combining erasure coding with consistent hashing algorithm further includes deleting a storage node on the hash ring, and when a migration condition is met, migrating data blocks of a data block sequence corresponding to the file, including the following steps:

[0017] The data blocks stored in the deleted storage node are migrated to the next storage node of the last storage node in the storage node sequence corresponding to the file; when the deleted storage node is not the tail node of the storage node sequence corresponding to the file, the tail identifier of the data block migrated to the next storage node is modified to the tail block.

[0018] In some embodiments, the migration condition is that the hash value of the storage node to be changed is not less than the hash value of the file name, and is not greater than the hash value of the tail node of the storage node sequence corresponding to the file.

[0019] In some embodiments, the data storage method combining erasure coding with consistent hashing algorithm further includes querying the data blocks stored in the hash ring, including the following steps:

[0020] According to the hash value of the file to be queried, the storage node sequence corresponding to the file to be queried on the hash ring is determined; for the queried storage node sequence, the corresponding data block is extracted according to the serial number identifier of the data block stored therein; if the extraction of the data block fails, the original file of the file to be queried is reconstructed by decoding the erasure code, and the corresponding data block is extracted from the reconstructed original file according to the serial number identifier of the data block.

[0021] In some embodiments, the data storage method combining erasure coding with consistent hashing algorithm further includes modifying the file blocks of the data block sequence corresponding to the file using an incremental update method, wherein the steps include:

[0022] The position of the file to be modified on the hash ring is located according to the hash value of the file to be modified, and the corresponding original file block is extracted from the data block sequence of the file to be modified according to the sequence number of the data block of the file to be modified; the updated data block corresponding to the extracted original file block is obtained, and the increments of the original file block and the updated file block are calculated; the calculated increment is sent to the storage node storing the redundant block in the storage node sequence corresponding to the file to be modified; according to the original redundant block corresponding to the file to be modified and the calculated increment, the updated redundant block is calculated by the incremental redundancy calculation formula, and the updated redundant block is stored in the corresponding redundant block storage node.

[0023] The incremental redundancy calculation formula is as follows:

[0024]

[0025] Among them, P′ j is the jth updated redundant block, P j is the original redundant block corresponding to the jth updated redundant block, ΔD k is the increment of the kth file block, G ji is the encoding value of the j-th row and i-th column of the generator matrix G, m is the total number of file blocks of the file to be modified, and n is the total number of redundant blocks of the file to be modified.

[0026] In some embodiments, the storage node further includes a virtual storage node, and determining a starting storage node on a hash ring according to a hash value of the file includes the following steps:

[0027] Marking virtual storage nodes and real storage nodes corresponding to the virtual nodes on the hash ring, calculating the hash value of the file, and determining the position of the file name of the file on the hash ring according to the calculated hash value;

[0028] Starting from the determined position, the first storage node is searched according to the set direction. If the first storage node found is not a virtual storage node, the first storage node found is the starting storage node; if the first storage node found is a virtual storage node, the real storage node corresponding to the virtual storage node is used as the starting storage node.

[0029] Based on the same inventive concept, in the second aspect, the present invention provides a terminal device for implementing the data storage method combining an erasure code with a consistent hashing algorithm as described above, comprising an input and output unit, a data block storage unit, a storage node change unit, a data block query unit, a data block modification unit and a communication unit, wherein the input and output unit is connected to the data block storage unit, the storage node change unit, the data block query unit and the data block modification unit, and the data block storage unit, the storage node change unit, the data block query unit and the data block modification unit are connected to the storage nodes on the hash ring via the communication unit.

[0030] The data block storage unit receives a storage file instruction and a corresponding storage file from the input-output unit, divides the received file into a plurality of file blocks, generates redundant blocks according to the divided file blocks, and stores the file blocks and redundant blocks on a hash ring; the storage node change unit receives an add or delete storage node instruction and corresponding add or delete storage node information from the input-output unit, adds or deletes a storage node on the hash ring according to the add or delete storage node information and the corresponding instruction, and performs data migration on a storage node sequence where the added or deleted storage node is located when a migration condition is met; the data block query unit receives a query file instruction and corresponding file query information from the input-output unit, and queries the data blocks stored on the hash ring according to the file query information and the corresponding instruction; the data block modification unit receives a modify file instruction and a corresponding modify file from the input-output unit, and modifies the file blocks of the data block sequence corresponding to the file by an incremental update method according to the modify file and the corresponding instruction; the file storage results, storage node change results, file query results, and file modification results corresponding to the data block storage unit, the storage node change unit, the data block query unit, and the data block modification unit are displayed through the input-output unit.

[0031] Based on the same inventive concept, in a third aspect, the present invention provides a terminal device, the terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the data storage method combining the erasure code with the consistent hash algorithm as described above are implemented;

[0032] Alternatively, the terminal device is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. When the program is executed by a processor, the steps of the data storage method combining the erasure code and the consistent hash algorithm as described above are implemented.

[0033] Implementing one of the above technical solutions of the present invention has the following advantages or beneficial effects:

[0034] The present invention applies the erasure code storage fault-tolerant strategy in the consistent hashing algorithm, and m data blocks generate n redundant blocks. At the same time, based on the consistent hashing algorithm, a data placement algorithm is further designed, and the file blocks and redundant blocks corresponding to the same file are placed on several consecutive nodes, so that the files are divided into blocks and placed on different real nodes to meet the MDS properties of the erasure code and the needs of the erasure code strategy, thereby greatly improving the storage efficiency. The concept of "end data block" is proposed to facilitate file query and rapid positioning when migrating data blocks, and facilitate rapid positioning and effective management of data. Therefore, the solution of the present invention is comprehensive in design, basically covering the situations that may occur in actual distributed storage systems, and the rule setting is concise and efficient.

[0035] This method does not require complex algorithms, but only involves relatively basic operations such as calculation, judgment, and migration. In addition, when a storage node changes, there is no need to traverse and compare the entire system. Instead, it is only necessary to find data blocks with specific tags and migrate them according to the migration conditions, thereby saving resources and avoiding the situation where the data in the entire hash ring is "flowing" when the storage node changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. It is obvious that the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0037] Figure 1 It is a flow chart of a data storage method combining erasure code and consistent hash algorithm according to an embodiment of the present invention;

[0038] Figure 2 is a schematic diagram of the storage of data blocks (without considering virtual nodes) according to an embodiment of the present invention;

[0039] Figure 3 is a schematic diagram of storage of data blocks (taking virtual nodes into consideration) according to an embodiment of the present invention;

[0040] Figure 4 It is a schematic diagram of the migration process (top) and migration result (bottom) of a data block when a new storage node is added to a hash ring according to an embodiment of the present invention;

[0041] Figure 5 It is a schematic diagram of the migration process (top) and migration result (bottom) of a data block when a storage node is deleted on a hash ring according to an embodiment of the present invention;

[0042] Figure 6 A schematic diagram of the structure of a terminal device according to an embodiment of the present invention;

[0043] Figure 7 A schematic diagram of another terminal device structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, wherein various exemplary embodiments that may be used to implement the present invention are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods, devices, etc. that are consistent with some aspects of the present disclosure as detailed in the attached claims, and other embodiments may also be used, or the embodiments listed herein may be modified in structure and function without departing from the scope and essence of the present invention.

[0045] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The terms "plurality" and "several" mean two or more. The terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0046] In order to illustrate the technical solution of the present invention, a specific embodiment is used below for description, and only the parts related to the embodiment of the present invention are shown.

[0047] Embodiment 1: A data storage method combining erasure code and consistent hash algorithm provided by the present invention, such as Figure 1 As shown, the following steps are included:

[0048] S100, dividing the received file into a plurality of file blocks, and generating corresponding redundant blocks according to the plurality of file blocks according to an erasure coding algorithm;

[0049] S200, marking each file block and redundant block according to the marking format of "name of the file + sequence number + tail block or non-tail block" to obtain marked data blocks;

[0050] S300. Determine the starting storage node on the hash ring according to the hash value of the file, and store the marked data blocks in the starting storage node and subsequent consecutive storage nodes in sequence according to the set direction to obtain a storage node sequence and a data block sequence corresponding to the file.

[0051] It should be noted that the erasure code algorithm can use RS (m, n) code, that is, m data blocks generate n redundant blocks, and both the erasure code and the replica mechanism have n redundancies. In this case, the storage efficiency of the system under the erasure code strategy is The storage efficiency of the replica strategy is A simple calculation shows that when m is greater than 1, the storage efficiency of the erasure coding scheme is higher than that of the multi-copy mechanism, which greatly improves the storage efficiency of existing data.

[0052] Furthermore, the data storage method sets a label for each data block (including redundant blocks) and proposes the concept of "tail data block", that is, the "tail block or non-tail block" mentioned above, so as to facilitate file query and rapid positioning when migrating data blocks. For the convenience of specific program design, "tail block or non-tail block" can be replaced by Y or N. That is, each data block will be assigned a unique label, which includes a file name identifier, a sequence number identifier, and a tail block identifier. The specific format is "file name + sequence number + Y / N (indicating whether it is the last data block)", where the label containing "Y" is the tail data block, indicating that it is the data block at the last node in the node sequence stored in the file. It is the main object of data migration when the node changes, and the tail data block is dynamically updated with the status of the file and the storage node. This label facilitates rapid positioning and effective management of data.

[0053] The data storage method of this embodiment applies the erasure code storage fault-tolerant strategy in the consistent hashing algorithm, and m data blocks generate n redundant blocks. At the same time, based on the consistent hashing algorithm, a data placement algorithm is further designed to place the file blocks and redundant blocks corresponding to the same file on several consecutive nodes, so that the files are divided into blocks and placed on different real nodes to meet the MDS properties of the erasure code and the needs of the erasure code strategy, thereby greatly improving the storage efficiency. The concept of "end data block" is proposed to facilitate file query and rapid positioning when migrating data blocks, and facilitate rapid positioning and effective management of data (such as extracting and querying data). Therefore, the scheme of the present invention is comprehensive in design, basically covering the situations that may occur in actual distributed storage systems, and the rule setting is concise and efficient.

[0054] Furthermore, multiple storage nodes are mapped on the hash ring. The hash ring can be formed by using a string hash algorithm to calculate the name of the storage node to obtain a corresponding number, and then the number is used to calculate the storage node name. 32 Modulo, get the hash value of the storage node. Arrange each storage node to form a hash ring according to the size of the hash value and the set direction. The set direction can be clockwise or counterclockwise.

[0055] In the above step S200, the specification of the RS erasure code is selected as (m, n), that is, m file blocks and n redundant blocks. The file to be stored (the received file described above) is evenly divided into m parts, and n redundant blocks are obtained according to the generation method of the RS erasure code. The file blocks and redundant blocks are collectively referred to as data blocks. These m+n data blocks are arranged, with m file blocks in front and n redundant blocks in the back, and are serially numbered according to their positions, that is, the first data block corresponding to the file is numbered 1, the second data block is numbered 2, and so on. The last data block is numbered m+n. At the same time, if the data block is the last data block, a mark "Y" is added, otherwise a mark "N" is added. In turn, each data block has a label of "file name + serial number + Y / N (tail mark)".

[0056] The storage node also includes a virtual storage node. In step S300, determining the starting storage node on the hash ring according to the hash value of the file includes the following steps:

[0057] The virtual storage nodes and the real storage nodes corresponding to the virtual nodes are marked on the hash ring, the hash value of the file is calculated, and the position of the file name on the hash ring is determined according to the calculated hash value; starting from the determined position, the first storage node is searched in the set direction, if the first storage node is not a virtual storage node, the first storage node is the starting storage node; if the first storage node is a virtual storage node, the real storage node corresponding to the virtual storage node is used as the starting storage node. The method of calculating the hash value of the file can be consistent with the method of calculating the hash value of the storage node.

[0058] It should be noted that in order to make the load of each storage node as balanced as possible, virtual nodes are used for adjustment in this embodiment.

[0059] As a specific example, the storage node is a server, and a hash algorithm is used to map K servers to a hash ring. Each server has its corresponding name. A currently public string hash algorithm, such as FNVHash, is selected to calculate the server name to obtain a number, and then the obtained number is used for 2 32 Modulo, get the final hash value and the position of the hash value on the hash ring. This position is the (logical) storage node mentioned above. Select the point with a hash value of 0 on the ring as the starting point, and name these storage nodes "node-1", "node-2", ..., "node-K" in a clockwise direction. Each storage node name corresponds to a server name. According to the rules of the consistent hashing algorithm, for storage node "node-j" (j = 1, 2, ...,

[0060] For node K), data with hash values ​​between "node-j-1" (not included) and "node-j" (inclusive) are all stored on "node-j" ("node-0" is defined as "node-K"), that is, the management scope of the storage node "node-j" is from "node-j-1" (not included) to "node-j" (inclusive).

[0061] After receiving the file to be stored, it is divided into blocks, named and stored. The RS (m, n) code (m+n<K) is used to divide the file into m parts to form file blocks. These m file blocks obtain n redundant blocks by generating a matrix. If the name of the file to be stored is "file", then these m+n data blocks are named "file-1-N", "file-2-N", "file-3-N", ..., "file-m+nY" in the form of "file name-number-N / Y" labels, where the data blocks numbered 1 to m are file blocks, and the data blocks numbered m+1 to m+n are redundant blocks. "N" indicates that the data block is not the last data block corresponding to the file, and "Y" indicates that the data block is the last data block of the file.

[0062] Calculate the hash value of the file (file name). The specific method is the same as calculating the hash value of the server name. Check which storage node the hash value is within the management scope of. Take the server corresponding to the storage node as the starting point, and place the data blocks in the server and the subsequent m+n consecutive servers in clockwise order according to the numbering sequence. At this time, the last data block, that is, the data block with "Y" in its name, is placed at the last one in this group of servers.

[0063] like Figure 2 As shown, the received file name is "file", and the RS (2, 2) code is used to divide the file into two parts, and two redundant blocks are obtained by the generator matrix. The four data blocks are named "file-1-N", "file-2-N", "file-3-N", and "file-4-Y". Among them, "file-1-N" and "file-2-N" are file blocks, "file-3-N" and "file-4-Y" are redundant blocks, and "file-4-Y" is the last data block. There are five storage nodes in the hash ring, namely "node-1", "node-2", ..., "node-5". If the hash value of the file "file" calculated by the hash algorithm used is 200, and the management range of "node-2" is 129 to 256, then the storage starting point of the file will belong to "node-2", and the four data blocks "file-1-N", "file-2-N", "file-3-N", and "file-4-Y" will be stored in the four consecutive storage nodes in the clockwise direction, namely "node-2", "node-3", "node-4", and "node-5" in order of numbering.

[0064] like Figure 3As shown, in the case of virtual nodes, first mark the virtual nodes and real nodes on the hash ring, then calculate the hash value of the file name to be stored and get its position on the hash ring, start from this position, and look for the first node clockwise. If it is an actual node, use this node as the starting point for storage; if it is a virtual node, use the real node corresponding to this virtual node as the starting point for storage. Furthermore, if the hash value of the file "file" calculated by the hash algorithm used is 200, the management range of "node-2" is 129 to 256, and if there are no other nodes between 200 and 256, then the file will use "node-2" as the storage node; if there are other nodes between 200 and 256, and the first node in the clockwise direction after 200 is a virtual node corresponding to "node-3", then the file will use "node-3" as the storage starting point, and the four data blocks "file-1-N", "file-2-N", "file-3-N", and "file-4-Y" will be stored in sequence in the four consecutive storage nodes in the clockwise direction, namely "node-3", "node-4", "node-5", and "node-1", in order of numbering.

[0065] In this embodiment, a data storage method combining erasure coding with a consistent hash algorithm further includes adding a new storage node on the hash ring, and when a migration condition is met, migrating data blocks of a data block sequence corresponding to a file, including the following steps:

[0066] Migrate the last data block in the data block sequence corresponding to the file to the new storage node to obtain a new storage node sequence corresponding to the file;

[0067] When the new storage node is not the tail node of the new storage node sequence corresponding to the file, the tail mark of the data block migrated to the new storage node is modified to a non-tail block, and the tail mark of the data block on the tail node of the new storage node sequence is modified to a tail block. Of course, when the new storage node is at the tail of the new storage node sequence corresponding to the file, the mark of the data block migrated to the new storage node is maintained without modification.

[0068] Among them, one migration condition is set as: the hash value of the storage node to be changed is not less than the hash value of the file name of the received file (i.e. the corresponding stored file), and is not greater than the hash value of the tail storage node of the storage node sequence corresponding to the file. The storage node to be changed refers to adding a new storage node on the hash ring.

[0069] like Figure 4As shown, as a specific example, the hash ring is set to the back in the clockwise direction and the front in the counterclockwise direction. When the hash position of the newly added storage node is located after the position of the file name hash of the received file on the ring, and before the last node of the continuous m+n storage nodes where the data block corresponding to the received file is stored, the data block located at the last node (that is, the data block marked as "Y" at the end) is migrated to the newly added node, and the "Y" in the label is changed to "N", and the label of the last data block in all the data blocks corresponding to the received file is changed to "Y" to complete the status update. In this way, the continuity of the data blocks and the balance of the data block storage are ensured, while the efficiency of data storage and retrieval is optimized.

[0070] Specifically, if the storage node is a server, the file name of the received file is "file", the hash value of the file is located at the point "file" on the hash ring, and the data blocks corresponding to the file are stored in the order of numbering on the four consecutive server nodes in the clockwise direction, namely "node-2", "node-3", "node-4", and "node-5". If the storage node "node-6" corresponding to the newly added server is located after "file" and before "node-5", the last data block "file-4-Y" in "node-1" is migrated to "node-6". If the data block is still at the end at this time, the tail mark "Y" is retained; otherwise, the tail mark is changed from "Y" to "N", indicating that the data block is no longer the last data block, and the tail mark of the last data block in the data block sequence corresponding to the file at this time is changed from "N" to "Y".

[0071] In this embodiment, a data storage method combining erasure coding with a consistent hash algorithm further includes deleting a storage node on a hash ring, and when a migration condition is met, migrating data blocks of a data block sequence corresponding to a file, including the following steps:

[0072] Migrate the data blocks stored in the deleted storage node to the next storage node of the last storage node in the storage node sequence corresponding to the file;

[0073] When the deleted storage node is not the tail node of the storage node sequence corresponding to the file, the tail mark of the data block migrated to the next storage node is modified to the tail block. Of course, when the deleted storage node is the last storage node in the storage node sequence corresponding to the file, the mark of the data block migrated to the next storage node is maintained without modification.

[0074] Among them, one migration condition is set as: the hash value of the storage node to be changed is not less than the hash value of the file name of the received file (i.e. the corresponding stored file), and is not greater than the hash value of the tail storage node of the storage node sequence corresponding to the file. The storage node to be changed refers to the storage node to be deleted on the hash ring.

[0075] like Figure 5 As shown, as a specific example, when a storage server is removed from the hash ring, or a storage server fails, the data in the server needs to be migrated. As specified, the clockwise direction on the hash ring represents the "backward" direction, and the counterclockwise direction represents the "forward" direction. If the hash value of the removed or failed storage node is located after the hash value of a certain accepted file on the ring (if virtual nodes are considered, it is after the first storage node in the storage node sequence corresponding to the file), and the data blocks of the file are located within the range of m+n nodes stored continuously on the ring, including the possible last node.

[0076] Still taking the file named "file" as mentioned above, the data blocks corresponding to the file are stored in "node-2", "node-3", "node-4", and "node-5" in sequence according to the numbering order, if the removed or invalid node is one of them. For example, if "node-4" fails, "file-2-N" stored on "node-4" is transferred to the next node after "node-5", that is, "node-1", and the tail identifier of "file-2-N" is modified to "Y", indicating that the data block has become the last data block in the sequence. At the same time, the tail identifier of the original last data block "file-4-Y" stored on "node-5" is modified to "N". Through the above steps, it can be ensured that when a node is deleted or a node fails, the storage and identification of the data block can be correctly updated to maintain the consistency and data integrity when the method deletes the node.

[0077] It is understandable that the data storage method of this embodiment applies the erasure code storage fault tolerance strategy in the consistent hash algorithm, which realizes the problem that the storage system maintains the fault tolerance capability unchanged after adding and deleting storage nodes. The consistent hash algorithm currently used in distributed file systems has an existing fault tolerance strategy that uses multiple copies. If the fault tolerance strategy is changed to an erasure code with higher storage efficiency, it must be ensured that the original fault tolerance capability of the erasure code remains unchanged after adding and deleting storage nodes.

[0078] In this embodiment, a data storage method combining erasure coding with a consistent hashing algorithm further includes querying a data block stored in a hash ring, including the following steps:

[0079] According to the hash value of the file to be queried, determine the storage node sequence corresponding to the file to be queried on the hash ring;

[0080] For the queried storage node sequence, the corresponding data block is extracted according to the serial number identifier of the data block stored therein;

[0081] If the data block extraction fails, the original file of the file to be queried is reconstructed by erasure code decoding, and the corresponding data block is extracted from the reconstructed original file according to the serial number identifier of the data block.

[0082] When you need to query the data of a file, determine which file the data to be queried is located in, and find the m+n continuous storage nodes where the corresponding data block is stored according to the hash value of the file name. If you know which file block the content to be queried is in, you can find the file block to be found according to the serial number in the label of each data block; if you cannot determine the specific file block, then the erasure code is decoded to restore the original file and query it. Through this step, it can be ensured that even when the data is stored in a dispersed manner, the required data can be effectively located and queried. This method not only improves the accuracy of data retrieval, but also enhances the fault tolerance of this method and ensures the integrity of the data.

[0083] It should be noted that reconstructing the original file of the to-be-queried file by decoding the erasure code can be achieved by using the existing technology, which will not be described in detail here.

[0084] In this embodiment, a data storage method combining erasure coding and consistent hashing algorithm is characterized in that it also includes modifying the file block of the data block sequence corresponding to the file by using an incremental update method, and the steps include:

[0085] Locate the position of the file to be modified on the hash ring according to the hash value of the file to be modified, and extract the corresponding original file block from the data block sequence of the file to be modified according to the sequence number of the file block of the file to be modified;

[0086] Obtaining the updated file block corresponding to the extracted original file block, and calculating the increments of the original file block and the updated file block;

[0087] Send the calculated increment to the storage node storing the redundant block in the storage node sequence corresponding to the file to be modified;

[0088] According to the original redundant block corresponding to the file to be modified and the calculated increment, the updated redundant block is calculated by the incremental redundancy calculation formula, and the updated redundant block is stored in the corresponding redundant block storage node.

[0089] It should be noted that calculating the increments between the original file block and the updated file block includes but is not limited to difference operations.

[0090] Furthermore, the incremental redundancy calculation formula is as follows:

[0091]

[0092] Among them, P′ j is the jth updated redundant block, P j is the original redundant block corresponding to the jth updated redundant block, ΔD k is the increment of the kth file block, G ji is the encoding value of the j-th row and i-th column of the generator matrix G, m is the total number of file blocks of the file to be modified, and n is the total number of redundant blocks of the file to be modified.

[0093] Furthermore, the received file is divided into m blocks D i , where i=1, 2...m, the coding matrix is ​​B, and B is a matrix of m+n rows and m columns.

[0094]

[0095] in,

[0096] There are n redundant blocks.

[0097] When a file block is modified, such as file block D1, the erasure coding process and result are:

[0098]

[0099] In actual operation, it is not necessary to modify every part of the erasure code encoding result. You can refer to the above formula to make targeted modifications to the generated m+n data blocks. i The total modification amount is Specifically, let G = [G ji ] n×m Represents the coding coefficient matrix, G and an m×m identity matrix form a generation matrix (the matrix B above). The identity matrix ensures that the data slices can be directly stored in the device. The generation of redundant blocks can be written as:

[0100] (P1,P2,…,P n ) T =G×(D1,D2,…,D m ) T (4)

[0101] If the client updates file block D k For D' k , the increment is ΔD k , the updated redundant block is P' j , then we get formula (1) using the incremental update method.

[0102] As a specific example, when a client sends a request to update the i-th file block in a specific stripe in the storage system, the specific steps are as follows:

[0103] Read the original file shard: First, you need to locate the storage node that stores the i-th file block and read the original file block from the storage node, denoted as D i ;

[0104] Get updated data: Get the updated file block sent by the client, denoted as D' i ;

[0105] Calculate the update increment: D will be used i and D' i To calculate the update increment, denoted as ΔD i . This delta represents the change from the original file block to the new file block.

[0106] Send update increment: calculated update increment ΔD i Will be sent to all storage nodes responsible for storing redundant blocks.

[0107] Reading original redundant blocks: On all storage nodes responsible for storing redundant blocks, the original redundant blocks corresponding to the updated files will be read out.

[0108] Calculate the new redundant block: According to the predetermined formula (1), all storage nodes responsible for storing redundant blocks will use the original redundant block P j and update increment ΔD i To calculate the new redundant block.

[0109] Persistence of new redundant blocks: After the calculation is completed, the new redundant blocks will be written to the storage medium to ensure the persistence of the data.

[0110] Persistence of updated file blocks: The updated file blocks will be persistently stored, the entire update process will be completed, and the client will be responded to.

[0111] In summary, compared with the multi-copy mechanism or the simple erasure coding strategy, the method combining erasure coding with the consistent hashing algorithm improves many system performances:

[0112] 1. Higher storage efficiency: Under the multi-copy scheme and the combined erasure coding scheme, if the same number of redundancies is used, set as n, the storage efficiency of the system under the multi-copy mechanism is Combined with the erasure code solution, the system storage efficiency is A simple calculation shows that when m is greater than 1, the storage efficiency combined with the erasure code scheme is higher than the multi-copy mechanism.

[0113] 2. When adding storage nodes, the amount of data migration is smaller: In theory, assuming that the location of each file on the hash ring is evenly distributed, the total size of the files to be stored is d, there are K nodes in total, and the added node is located between two existing nodes, then when the system uses RS (m, n) erasure code, the amount of data migration is When the three-copy strategy is used for storage, the amount of data migration is Furthermore, if the n+1 copy strategy is used for storage, the amount of data migration is At this time, the storage efficiency ratio of the two Substituting r into the data migration amount of the system using RS (m, n) erasure code is It is calculated that when r>1, when adding storage nodes, the amount of data migration of the system using the erasure code strategy for storage is less than the migration amount using the n+1 copy strategy.

[0114] 3. Improved data fault tolerance and security: Compared with traditional replication strategies, the utilization rate of multiple hard disks on a single server is enhanced. By applying erasure coding technology to divide files into blocks, the security of stored files is ensured. Even if some of the stored data blocks fail, the original files can be restored to a certain extent. RS (m, n) code can tolerate the failure of up to n data blocks. Compared with solutions with the same storage efficiency, this method provides stronger fault tolerance.

[0115] 4. The incremental update method reduces the amount of modification during the update and saves resources. Compared with the re-encoding method, the incremental update method can reduce the amount of modification of file blocks and redundant blocks, save a lot of network bandwidth and hard disk I / O resources, and greatly reduce the delay of update requests of the erasure code storage system.

[0116] The method of this embodiment does not require complex algorithms, but only involves relatively basic operations such as calculation, judgment, and migration. In addition, when a storage node changes, there is no need to traverse and compare the entire system, but only to find data blocks with specific tags for migration, thereby saving resources and avoiding the situation where the data in the entire hash ring is "flowing" when the storage node changes.

[0117] Embodiment 2: Based on the same inventive concept, the second embodiment of the present invention further provides a terminal device for implementing the data storage method combining erasure code and consistent hash algorithm described in embodiment 1. Figure 6As shown, it includes an input-output unit 801, a data block storage unit 802, a storage node change unit 803, a data block query unit 804, a data block modification unit 805, and a communication unit 806. Specifically, the input-output unit 801 is connected to the data block storage unit 802, the storage node change unit 803, the data block query unit 804, and the data block modification unit 805, and the data block storage unit 802, the storage node change unit 803, the data block query unit 804, and the data block modification unit 805 are connected to the storage nodes on the hash ring through the communication unit 806.

[0118] Furthermore, the data block storage unit 802 receives the file storage instruction and the corresponding storage file from the input / output unit 801, divides the received file into multiple file blocks, generates redundant blocks according to the divided file blocks, and stores the file blocks and redundant blocks on the hash ring, that is, implementing steps S100-S300 in the first embodiment.

[0119] The storage node change unit 803 receives the instruction to add or delete storage nodes and the corresponding information to add or delete storage nodes from the input / output unit 801, and according to the instruction to add or delete storage nodes and the information to add or delete storage nodes, adds a new storage node on the hash ring or deletes a storage node on the hash ring, and performs data migration in the storage node sequence where the added or deleted storage node is located when the migration condition is met. The steps related to "deleting a storage node on the hash ring, and migrating data blocks of a data block sequence corresponding to a file when the migration condition is met" and "adding a new storage node on the hash ring, and migrating data blocks of a data block sequence corresponding to a file when the migration condition is met" in the first embodiment are implemented.

[0120] The data block query unit 804 receives the query file instruction and the corresponding file query information from the input / output unit 801, and queries the data blocks stored on the hash ring according to the query file instruction and the file query information, thereby implementing the steps related to "querying the data blocks stored on the hash ring" in the first embodiment.

[0121] The data block modification unit 805 receives the file modification instruction and the corresponding modification file from the input / output unit 801, and modifies the file blocks of the data block sequence corresponding to the file using the incremental update method according to the file modification instruction and the modification file, thereby implementing the steps related to "modifying the file blocks of the data block sequence corresponding to the file using the incremental update method" in the first embodiment.

[0122] As an implementation mode, the input-output unit 801 includes an instruction transmission module and a data transmission module. The instruction input module is used to receive input file storage instructions, file modification instructions, file query instructions, and storage node change instructions, and transmit the instructions to the corresponding processing unit. The data transmission module is used to transmit the received input file to the corresponding processing unit, and display the file storage results, storage node change results, file query results, and file modification results corresponding to the data block storage unit, storage node change unit, data block query unit, and data block modification unit.

[0123] Embodiment 3: Based on the same inventive concept, the third embodiment of the present invention also provides another terminal device. Figure 7 As shown, it includes a memory 901, a processor 902, and a computer program stored in the memory 901 and executable on the processor 902. When the processor 902 executes the program, the steps of the data storage method combining the erasure code and the consistent hash algorithm are implemented.

[0124] Among them, the bus architecture in the figure (represented by bus 906), bus 906 may include any number of interconnected buses and bridges, and bus 906 links various circuits including one or more processors represented by processor 902 and memory represented by memory 901. Bus 906 can also link various other circuits such as peripherals, voltage regulators, and power management circuits together, which are all well known in the art, so they are not further described herein. Bus interface 903 provides an interface between bus 906 and receiver 904 and transmitter 905. Receiver 904 and transmitter 905 can be the same element, namely a transceiver, which provides a unit for communicating with various other devices on a transmission medium. Processor 902 is responsible for managing bus 906 and general processing, while memory 901 can be used to store data used by processor 902 when performing operations.

[0125] Embodiment 4: Based on the same inventive concept, the fourth embodiment of the present invention further provides another terminal device, which is a computer-readable storage medium on which a computer program is stored. When the program is executed by the processor, the steps of the data storage method combining the erasure code and the consistent hash algorithm described above are implemented.

[0126] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0128] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0129] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0130] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flow involved in the above embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0131] The above description is only the preferred embodiment of the present invention. It is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the protection scope of the present invention.

Claims

1. A data storage method combining erasure code and consistent hash algorithm, characterized in that: The steps include: Dividing the received file into a plurality of file blocks, and generating corresponding redundant blocks according to the plurality of file blocks according to an erasure coding algorithm; Marking each of the file blocks and the redundant blocks according to the marking format of "name of the file to which it belongs + sequence number + tail block or non-tail block" to obtain marked data blocks; The starting storage node is determined on the hash ring according to the hash value of the file, and the marked data blocks are stored in the starting storage node and subsequent consecutive storage nodes in sequence according to the set direction to obtain the storage node sequence and data block sequence corresponding to the file.

2. According to claim 1, a data storage method combining erasure code and consistent hash algorithm is characterized in that: The method further includes adding a new storage node on the hash ring, and migrating data blocks of the data block sequence corresponding to the file when a migration condition is met, including the following steps: Migrating the last data block in the data block sequence corresponding to the file to the new storage node to obtain a new storage node sequence corresponding to the file; When the new storage node is not the tail node of the new storage node sequence corresponding to the file, the tail identifier of the data block migrated to the new storage node is modified to a non-tail block, and the tail identifier of the data block on the tail node of the new storage node sequence is modified to a tail block.

3. According to claim 1, a data storage method combining erasure code and consistent hash algorithm is characterized in that: The method further includes deleting a storage node on the hash ring, and migrating data blocks of a data block sequence corresponding to the file when a migration condition is met, including the following steps: Migrating the data block stored in the deleted storage node to the next storage node of the last storage node in the storage node sequence corresponding to the file; When the deleted storage node is not the tail node of the storage node sequence corresponding to the file, the tail identifier of the data block migrated to the next storage node is modified to the tail block.

4. A data storage method combining erasure code and consistent hash algorithm according to claim 2 or 3, characterized in that: The migration condition is that the hash value of the storage node to be changed is not less than the hash value of the file name, and is not greater than the hash value of the tail node of the storage node sequence corresponding to the file.

5. According to claim 1, a data storage method combining erasure code and consistent hash algorithm is characterized in that: The method further includes querying the data blocks stored in the hash ring, including the following steps: Determine, according to the hash value of the file to be queried, a storage node sequence corresponding to the file to be queried on the hash ring; For the queried storage node sequence, the corresponding data block is extracted according to the serial number identifier of the data block stored therein; If the data block extraction fails, the original file of the file to be queried is reconstructed by erasure code decoding, and the corresponding data block is extracted from the reconstructed original file according to the serial number identifier of the data block.

6. According to claim 1, a data storage method combining erasure code and consistent hash algorithm is characterized in that: The method further includes modifying the file block of the data block sequence corresponding to the file by using an incremental update method, wherein the steps include: According to the hash value of the file to be modified, locate the position of the file to be modified on the hash ring; according to the sequence number of the data block of the file to be modified, extract the corresponding original file block from the data block sequence of the file to be modified; Obtaining the updated data block corresponding to the extracted original file block, and calculating the increments of the original file block and the updated file block; Send the calculated increment to the storage node storing the redundant block in the storage node sequence corresponding to the file to be modified; According to the original redundant block corresponding to the file to be modified and the calculated increment, the updated redundant block is calculated by the incremental redundancy calculation formula, and the updated redundant block is stored in the corresponding redundant block storage node.

7. The data storage method combining erasure code and consistent hash algorithm according to claim 6, characterized in that: The incremental redundancy calculation formula is as follows: Among them, P ′ j is the jth updated redundant block, P j is the original redundant block corresponding to the jth updated redundant block, ΔD k is the increment of the kth file block, G ji is the encoding value of the j-th row and i-th column of the generator matrix G, m is the total number of file blocks of the file to be modified, and n is the total number of redundant blocks of the file to be modified.

8. The data storage method combining erasure code and consistent hash algorithm according to claim 1, characterized in that: The storage node also includes a virtual storage node, and determining a starting storage node on the hash ring according to the hash value of the file includes the following steps: Marking virtual storage nodes and real storage nodes corresponding to the virtual nodes on the hash ring, calculating a hash value of the file, and determining a position of the file on the hash ring according to the calculated hash value; Starting from the determined position, searching for a first storage node in the set direction, if the first storage node to be searched is not a virtual storage node, the first storage node to be searched is the starting storage node; If the first storage node to be found is a virtual storage node, the real storage node corresponding to the virtual storage node is used as the starting storage node.

9. A terminal device, used to implement a data storage method combining an erasure code and a consistent hash algorithm as described in any one of claims 1 to 8, comprising an input-output unit, a data block storage unit, a storage node change unit, a data block query unit, a data block modification unit and a communication unit, wherein the input-output unit is connected to the data block storage unit, the storage node change unit, the data block query unit and the data block modification unit, and the data block storage unit, the storage node change unit, the data block query unit and the data block modification unit are connected to the storage nodes on the hash ring through the communication unit; The data block storage unit receives a storage file instruction and a corresponding storage file from the input and output unit, divides the received file into a plurality of file blocks, generates redundant blocks according to the divided file blocks, and stores the file blocks and the redundant blocks on a hash ring; The storage node change unit receives an instruction to add or delete a storage node and corresponding information to add or delete a storage node from the input / output unit, adds or deletes a storage node on the hash ring according to the information to add or delete a storage node and the corresponding instruction, and performs data migration on the storage node sequence where the added or deleted storage node is located when the migration condition is met; the data block query unit receives an instruction to query a file and corresponding file query information from the input / output unit, and queries the data blocks stored on the hash ring according to the file query information and the corresponding instruction; the data block modification unit receives an instruction to modify a file and a corresponding modification file from the input / output unit, and modifies the file blocks of the data block sequence corresponding to the file by an incremental update method according to the modification file and the corresponding instruction; The file storage results, storage node change results, file query results and file modification results corresponding to the data block storage unit, storage node change unit, data block query unit and data block modification unit are displayed through the input and output unit.

10. A terminal device, characterized in that: The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the data storage method combining the erasure code and the consistent hash algorithm as claimed in any one of claims 1 to 8 are implemented; Or, the terminal device is a computer-readable storage medium, and a computer program is stored on the readable storage medium, which, when executed by a processor, implements the steps of the data storage method combining the erasure code and the consistent hash algorithm as described in any one of claims 1 to 8.

Images