Ultra-large capacity data storage erasure code redundancy device and method

Abstract

The invention provides an ultra-large-capacity data storage erasure code redundancy device. The ultra-large-capacity data storage erasure code redundancy device comprises a plurality of system nodes and at least two check nodes, each system node comprises a plurality of system data blocks, and the check node comprises a first check data block and a second check data block; the stored data are separated in the form of stripes, and each stripe comprises a data block in each system node and a data block in each check node; the system data block is used for storing original data, the first check data block is used for storing a first summation function corresponding to the original data in a stripe, and the second check data block is used for storing a second summation function corresponding to the original data in the same stripe and summing a plurality of additional data; the additional data are corresponding original data in other strips; and when any data block is damaged, determining a recovery mode according to the damaged position. According to the invention, the number of storage nodes required for optimizing erasure coding is reduced, so that better performance is provided in the aspect of bandwidth.

Description

Technical Field

[0001] The present invention belongs to the technical field of erasure coding, and relates to an ultra-large capacity data storage erasure coding redundancy device and method for path tracking problems. Background Art

[0002] Erasure codes are forward error correction codes used for data protection in storage systems. Erasure codes divide data into k data blocks and generate m redundant blocks, forming a total of n = k + m blocks, which are stored on different storage nodes. Even if up to m blocks are lost or damaged, the original data can still be recovered from the remaining blocks. System-level erasure codes are erasure codes that convert the input original information encoding. The converted message includes redundant check data stored in different locations, which can be used to recover the lost data. Maximum distance separable erasure codes can recover the original data from k blocks out of n data blocks.

[0003] With the advancement of storage systems, erasure coding technology is also constantly improving. In particular, the emergence of distributed storage systems has promoted the development of new erasure coding technologies. When using erasure codes to recover data in distributed storage, data transmission is required on some nodes, which brings bandwidth issues. Therefore, erasure codes need to be improved urgently to meet the increasingly important status quo of distributed storage. Summary of the invention

[0004] Aiming at the problem that the existing erasure coding technology needs to be transmitted on nodes in distributed storage, this paper provides a solution of improved erasure coding technology, which reduces the number of storage nodes required for optimizing erasure coding, thereby providing better performance in terms of bandwidth.

[0005] The present invention discloses an ultra-large capacity data storage erasure code redundancy device, characterized in that the ultra-large capacity data storage erasure code redundancy device comprises a plurality of system nodes and at least two check nodes;

[0006] Each system node includes a plurality of system data blocks, and the check node includes a first check data block and a second check data block;

[0007] The stored data is divided into stripes, each stripe includes a data block in each system node and each check node;

[0008] The system data block is used to store original data, the first check data block stores a first sum function corresponding to the original data in the stripe, and the second check data block stores a second sum function corresponding to the original data in the same stripe and sums with a plurality of additional data; the additional data is the original data corresponding to other stripes;

[0009] When any data block is damaged, the original data recovery method is determined based on the damaged location and the sum function.

[0010] Furthermore, the check node further includes a first check data block, and the first check data block is used to store a first summation function corresponding to the original data in the same stripe.

[0011] Furthermore, the number of stripes is:

[0012]

[0013] Where w is the number of stripes, k ′ is the number of system data blocks, n is the number of data nodes, and r is the number of check nodes;

[0014] The total amount of extra data in all stripes is:

[0015] e t =(w×n)×(d-1)

[0016] e t is the total amount of additional data, w is the number of stripes, and d represents the data duplication;

[0017] The amount of extra data in the check node containing extra data is:

[0018] e sw =e t / (r-1)

[0019] e sw is the amount of additional data stored by the check nodes storing additional data, and r represents the number of check nodes.

[0020] Furthermore, the number of system data blocks as additional data in each stripe is:

[0021]

[0022] Among them, e t is the total amount of additional data, e c The number of system data blocks in each stripe as additional data;

[0023] The number of system data blocks as additional data on each system data node is:

[0024]

[0025] e r is the number of system data blocks used as additional data on each system data node, and n is the number of system data nodes.

[0026] Furthermore, the ultra-large capacity data storage erasure code redundancy device includes a plurality of computing nodes and a plurality of storage nodes;

[0027] The storage node includes at least a non-volatile memory or a combination of a volatile memory and a non-volatile memory for storing system data; the computing node is configured to receive access commands from a client node, access all storage nodes to read data stored in the storage nodes, and write data to one or more specified parts stored in the storage nodes.

[0028] Furthermore, each of the computing nodes is used to store the check data in one or more stripes in the storage node.

[0029] A method for erasure code redundancy of ultra-large capacity data storage is also provided, and the method for erasure code redundancy of ultra-large capacity data storage comprises the following steps:

[0030] Step S210, identifying a system data block for erasure coding;

[0031] The system data includes system data blocks in one or more stripes, and also includes system data blocks that have been stored in one or more stripes;

[0032] Step S220, generating check data for each stripe based on the identified erasure coded data;

[0033] The check data includes one or more check data blocks, each check data block being a function of a system data block in the same stripe and / or one or more additional data in other stripes;

[0034] In step S230, the verification data is stored on the storage node.

[0035] Furthermore, the ultra-large capacity data storage erasure code redundancy method comprises the following steps:

[0036] Step S240: When a storage node fails, data of the failed storage node can be restored based on the distributed data.

[0037] Furthermore, in step S220, the plurality of check data blocks use different summation functions to represent functions of corresponding system data blocks and additional data blocks.

[0038] Furthermore, in step S220, the number of the additional data blocks is determined according to the set data repetition degree;

[0039] e t =(w×n)×(d-1)

[0040] e tis the total number of additional data blocks, w is the number of stripes, n is the number of data nodes, and d represents the data duplication.

[0041] The beneficial effects achieved by the present invention are:

[0042] The present invention provides an erasure coding technology that reduces the amount of data that needs to be distributed to nodes in a distributed storage system. Specifically, this technology reduces the number of stripes and the number of stripes that need to ensure sufficient redundancy, and the amount of redundant data that needs to be stored is also reduced. Because the number of data blocks that need to be read is reduced, the bandwidth required for erasure coding during reconstruction is optimized. Therefore, the case allows erasure coding to be performed on a large number of storage nodes, while significantly reducing storage overhead.

[0043] Compared with the solutions in the prior art, the solution provided by the present invention achieves stronger fault tolerance by adjusting the data and calculation method in the check nodes without increasing the number of check nodes and slightly increasing the calculation overhead. In the prior art, in order to achieve the improvement of fault tolerance, it is usually necessary to significantly increase the number of check nodes to achieve the same redundancy effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Example network diagram of an erasure coding redundancy device for storing very large amounts of data.

[0045] Figure 2 Flowchart of the erasure coding redundancy algorithm for storing very large data volumes.

[0046] Figure 3 An example diagram of a computing node in the erasure coding redundancy algorithm for ultra-large-capacity data storage. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with specific embodiments, and the advantages and features of the present invention will become clearer as the description proceeds. However, these embodiments are exemplary only and do not constitute any limitation to the scope of the present invention. It should be understood by those skilled in the art that the details and forms of the technical solution of the present invention may be modified or replaced without departing from the spirit and scope of the present invention, but these modifications and replacements all fall within the scope of protection of the present invention.

[0048] Embodiment 1

[0049] Existing solutions for erasure coding in distributed storage systems need to ensure a sufficient number of check blocks to optimize bandwidth usage. For example, taking the Reed-Solomon (RS) erasure coding solution as an example, when data is distributed on 150 drives, a typical configuration requires adding 50 check blocks.

[0050] Therefore, providing an optimal erasure coding scheme is impractical for most real-world solutions. Embodiments herein provide a technique that reduces the number of storage nodes required to implement erasure coding, thereby providing better performance in terms of storage and bandwidth.

[0051] Various disclosed embodiments include techniques for improving erasure coding in distributed storage systems. The disclosed embodiments utilize a mirroring scheme implemented by allocating data block groups among data nodes. Each data node includes a storage node, such as (but not limited to) a disk or a drive. More specifically, the data nodes include a system node storing system data and a check node storing check data. The data block group includes data blocks stored in the system node, the data blocks used to store the system node are system data blocks, and the data blocks used to store the check node are check data blocks, and the check data blocks are functions of part of the system data blocks.

[0052] Each data block is a portion of data, including one block or multiple other minimum storage data units.

[0053] More specifically, the data in the storage nodes are distributed in the form of stripes. Each stripe includes data blocks distributed in the system nodes and the check nodes. Each stripe further includes one or more rows. Each row is a group of data blocks located in multiple nodes. As a non-limiting example, when the storage node includes 4 system nodes and 2 check nodes, a row may include a data block stored in each node, a total of 6 data blocks, and a stripe includes one of the rows.

[0054] The data block in the check node is called the verification data block, also known as the summation data block, which is a function of the system data block and one or more additional data in the same stripe. The additional data refers to the system data block stored in different stripes. The number of additional data that the summation data block depends on needs to be determined according to the number of check nodes and stripes used.

[0055] Table 1 shows an example data block layout for describing summed data as described herein.

[0056] Table 1

[0057]

[0058] In Table 1, data is distributed in storage nodes including system nodes (SN) 0 to 3 and check nodes (PN) 0 and 1. Each column of Table 1 corresponds to a storage node, and each row of Table 1 corresponds to a stripe. Therefore, Table 1 shows a stripe containing 4. System nodes 0 to 3 include system data blocks a 0,0 to a 3,3Functions f and g are both summation functions. In one embodiment, f and g are different functions. In one example implementation, f and g are functions based on Reed-Solomon error correction codes.

[0059] In Table 1, check node 0 contains data blocks that are functions of the systematic data in its respective stripe. In one example implementation, function f is the sum of the systematic data blocks in the same stripe. For example, the data blocks stored in check node 0 belonging to the first stripe are stored as f(a 0,0 ,a 0,1 ,a 0,2 ,a 0,3 ).

[0060] In Table 1, check node 1 contains summed data blocks that are a function of the system data blocks in their respective stripes and additional data in one or more other stripes. For example, the data blocks stored in check node 1 belonging to the first stripe are stored as In one example implementation, function g is the sum of the system data blocks in the same stripe, and operation ⊕ is the sum of the additional data (e.g., for the first stripe of check node 1, the summed data block is the additional data a 3,0 and a 1,2 More specifically, in one example implementation, these operations are sums over a Galois field.

[0061] Storing checksum data as a function of the additional data in other stripes allows reconstruction while reducing the amount of redundant data required. For example, when a system node 0 fails, the data in check node 1 can be reconstructed by subtracting g(a 0,0 ,a 0,1 ,a 0,2 ,a 0,3 ) and a 1,2 To recover data block a 3,0 .

[0062] For example, when a 0,0 When a data block fails, it can be 0,0 ,a 0,1 ,a 0,2 ,a 0,3 ) and a 0,1 ,a 0,2 ,a 0,3 To recover data block a 0,0 , when data block a 0,0 and a 0,1 When a data block fails, it can be 0,0 ,a 0,1 ,a 0,2 ,a 0,3 )、g(a 0,0,a 0,1 ,a 0,2 ,a 0,3 ), a 0,2 、a 0,3 、a 1,2 、a 3,0 To recover. Or, through g(a 2,0 ,a 2,1 ,a 2,2 ,a 2,3 ) and a 3,2 Restore a 0,1 , and then through f(a 0,0 ,a 0,1 ,a 0,2 ,a 0,3 ) and a 0,1 ,a 0,2 ,a 0,3 Restore a 0,0 .

[0063] Furthermore, when a 0,0 , a 0,1 , a 0,2 , a 0,3 When all of them fail, the data can still be recovered because a 0,1 , a 0,3 data.

[0064] Obviously, this recovery capability is not unlimited, but considering that it is an obvious waste to consider extreme cases under limited costs, this solution is still an effective means to ensure fault tolerance. In comparison, if a general erasure code solution wants to have enough capacity to recover 4 nodes, it needs to set up 4 check nodes in a stripe.

[0065] Furthermore, in order to fully utilize the advantages of this solution, the system data blocks, which are the source of additional data, should be distributed as evenly as possible. On the one hand, this ensures that different stripes have consistent fault tolerance performance, and on the other hand, it avoids damage to storage caused by regional failures.

[0066] Each data block in a group has a role, where the role of the data block corresponds to its relative position within the group. Every data block in a group has the same role as every data block with the same relative position in other groups. Table 2 shows an example data block layout for describing a data block group.

[0067] Table 2

[0068]

[0069] In Table 2, the layout in Table 1 is further expanded, and system nodes 4 to 7 are added. Functions f and g are summation functions, which are the same as the functions described in Table 1, and are functions of the system data blocks in their respective stripes. For example, f0 is the function f(a 0,0 ,a 0,1 ,a 0,2 ,a 0,3 ,a 0,4 ,a 0,5 ,a 0,6 ,a 0,7 ), g0 is the function g(a 0,0 ,a 0,1 ,a 0,2 ,a 0,3 ,a 0,4 ,a 0,5 ,a 0,6 ,a 0,7 ). The summed data block stored in check node 1 is also a function of one or more additional data. In Table 2, check node 1 in the first stripe, in addition to the additional data used by check node 1 in the first stripe in Table 1, also contains a 2,4 and a 1,6 The sum of .

[0070] In one embodiment, the mirroring scheme defines groups of 4 system data blocks. Assume that the first group includes a 0,0 to a 0,3 , the second group includes a 0,4 to a 0,7 The relative position in the group can be expressed as the first to fourth position (for example, a 0,0 is located in the first position of the group, and a 0,3 The first position corresponds to the first system node, and the fourth position corresponds to the fourth system node. 0,0 and a 0,4 Both are in the first position of their respective groups and therefore have the same role.

[0071] It is worth noting that there is a special case, that is, when a 1,2 and a 1,6 When the backup mechanism fails, it will lose its function because the calculation result of pn1 of the first strip is This does not help with data recovery (a 1,2 , a 1,6 can still be restored based on the second strip), further, when a 1,2 , a 1,6 and a 1,3 / a 1,7When the data fails, it cannot be restored. At this time, the information in the second strip is not enough to restore the data, and the backup mechanism is 1,2 and a 1,6 At the same time, failure also makes it useless.

[0072] Furthermore, when the number of redundant data blocks increases, the recovery capability will decrease. For example, when a 0,2 When (any data block that has not been used as the source of additional data blocks) is also backed up in the fourth stripe PN1, the external data blocks provided are a 0,6 , The only valid information is a 0,6 , at most three failed nodes can be rebuilt.

[0073] Therefore, taking Table 1 as an example, this embodiment has a small probability of not being able to rebuild three nodes, and further adding redundant data blocks has a high probability of not being able to rebuild three failed nodes. In particular, if a more complex calculation method is adopted, this problem can be solved, but this will bring significant overhead (increasing the overhead each time a node fails). Therefore, for the comprehensive consideration of efficiency and fault tolerance, we recommend adopting this implementation plan.

[0074] In a specific embodiment, the additional data does not include all system data blocks. More specifically, in the following embodiments, the setting of the system redundancy is related to the number of stripes, and the system data blocks as additional data are positively correlated with the data redundancy requirement.

[0075] In a specific embodiment, the number of stripes used is based on the number of system data nodes and check nodes, and the specific formula is as follows:

[0076]

[0077] Where w is the number of stripes, k ′ is the number of system data blocks, n is the number of data nodes, and r is the number of check nodes. The number of system data blocks needs to be determined based on actual storage requirements and the amount of data nodes, while the amount of data nodes and the number of data nodes need to consider actual application requirements. Reducing the amount of data and the number of data nodes can significantly improve the recovery speed when errors occur, but it will increase storage overhead, so it needs to be weighed according to actual needs. The number of check nodes is generally 2. When a larger value is taken, the storage overhead will increase, but the recovery speed can be improved to a certain extent.

[0078] The amount of data stored in the check node is determined according to the data duplication. The data duplication in Table 1 and Table 2 is 1.5, which means that the additional data contained in a stripe is 0.5*system data blocks.

[0079] In particular,

[0080] e t =(w×n)×(d-1)

[0081] e t is the total number of additional data blocks, w is the number of stripes, and d represents the data duplication;

[0082] Then the amount of extra data in the check node containing extra data is:

[0083] e sw =e t / (r-1)

[0084] e sw is the number of additional data blocks stored by the check nodes storing additional data, e t is the total number of additional data blocks, and r represents the number of check nodes. For example, as shown in Table 2, e sw Equal to 4 (ie, check node 1 is the only check node storing additional data, containing 4 additional data, which are averagely derived from other stripes).

[0085] The extra data in a stripe comes from other stripes. For the stability and robustness of the method, the extra data sources should be evenly distributed. An optional approach is given below.

[0086]

[0087]

[0088] Among them, e t is the total amount of additional data, e c is the number of system data blocks in each stripe as additional data, e r is the number of system data blocks used as additional data on each system data node, w is the number of stripes, and n is the number of system data nodes.

[0089] The case provides an erasure coding technique that reduces the amount of data that needs to be distributed to nodes in a distributed storage system. Specifically, this technique reduces the number of stripes and the number of stripes that need to ensure sufficient redundancy, and therefore, the amount of redundant data that needs to be stored is also reduced. Finally, because the number of data blocks that need to be read is reduced, the bandwidth required for erasure coding during reconstruction is optimized. Therefore, the case allows erasure coding to be performed on a large number of storage nodes while significantly reducing storage overhead.

[0090] Compared with the solutions in the prior art, the solution provided by the present invention achieves stronger fault tolerance by adjusting the data and calculation method in the check nodes without increasing the number of check nodes and slightly increasing the calculation overhead. In the prior art, in order to achieve the improvement of fault tolerance, it is usually necessary to significantly increase the number of check nodes.

[0091] Embodiment 2

[0092] Figure 1 The network diagram 100 is used as an example of a distributed storage system to describe a practical case. The network diagram 100 includes a distributed storage system 130, a network 110, and client nodes 120-1 to 120-p (collectively referred to as client nodes 120 for simplicity).

[0093] The distributed storage system 130 includes computing nodes 132 - 1 through 132 - g (collectively referred to as computing nodes 132 for simplicity) and storage nodes 134 - 1 through 134 - m (collectively referred to as storage nodes 134 for simplicity).

[0094] The network 110 may be (but not limited to) a wireless, cellular or wired network, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), the Internet, the World Wide Web (WWW), similar networks, and any combination thereof. The client node 120 may be (but not limited to) a server, a personal computer, a laptop, a tablet computer, a smartphone, or any other device configured to store data, access data, or both.

[0095] The storage node 134 includes at least a non-volatile memory (e.g., Flash, 3D Xpoint), and may be implemented by a combination of volatile (e.g., RAM) and non-volatile memory. The non-volatile memory may be included in, for example, a hard disk drive (HDD), a solid-state drive (SSD), or a combination thereof. In the case, each storage node 134 includes one or more drives including a disk.

[0096] Node 132 is configured to access data in storage node 134. In some implementations, node 132 may be configured as a distributed cooperative transaction manager to access data in storage node 134. An example schematic diagram of node 132 is described below. Figure 3 Described in.

[0097] Client node 120 is configured to send access commands to distributed storage system 130 via network 110. Node 132 is configured to receive access commands from client node 120 and access storage 134 according to the received commands. Accessing may include converting the received commands into a format supported by storage node 134.

[0098] In one embodiment, each node 132 can access all storage nodes 134. In a further example implementation, each node 132 is configured to read data stored in any storage node 134 and is configured to write data to one or more designated portions stored in a storage node 134.

[0099] In one embodiment, the nodes 132 store erasure coded data in the storage nodes 134 using the mirroring scheme described herein. To this end, the nodes 132 are responsible for storing one or more data sets (e.g., stripes or rows) for erasure coding in the storage nodes 134, and maintaining a mapping of each of its erasure coded data sets to any free user data locations in these data sets in the storage nodes 134. Each node 132 may also maintain a record of whether each erasure coded data set allocated contains free user data. The mapping and indication data are stored in the corresponding node 132 or storage node 134.

[0100] In one embodiment, each node 132 is configured to distribute data using the erasure coding scheme herein, which reduces space usage and optimizes the locality of erasure coded stripes. Erasure coded data, including system data and parity data, is divided into data blocks and stored on storage nodes 134. In one example implementation, the erasure coded data is distributed by stripes, such that each stripe includes data blocks stored in multiple storage nodes.

[0101] Each stripe is distributed across n storage nodes and contains one or more rows. Each row includes a data block on each of the n storage nodes. To this end, for each row, its system data block is distributed across k system nodes (each system node stores a block of system data for that row). A check data block is generated and distributed across r check nodes, providing redundancy for the system data such that nk=r.

[0102] In an embodiment, data is distributed in each row according to a mirroring scheme. The mirroring scheme is based on the grouping of data blocks in the row. Each data block has a role corresponding to its relative position in the group, so that data blocks with the same role have the same relative position in their respective groups. The check data in each row includes one or more data blocks that are functions of the data in the row, and one or more data blocks that are functions of the data blocks in the row and additional data in one or more other rows.

[0103] The amount of extra data used may vary between check data blocks (eg, the first check data block contains no extra data, while the second check data block is a function of 4 extra data). The total amount of extra data required may be based on data repetition requirements.

[0104] It should be noted that the embodiments described herein are not limited to Figure 1 For example, the storage system to which the disclosed erasure coding technology can be applied is not limited to the specific configuration of the distributed storage system 130. In particular, a storage system that uses a single computing node to manage multiple storage nodes can also use the improved erasure coding technology described in this article, which is also one of the contents of this article.

[0105] Embodiment 3

[0106] Figure 2 is an example flow chart illustrating how free space can be used to improve the locality of erasure codes according to one embodiment. In one embodiment, Figure 1 The middle computing node 132 may be configured to execute the method for corresponding rows containing erasure coded data and check data, so that multiple computing nodes control data distribution according to a mirroring scheme.

[0107] Step S210: Identify system data for erasure coding. The identified data includes system data blocks to be included in one or more rows, and may also include system data blocks already stored in one or more rows.

[0108] Step S220, based on the identified erasure coded data, generate check data for each row. The check data includes one or more check data blocks, which are functions of the system data in the same row. The check data also includes one or more check data block sets, each block is a function of the system data in the same row and one or more additional data in other rows. In general, in this embodiment, step S220 includes calculating the check data based on the identified erasure coded data.

[0109] In step S230, data is stored on the storage node. The stored data includes the generated verification data and may include at least a portion of the identified system data (when the system data needs to be stored). The system data blocks within the group are distributed according to the mirroring scheme described herein. The system data blocks have roles corresponding to their relative positions in their respective groups, and the mirroring scheme defines the groups so that system data blocks in the same relative position in their respective groups have the same roles.

[0110] In optional step S240, when a storage node fails, data of the failed storage node can be restored based on the distributed data. In an implementation example, one or more data blocks stored in the failed storage node can be restored by subtracting one or more additional data from the check data as a function of the additional data.

[0111] Embodiment 4

[0112] Figure 3is a schematic diagram of an implementation of a computing node 132. The node 132 includes a processing circuit 310, a memory 320, a storage device 330, and a network interface 340, and communicates via a bus 350.

[0113] The processing circuit 310 may be implemented by one or more hardware logic components and circuits. The types of hardware logic components that may be used include, but are not limited to, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), general-purpose microprocessors, microcontrollers, digital signal processors (DSPs), etc., or any other hardware logic components capable of performing information calculations or other operations.

[0114] Memory 320 may be volatile (eg, RAM, etc.), non-volatile (eg, ROM, flash memory, NVRAM, etc.), or a combination of both. Storage 320 stores computer-readable instructions of the embodiments herein.

[0115] In another embodiment, memory 320 is configured to store software. The definition of software should be broadly understood as any type of instructions, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Instructions may include code (e.g., in source code format, binary code format, executable code format, or any other suitable code format). When these instructions are executed by processing circuit 310, processing circuit 310 is caused to perform the various processes described herein.

[0116] The storage device 330 may be magnetic storage, optical storage, etc., and may be implemented as a flash memory, a CD-ROM, a digital versatile disc (DVD), or any other medium that can store the required information.

[0117] Network interface 340 allows computing node 120 to receive access commands and communicate with computing node 120 through, for example, network 110 (see Figure 1 ) to send data.

[0118] For simplicity, the embodiments in this article are discussed with respect to stripes and rows distributed on storage nodes (such as disks). The embodiments are not limited to Figure 3 Although the specific architecture shown in the figure is not intended, other architectures can be used and should be included in the scope of this article.

[0119] The embodiments herein can be implemented as hardware, firmware, software or any combination thereof. In addition, the software is preferably implemented as an application on a program storage unit or a computer-readable medium composed of a specific device, which can be uploaded to and executed by a machine of any suitable architecture. The embodiment is preferably implemented on a platform with hardware, such as one or more central processing units ("CPU"), memory and input / output interface, and the computer platform also includes an operating system and microinstruction code. The various processes and functions described herein can be part of the microinstruction code, or part of the application program, or any combination thereof, and can be executed by the CPU. In addition, other peripheral devices can be connected to the computer platform, such as additional data storage devices and printing devices.

[0120] The above are only specific steps of the present invention and do not constitute any limitation to the protection scope of the present invention; any technical solutions formed by equivalent transformation or equivalent replacement shall fall within the protection scope of the present invention; some parts not elaborated in detail in the present invention belong to the known technologies of those skilled in the art.

Claims

1. An erasure code redundancy device for storing ultra-large capacity data, characterized in that: The ultra-large capacity data storage erasure code redundancy device comprises a plurality of system nodes and at least two check nodes; Each system node includes a plurality of system data blocks, and the check node includes a first check data block and a second check data block; The stored data is divided into stripes, each stripe includes a data block in each system node and each check node; The system data block is used to store original data, the first check data block stores a first sum function corresponding to the original data in the stripe, and the second check data block stores a second sum function corresponding to the original data in the same stripe and sums with a plurality of additional data; the additional data is the original data corresponding to other stripes; When any data block is damaged, the original data recovery method is determined based on the damaged location and the sum function.

2. The ultra-large capacity data storage erasure code redundancy device according to claim 1, characterized in that: The check node further includes a first check data block, where the first check data block is used to store a first summation function corresponding to the original data in the same stripe.

3. The ultra-large capacity data storage erasure code redundancy device according to claim 1, characterized in that: The number of strips is: Where w is the number of stripes, k ′ is the number of system data blocks, n is the number of data nodes, and r is the number of check nodes; The total amount of extra data in all stripes is: e t =(w×n)×(d-1) e t is the total amount of additional data, w is the number of stripes, and d represents the data duplication; The amount of extra data in the check node containing extra data is: And sw =and t / (r-1) e sw is the amount of additional data stored by the check nodes storing additional data, and r represents the number of check nodes.

4. The ultra-large capacity data storage erasure code redundancy device according to claim 3, characterized in that: The number of system data blocks as additional data in each stripe is: Among them, e t is the total amount of additional data, e c The number of system data blocks in each stripe as additional data; The number of system data blocks as additional data on each system data node is: e r is the number of system data blocks used as additional data on each system data node, and n is the number of system data nodes.

5. The ultra-large capacity data storage erasure code redundancy device according to claim 1, characterized in that: The ultra-large capacity data storage erasure code redundancy device comprises a plurality of computing nodes (132) and a plurality of storage nodes (134); The storage node (134) includes at least a non-volatile memory or a combination of a volatile memory and a non-volatile memory for storing system data; the computing node (132) is configured to receive an access command from a client node, access all storage nodes (134) to read data stored in the storage nodes (134), and write data to one or more specified parts stored in the storage nodes (134).

6. The ultra-large capacity data storage erasure code redundancy device according to claim 5, characterized in that: Each of the computing nodes (132) is used to store the check data in one or more stripes in the storage node (134).

7. A method for erasure code redundancy of ultra-large capacity data storage based on the ultra-large capacity data storage erasure code redundancy device according to any one of claims 1 to 6, characterized in that: The ultra-large capacity data storage erasure code redundancy method comprises the following steps: Step S210, identifying a system data block for erasure coding; The system data includes system data blocks in one or more stripes, and also includes system data blocks that have been stored in one or more stripes; Step S220, generating check data for each stripe based on the identified erasure coded data; The check data includes one or more check data blocks, each check data block being a function of a system data block in the same stripe and / or one or more additional data in other stripes; In step S230, the verification data is stored on the storage node.

8. The method for storing erasure codes for ultra-large capacity data according to claim 7, characterized in that: The ultra-large capacity data storage erasure code redundancy method comprises the following steps: Step S240: When a storage node fails, data of the failed storage node can be restored based on the distributed data.

9. The method for storing erasure codes for ultra-large capacity data according to claim 7, characterized in that: In step S220, the plurality of check data blocks use different summation functions to represent functions of corresponding system data blocks and additional data blocks.

10. The method for storing erasure codes for ultra-large capacity data according to claim 7, characterized in that: In step S220, the number of the additional data blocks is determined according to the set data repetition degree; e t =(w×n)×(d-1) e t is the total number of additional data blocks, w is the number of stripes, n is the number of data nodes, and d represents the data duplication.

Images