Distributed storage system and data sharing method

Abstract

To shorten a time required for rebuilding data and / or a redundant code of the data when a failure occurs.SOLUTION: When rebuilding the data blocks and the second redundancy code having the same contents as the data blocks and the second redundancy code stored in the storage device of at least one of the plurality of nodes on another node based on the first redundancy code, the controller rebuilds the data blocks and the second redundancy code in a node different from a substitute node used to substitute for the node storing the data blocks to be rebuilt or the second redundancy code, and transfers the rebuilt data blocks and the rebuilt second redundancy code to the substitute node and stores the rebuilt data blocks and the rebuilt second redundancy code in a storage device of the substitute node.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a distributed storage system and a data sharing method, and is suitable for application to, for example, a distributed storage system relating to a technology for storing data in a redundant manner. [Background technology]

[0002] In order to analyze and utilize the vast amounts of data generated in social and corporate activities, storage systems for storing this data are essential. However, the more important the stored data, the greater the impact, such as lost opportunities for maintaining social life and lost business activities, if that data is lost or the necessary data cannot be accessed at the required time due to hardware failure or network failure. To address this issue, in order to minimize the impact of storage system failures and improve availability, redundancy is sometimes incorporated to prevent data loss in the event of a node or device failure. To increase the availability of a storage system, it is important to quickly restore data redundancy after a failure occurs and return the storage system to a healthy state.

[0003] Methods for making data redundant include, for example, the mirroring method, which prepares a copy of data, and a redundancy method called erasure coding, which creates a redundant code from data used for redundancy. In a distributed storage system consisting of multiple nodes, the erasure coding method is sometimes applied because it can save disk space during redundancy compared to the mirroring method. In this erasure coding method, when a failure occurs in a device such as a node or disk, data that is stored in the failed device and has become inaccessible is restored from data and redundant code distributed to other devices, and the restored data is stored in a specified node and redundancy is restored again to restore availability (hereinafter also referred to as "rebuild"). Erasure coding methods have been disclosed as methods for improving the efficiency of access to data distributed across multiple nodes, for example, by increasing data locality and optimizing network paths when applications access storage (see, for example, Patent Documents 1, 2, and 3). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6815378 [Patent Document 2] Patent No. 6798007 [Patent Document 3] Patent No. 6547057 Summary of the Invention [Problem to be solved by the invention]

[0005] In the methods disclosed in Patent Document 1 and Patent Document 2, in order to rebuild data that has been lost or becomes inaccessible due to a device failure or other reason, data necessary for the rebuild is collected at a node (hereinafter also referred to as an "alternate node") having a recovery device that will be the destination of the rebuild. The data necessary for the rebuild is collected from other nodes via a communication path between nodes. Therefore, when the bandwidth performance of the communication path between nodes is low, the time it takes to collect the data necessary for the rebuild significantly affects the rebuild performance. In the methods disclosed in Patent Document 1 and Patent Document 2, when the alternative node rebuilds redundant code that has become inaccessible and is stored in a failed device, all data necessary for the rebuild is transferred to the alternative node via a communication path between nodes, which means it takes time for the alternative node to collect the data. In addition, because the processing load for the rebuild is concentrated on the alternative node, there is a problem that the I / O performance of the alternative node is affected by the processing load for the rebuild. Furthermore, in Patent Document 2, rebuilds are distributed to multiple operating non-faulty nodes in a distributed storage system, but there is a problem that if a rebuild is performed on a specific non-faulty node that is heavily loaded due to I / O from the host server, the efficiency of the rebuild execution decreases.

[0006] The present invention has been made in consideration of the above points, and aims to propose a distributed storage system and data sharing method that can shorten the time required to rebuild data and / or redundant code for that data when a failure occurs. [Means for solving the problem]

[0007] In order to solve the above problem, the present invention provides a distributed storage system having a plurality of nodes, each including a storage device for storing data and a controller for making the data stored in the storage device redundant, wherein the controller divides data related to a received write request into a plurality of data blocks and writes the data to the storage device, generates a first redundancy code from the plurality of data blocks, and transmits the plurality of data blocks and the first redundancy code to different other nodes, the controller of the other nodes generates a second redundancy code from the plurality of data blocks and the first redundancy code received from the plurality of nodes and stores the second redundancy code in the storage device, rebuilds the data blocks and second redundancy code stored in the storage device of the node in another node, and the alternative node that rebuilt the data blocks uses the restored data When storing the block in a storage device, processing read requests and write requests from the host server, and rebuilding the data block and the second redundancy code, any one of the nodes restores the data block to be rebuilt based on the data block and the second redundancy code stored in any one of the plurality of nodes, generates a first redundancy code based on the plurality of data blocks stored in any one node, restores the second redundancy code to be rebuilt based on the generated first redundancy code and the data block stored in any one of the nodes, and stores the restored data block to be rebuilt and the second redundancy code in the storage device, and the node that restores the second redundancy data is a specific node different from the alternative node that stores the restored data block.

[0008] Furthermore, in the present invention, there is provided a data sharing method for a distributed storage system having a plurality of nodes, each including a storage device for storing data and a controller for making the data stored in the storage device redundant, wherein the controller divides data related to a received write request into a plurality of data blocks and writes the data into the storage device, generates a first redundancy code from the plurality of data blocks, and transmits the plurality of data blocks and the first redundancy code to different other nodes, the controller of the other nodes generates a second redundancy code from the plurality of data blocks received from the plurality of nodes and stores it in the storage device, the controller rebuilds, on another node, a data block and second redundancy code with the same contents as the data block and second redundancy code stored in the storage device of the previous node, and the alternative node that rebuilt the data block When storing the restored data block in a storage device, processing read requests and write requests from the host server, and rebuilding the data block and the second redundancy code, any one of the nodes restores the data block to be rebuilt based on the data block and the second redundancy code stored in any one of the plurality of nodes, generates a first redundancy code based on the plurality of data blocks stored in any one node, restores the second redundancy code to be rebuilt based on the generated first redundancy code and the data block stored in any one of the nodes, stores the restored data block to be rebuilt and the second redundancy code in the storage device, and the node that restores the second redundancy data is set to a specific node different from the alternative node that stores the restored data block. [Effects of the Invention]

[0009] According to the present invention, when a failure occurs, it is possible to reduce the time required to rebuild data and / or redundant code for that data. [Brief explanation of the drawings]

[0010] [Figure 1]1 illustrates an example of the configuration of a distributed storage system according to an embodiment of the present invention. [Figure 2] 1 shows an example of the configuration of a storage control program according to the present embodiment. [Figure 3] 1 shows an example of a two-stage redundancy scheme according to this embodiment. [Figure 4] 1 shows the relationship between data, data blocks, and class 1 redundancy codes according to this embodiment. [Figure 5] 2 shows the relationship between a data block, a class 1 redundant code, and a class 2 redundant code according to this embodiment. [Figure 6] 10 shows a first example of a method for rebuilding a data block according to this embodiment. [Figure 7] 10 shows a first example of a method for rebuilding another data block according to this embodiment. [Figure 8] 1 shows a first example of a method for rebuilding class 2 redundancy code according to this embodiment. [Figure 9] 10 shows a second example of a method for rebuilding class 2 redundancy code according to this embodiment. [Figure 10] 10 shows a second example of a method for rebuilding a data block according to this embodiment. [Figure 11] 10 shows a second example of a method for rebuilding another data block according to the present embodiment. [Figure 12] 10 shows a third example of a method for rebuilding class 2 redundancy code according to this embodiment. [Figure 13] 10 shows a procedure for selecting a node where a rebuild is to be performed based on the load status of the node according to this embodiment. [Figure 14] 10 shows a fourth example of a method for rebuilding class 2 redundancy code according to this embodiment. [Figure 15] FIG. 10 illustrates an example of a rebuild node selection process. [Figure 16] FIG. 10 is a diagram illustrating an example in which the rebuilt class 2 redundant code is not transferred to an alternative node. DETAILED DESCRIPTION OF THE INVENTION

[0011] In the following description, an "interface apparatus" may refer to one or more interface devices. The one or more interface devices may be at least one of the following: One or more I / O (Input / Output) interface devices. The I / O (Input / Output) interface devices are interface devices to at least one of the I / O devices and a remote display computer. The I / O interface device to the display computer may be a communications interface device. The at least one I / O device may be a user interface device, for example, either an input device such as a keyboard and a pointing device, or an output device such as a display device. One or more communication interface devices. The one or more communication interface devices may be one or more homogeneous communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more heterogeneous communication interface devices (e.g., a NIC and an HBA (Host Bus Adapter)).

[0012] In the following description, "memory" refers to one or more memory devices, which are an example of one or more storage devices, and may typically be a primary storage device. At least one memory device in the memory may be a volatile memory device or a non-volatile memory device.

[0013] In the following description, a "persistent storage device" may be one or more persistent storage devices, which are an example of one or more storage devices. A persistent storage device may typically be a non-volatile storage device (e.g., an auxiliary storage device), and more specifically, may be, for example, a hard disk drive (HDD), a solid state drive (SSD), a non-volatile memory express (NVMe) drive, or a storage class memory (SCM). In the following description, a "storage device" may be at least a memory and a persistent storage device.

[0014] Furthermore, in the following description, a "processor" may refer to one or more processor devices. The at least one processor device may typically be a microprocessor device such as a CPU (Central Processing Unit), but may also be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may also be a processor core. The at least one processor device may also be a processor device in a broader sense, such as a circuit that is a collection of gate arrays written in a hardware description language that performs some or all of the processing (e.g., an FPGA (Field-Programmable Gate Array), a CPLD (Complex Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit)), or a dedicated hardware circuit.

[0015] In the following description, information that provides an output for an input may be described in terms of a table such as an "xxx table." However, this information may be data of any structure (for example, structured data or unstructured data), or may be a learning model such as that used in a neural network, genetic algorithm, or random forest that generates an output for an input. Therefore, the "xxx table" may be referred to as "xxx information." In the following description, the structure of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table.

[0016] Furthermore, in the following description, functions may be described using the expression "yyy unit." However, the functions may be realized by one or more computer programs executed by a processor, by one or more hardware circuits (e.g., FPGAs or ASICs), or by a combination thereof. When a function is realized by a program executed by a processor, the specified processing is performed using a storage device and / or an interface device, etc., as appropriate, and therefore the function may be considered to be at least a part of the processor. Processing described using a function as the subject may be processing performed by a processor or a device having the processor. A program may be installed from a program source. The program source may be, for example, a computer from which the program is distributed or a computer-readable recording medium (e.g., a non-transitory recording medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0017] In addition, in the following description, when describing elements of the same type without distinguishing between them, common reference symbols will be used, and when describing elements of the same type with distinction between them, reference symbols will be used.

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the following description and drawings are merely examples for explaining the present invention, and have been omitted or simplified as appropriate for clarity of explanation. The present invention can be embodied in various other forms, and unless otherwise specified, each component may be singular or plural.

[0019] Furthermore, the examples described below do not limit the invention according to the claims, and not all of the combinations of elements described in the examples are necessarily essential to the solution of the invention.

[0020] A distributed storage system according to an embodiment of the present invention will be described below. FIG. 1 shows an example of the configuration of a computer system according to this embodiment.

[0021] The computer system includes a user terminal 100, a host server 110, a management server 120, and a distributed storage system 130. These can communicate via a network 160. The distributed storage system 130 employs a distributed configuration consisting of multiple storage nodes 140. The host server 110, the management server 120, and the storage nodes 140 may be configured using not only physical computers (bare machines), but also virtual machines (VMs) or containers virtually configured on physical computers or in the cloud. While only three storage nodes are shown in the figure for simplification, the number of storage nodes constituting the distributed storage system is not limited and may be added or removed as needed. In this embodiment, storage nodes may be abbreviated as nodes. The distributed storage system can be configured using existing technology. The number of each of these components may be arbitrary. The user terminal 100 may also include the functions of the host server 110 and / or the management server 120.

[0022] Network 160 may be, for example, a local area network (LAN) or a storage area network (SAN). Host server 110 and management server 120 may access distributed storage system 130 via different networks, and user terminal 100 may access host server 110 or management server 120 via a network different from network 160.

[0023] The host server 110, management server 120, distributed storage system 130, and multiple storage nodes 140 that make up the distributed storage system may each be located within the same site, or some or all of them may be located at different sites. Furthermore, some or all of them may be located on the cloud. In this case, the sites and / or the sites and the cloud may be connected via, for example, a wide area network (WAN).

[0024] The user terminal 100 is a device that allows a user to access a computer system. The user terminal 100 may have, for example, a general computer configuration, including an interface device, a storage device, and a processor connected thereto. The user terminal 100 may also include hardware dedicated to specific processing. The user terminal 100 may have I / O devices (e.g., a keyboard, a pointing device, and a display device).

[0025] The host server 110 is a host machine on which user applications and the like run. The host server 110 may have, for example, a general computer configuration, including an interface device, a storage device, and a processor connected to them. The host server 110 may also include hardware dedicated to specific processes. The host server 110 can execute various software programs, such as databases and web services, and write and / or read data created thereby to the distributed storage system 130 via the network 160.

[0026] The management server 120 manages the distributed storage system 130. The management server 120 may have, for example, a general computer configuration, including an interface device, a storage device, and a processor connected thereto. The management server 120 may also include hardware dedicated to specific processing.

[0027] The storage system of this embodiment has multiple storage nodes 140, each including a storage device (corresponding to an SSD (Solid State Drive) 149 or HDD (Hard Disk Drive) 148, described below) that stores data, and a controller that makes the data stored in the storage device redundant.The storage system is a distributed storage system in which each piece of data, which is made up of multiple data blocks, is distributed and managed by multiple storage nodes 140, and receives input / output requests and write data that request reading or writing of data from or to a host server, and provides read data to the host server.

[0028] The storage node 140 may have, for example, a general computer configuration, and includes an interface device, a storage device, and a processor connected thereto. The storage node 140 may also include hardware dedicated to a specific process. The storage node 140 includes a controller 141 and a drive box 147. The controller 141 includes a host interface 142, a management interface 133, a drive interface 144, a memory 146, and a processor 145 connected thereto. The number of these components is arbitrary.

[0029] The host interface 142 is an interface device for communication with the host server 110. The management interface 143 is an interface device for communication with the management server 120. The drive interface 144 is an interface device for communication with the drive box 147.

[0030] The drive box 147 contains one or more non-volatile or volatile storage drives that store various data used by application programs of the host server 110. The drive box 147 is connected to the drive interface 144 of the controller 141.

[0031] 1, the drive box 147 includes at least one of a plurality of hard disk drives (HDDs) 148 and a plurality of solid state drives (SSDs) 149. The plurality of drives 148, 149 may form a group for data redundancy such as a redundant array of independent disks (RAID).

[0032] The controller 141 controls the distributed storage system 130. The controller 141 provides the host server 110 with a logical volume for storing data that the host server 110 reads and writes. The controller 141 allocates physical storage areas of the drives 148 and 149 to the volume and stores the data in the drives 148 and 149. The controller 141 provides the host server 110 with a storage function. The controller 141 has a function for controlling the cooperative operations between storage nodes so that the distributed storage system 130 functions as a storage system.

[0033] In response to a read request or a write request from the host server 110, the processor 145 issues an instruction to transfer or change data stored in the corresponding drive box 147. The memory 146 of the controller 141 is configured, for example, by a semiconductor memory such as an SDRAM (Synchronous Dynamic Random Access Memory). The memory may be configured by combining a volatile memory and a non-volatile memory.

[0034] The processor 145 executes processes for controlling the distributed storage system 130 and for communicating with the host server 110, the management server 120, and the drive box 147. The memory 146 serves as the main memory of the processor 145 and stores programs and various data for control and communication. The memory 146 is also used as a disk cache (cache memory) for the controller 141. The processor 145 realizes predetermined functions by executing programs, including instruction codes, stored in the memory 146.

[0035] A plurality of controllers 141 may be implemented for redundancy. The plurality of controllers 141 communicate via a network within the distributed storage system 130. The controllers 141 perform redundancy of written data, sharing of metadata, and the like via the network within the distributed storage system 130. Even if one controller 141 is blocked due to maintenance, a failure, or the like, the other controller 141 can continue storage processing. The distributed storage system 130 may be configured using a general server computer, or may include hardware dedicated to specific processing. The distributed storage system 130 can be configured using existing technology.

[0036] The computer system may include other components than those shown here. For example, the network may be connected to network devices such as switches and routers. The computer system may also be configured to connect to a storage service on a public cloud via an external network.

[0037] 2 shows an example of the software configuration of a storage control program 210 that is processed by a processor in a controller 141 included in a storage node 140 that constitutes the distributed storage system 130 in Fig. 1 and controls the distributed storage system 130. The storage control program 210 includes a host I / O processing unit 220 and a redundancy processing unit 250, and preferably further includes a cluster cooperation management unit 230 and a volume management unit 240.

[0038] The host I / O processing unit 220 receives I / O requests for reading or writing data from the host server 110 in FIG. 1, as well as write data, and processes them in a format and at a timing that can be executed by the storage system.

[0039] The cluster cooperation management unit 230 has a function of determining whether an I / O request from a host processed by the host I / O processing unit 220 is a request for a volume managed by its own storage node or a request for a volume managed by another storage node, and if the request is for a volume managed by its own storage node, controlling the request so that it is processed by the volume management unit 240 of its own storage node, and if the request is for a volume managed by another storage node, cooperating with the cluster cooperation management unit 230 of the other storage node and controlling the request so that it is processed by the volume management unit 240 of the other storage node.

[0040] If an I / O request from the host server 110 is a read request for data stored in a volume managed by the storage node itself, the volume management unit 240 of the storage node itself controls the request to be processed. If an I / O request from the host is a read request for data stored in a volume managed by another storage node, the cluster cooperation management unit 230 of that storage node cooperates with the cluster cooperation management unit 230 to receive the data transferred from the other storage node. The host I / O processing unit 220 transfers the data to be read to the host server 110.

[0041] If an I / O request from the host server 110 is a write request to a volume managed by its own storage node, the request is processed by the volume management unit 240 of its own storage node, and the write request data is written to the volume managed by its own storage node. If an I / O request from the host is a write request to a volume managed by another storage node, the data is written to the volume managed by that storage node in cooperation with the cluster cooperation management unit 230 of that storage node.

[0042] The volume management unit 240 manages volumes, reads target data from a volume based on a read request to the volume, and writes data to the volume based on a write request to the volume.

[0043] The redundancy processing unit 250 makes the data written to the volume redundant based on a redundancy method described below. The redundancy processing unit 250 distributes the data to be made redundant and the redundant code generated by the redundancy processing to other storage nodes via the cluster cooperation management unit 230. The redundancy processing unit 250 may include a function to rebalance the distributed data and redundant code among storage nodes as necessary.

[0044] The redundancy processing unit 250 divides the data related to the received write request into multiple data blocks and writes them to the storage device, and also generates a first redundancy code from the multiple data blocks and transmits the multiple data blocks and a class 1 redundancy code as an example of the first redundancy code to different other nodes, and the controller of the other storage node generates a class 2 redundancy code as an example of a second redundancy code from the multiple data blocks received from the multiple storage nodes and stores it in a storage device (hereinafter also collectively referred to as a "drive"). The drive is, for example, at least one of an HDD 148 and an SSD 149.

[0045] The controller of the other storage node generates a class 2 redundancy code from the multiple data blocks and class 1 redundancy code received from the multiple storage nodes and stores the generated class 2 redundancy code in a storage device, rebuilds the class 2 redundancy code stored in the storage device of the storage node on another storage node, the alternative node that rebuilt the data block stores the restored data block in a storage device and processes read requests and write requests from the host server, and when rebuilding the data block and class 2 redundancy code, one of the storage nodes restores the data block to be rebuilt based on the data block and class 2 redundancy code stored in any of the multiple storage nodes, generates a class 1 redundancy code based on the multiple data blocks stored in any one node, restores the class 2 redundancy code to be rebuilt based on the generated class 1 redundancy code and the data block stored in any one of the nodes, and stores the restored data block and class 2 redundancy code to be rebuilt in a storage device, and the storage node that restores the class 2 redundancy code is a specific storage node different from the alternative node that stores the restored data block. Furthermore, in this embodiment, the storage node that restores the data block to be rebuilt based on the data block and the class 2 redundancy code is a specific storage node different from the alternative node that stores the restored data block. The restored class 2 redundancy code is stored in the alternative node. The storage node that stores the restored class 2 redundancy code is a storage node different from the alternative node that stores the restored data block.

[0046] The storage control program 210 may also include a node information acquisition unit 260 that acquires node information indicating the states of multiple storage nodes. The node information acquisition unit 260 acquires, for example, the load states of multiple storage nodes as the states of the multiple storage nodes.

[0047] Specifically, the node information acquisition unit 260 acquires information about the storage node, such as CPU usage, memory usage, network bandwidth usage between storage nodes, network bandwidth usage between the host server and the storage node, volume usage, and / or any combination thereof, using a measurement function provided in the operating system of the computer used as the storage node, a known monitor program, and / or physical sensors. The information about the storage node may include physical information such as CPU temperature and operating frequency, voltage supplied to the computer, fan rotation speed, etc. The operation of the storage control program 210 may be controlled using this node information.

[0048] The controller acquires node information indicating the states of the nodes, and selects specific storage nodes to restore the data blocks and the second redundancy code based on the acquired node information. The node information includes load states of the multiple nodes.

[0049] For simplicity, storage nodes may be referred to simply as nodes below. Figure 3 illustrates an example of two-stage redundancy, enabling all data to be recovered in the event of two node failures, among four nodes constituting a distributed storage system: a first node 310, a second node 320, a third node 330, and a fourth node 340. In the illustrated example, the area above the horizontal dashed line is memory, and the area below the horizontal dashed line is drive. For simplicity's sake, in the following description, including Figure 3, some letters and numbers are represented as normal letters and numbers, rather than subscripts and numbers, or superscripts and numbers. In this redundancy scheme, for example, in the case of 2D2P, which ultimately generates two redundancy codes from two data blocks as described below, only one class 1 redundancy code is generated, and the two data blocks and one parity are distributed to other nodes. Each node then generates a class 2 redundancy code using the two data blocks and one class 1 redundancy code collected from the other nodes.

[0050] In this embodiment, the redundancy processing unit 250 first generates one redundant code from two data blocks as the first stage of redundancy. For example, a method such as parity generation is used for the redundant code. The first stage redundant code is called a "class 1 redundant code" as an example of a first redundant code. Note that dotted arrows in the figure represent data transfer between nodes, and solid arrows represent data transfer between memory and drives within a node.

[0051] In FIG. 3, the redundancy processing unit 250 described above first divides data DN1 311, which is requested to be written to the first node 310 by a host server application or the like, into two data blocks D1N1 312 and D2N1 313 in memory, and then generates a class 1 redundancy code C1N1 314 from D1N1 312 and D2N1 313.

[0052] Next, D1N1 312, D2N1 313, and C1N1 314 are transferred to the second node 320, the third node 330, and the fourth node 340, respectively, which are nodes different from the first node 310. The same processing as for DN1 311 is also performed on data DN2 321, DN3 331, and DN4 341 that have been requested to be written to the second node 320, the third node 330, and the fourth node 340, respectively.

[0053] Next, the first node 310 generates second-stage redundancy codes (second redundancy codes), C2N11 315 and C2N12 316, using data D2N3 332 and D1N4 342 transferred to the first node 310 from the second node 320, third node 330, and fourth node 340, respectively, for redundancy, and class 1 redundancy code C1N2 322 generated by the second node 320 and transferred to the first node 310. These codes are stored in the drive 317 of the first node 310 together with D1N1 312 and D2N1 313. C2N11 315 and C2N12 316 are referred to as "class 2 redundancy codes."

[0054] The group of data blocks D1N1 312, D2N1 313 and class 2 redundancy codes C2N11 315, C2N12 316 is called a redundancy group. The second node 320, third node 330, and fourth node 340 perform the same process as the first node 310, and store the data blocks and class 2 redundancy codes in their respective drives.

[0055] As a result, the data DN1 311, DN2 321, DN3 331, and DN4 341 written to the distributed storage system consisting of four nodes are made redundant and distributed to multiple nodes, including the redundant codes, making it possible to recover all data even if up to two nodes fail. Class 2 redundancy codes are updated each time the data blocks used to generate them are updated by a write process.

[0056] While this procedure illustrates an example in which class 1 redundancy codes are generated each time a rebuild is performed, class 1 redundancy codes are generated during the process of generating class 2 redundancy codes for redundancy. Therefore, by storing class 1 redundancy codes in memory or on a drive when they are generated and updating them simultaneously when the data block used to generate the class 1 redundancy code is updated, the process of generating class 1 redundancy codes when a failure occurs is unnecessary. In the rebuild procedure described below, class 1 redundancy codes are generated from data blocks each time. However, if the latest class 1 redundancy code based on the latest value of the data block from which the code was generated is stored in memory or on a drive, the stored class 1 redundancy code may be used instead of generating class 1 redundancy codes each time; this is not described in the rebuild procedure described below. If class 1 redundancy codes are stored on a drive or in memory, the drive or memory must have sufficient capacity to store the class 1 redundancy codes.

[0057] In the redundancy method described above, data blocks are stored directly in the node where they were written, so when reading the same data, the same node can be accessed as the node where it was written. Therefore, no data transfer between nodes occurs when reading, and the resulting performance degradation does not occur.

[0058] Figure 4 shows an example of the relationship between data 402 requested to be written to each node 401 shown in Figure 3, the first data block 403 and the second data block 404 into which the data is divided, and a class 1 redundancy code 405 generated from the first data block 403 and the second data block 404.

[0059] For example, data DN2 in the second node 320 is divided into data block D1N2 and data block D2N2, and a class 1 redundancy code C1N2 is generated from data block D1N2 and data block D2N2. Due to the redundancy, even if any one of the first data block 403, the second data block 404, or the class 1 redundancy code 405 is lost in each of the first node 310 to the fourth node 340, the lost data block or class 1 redundancy code can be restored from the remaining two of these.

[0060] The storage control program 210 shown in Fig. 2 may store information showing the relationship between the data, data blocks, and class 1 redundant codes shown in Fig. 4 in a memory or drive as a table, and may refer to the table to divide the data into data blocks and generate class 1 redundant codes. Also, the table may be assigned a code indicating the correspondence of the storage destination node for each data block shown in the table, and control may be performed to store the data blocks in that node.

[0061] Figure 5 shows an example of the relationship between data blocks and class 2 redundant codes when a class 2 redundant code is further generated from the first data block 403, the second data block 404, and the class 1 redundant code 405 of the data blocks shown in Figure 4.

[0062] A first first class 2 redundancy code 505 and a second second class 2 redundancy code 506 are generated from the first data block 502, second data block 503, and class 1 redundancy code 504 transferred to each node 501. For example, in the second node 320, a first first class 2 redundancy code C2N21 and a second second class 2 redundancy code C2N22 are generated from D1N1 of the first data block 502, D2N4 of the second data block 503, and class 1 redundancy code C1N3.

[0063] The storage control program shown in Fig. 2 may store information showing the relationships between the data, data blocks, class 1 redundant codes, and class 2 redundant codes shown in Fig. 5 as a table in memory or on a drive, and may refer to the table to generate class 2 redundant codes from the data blocks and class 1 redundant codes. The table may also be assigned a code indicating the correspondence of the storage destination node for each data block and class 2 redundant code shown in the table, and control may be exercised to store the data blocks and class 1 redundant codes in the node.

[0064] 3, 4, and 5 show the case where one redundancy group is arranged in one node for simplicity of explanation, but multiple redundancy groups may be arranged in one node. In this case, the data blocks to be redundancy-processed and the class 1 redundancy codes generated from the data blocks may be transferred to different nodes for each redundancy group, and class 2 redundancy codes may be generated in each node. Furthermore, information indicating the relationship between the data blocks and the redundancy codes shown in FIGS. 4 and 5 may be stored in a table or the like for use in the redundancy process.

[0065] The example in Figure 3 is called a 2D2P configuration because two redundancy codes are ultimately generated from two data blocks. D represents a data block, and P represents a redundancy code. Data redundancy can be expanded to mDnP, which uses m data blocks and n redundancy codes, based on techniques such as erasure coding. For example, in a 3D2P configuration, data requested to be written to a node is divided into three parts, and one class 1 redundancy code is generated from the three data blocks and distributed to nodes other than the node where the data was written.

[0066] Each node generates two class 2 redundancy codes from the data blocks transferred from other nodes and the class 1 redundancy codes, and stores the data requested to be written to each node along with the three divided data blocks on the drive of that node.

[0067] Therefore, in the case of a 3D2P configuration, the distributed storage system must have at least five nodes. Class 2 redundancy codes are generated from the distributed data blocks and class 1 redundancy codes using the same procedure as in the 2D2P configuration, and stored on the drives, providing protection against two-node failures. The same applies when the number of data blocks m and the number of redundancy codes n are other than 3 and 2.

[0068] Generally, the larger the ratio of the number of data blocks used for redundancy to the generated redundant code, the smaller the drive capacity required to store the redundant code becomes, improving capacity efficiency; however, the number of distributions increases, requiring a larger number of nodes.

[0069] While Figure 3 shows an example of data redundancy for node failures, it goes without saying that similar data protection is possible for drive-level failures by replacing a node with a drive. This process can also be applied when a node or drive needs to be stopped for a certain period of time for maintenance, in addition to when a node or drive fails.

[0070] <Rebuild data blocks on alternate nodes> Fig. 6 shows a method for rebuilding, in the event of a failure in one node, a data block with the same contents as the data block stored in the failed node on an alternative node that functions as a storage node in the data redundancy method shown in Fig. 3. Note that in the following explanation, explanations of parts that are the same as those in the previous embodiment will be omitted.

[0071] 6 illustrates a process for rebuilding data block D1N1 312 stored in the first node 310 using data blocks and class 2 redundancy codes stored in other nodes when a single node failure occurs in the first node 310. Assume that data block D1N1 312, data block D2N1 313, class 2 redundancy code C2N11 315, and class 2 redundancy code C2N12 316 stored in the first node 310 are inaccessible due to the node failure. In this case, D1N1 312 is rebuilt using the following procedure.

[0072] Step (1): The redundancy processing unit 250 reads the class 2 redundancy code C2N21 601 and the class 2 redundancy code C2N22 602 from the drive 327 of the second node 320, transfers them to memory, and then transfers them to the fifth node 350, which is an alternative node for the failed first node 310 and has the function of a storage node.

[0073] Step (2): The redundancy processing unit 250 reads the data block D2N4 603 from the drive 347 of the fourth node 340, transfers it to memory, and further transfers it to the fifth node 350.

[0074] Step (3): In the fifth node 350, the redundancy processing unit 250 restores a data block D1N1 604 having the same contents as the data block D1N1 312 stored in the first node 310 where the failure occurred, from the class 2 redundancy codes C2N21 601, C2N22 602, and the data block D2N4 603.

[0075] Step (4): The redundancy processing unit 250 stores the data block D1N1 604 of the fifth node 350 in the drive 357 of the same node.

[0076] Therefore, when restoring a data block, inter-node data transfer for one data block and two class 2 redundancy codes occurs to the alternative node where the rebuild is to be performed. After the data and class 2 redundancy code stored in the failed node are rebuilt on the alternative node, the storage control program 210 changes the distributed storage system configuration and performs processing to switch from the failed node to the alternative node. The processing for switching from the failed node to the alternative node is similar in the following, so a description thereof will be omitted.

[0077] FIG. 7 shows a method for rebuilding the data block D2N1 313 stored in the first node 310 using data blocks stored in other nodes and class 2 redundancy code when a single node failure occurs in the first node 310, similar to FIG. 6, in the data redundancy scheme shown in FIG. 3.

[0078] 7 shows the state after the restoration of data block D1N1 has already been completed according to the procedure in Fig. 6 and the data has been stored in the drive 357 of the fifth node 350, which is the alternative node to rebuild. In this embodiment, the restoration of data block D1N1 is performed first, but any of the rebuild targets may be rebuilt first unless the order in which the rebuilds are performed affects the execution of the rebuilds, such as when multiple nodes fail, as will be described later.

[0079] Step (1): The redundancy processing unit 250 reads the data block D1N2 701 from the drive 327 of the second node 320, transfers it to memory, and then transfers it to the fifth node, which is an alternative node for the failed first node 310 and has the function of a storage node.

[0080] Step (2): The redundancy processing unit 250 reads the class 2 redundancy code C2N31 702 and the class 2 redundancy code C2N32 703 from the drive 337 of the third node 330 and transfers them to memory, and then transfers them to the fifth node 350.

[0081] Step (3): In the fifth node 350, the redundancy processing unit 250 restores data block D2N1 704 with the same content as data block D2N1 313 stored in the failed first node from data block D1N2 701 and class 2 redundancy codes C2N31 702 and C2N32 703.

[0082] Step (4): The redundancy processing unit 250 stores the data block D2N1 704 of the fifth node 350 in the drive 357 of the same node.

[0083] As in Figure 6, when restoring a data block, inter-node data transfer for one data block and two class 2 redundancy codes occurs to the alternative node where the data is to be rebuilt.

[0084] Combining Figures 6 and 7, in order to restore the data blocks stored in the failed node, m+2n data transfers are required in the redundancy of the mDnP configuration.

[0085] <Rebuild class 2 redundant code on an alternate node> FIG. 8 illustrates a procedure for rebuilding the class 2 redundancy code C2N11 315 and the class 2 redundancy code C2N12 316 stored in the first node 310 using data blocks and class 2 redundancy codes stored in other nodes when a single node failure occurs in the first node 310, in the data redundancy scheme shown in FIG. 3, similar to FIGS. 6 and 7.

[0086] 8 shows the state after the restoration of data blocks D1N1 and D2N1 has already been completed according to the procedures in Figures 6 and 7 and they have been stored in the drive 357 of the fifth node 350, which is the alternative node to rebuild. In this embodiment, the restoration of data blocks D1N1 and D2N1 is performed first, but any of the rebuild targets may be rebuilt first unless the order in which the rebuilds are performed affects the execution of the restoration, such as when multiple nodes fail, as will be described later.

[0087] Step (1): The redundancy processing unit 250 reads the data blocks D1N2 701 and D2N2 801 from the drive 327 of the second node 320, transfers them to memory, and generates the class 1 redundancy code C1N2 802 using these.

[0088] Step (2): The redundancy processing unit 250 transfers the generated class 1 redundancy code C1N2 802 from the second node 320 to the fifth node 350, which is an alternative node with storage node functionality for the failed first node 310.

[0089] Step (3): The redundancy processing unit 250 reads the data block D2N3 803 from the drive 337 of the third node 330, transfers it to memory, and further transfers it to the fifth node 350, which is the alternative node.

[0090] Step (4): The redundancy processing unit 250 reads the data block D1N4 603 from the drive 347 of the fourth node 340, transfers it to the memory, and further transfers it to the fifth node 350, which is the alternative node.

[0091] Step (5): In the fifth node 350, the redundancy processing unit 250 restores class 2 redundancy code C2N11 804 and class 2 redundancy code C2N12 805, which have the same contents as class 2 redundancy code C2N11 315 and class 2 redundancy code C2N12 316 stored in the first node 310 where the failure occurred, from class 1 redundancy code C1N2 802, data block D2N3 803, and data block D1N4 603, and stores them in the drive 357 of the fifth node 350.

[0092] In the case of redundancy in the 2D2P configuration shown in Figure 8, when restoring a class 2 redundancy code, two data blocks and one class 1 redundancy code are transferred between nodes to the replacement node where the rebuild is to be performed. In the case of redundancy in the mDnP configuration, m+n-1 data transfers are required.

[0093] By the procedures of Figures 6, 7, and 8, the data blocks stored in the failed node and data with the same contents as the class 2 redundancy code are rebuilt on the fifth node 350 as an alternative node.

[0094] The mDnP redundancy method described above makes it possible to restore data and class 2 redundant code even when n nodes fail simultaneously. Figures 6, 7, and 8 show the rebuild procedure when one node fails (one node failure), but even when n nodes fail, the data blocks and class 2 redundant code stored in the failed node can be rebuilt by executing the same rebuild procedure from data blocks and class 2 redundant code stored in nodes other than the failed node.

[0095] For example, a rebuild procedure will be described when a failure occurs in the first node 310 or the second node 320 in Figures 3, 6, 7, and 8, making them inaccessible. Note that in this embodiment, reference numerals may be omitted as appropriate.

[0096] Step (1): The redundancy processing unit 250 generates a class 1 redundancy code C1N4 from D1N4, the first data block 403 stored in the fourth node 340, and D2N4, the second data block 404. The data required for restoration is collected in the memory of one of the nodes by inter-node or intra-node transfer, and the restoration operation is performed in the memory of that node. Below, a description of the transfer operation is omitted.

[0097] Step (2): The redundancy processing unit 250 restores the data block D1N2 of the second node 320 and the data block D2N1 of the first node from, for example, the class 1 redundancy code C1N4 and the class 2 redundancy code C2N31 and class 2 redundancy code C2N32 stored in the third node 330.

[0098] Step (3): The redundancy processing unit 250 restores the data block D2N2 and the class 1 redundancy code C1N1 of the second node 320 from the data block D1N3 stored in the third node, the class 2 redundancy code C2N41 stored in the fourth node 340, and C2N42.

[0099] Step (4): The redundancy processing unit 250 restores the data block D1N1 of the first node 310 from the data block D2N1 of the first node 310 and the class 1 redundancy code C1N1 that have been restored in the above-mentioned step (2).

[0100] Step (5): The redundancy processing unit 250 generates a class 1 redundancy code C1N3 from the data blocks D1N3 and D2N3 of the third node 330.

[0101] Step (6): The redundancy processing unit 250 restores the class 2 redundancy code C2N21 and the class 2 redundancy code C2N22 of the second node 320 from the data block D1N1 of the first node 310 restored in step (4), the data block D2N4 of the second node 320, and the class 1 redundancy code C1N3 generated in step (5).

[0102] Step (7): The redundancy processing unit 250 generates a class 1 redundancy code C1N2 from the data block D1N2 of the second node that was rebuilt in step (2) above and the data block D2N2 of the second node 320 that was restored in step (3) above.

[0103] Step (8): The redundancy processing unit 250 restores the class 2 redundancy code C2N11 and the class 2 redundancy code C2N12 of the first node 310 from the data block D1N4 stored in the fourth node 340, the data block D2N3 stored in the third node, and the class 1 redundancy code C1N2 generated in step (7).

[0104] As a result, even if two nodes fail, the data blocks and class 2 redundancy codes stored in the failed nodes can be restored sequentially. While the examples in Figures 6, 7, and 8 show a case where failures occur in the first node 310 and the second node 320, the rebuild procedure is similar even if the failed node is another node. Furthermore, even if the redundant configuration is, for example, mDnP, rebuilds can be performed using the same procedure.

[0105] 6, 7, and 8, data blocks stored in a failed node, data blocks required for restoring the class 2 redundancy code, and class 1 redundancy code are transferred between nodes to a node (alternative node) that functions as a storage node and replaces the failed node, and the data blocks and class 2 redundancy code are restored on the alternative node. For example, to restore data block D1N1 stored in a failed first node, as shown in FIG. 6, the second data block D2N4 603 stored in the fourth node 340 and the class 2 redundancy codes C2N21 601 and C2N22 602 stored in the second node are transferred to the alternative fifth node 350, and data block D1N1 604 is restored on the fifth node.

[0106] 7, to restore data D2N1 stored in the failed first node 310, the first data block D1N2 701 stored in the second node 320 and the class 2 redundancy codes C2N31 702 and C2N32 703 stored in the third node are transferred to the alternative node, the fifth node 350, and data block D1N1 704 is restored in the fifth node 350. In mDnP, when restoring m data blocks stored in the first node 310, the amount of data transferred to the alternative node is m+2n, combining the transfer of m data blocks and the transfer of 2n class 2 redundancy codes.

[0107] 8, to restore the class 2 redundancy code, the second data block D2N3 803 stored in the third node 330, the second data block D2N4 603 stored in the fourth node 340, the first data block D1N2 701 stored in the second node 320, and the class 1 redundancy code C1N2 802 generated from the second data block D2N2 801 are transferred to the alternative node, the fifth node 350, and the class 2 redundancy code C2N11 804 and the class 2 redundancy code C2N12 805 are restored in the fifth node 350. In mDnP, when restoring the class 2 redundancy code, the transfer amount to the alternative node is m+n-1, combining the transfer of m data blocks and the transfer of n-1 class 1 redundancy codes.

[0108] From the above, the amount of data transfer to the fifth node 350 as the replacement node required to restore the data blocks per redundancy block stored in the first node 310 and the class 2 redundancy code is 2m+3n-1, as shown in Figures 6, 7, and 8.

[0109] If there are multiple redundancy groups stored in the first node 310, the data blocks belonging to all of the redundancy groups, the data blocks required to restore the class 2 redundancy code, and the class 1 redundancy code are transferred to the fifth node 350, which serves as the alternative node and the destination for the rebuild. The amount of data required to transfer to the alternative node for the rebuild is the number of redundancy groups × (2m + 3n - 1). For this reason, when a rebuild is performed, data transfers are concentrated on the alternative node, which is the destination for the rebuild. In addition, the rebuild processing load for the class 2 redundancy code is concentrated on the alternative node.

[0110] In contrast, Fig. 9 shows a rebuild method according to this embodiment that reduces the amount of data flowing into an alternative node when rebuilding a data block in the data redundancy scheme shown in Fig. 3. The controller determines whether to restore the data blocks and class 2 redundancy code at an alternative node, and if it is selected to restore the data at the alternative node, it transfers the data required for restoration from the storage devices of the multiple nodes to the alternative node and performs the restoration at the alternative node. If it is selected not to restore the data at the alternative node, it selects a specific node to perform the restoration, transfers the data required for restoration from the storage devices of the multiple nodes to the specific node, performs the restoration at the specific node, and stores the restored class 2 redundancy code in the alternative node.

[0111] In the above-described FIG. 6, when a single node failure occurs in the first node 310, the process of rebuilding the data block D1N1 312 stored in the first node 310 using data blocks stored in other nodes and class 2 redundancy code was explained.

[0112] Assume that the data block D1N1 312, data block D2N1 313, class 2 redundancy code C2N11 315, and class 2 redundancy code C2N12 316 stored in the first node 310 are inaccessible due to a node failure. In this case, the data block D1N1 312 stored in the first node 310 is rebuilt according to the following procedure.

[0113] Step (1): The redundancy processing unit 250 reads the class 2 redundancy code C2N21 601 and the class 2 redundancy code C2N22 602 from the drive 327 of the second node 320 and transfers them to memory.

[0114] Step (2): The redundancy processing unit 250 reads the data block D2N4 603 from the drive 347 of the fourth node 340, transfers it to the memory, and then transfers it to the second node 320.

[0115] Step (3): In the second node 320, the redundancy processing unit 250 restores a data block D1N1 901 having the same contents as the data block D1N1 312 stored in the first node 310 where the failure occurred, from the class 2 redundancy codes C2N21 601, C2N22 602, and the data block D2N4 603.

[0116] Step (4): The redundancy processing unit 250 transfers the data block D1N1 901 to the fifth node 350, which is an alternative node having the function of a storage node for the failed first node 310, and stores it in the drive 357.

[0117] Therefore, when restoring a data block, inter-node data transfer of one data block occurs to the alternative node to which the rebuild is to be performed. After the data and class 2 redundancy code stored in the failed node are rebuilt on the alternative node, the storage control program 210 changes the distributed storage system configuration and performs a process to switch from the failed node to the alternative node. The process of switching from the failed node to the alternative node is similar in the following sections, so a description of this process will be omitted.

[0118] <Rebuild data blocks on alternate nodes> Fig. 10 shows a rebuild method according to this embodiment that reduces the amount of data flowing into an alternative node during a data block rebuild when a single node failure occurs in the first node 310 in the data redundancy scheme shown in Fig. 3, similar to Fig. 9. Note that Fig. 10 shows the state after data block D1N1 has already been restored according to the procedure in Fig. 9 and stored in the drive 357 of the fifth node 350, which is the alternative node to be rebuilt.

[0119] In this embodiment, the restoration of data block D1N1 is performed first, but any of the rebuild targets may be restored first, except in cases where the order in which restoration is performed affects the execution of restoration, such as when failures occur in multiple nodes, as described below.

[0120] Step (1): The redundancy processing unit 250 reads the data block D 1 N 2 701 from the drive 327 of the second node 320 , transfers it to memory, and then transfers it to the third node 330 .

[0121] Step (2): The redundancy processing unit 250 reads the class 2 redundancy code C2N31 702 and the class 2 redundancy code C2N32 703 from the drive 337 of the third node 330 and transfers them to memory.

[0122] Step (3): In the third node 330, the redundancy processing unit 250 restores a data block D2N1 1001 having the same contents as the data block D2N1 313 stored in the failed first node from the data block D1N2 701 and the class 2 redundancy codes C2N31 702 and C2N32 703.

[0123] Step (4): The redundancy processing unit 250 transfers the data block D2N1 1001 to the fifth node 350, which is an alternative node having the function of a storage node for the failed first node 310, and stores it in the drive 357.

[0124] As in Figure 9, when restoring a data block, inter-node data transfer of one data block occurs to the alternative node where the data is to be rebuilt.

[0125] Combining Figures 9 and 10, in order to restore the data blocks stored in the failed node, m data transfers occur in the redundancy of the mDnP configuration.

[0126] Fig. 11 is a diagram showing a rebuild method according to this embodiment that reduces the amount of data flowing into an alternative node when rebuilding redundancy codes. Fig. 11 shows a procedure for restoring the class 2 redundancy code C2N11 315 and the class 2 redundancy code C2N12 316 stored in the first node 310 using data blocks and class 2 redundancy codes stored in other nodes when a single-node failure occurs in the first node 310, in the data redundancy scheme shown in Fig. 3, similar to Fig. 8.

[0127] 8, Fig. 11 shows the state after the restoration of data blocks D1N1 and D2N1 has already been completed according to the procedures in Fig. 9 and 10 and they have been stored in the drive 357 of the fifth node 350, which is the alternative node to be rebuilt. In this embodiment, the restoration of data blocks D1N1 and D2N1 is performed first, but any of the rebuild targets can be performed first, except in cases where the order in which the restoration is performed affects the execution of the restoration, such as when multiple nodes fail, as will be described later.

[0128] <Rebuild class 2 redundant code on an alternate node> In FIG. 11, when a failure occurs in the first node 310, the class 2 redundancy code C2N11 315 and the class 2 redundancy code C2N12 316 stored in the failed first node 310 are rebuilt on the fifth node 350, which is an alternative node, by the following procedure.

[0129] Step (1): Following the same procedure as in FIG. 8, the redundancy processing unit 250 first reads data blocks D1N2 701 and D2N2 801 from the drive 327 of the second node 320, transfers them to memory, and uses them to generate class 1 redundancy code C1N2 802.

[0130] Step (2): The redundancy processing unit 250 transfers the data block D2N3 803 stored in the drive 337 of the third node 330 to the memory, and then transfers it to the second node 320.

[0131] Step (3): The redundancy processing unit 250 transfers the data block D1N4 603 stored in the drive 347 of the fourth node 340 to the memory, and then transfers it to the second node 320.

[0132] Step (4): In the second node 320, the redundancy processing unit 250 restores the class 2 redundancy code C2N11 1101 and the class 2 redundancy code C2N12 1102 from the generated class 1 redundancy code C1N2 802, the data block D2N3 803 transferred from the third node 330, and the data block D1N4 603 transferred from the fourth node 340.

[0133] Step (5): The redundancy processing unit 250 transfers the restored class 2 redundancy code C2N11 1101 and class 2 redundancy code C2N12 1102 to the fifth node, which is the alternative node to be rebuilt, and stores them in the drive 357 of the fifth node.

[0134] In this case, the class 2 redundancy code C2N11 1101 and the class 2 redundancy code C2N12 1102 are restored by the second node, so only the class 2 redundancy code is transferred to the fifth node 350, which serves as the replacement node. In the mDnP redundancy configuration, the amount of data transferred to the replacement node required for restoration is m+n, which is the sum of the data transfer amount m required to restore the data blocks described in FIGS. 9 and 10 and the class 2 redundancy code transfer amount n described in FIG. 11. When multiple redundancy groups exist, the amount of data transferred to the replacement node is the number of redundancy groups × (m+n). Compared to the method of FIG. 8, the method of FIG. 11, which is the present embodiment, reduces the amount of data transfer to (m+n) / (2m+3n-1). For example, in the case of 4D2P, the amount of data transfer is reduced to (4+2) / (8+6-1)=6 / 13.

[0135] Furthermore, when there are multiple redundancy groups, as described above, the data blocks to be redundancy-enhanced and the class 1 redundancy codes generated from the data blocks are transferred to different nodes for each redundancy group, and each node is arranged to generate class 2 redundancy codes. This distributes the restoration process of the class 2 redundancy codes across multiple nodes, thereby reducing the rebuild process load on the alternative node.

[0136] <Rebuild data blocks on alternate nodes> Figure 12 shows an example of selecting a node to rebuild a data block based on a specific condition in Figure 9. In Figure 9, the node that performs the reconstruction of data block D1N1 312 is the second node 320, which stores the class 2 redundancy code used for the reconstruction.

[0137] However, the nodes that rebuild data blocks or class 2 redundancy codes are not limited to these examples. Figure 12 shows a case where the number of nodes is eight, for example. In this case, for example, multiple redundancy groups are allocated to each node, and data blocks stored in drives 317, 327, 337, and 347 (examples of storage devices) of the first node 310, second node 320, third node 330, and fourth node 340 shown in Figures 3, 6, 7, 8, 9, 10, and 11, as well as data blocks belonging to a redundancy group different from the class 2 redundancy code, and class 2 redundancy code are allocated to drives 357, 1067, and 1077 of the fifth node 350, sixth node 1060, and seventh node 1070, respectively. This distributes the data blocks, enabling I / O processing from the host server to be distributed. For the drives 357, 1067, 1077 of the fifth node 350, the sixth node 1060, and the seventh node 1070, new nodes having functions as storage nodes may be prepared and used for rebuilding.

[0138] In FIG. 9, the rebuild of data block D1N1 stored on drive 317 of the first node where the failure occurred was performed by the second node 320, but in the example of FIG. 12, the rebuild is performed by the fifth node 350 based on the following procedure.

[0139] Step (1): The redundancy processing unit 250 reads the class 2 redundancy code C2N21 601 and the class 2 redundancy code C2N22 602 from the drive 327 of the second node 320 and transfers them to memory, and then transfers them to the fifth node 350.

[0140] Step (2): The redundancy processing unit 250 reads the data block D2N4 603 from the drive of the fourth node, transfers it to memory, and further transfers it to the fifth node 350.

[0141] Step (3): In the fifth node 350, the redundancy processing unit 250 restores a data block D1N1 1201 having the same contents as the data block D1N1 312 stored in the first node 310 where the failure occurred, from the class 2 redundancy codes C2N21 601, C2N22 602, and the data block D2N4 603.

[0142] Step (4): The redundancy processing unit 250 transfers the data block D1N1 1201 to the eighth node 1050, which is an alternative node having the function of a storage node for the failed first node 310, and stores it in the drive 1087.

[0143] <Rebuild selected data blocks based on specific conditions on an alternative node> Fig. 13 shows an example of selecting a node to rebuild a data block in Fig. 10 based on specific conditions. Like Fig. 12, Fig. 13 shows an example of rebuilding a data block in an eight-node configuration. Fig. 10 shows an example in which the node that restores data block D2N1 313 is the third node 330, which stores the class 2 redundancy code used for the rebuild. In the example of Fig. 13, the sixth node 360 ​​performs the following procedure.

[0144] Step (1): The redundancy processing unit 250 reads the data block D1N2 701 from the drive 327 of the second node 320, transfers it to memory, and further transfers it to the sixth node 1060.

[0145] Step (2): The redundancy processing unit 250 reads the class 2 redundancy code C2N31 702 and the class 2 redundancy code C2N32 703 from the drive 337 of the third node 330 and transfers them to memory, and further transfers them to the sixth node 1060.

[0146] Step (3): In the sixth node 1060, the redundancy processing unit 250 restores a data block D2N1 1301 having the same contents as the data block D2N1 313 stored in the first node where the failure occurred, from the data block D1N2 701 and the class 2 redundancy codes C2N31 702 and C2N32 703.

[0147] Step (4): The redundancy processing unit 250 transfers the data block D2N1 1301 to the eighth node 1080, which is an alternative node having the function of a storage node for the failed first node 310, and stores it in the drive 357.

[0148] <Rebuilds selected class 2 redundant code on an alternative node based on specific conditions> Fig. 14 shows an example in Fig. 11 in which the redundancy processing unit 250 selects a specific node for rebuilding the class 2 redundancy code based on specific conditions. Fig. 14 shows an example of rebuilding the class 2 redundancy code in an eight-node configuration, similar to Figs. 12 and 13. In Fig. 11, the second node 320, which stores the data blocks used to generate the class 1 redundancy code, is used as the node that performs the restoration of the class 2 redundancy code C2N11 315 and the class 2 redundancy code C2N12 316. However, in the example in Fig. 14, the restoration is performed by the seventh node 1070 based on the following procedure.

[0149] Step (1): Following the same procedure as in FIG. 11, the redundancy processing unit 250 first reads data blocks D1N2 701 and D2N2 801 from the drive 327 of the second node 320, transfers them to memory, and uses them to generate class 1 redundancy code C1N2 1401.

[0150] Step (2): The redundancy processing unit 250 transfers the class 1 redundancy code C1N2 1201 to the seventh node 1070.

[0151] Step (3): The redundancy processing unit 250 transfers the data block D2N3 803 stored in the drive 337 of the third node 330 to the memory, and then transfers it to the seventh node 1070.

[0152] Step (4): The redundancy processing unit 250 transfers the data block D1N4 603 stored in the drive 347 of the fourth node 340 to the memory, and then transfers it to the seventh node 1070.

[0153] Step (5): At the seventh node 1070, the redundancy processing unit 250 restores the class 2 redundancy code C2N11 1402 and the class 2 redundancy code C2N12 1403 from the class 1 redundancy code C1N2 1401 transferred from the second node 320, the data block D2N3 803 transferred from the third node 330, and the data block D1N4 603 transferred from the fourth node 340.

[0154] Step (6): The redundancy processing unit 250 transfers the restored class 2 redundancy code C2N11 1402 and class 2 redundancy code C2N12 1403 to the eighth node 1080, which is an alternative node that functions as a storage node to be rebuilt, and stores them in the drive 1087.

[0155] In this case, the redundancy processing unit (250) selects, for example, based on the load status of the nodes, a specific node (the fifth node 350 in this embodiment) to restore the data block D1N1 1201 in FIG. 12, a specific node (the sixth node 1060 in this embodiment) to restore the data block D2N1 1301 in FIG. 13, and a specific node (the seventh node 1070 in this embodiment) to restore the class 2 redundancy code C2N11 1402 and the class 2 redundancy code C2N12 1403 in FIG. 14.

[0156] The node status may be acquired by the node information acquisition unit 260 included in the storage control program 210 of Fig. 2. In this embodiment, part or all of the restoration of the data blocks described in Fig. 12 and Fig. 13 and the class 2 redundancy code described in Fig. 14 may be performed by the same node.

[0157] <Selecting data blocks and class 2 redundancy codes based on node load status> FIG. 15 shows an example of a procedure for selecting a node for restoring a data block and a class 2 redundancy code based on the load status of the node.

[0158] First, the redundancy processing unit 250 determines whether to execute restoration on an alternative node (step S1510). If execution on an alternative node is selected, the redundancy processing unit 250 transfers data required for restoration to the alternative node and executes restoration on the alternative node (step S1520). On the other hand, if restoration is not to be executed on an alternative node, the redundancy processing unit 250 determines whether to select a specific node on which to execute restoration (step S1530).

[0159] If the redundancy processing unit 250 does not select a specific node to perform the restoration, it performs the restoration at a node designated in a predetermined manner based on, for example, data placement, as shown in Figures 9, 10, and 11 (step S1540).

[0160] When selecting a specific node to perform restoration, the redundancy processing unit 250 acquires node information about each node, for example, in the node information acquisition unit 260 included in the storage control program 210 shown in FIG. 2 (step S1550).

[0161] Examples of node information include hardware operation information such as CPU usage, memory usage, bandwidth usage of network hardware such as a NIC (Network Interface Card) equipped on the node, drive usage, CPU temperature, operating frequency, voltage supplied to the computer, fan rotation speed, etc., obtained at the time of starting restoration, or a combination of these.

[0162] The node information may include, for example, the number of CPU cores, the specified bandwidth performance of the NIC, and other node hardware performance, in addition to the node information acquired by the node information acquisition unit 260. The redundancy processing unit 250 uses at least one of these to calculate the node loads for the operating or standby storage nodes, excluding the failed node, among the storage nodes constituting the distributed storage system, and selects a specific storage node according to the node loads (step S1560).

[0163] The node load may be calculated using, for example, one of the above node information acquired by the node information acquisition unit 260. Alternatively, the node load may be calculated using a combination of multiple pieces of information, such as some or all of the node information. Examples of using node information alone include using the average CPU utilization rate for each CPU core of each node or the CPU utilization rate of the CPU core with the lowest operating rate as a criterion for determining the node load, or using the network bandwidth utilization rate, which indicates the proportion of network bandwidth performance used by I / O from the host server or I / O between storage nodes.

[0164] Alternatively, methods for using these pieces of information in a composite manner may include, for example, applying preset weights to CPU utilization rates and network utilization rates to calculate arithmetic or geometric means, or using mathematical formulas based on statistical analysis of measured values ​​of load trends, or judgment formulas based on machine learning, etc. Also, a threshold may be set for the value of each piece of information, and nodes with values ​​exceeding the threshold may not be selected as restoration destinations.

[0165] The redundancy processing unit 250 compares the loads for each node calculated using these methods between the nodes that are the targets of restoration execution, and ranks the nodes in order of lowest load (step S1570). Furthermore, the redundancy processing unit 250 counts the number of data blocks to be rebuilt and the number of class 2 redundant codes.

[0166] If multiple redundancy groups are used, the redundancy processing unit 250 obtains the number of restorations for all redundancy groups to be rebuilt (step S1580). Data blocks and / or class 2 redundancy codes to be rebuilt are assigned to the nodes to be rebuilt starting with the node with the lowest load, and restoration is performed in a distributed manner across each node (step S1590). When assigning the nodes to be rebuilt, the nodes with the heaviest restoration processing load may be assigned to the nodes with the lightest node load.

[0167] <Do not transfer rebuilt class 2 redundant code to alternate node> 16 shows an example in which the restored class 2 redundant code is not transferred to the alternative node. The class 2 redundant code is not subject to I / O from the application. Therefore, even if the restored class 2 redundant code is stored in a node other than the alternative node, I / O from the application is not affected.

[0168] Therefore, when restoring class 2 redundant code as an example of second redundant code stored in the storage device of one or more of the multiple nodes 140, the redundancy processing unit 250 restores the class 2 redundant code to be rebuilt in a specific node other than the alternative node and stores it in the storage device of that specific node. This makes it possible to reduce the amount of data transferred to the alternative node. Note that the node that performs the rebuild and stores the rebuilt class 2 redundant code may be selected based on the load status of the node using the method shown in FIG. 13.

[0169] In the example of FIG. 16, the seventh node 1070 executes the rebuild based on the following procedure. Step (1): Following the same steps as in Figures 11 and 14, the redundancy processing unit 250 first reads data blocks D1N2 701 and D2N2 801 from the drive 327 of the second node 320 and transfers them to memory, and then uses these to generate class 1 redundancy code C1N2 1401.

[0170] Step (2): The redundancy processing unit 250 transfers the class 1 redundancy code C 1 N 2 1401 of the second node 320 to the seventh node 1070 .

[0171] Step (3): The redundancy processing unit 250 transfers the data block D2N3 803 stored in the drive 337 of the third node 330 to the memory, and then transfers it to the seventh node 1070.

[0172] Step (4): The redundancy processing unit 250 transfers the data block D1N4 603 stored in the drive 347 of the fourth node 340 to the memory, and then transfers it to the seventh node 1070.

[0173] Step (5): At the seventh node 1070, the redundancy processing unit 250 restores the class 2 redundancy code C2N11 1402 and the class 2 redundancy code C2N12 1403 from the class 1 redundancy code C1N2 1401 transferred from the second node 320, the data block D2N3 803 transferred from the third node 330, and the data block D1N4 603 transferred from the fourth node 340.

[0174] Step (6): The redundancy processing unit 250 stores the restored class 2 redundancy code C2N11 1402 and class 2 redundancy code C2N12 1403 in the drive 1077 of the seventh node 370.

[0175] In this case, by restoring the class 2 redundant code to a node other than the node where the data block used to restore the class 2 redundant code and / or the class 2 redundant code is stored, it becomes possible to rebuild the class 2 redundant code even if a failure occurs in the node. In this case, it is assumed that the drive 1077 where the class 2 redundant code is stored has a data capacity sufficient to store the class 2 redundant code. This allows the distributed storage system according to this embodiment to change the storage location of the class 2 redundant code.

[0176] The distributed storage system 130 according to this embodiment has a plurality of nodes, each including a storage device for storing data and a controller 141 for making the data stored in the storage device redundant. Data consisting of a plurality of data blocks is distributed and managed by the plurality of nodes 140, and the distributed storage system receives input / output requests (I / O requests) and write data for reading or writing data from or to a host server 110, and provides the read data to the host server 110. The controller 141 divides data related to the received write request into a plurality of data blocks and writes them to the storage device, generates a class 1 redundancy code as an example of a first redundancy code from the plurality of data blocks, and transmits the plurality of data blocks and the class 1 redundancy code to different other nodes. The controllers of the other nodes generate the plurality of data blocks and class 1 redundancy code received from the plurality of nodes 140 and store them in the storage device, and a redundancy processing unit 2 50 rebuilds the data block and class 2 redundancy code stored in the storage device of the node in another node, and the alternative node that rebuilt the data block stores the restored data block in a storage device and processes read requests and write requests from the host server, and when rebuilding the data block and second redundancy code, any one of the nodes restores the data block to be rebuilt based on the data block and class 2 redundancy code stored in any one of multiple nodes, generates a class 1 redundancy code based on multiple data blocks stored in any one node, restores the class 2 redundancy code to be rebuilt based on the generated class 1 redundancy code and a data block stored in any one of the nodes, and stores the restored data block and class 2 redundancy code to be rebuilt in a storage device, and the node that restores the class 2 redundancy code is a specific node different from the alternative node that stores the restored data block. This makes it possible to reduce the load on the alternative node that stored the data block in processing host I / O after the rebuild (and during the rebuild process).

[0177] As described above, in the distributed storage system according to this embodiment, when rebuilding a data block and class 2 redundant code with the same contents as those stored in a failed node on an alternative node that replaces the failed node in the above-described two-stage redundancy scheme, the class 2 redundant code is restored at a specific node other than the alternative node, then transferred to the alternative node, and stored there, thereby making it possible to reduce the amount of data transferred to the alternative node. Therefore, when a failure occurs, the time required to rebuild the data block and its class 2 redundant code can be shortened.

[0178] Furthermore, according to this embodiment, data in a distributed storage system is made redundant, and in an erasure coding system, data locality is maintained while the amount of data flowing to an alternative node during a redundant code rebuild can be reduced, thereby shortening the time required for the rebuild. Furthermore, according to this embodiment, rebuilding can be performed efficiently by selecting a node with a low processing load. Furthermore, the impact of the rebuild process on host I / O performance can be reduced.

[0179] In this embodiment, the node that restores the data block involved in the rebuild based on the data block and the class 2 redundancy code is a specific node that is different from the alternative node that stores the restored data block.

[0180] In this embodiment, the restored class 2 redundancy code is stored in an alternative node.

[0181] In this embodiment, the node that stores the restored class 2 redundancy code is a node that is different from the alternative node that stores the restored data block.

[0182] The distributed storage system 130 according to this embodiment includes a node information acquisition unit 260 that acquires node information indicating the states of the multiple nodes 140. When rebuilding data blocks and second redundancy codes stored in storage devices of one or more of the multiple nodes 140 on an alternative node, the redundancy processing unit 250 selects a specific node from which to restore the data blocks and class 2 redundancy codes based on the states of the multiple nodes 140 acquired by the node information acquisition unit 260. In this way, a suitable specific node that reduces inter-node transfers can be selected based on the states of the multiple nodes 140 acquired by the node information acquisition unit 260 and used for restoration.

[0183] In this embodiment, the node information acquisition unit 260 acquires the load states of the multiple nodes as the states of the multiple nodes. In this way, the redundancy processing unit 250 can more appropriately select a specific node from which to restore the data block and class 2 redundancy code based on the load states of the multiple nodes 140 acquired by the node information acquisition unit 260, and restore the data block and second redundancy code stored in the storage device of one or more of the multiple nodes 140 to an alternative node.

[0184] In this embodiment, the redundancy processing unit 250 determines whether to restore the data blocks and class 2 redundancy code at an alternative node, and if it is selected to restore the data at the alternative node, it transfers the data required for restoration from the storage devices of the multiple nodes to the alternative node and performs restoration at the alternative node.If it is selected not to restore the data at the alternative node, it selects a specific node to perform restoration, transfers the data required for restoration from the storage devices of the multiple nodes to the specific node, performs restoration at the specific node, and stores the restored class 2 redundancy code in the alternative node.In this way, even if a specific node to perform restoration is not selected, it is possible to select a specific node to perform restoration and restore the data blocks and class 2 redundancy code.

[0185] In the distributed storage system 130 according to this embodiment, the redundancy processing unit 250 uses node information from multiple nodes to calculate node loads for operating or standby storage nodes, excluding the node where a failure occurred, and selects a specific node that is suitable according to the node load. In this way, a specific node that is suitable according to the node load for an operating or standby storage node can be selected.

[0186] In this embodiment, the node information acquisition unit 260 acquires hardware operation information for each node when the rebuild is started. This allows a specific suitable node to be selected based on the hardware operation information for each node acquired when the rebuild is started.

[0187] In this embodiment, the node information acquisition unit 260 acquires, as the above-mentioned hardware operation information, any one or any combination of the following information: CPU usage rate, memory usage rate, bandwidth usage rate of network hardware equipped in the node, drive usage rate, CPU temperature, operating frequency, voltage supplied to the computer, and fan rotation speed. In this way, it is possible to select a suitable specific node according to this hardware operation information and restore the data block and class 2 redundancy code.

[0188] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, the elements described in parallel in the present embodiment may be configured such that at least one of the elements is connected in series to the other elements. [Industrial Applicability]

[0189] The present invention can be applied to a distributed storage system related to a technology for storing data in a redundant manner. [Explanation of symbols]

[0190] 100...user terminal, 110...host server, 120...management server, 130...storage system, 140...storage node, 210...storage control program, 310, 320, 330, 340, 350, 1060, 1070, 1080...node, 317, 327, 337, 347, 357, 1067, 1077, 1087...drive

Claims

1. A distributed storage system having a plurality of nodes, each including a storage device that stores data and a controller that makes the data stored in the storage device redundant, The controller Dividing data related to the received write request into a plurality of data blocks and writing the data into the storage device, generating a first redundancy code from the plurality of data blocks, and transmitting the plurality of data blocks and the first redundancy code to different other nodes, the controller of the other node generates a second redundancy code from the plurality of data blocks and the first redundancy code received from the plurality of nodes, and stores the second redundancy code in the storage device; The data block and the second redundancy code stored in the storage device of the node are rebuilt in another node, and the alternative node that rebuilt the data block stores the restored data block in a storage device and processes read requests and write requests from the host server; When rebuilding the data block and the second redundancy code, any of the nodes may: restoring the data block to be rebuilt based on the data block and the second redundancy code stored in each of the plurality of nodes; generating a first redundant code based on the plurality of data blocks stored in any one of the nodes, and restoring a second redundant code for rebuilding based on the generated first redundant code and the data blocks stored in any one of the nodes; storing the restored data block related to the rebuild and the second redundancy code in the storage device; The node that restores the second redundant data is a specific node that is different from the alternative node that stores the restored data block. A distributed storage system characterized by:

2. The node that restores the data block involved in the rebuild based on the data block and the second redundancy code is a specific node that is different from the alternative node that stores the restored data block. The distributed storage system according to claim 1 .

3. storing the restored second redundant data in the alternative node; The distributed storage system according to claim 1 .

4. The node that stores the restored second redundant data is a node different from the replacement node that stores the restored data block. The distributed storage system according to claim 1 .

5. The controller acquiring node information indicating the state of the node; Selecting specific nodes that will restore the data block and the second redundancy code based on the acquired node information. The distributed storage system according to claim 1 .

6. The node information includes load conditions of the plurality of nodes. The distributed storage system according to claim 4 .

7. The controller determining whether to perform recovery of the data block and the second redundancy code at the alternative node; If execution at the alternative node is selected, data required for restoration is transferred from each storage device of the plurality of nodes to the alternative node, and restoration is executed at the alternative node; If it is selected not to execute the process on the alternative node, a specific node for performing the restoration is selected, data required for the restoration is transferred from each storage device of the plurality of nodes to the specific node, the restoration is performed on the specific node, and the restored second redundant code is stored in the alternative node. The distributed storage system according to claim 3 .

8. The controller Using the node information of the plurality of nodes, the node load is calculated for the storage nodes in operation or on standby, excluding the node in which the failure occurred, and the specific node is selected according to the node load. The distributed storage system according to claim 2 .

9. The controller The node information includes hardware operation information for each node. The distributed storage system according to claim 6 .

10. The node information acquisition unit As the hardware operation information, information on any one of CPU (Central Processing Unit) usage rate, memory usage rate, bandwidth usage rate of network hardware provided in the node, drive usage rate, CPU temperature, operating frequency, voltage supplied to the computer, and fan rotation speed, or any combination thereof, is acquired. The distributed storage system according to claim 7 .

11. A data sharing method for a distributed storage system having a plurality of nodes, each including a storage device that stores data and a controller that makes the data stored in the storage device redundant, comprising: the controller divides data related to the received write request into a plurality of data blocks and writes the data into the storage device, generates a first redundancy code from the plurality of data blocks, and transmits the plurality of data blocks and the first redundancy code to different other nodes, the controller of the other node generates a second redundancy code from the plurality of data blocks received from the plurality of nodes and stores the second redundancy code in the storage device; The controller rebuilds the data block and the second redundancy code stored in the storage device of the node on another node; the replacement node that rebuilt the data block stores the restored data block in a storage device and processes read and write requests from the host server; When rebuilding the data block and the second redundancy code, any one of the nodes restores the data block to be rebuilt based on the data block and the second redundancy code stored in any one of the plurality of nodes, respectively; generating a first redundant code based on the plurality of data blocks stored in any one of the nodes, and restoring a second redundant code for rebuilding based on the generated first redundant code and the data blocks stored in any one of the nodes; storing the restored data block related to the rebuild and the second redundancy code in the storage device; The node that restores the second redundant data is set to a specific node that is different from the alternative node that stores the restored data block. A data sharing method comprising:

Images