Distributed Storage System

The distributed storage system addresses network bottlenecks in Server SAN systems by storing data within nodes and generating redundant codes across nodes, ensuring high reliability and reduced latency.

JP7595711B2Active Publication Date: 2024-12-06HITACHI VANTARA LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023110180
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2014-09-30
Filing Date
2023-07-04
Publication Date
2024-12-06
Estimated Expiration
2035-09-30

AI Technical Summary

Technical Problem

Conventional Server SAN storage systems face network throughput bottlenecks and increased latency due to data distribution across multiple server nodes, which affects data access performance.

Method used

A distributed storage system where data is stored and redundant codes are generated within the same node, then transferred to other nodes, minimizing network communication during read operations and ensuring data protection with redundant codes across different nodes.

Benefits of technology

This approach enhances data reliability and reduces network traffic, maintaining high read performance and reducing storage costs while optimizing data layout to avoid network bottlenecks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007595711000001
    Figure 0007595711000001
  • Figure 0007595711000002
    Figure 0007595711000002
  • Figure 0007595711000003
    Figure 0007595711000003
Patent Text Reader

Abstract

To provide a distributed storage system that achieves high reliability in storage systems.SOLUTION: A first node of a first site receives user data along with a write request from a host device, stores the received user data in a storage device of the first site, and transmits data to a second site for storing redundant data of the user data in another site. A second node of a second site stores redundant data obtained from data for storing the redundant data in another site in a storage device of the second site. The first node receives a read request for the user data from the host device, reads the user data from the storage device of the first site, and transmits the user data to the host device.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

Incorporation by Reference

[0001] This application claims priority to international application PCT / JP2014 / 076105, filed September 30, 2014, the contents of which are incorporated herein by reference. [Technical field]

[0002] The present invention relates to a distributed storage system. [Background technology]

[0003] While IT investments are leveling off, data volumes are increasing. Reducing storage costs is becoming more and more important. For example, ServerSAN-type storage systems, which connect many general-purpose servers over a network to create a storage pool, are expected to become widespread in the future as a distributed storage system. ServerSAN-type storage systems are considered to be an effective solution, particularly for systems that aim for high-performance analysis by equipping server nodes with high-speed SSDs for large-scale big data analysis.

[0004] As a background technique in this technical field, there is U.S. Patent No. 7,546,342 (Patent Document 1). This publication states that "the relative importance of each file associated with a Web site is calculated. The relative importance is used to compute multiple subsets of the content that are distributed across multiple devices in a computer cluster, such as a server array or a peer-to-peer network. The subsets may include encoded messages created using an erasure coding scheme on packets containing portions of one or more files. When a file is retrieved, a certain number of distinctly identifiable encoded messages are retrieved from the device based on this scheme. The file is then recreated using these messages. By storing content on multiple devices, the website can be retrieved significantly faster and more reliably without requiring large amounts of storage or bandwidth on any one computing device." (See abstract.) [Prior art documents] [Patent documents]

[0005] [Patent Document 1] U.S. Patent No. 7,546,342 Summary of the Invention [Problem to be solved by the invention]

[0006] Conventional ServerSAN type storage systems use local storage devices directly connected to each server node as the final storage location, and protect data by distributing write data and its redundant data among multiple server nodes. Specifically, write data from a host is divided into multiple data blocks, redundant codes are generated from the divided data blocks using erasure coding, and the divided data blocks and redundant codes are evenly distributed among multiple server nodes.

[0007] In this way, conventional ServerSAN type storage systems distribute write data received from a host to multiple server nodes. Therefore, when an application program reads data from a ServerSAN type storage, the data block is transferred over the network between the server nodes. This can cause network throughput to become a bottleneck, increasing the latency of accessing data compared to when the data is accessed without going through a network. [Means for solving the problem]

[0008] A representative example of the present invention is a distributed storage system including a plurality of sites communicating via a network, each of the plurality of sites including a plurality of nodes and a storage device for storing data, a first node of a first site among the plurality of sites receives user data together with a write request from a host device, stores the received user data in a storage device of the first site, and transmits data for storing redundant data of the user data to be stored in the first site in another site to a second site among the plurality of sites, a second node of the second site receives data for storing the redundant data in the other site from the first node of the first site, and stores redundant data obtained from the data for storing the redundant data in the other site in a storage device of the second site, and the first node of the first site receives a read request for the user data from the host device, reads the user data from the storage device of the first site, and transmits the user data to the host device. Effect of the Invention

[0009] According to one aspect of the present invention, high reliability can be achieved in a storage system. [Brief description of the drawings]

[0010] [Figure 1] 1 shows an overview of write processing in a distributed storage system. [Diagram 2] 1 shows an example of a mapping image of multiple protection layers in a distributed storage system. [Diagram 3] 1 illustrates an example of a system configuration of a distributed storage system. [Figure 4] This shows information for controlling a distributed storage system. [Figure 5A] 13 shows an example of the configuration of a virtual volume management table. [Figure 5B] 13 shows an example of the configuration of a pool volume management table. [Figure 5C] 13 shows an example of the configuration of a drive management table. [Figure 5D] 13 shows an example of the configuration of a drive status management table. [Figure 5E] 13 shows an example of the configuration of a node status management table. [Figure 5F] 13 shows an example of the configuration of a site status management table. [Figure 6A] 1 shows an example of the configuration of a page mapping table. [Figure 6B] 1 shows an example of the configuration of a page load frequency table. [Figure 6C] 1 shows an example of the configuration of a page load distribution table. [Figure 7A] 1 shows an example of the configuration of a static mapping table. [Figure 7B] 1 shows an example of the configuration of a geo-static mapping table. [Figure 7C] 1 shows an example of the configuration of a consistent hashing table. [Figure 8] 13 shows an example of the configuration of a log-structured mapping table. [Figure 9] 1 shows an example of the configuration of a local area control table 214. [Figure 10] An example of cache information is shown below. [Figure 11] A mapping image of the site protection layer is shown. [Figure 12A] 1 shows state transitions of a node in a distributed storage system. [Figure 12B] 1 shows a state transition of a site in a distributed storage system. [Figure 13] 1 illustrates an example of a logical configuration of a virtual provisioning layer in one node of a distributed storage system. [Figure 14] 1 illustrates an example of page mapping for multiple nodes in a distributed storage system. [Figure 15] 13 shows a flowchart of a read process in the distributed storage system. [Figure 16] 13 shows a flowchart of a synchronous write process. [Figure 17] 1 shows a flowchart of an asynchronous write process. [Figure 18] 13 shows a flowchart of a destage process. [Figure 19] 1 shows a flowchart of a process for managing capacity exhaustion. [Figure 20] 1 illustrates the concept of capacity exhaustion management processing. [Figure 21] 13 shows a flowchart of a save rebuild process. [Figure 22] 1 shows a flowchart of a data resync process. [Diagram 23] 1 shows a flowchart of a relocation and rebalancing process. [Figure 24A] An example of a method for determining a self-threshold value in rearrangement will be described. [Figure 24B] An example of a method for determining a self-threshold value in rearrangement will be described. [Figure 25A] 1 shows a flowchart of a configuration change process. [Figure 25B] An example of adding a stripe type and rearranging stripes when a node is added will be shown. [Figure 26] An example of a command line management I / F is shown below. [Figure 27] 1 shows an example of a management interface for the GUI of a distributed storage system. [Figure 28] 1 illustrates an example of a hardware configuration of a distributed storage system. [Figure 29] In the second embodiment, a method for improving the efficiency of transfer between nodes for redundancy will be described. [Diagram 30] In the second embodiment, a data restoration method in the method for improving the efficiency of transfer between nodes for redundancy described with reference to FIG. 29 will be described. [Diagram 31] In the third embodiment, an example of the hardware configuration of a distributed storage system is shown. [Diagram 32] An outline of the third embodiment is shown below. [Diagram 33] In the third embodiment, a table structure managed by the drive for controlling the storage system is shown. [Diagram 34]In embodiment 3, a communication interface between a computer node and a flash drive is shown. [Diagram 35] 13 shows a flowchart of the process in which a computer node reads the latest data from the D drive in embodiment 3. [Diagram 36] In the third embodiment, the read process of old data is shown. [Figure 37] 13 shows a flowchart of the process in which a computer node writes data to a D drive in embodiment 3. [Figure 38] 13 shows a processing flow when data write processing is performed in parallel to each drive in synchronous write processing in the third embodiment. [Figure 39] 13 shows a flowchart of garbage collection processing in the third embodiment. [Diagram 40] In the fourth embodiment, an example of the hardware configuration of a distributed storage system is shown. [Diagram 41] An outline of the fourth embodiment is shown below. [Diagram 42] In the fourth embodiment, a communication interface between a computer node and a drive is shown. [Diagram 43] 13 shows a flowchart of a synchronous write process in the fourth embodiment. [Diagram 44] 13 shows a flowchart of an asynchronous write process in the fourth embodiment. [Diagram 45] 13 shows a flowchart of a garbage collection process in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The following description of the preferred embodiments of the present invention will be given with reference to the accompanying drawings. Note that the following description of the preferred embodiments does not limit the scope of the invention, and not all of the combinations of features described in the preferred embodiments are necessarily essential to the solution of the invention.

[0012] In the following explanation, various information may be explained using expressions such as "table," "list," and "queue," but the various information may be expressed in other data structures. To indicate independence from the data structure, "XX table," "XX list," and the like may be referred to as "XX information." When explaining the content of each piece of information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable.

[0013] <Embodiment 1> overview This embodiment discloses a distributed storage system. The distributed storage system is configured by connecting a plurality of computer nodes, each including a storage device, via a network. The distributed storage system realizes a virtual storage system that realizes a storage pool by the storage devices of the plurality of computer nodes.

[0014] In one example of a distributed storage system, a computer node stores write data from a host in its own storage device, and then transfers the write data to another computer node in order to protect the data in the event of a computer node failure. Here, the other computer node is called a destination computer node.

[0015] The destination computer node generates a redundant code from the write data transferred from the plurality of different computer nodes, and stores the generated redundant code in a storage device of the destination computer node.

[0016] In this way, by placing data as much as possible in the node that received the write request, inter-node communication during reading is unnecessary, enabling high-speed reading. Meanwhile, by generating redundant code between computer nodes in a node different from the node that received the write request, data protection with small overhead is realized. In particular, when building a distributed storage system with many unreliable nodes, the configuration of the present application, which ensures redundancy while maintaining read performance, is effective.

[0017] In particular, when analytical applications are run on the distributed storage system of the present invention, each computer node is often able to store the data to be analyzed in its own storage area, which reduces the load time for data analysis, improves business agility, and reduces storage costs.

[0018] In one example, the distributed storage system provides a virtual volume to a host. The distributed storage system allocates a logical page from a pool volume to a virtual page that has been write-accessed. The pool volume is a logical volume, and a physical storage area of ​​a storage device is allocated to a logical storage area of ​​the pool volume.

[0019] A computer node selects a virtual page to which a logical page is to be allocated from its own storage device based on the network bandwidth of the distributed storage system and the frequency of access to the computer node for each virtual page from the host. For example, a computer node determines a threshold based on the network bandwidth of the distributed storage system, and places logical pages that are accessed more frequently than the threshold in its own storage device. This makes it possible to avoid network bottlenecks and realize page placement that allows high-speed page access.

[0020] In one example, a computer node has an interface for an application program or a user to specify the location of a virtual page. The virtual page is specified, for example, by a logical address of a virtual volume. The location of the virtual page is indicated by a computer node where the data of the virtual page is stored. By providing an interface for specifying the location of the virtual page, it is possible to implement a page arrangement optimized for the virtual page destination.

[0021] In this embodiment, the distributed network system may include all of the above configuration examples at the same time, or may include only some of the configurations.

[0022] Explanation of terms In this disclosure, a storage device includes a single storage drive such as an HDD or SSD, a RAID device including multiple storage drives, and multiple RAID devices. A stripe or stripe data is a data unit that is the source of generating a redundancy code for data protection. A stripe is sometimes called user data to differentiate it from a redundancy code. A stripe is stored in a storage device in a computer node and is used in generating a redundancy code in other computer nodes.

[0023] A stripe type is a class of stripes that generate redundancy code. The stripe type to which a stripe belongs is determined, for example, by the logical address of the stripe and the computer node that stores the stripe. A stripe type number, which is an identifier of a stripe type, indicates a group of corresponding computer nodes. One stripe can belong to each stripe type of different protection layers. A host is a computer that accesses the storage system, a processor that operates on the computer, or a program that the processor executes.

[0024] FIG. 1 shows an overview of a write process of a distributed storage system according to an example of this embodiment. Computer nodes 101A, 101B, and 101C are included in the same computer domain (hereinafter also referred to as a domain). In the example described below, it is assumed that a domain corresponds to a site. Computer node 101D and computer node 101E are located in a site different from other computer nodes. Computer nodes 101A to 101E communicate via a network. Below, computer nodes are also simply referred to as nodes.

[0025] Each of the computer nodes 101A to 101E includes a cache 181 and a storage drive 113. Each of the nodes 101A to 101E provides a volume 1303.

[0026] The node 101A stores the write data DATA1 (1501A) received from the host in its own cache 181, and further stores it in its own storage drive 113. The write data DATA1 is a stripe.

[0027] The node 101A generates a node redundancy code P from the write data DATA1 and stores it in its own storage drive 113. The node redundancy code is a redundancy code generated from a data unit stored in its own storage device, and is indicated by the symbol P. The node 101A transfers the write data DATA1 in its own cache 181 to the cache 181 of another node 101B.

[0028] The node 101C stores the write data DATA2 (1501B) received from the external device in its own cache 181, and further stores it in its own storage drive 113. The write data DATA2 is a stripe. The node 101C generates intra-node redundant code P from the write data DATA2 and stores it in its own storage drive 113. The node 101C transfers the write data DATA2 in its own cache 181 to the cache 181 of another node 101B.

[0029] In order to protect data in the event of a node failure, node 101B generates a site redundancy code Q (1502B) from data DATA1 and DATA2 stored in its own cache 181 and stores it in its own storage drive 113. The site redundancy code is an inter-node redundancy code within a site, and is indicated by the symbol Q. The site redundancy code Q belongs to a different protection layer than the node redundancy code P.

[0030] The node 101E stores the write data DATA3 (1501C) received from the host in its own cache 181, and further stores it in its own storage drive 113. The write data DATA3 is a stripe. The node 101E generates a node redundancy code P from the write data DATA3, and stores it in its own storage drive 113.

[0031] The node 101A transfers the write data DATA1 in the cache 181 of its own system to the cache 181 of the other node 101D. The node 101E transfers the write data DATA3 in the cache 181 of its own system to the cache 181 of the other node 101D.

[0032] In order to protect data in the event of a node failure, node 101D generates a geo redundancy code R (1502C) from data DATA1 and DATA3 stored in its own cache 181 and stores it in its own storage drive 113. The geo redundancy code is a redundancy code between nodes at different sites, and is indicated by the symbol R. The geo redundancy code R belongs to a different protection layer from the node redundancy code P and the site redundancy code Q.

[0033] Figure 2 shows an example of a mapping image of multiple protection layers in a distributed storage system. Figure 2 shows an image of implementing redundancy between nodes at the same site while also implementing redundancy between sites. For example, a first redundancy is implemented between nodes in a data center, and further, redundancy is implemented between other bases, thereby protecting data in multiple layers and improving the reliability of the system. In Figure 2, only some elements are indicated by symbols, and some of the symbols for elements of the same type are omitted. In Figure 2, square pillars represent nodes, dashed rectangles represent sites (domains), and rectangles within nodes represent stripes or addresses of stripes. (data location). Figure 2 shows four sites 102, with four nodes located at each site. Figure 2 does not show the redundant code generated from multiple stripes.

[0034] The numbers (X_Y) in a stripe 103 indicate the identifier of the stripe type to which the stripe 103 belongs. X is the identifier of the inter-node stripe type within a site (site stripe type), and Y is the identifier of the inter-site stripe type (geo stripe type).

[0035] One stripe 103 belongs to one site stripe type and one geo stripe type. For example, stripe 1_A stored in node 101A1 belongs to site stripe type 1001 and geo stripe type 1002.

[0036] The stripes belonging to the site stripe type 1001 are stripe 1_A of node 101A1, stripe 1_D of node 101A2, and stripe 1_C of node 101A3. Node 101A4 in the same site, which does not hold these stripes, generates and holds redundant codes for these stripes.

[0037] The stripes belonging to the geo stripe type 1002 are stripe 1_A of node 101A1, stripe 1_A of node 101B1, and stripe 2_A of node 101C2. Node 101D4, which is at a different site from these nodes, generates and holds the redundant codes of these stripes.

[0038] In the above configuration, multiple nodes each receive and transfer the stripes (data units) they hold to one destination node, which then generates and holds a redundant code from the transferred data units. The stripes and redundant code are stored in different nodes, providing data protection against node failure.

[0039] A node that receives a host command transmits the received write data to another node without reading old data to generate a site redundancy code or a geo-redundancy code. This improves response performance to write commands. Also, since stripe movement for generating redundancy code is performed between caches and does not involve the drive 113, if the drive 113 uses a flash medium, the lifespan can be improved by reducing the amount of writing.

[0040] Since the node stores the stripe received from the host in its own storage device without dividing it, the response time and network traffic during reads are reduced. In addition, the transfer of redundant code is unnecessary, which reduces network traffic.

[0041] Furthermore, the above configuration can improve the fault tolerance of the system because one stripe belongs to multiple protection layers. The distributed storage system may be configured with a single protection layer that generates only inter-node redundancy code within a site or between sites.

[0042] 3 shows an example of the system configuration of a distributed storage system. The node 101 has, for example, the configuration of a general server computer. The hardware configuration of the node 101 is not particularly limited. The node 101 is connected to other nodes 101 through a port 106 via a network 103. The network 103 is configured, for example, by InfiniBand, Ethernet (registered trademark), or the like.

[0043] A number of nodes 101 form a domain 102. The domain 102 may correspond, for example, to a geographical region or to the topology of a virtual or physical network 103. A network 104 connects multiple domains 102. In the following, a domain is assumed to correspond to a geographically separate site.

[0044] The internal configuration of the node 101 connects the port 106, a processor package 111, and a disk drive (hereinafter also referred to as a drive) 113 via an internal network 112. The processor package 111 includes a memory 118 and a processor 119.

[0045] Memory 118 stores control information necessary for processor 119 to process read and write commands and execute storage functions, and also stores storage cache data. Memory 118 also stores, for example, programs executed by processor 119. Memory 118 may be a volatile DRAM, or a non-volatile SCM (Storage Class Memory), etc.

[0046] The drive 113 is configured by, for example, a hard disk drive having an interface such as FC (Fibre Channel), SAS (Serial Attached SCSI), or SATA (Serial Advanced Technology Attachment), or an SSD (Solid State Drive).

[0047] SCM such as NAND, PRAM, ReRAM, etc. may be used, or volatile memory may be used. When using volatile memory, the storage device may be made non-volatile by a battery.

[0048] The various types of drives mentioned above have different performance. For example, SSDs have higher throughput performance than HDDs. The node 101 includes multiple types of drives 113. The node 101 in this example classifies the different types of drives 113 into groups of drives with similar performance to form tiers 115 and 116.

[0049] The relationship between tiers is defined by the performance of the tier. Performance includes access performance and fault tolerance performance. In the example described below, the access performance of the tiers decreases in the order of Tier 1 to Tier 2 to Tier 3. Also, in the example described below, each of the drive groups in each tier constitutes a RAID. Note that while the number of tiers shown in the example of FIG. 3 is two, the number of tiers depends on the design. Also, a tier with high access performance may be used as a cache. The drives, RAID, tiers, and groups thereof are each a storage device.

[0050] Fig. 4 shows information for controlling the distributed storage system. Memory 118 stores various programs including a storage program, an OS, and an interface program that realizes a storage function, in addition to the information shown in Fig. 4. Memory 118 may further store an application program that executes a task.

[0051] Protection layer information 201 is information related to data protection. Virtualization provisioning information 202 is information related to provisioning of virtual volumes. Cache information 204 is information related to the cache 181. Configuration information 203 is information related to the configuration of the distributed storage system.

[0052] The protection layer information 201 includes static mapping tables 210, 211, and 212 for protection layers No. 1, No. 2, and No. 3, respectively. The protection layer information 201 further includes a log-structured mapping table 213 and a local region control table 214.

[0053] The virtualization provisioning information 202 includes a page mapping table 215, a page load frequency table 216, and a page load distribution table 217. The configuration information 203 includes a virtual volume management table 218, a pool volume management table 219, and a drive management table 220. The configuration information 203 further includes a drive status management table 221, a node status management table 222, and a site status management table 223.

[0054] A copy of all or part of the above information may be stored synchronously or asynchronously in the drive 113. The node 101 may hold the above information for each pool, for example. One pool is made up of one or more logical volumes. This logical volume is also called a pool volume. One pool is made up of one or more hierarchies. In the example described below, the pool is made up of three hierarchies, that is, it is made up of three hierarchies of pool volumes. The substance of the pool volume is the storage area of ​​the drive 113. It is also possible for the storage area of ​​a drive of another node 101 to be allocated to the pool volume.

[0055] Below, an example of the configuration of tables showing information held by node 101 is described. In each table, only some entries are shown. In each table, blank cells are cells in which data is omitted. In table cells, "0x" indicates a hexadecimal number. Drive numbers are unique within a node, and node numbers are unique within a site. Site numbers are unique within the system.

[0056] Figures 5A to 5F show example configurations of tables showing information included in the configuration information 203. Figures 5A to 5C show management information for different storage resource types. Figure 5A shows an example configuration of a virtual volume management table 218. The virtual volume management table 218 shows information on virtual volumes.

[0057] In this example, the virtual volume management table 218 indicates information on virtual volumes provided by the node 101 that holds the information 218. The node 101 accepts access to the virtual volumes it provides. The virtual volume management table 218 may hold information on virtual volumes that the node itself is not the owner of, in preparation for the occurrence of a failure.

[0058] The virtual volume management table 218 includes the size (capacity) of each virtual volume and a list of node numbers of nodes (owner nodes) that provide each virtual volume. It also includes information indicating whether the generation and writing of redundant code for each protection layer is synchronous or asynchronous with the writing of write data to a storage device of its own system. The size of a virtual volume indicates the virtual capacity (maximum capacity) of the virtual volume, not the total amount of allocated logical pages. Synchronous / asynchronous information is provided for each protection layer.

[0059] 5B shows an example of the configuration of the pool volume management table 219. The pool volume management table 219 shows information on pool volumes. In this example, the pool volume management table 219 shows information on pool volumes provided by the node 101 that holds the information 219 and the other node 101 to which the node 101 belongs. The pool volume management table 219 includes information on the size (capacity) of each pool volume and the node number of the node that provides each pool volume.

[0060] 5C shows an example of the configuration of the drive management table 220. The drive management table 220 shows drives allocated to each pool volume. In this example, the drive management table 220 shows information on the drives 113 of the local system included in the node 101 that holds the information 220.

[0061] The drive management table 220 has information on the type of drive (SSD, NL-SAS drive, etc.), the set of drive numbers being striped (the set of drive numbers that make up the RAID), and the size (capacity) of the drive for each pool volume. If not implemented, only one drive is assigned to a pool volume, but different regions of one drive can be assigned to different pool volumes.

[0062] 5D to 5F show failure management information in the distributed storage system, which is held by each of the nodes 101. FIG.

[0063] 5D shows an example of the configuration of the drive status management table 221. The drive status management table 221 shows the status and error count of each drive 113 in the node 101's own system.

[0064] 5E shows an example of the configuration of the node status management table 221. The node status management table 221 shows the status and error count of each of the other nodes 101 in the own site 102. The own site 10c of the node 101 is the site 102 to which the node 101 belongs. When the node 101 detects an error in communication with the other node 101, it increments the error count.

[0065] 5F shows an example of the configuration of the site status management table 223. The site status management table 222 shows the status and error count for each site. In this example, it is assumed that the node 101 can only communicate with the representative node of the other site 102. Therefore, an error in the representative node 101 means an error in the site.

[0066] When the processor 119 of the node 101 detects an error in communication with the drive 113 of its own system or with another node 101, the processor 119 increments the error count in the management information 221-223 held therein.

[0067] When the error count in any hardware resource (drive, node, or site) reaches a first threshold, processor 119 changes the state of the resource from a normal state to a warning state. Furthermore, when the error count reaches the first threshold, processor 119 changes the state of the resource from a warning state to a blocked state. The warning state and the blocked state are abnormal states.

[0068] When each node 101 detects an abnormal state of any of the hardware resources, it notifies the other nodes 101 of that information. Specifically, the node 101 notifies all nodes 101 in its own site 102 and the representative nodes 101 of the other sites 102. The representative node 101 notifies the other nodes in its own site 102 of that information. This allows information on hardware resources in an abnormal state to be shared between nodes. Information on drives in an abnormal state does not need to be shared between nodes.

[0069] The nodes 101 may share information on error counts. For example, when each node 101 detects a communication error with another node or another site, it updates its own management information and broadcasts the updated information to the other nodes 101. The node 101 may determine an abnormal state based on the error counts of the other nodes 101 in addition to the error counts of its own node.

[0070] In a configuration in which a node 101 communicates with each node 101 at other sites 102, the node 101 may count communication errors with the nodes 101 at the other sites 102. The site error count may be, for example, the aggregate of the error counts of all nodes 101 at that site 102.

[0071] 6A to 6C show information included in the virtualization provisioning information 202. 1 shows an example of the configuration of the page mapping table 215. The page mapping table 215 holds the correspondence between the virtual pages of the virtual volume and the logical pages of the pool volume.

[0072] In this example, the page mapping table 215 holds information on virtual volumes provided by the node 101 that holds the information 215. A virtual page may be allocated to a logical page of a pool volume 1303B of another node 101 directly or via a local pool volume 1303C, which will be described later. The page mapping table 215 indicates the relationship between the virtual page and the local pool volume 1303C or the pool volume 1303B of the other node 101.

[0073] The page mapping table 215 holds the first LBA (Logical Block Address) and address range of the virtual page of the virtual volume, and the first LBA of the logical page of the pool volume corresponding to the first LBA of the virtual page.

[0074] 6B shows an example of the configuration of the page load frequency table 216. The page load frequency table 216 holds a history of the I / O frequency (access frequency) for each virtual page. Specifically, it holds the first LBA and address range of the virtual page of the virtual volume, and the access frequency for that area.

[0075] The page load frequency table 216 holds information on virtual pages to which logical pages that store user data (write data) are allocated from a pool volume. Therefore, the page load frequency table 216 also indicates the access frequency of the logical pages allocated to the virtual pages. The page load frequency table 216 holds information on virtual volumes provided by the node 101 that holds the table 216. In addition, the page load frequency table 216 holds information on accesses received by the node that holds the table 216 from its own node and other nodes.

[0076] The access frequency information may be acquired and managed for each access source node, or may be acquired and managed separately for read access and write access. The node 101 may acquire and manage the access frequency information by separating sequential access and random access, or may acquire and manage the access frequency information at multiple measurement periods.

[0077] 6C shows an example of the configuration of the page load distribution table 217. The page load distribution table 217 divides the access frequency of each virtual page into a plurality of levels, and shows the page amount for each level. In other words, the page load distribution table 217 shows the distribution of page amounts relative to the access frequency (I / O frequency). The page load distribution table 217 shows the history of page load distribution.

[0078] The node 101 holds a page load distribution table 217 for each of a plurality of protection layers. For example, the page load distribution table 217 may hold information on the access frequency of each page in the node, information on the access frequency of each page in all nodes in a site, and information on the access frequency of each page in all nodes in a system spanning multiple sites. The node 101 can generate the page load distribution table 217 from the page load frequency tables 216 acquired from its own node and other nodes.

[0079] For example, when multiple nodes 101 provide one virtual volume, each of the multiple nodes 101 receives access to the same virtual page. Therefore, the total number of accesses to one virtual page in all owner nodes of the virtual volume represents all accesses to the virtual page.

[0080] The page load distribution table 217 has a smaller amount of information than the page load frequency table 216, and is basically independent of the storage capacity (logical page amount) of the node 101. Therefore, the page load distribution table 217 can be shared among a large number of nodes 101. Furthermore, by adding up the number of pages for each access frequency level in multiple nodes 101, it is possible to generate a page load distribution across multiple nodes 101, such as page load distribution information for the entire site or the entire system. A page load distribution table 217 may be created for each access source node 101.

[0081] It is efficient to configure the page load frequency table 216 with two types of lists: a top list of pages with high access frequency (high load) (for example, using the Lossy Count Method) and a list of access frequency (page load) for each divided section obtained by dividing the storage area of ​​a node or a group of nodes into a predetermined number of sections. If there is only a top list of high load pages, the top list may become saturated and pages that should be included in the list may not be included when the random load often seen in OLTP databases is wide.

[0082] On the other hand, if the load on a particular page is very high and the number of sections is small due to memory constraints, the width of the section may be too wide, smoothing out the page load and losing the characteristics of the load on each page. Therefore, it is more efficient to have both types of lists.

[0083] The node 101 may have history tables 216, 217 for each predetermined period (for example, one week). This example describes a mapping table (managed by LBA) in block storage, but the node 101 can also hold similar information in commonly known file storage (for example, NFS / CIFS: Network File System / Common Internet File System) or object storage (for example, REST: Representational State Transfer).

[0084] In file storage, the management information may associate files with pages, or may associate small areas into which a file is divided with pages. Also, in object storage, the management information may associate objects with pages, or may associate small areas into which an object is divided with pages.

[0085] 7A to 7C show examples of static mapping tables in the data protection layer information 201. Protection layer number 1 is a protection layer within node 101, and each node 101 holds its own node static mapping table 210. Illustration of the node static mapping table 210 is omitted. The tables in FIGS. 7A to 7C are held, for example, in node 101 with node number 0 which belongs to the site with site number 0.

[0086] 7A shows an example of the configuration of the static mapping table 211 for protection layer number 2 (site). The site static mapping table 211 is information shared between the nodes 101 in the site 102. For each site stripe type number, the site static mapping table 211 holds the relationship between the node number of the data node that stores the corresponding stripe (user data / write data) and the node number of the redundant code node that stores the redundant code generated from the stripe.

[0087] The site stripe type number is identification information of the stripe type within the site. A stripe type is a class of stripes, and one or more redundancy codes are generated from multiple stripes within a stripe type. A stripe is a data unit of a predetermined size.

[0088] The method of determining the stripe type to which each stripe belongs and the method of generating the redundant code will be described later. The stripe type number also indicates the group of nodes 101 that stores the user data and redundant code contained in the stripe type.

[0089] Redundancy codes are generated from multiple stripes from different data nodes belonging to a site stripe. In the example of FIG. 7A, two redundant codes are generated and stored in different nodes 101. The number of redundant codes depends on the design. Multiple redundant codes are generated by, for example, erasure coding. The site static mapping table 211 may be shared between sites if there are no memory or security constraints.

[0090] In this example, one stripe belongs to a single site stripe type. As illustrated in Fig. 7A, stripes stored by one node may belong to different stripe types. For example, in the example of Fig. 7A, one stripe stored by node 0x00 belongs to site stripe type 0x0000, and another stripe belongs to site stripe type 0x0001.

[0091] 7B and 7C show configuration examples of a geo static mapping table 212A and a consistent hashing table 212B included in the static mapping table 212 of protection layer number 3 (geo). The geo static mapping table 212A basically has the same configuration as the site static mapping table 211. The geo static mapping table 212A is shared between sites.

[0092] The geo static mapping table 212A holds, for each geo stripe type number, the relationship between the site number of the data site where the corresponding stripe is located and the site number of the redundant code site where the redundant code is located. One node 101 at each data site stores the stripe. Also, one node 101 at each redundant code site stores the redundant code.

[0093] The consistent hashing table 212B indicates information for identifying the node 101 that stores the redundant code within the redundant code site. Each site 102 holds its own consistent hashing table 212B. The information in the consistent hashing table 212B differs for each site.

[0094] The consistent hashing table 212B indicates the relationship between the node number of the node 101 in the redundant code site, the hash value when the node 101 stores redundant code (1), and the hash value when the node 101 stores redundant code (2). The hash value is calculated based on information about the transfer source that is transferred together with the stripe from another site 102. The stripe is transferred to the node 101 associated with the calculated hash value, and the destination node 101 generates and stores a redundant code.

[0095] 7A to 7C are changed when the storage destination of user data (stripes) and redundant code is changed to a spare area in the event of a node / site failure, and are further changed when nodes or sites are added or removed.

[0096] The nodes 101 may share the same computation logic so that they can uniquely change the static mapping table from the information of the failed node / site. This eliminates the need for the nodes 101 to multicast the static mapping table they hold after changing it, thereby reducing the load on the network.

[0097] By defining in advance which nodes belong to which stripe type in the static mapping table, it is possible to realize a suitable redundant configuration in terms of data recovery. By including different stripe types in the data of one node and specifying the number of stripe types to which one node belongs, it is possible to increase the possibility of data recovery when a node is blocked. The method of using the site static mapping table 211 will be described later with reference to FIG. 11.

[0098] 8 shows an example of the configuration of the log-structured mapping table 213 in the data protection layer information 201. In Fig. 8, the arrows represent pointers. The log-structured mapping table 213 includes a data mapping table 701, a geo / site / node code mapping table 702, and a reverse mapping table 703.

[0099] The data mapping table 701 manages user data (stripes) stored in a storage device (drive 113) of the node 101 that holds the table 701. The node 101 can know the storage address (physical address) of the drive 113 (physical storage device) that stores the stripe from the storage address (logical address) of the pool volume of the stripe.

[0100] The data mapping table 701 associates the storage address (logical address) of user data (stripe) in the pool volume with the storage address (physical address) in the physical storage area of ​​the drive 113.

[0101] The storage address of a stripe in a pool volume is specified by the LDEV number of the pool volume and the stripe number of the stripe, and each block in the stripe is specified by an LBA offset. The stripe size is constant. The stripe number is calculated, for example, by floor (LBA / stripe length). The storage address of a physical storage area is specified by the drive number, LBA, and data length.

[0102] 8, one stripe is stored divided into two physical areas (blocks). The data mapping table 701 shows that the data with LDEV number 0, stripe number 0, and LBA offset 0 in the stripe is stored in an area with drive number 0x43, LBA 0x0003, and data length 8. Furthermore, it shows that the data with LDEV number 0, stripe number 0, and LBA offset 1 in the stripe is stored in an area with drive number 0x42, LBA 0x0007, and data length 8.

[0103] The physical storage area further stores information indicating the state of the stored data. The state information indicates whether the data has been copied (transferred) to the corresponding redundant code node. As will be described later, the write data (stripe) is transferred to the redundant code node for redundant code generation synchronously or asynchronously with the host write process of the write data (stripe) according to the Sync / Async setting.

[0104] The redundancy code mapping table 702 manages the redundancy codes stored in the storage devices (drives 113) of the node 101 that holds this table 701. The managed redundancy codes include inter-site redundancy codes (geo redundancy codes R), intra-site redundancy codes (site redundancy codes Q), and intra-node redundancy codes (node ​​redundancy codes Q). The node 101 can know the physical address of the redundancy code of a stripe from the logical address of the pool volume that stores the stripe.

[0105] The redundancy code mapping table 702 associates the logical address of the pool volume of the stripe used to generate the redundancy code with the physical address of the physical storage area of ​​the drive 113 (the storage device of the own system). It is generated by an operation (e.g., xor). Therefore, the logical addresses of multiple stripes are associated with the physical address of the redundant code.

[0106] Fig. 8 shows an example of generating one redundancy code from two stripes. In the example of Fig. 8, the redundancy code mapping table 702 shows the relationship between the physical address of one geo redundancy code and the logical addresses of the two stripes used to generate the geo redundancy code. The logical address of the stripe is indicated by the site, node, pool volume identifier, and intra-volume address. The geo redundancy code is stored in two separate address areas (blocks) in the physical storage area.

[0107] For example, the block of geo-redundancy code generated from the block at LBA offset 0 in the stripe at site number 4, node number 3, LDEV number 7, and stripe number 8, and the block at LBA offset 0 in the stripe at site number 6, node number 5, LDEV number 4, and stripe number 13, is stored in the area with drive number 0x40, LBA 0x0020, and data length 8.

[0108] The distributed storage system of this example stores data in a log-structured manner. In the log-structured manner, when data at a logical address is updated with new data, the new data is added to a new physical address without updating the data at the physical address with the new data. Unnecessary data is erased as appropriate. The log-structured manner eliminates the need to read data to update the node redundancy code P, and can reduce the time required for write processing to the drive 113. The distributed storage system does not need to implement the log-structured manner.

[0109] Therefore, old data and new data can be stored in the physical storage area as data of a logical address. The log-structured mapping table 213 holds information on the relationship between the logical address and the physical address of the latest data, as well as information on the relationship between the logical address and the physical address of the old data, and generation management information for data. The generation management information for the redundant code generated from multiple stripes indicates generation information for each stripe used in generating the redundant code.

[0110] Furthermore, a data guarantee code (write sequence number, CRC, etc.) may be added to the data mapping table 701 and the redundancy code mapping table 702. By adding this information, data consistency can be checked by simply referring to the information in this mapping table once during address conversion.

[0111] The reverse mapping table 703 is a reverse conversion table of the above tables 701 and 702. That is, the reverse mapping table is referred to for conversion from a physical area address to a pool volume address. The reverse mapping table 703 includes a table 732 indicating logical addresses corresponding to each of address areas 731 that store data in the physical area.

[0112] Each of the tables 732 includes a data type (stripe / geocode / site code / node code), an index number (number of references), an update date and time, and a reference (corresponding pool volume area, site number, node number, etc.).

[0113] For example, Fig. 8 shows information of logical addresses corresponding to physical addresses storing geo-redundancy codes as an example. This example is consistent with the example of the redundancy code mapping table 702 in Fig. 8. The data type is geo-redundancy code, and the index number is 2. This indicates that two stripes are used to generate the geo-redundancy code.

[0114] The reference is the logical address where the stripe used to generate the geo-redundant code is stored. The logical address is indicated by a site number, a node number, an LDEV number, a stripe number, and an offset.

[0115] As described above, by managing the transfer source address of each stripe that is the source of the redundant code in association with the physical address of the redundant code, it is possible to appropriately manage redundant codes for various combinations of stripes.

[0116] If the drive 113 includes a non-volatile medium, the node may add update information to the deferred mapping table 703 in synchronization with the drive write of the user data. This enables data recovery in the event of an unexpected power loss. The node 101 may store the deferred mapping table 703 in the memory 118 and update the deferred mapping table 703 in the drive 113 asynchronously with the drive write of the user data. In order to enable data recovery in the event of an unexpected power loss, the deferred mapping table 703 may hold a write sequence number. The deferred mapping table 703 may hold information on old data in addition to information on the latest data.

[0117] Fig. 9 shows an example of the configuration of the local area control table 214. In Fig. 9, the arrows represent pointers. The local area control table 214 includes a valid list 801A, an invalid list 801B, a free list 801C, and a local area amount table 802. The local area control table 214 manages the areas of the drive 113 in the node 101. The arrows in the lists 801A to 801 represent pointers. In the lists 801A to 801, each area is indicated by a drive number and an LBA in the drive.

[0118] The valid list 801A is a list of valid areas. A valid area is an area that stores the latest user data or the latest redundant data. In the example of Fig. 9, in the drive 113 with drive number 0, the blocks of LBA 0, 4, and 5 each store valid data.

[0119] The invalid list 801B is a list of invalid areas. An invalid area is an area that stores old user data or old redundant code. An old, invalid redundant code is a redundant code in which all stripes used to generate the redundant code are invalid. In the example of FIG. 9, in the drive 113 with drive number 0, blocks at LBAs 1, 3, and 7 each store invalid data. The free list 801C is a list of unused areas.

[0120] The local area amount table 802 manages each stripe type, node redundancy code, site redundancy code, geo redundancy code, target usage amount of spare area, actual usage amount, and available area amount. The node 101 holds a local area amount table 802 for each tier. Each entry in the local area amount table 802 may indicate the total amount for all tiers. By managing the amount of each stripe type and redundancy code individually, the amounts for various data can be appropriately controlled. The processor 119 updates the local area control table 214 synchronously or asynchronously with host I / O.

[0121] For example, the local area capacity table 802 holds only entries of the stripe type to which the local node 101 belongs. The local area capacity table 802 may include entries for data of a stripe type to which the local node 101 does not belong in order to manage the usage amount of data transferred from other nodes 101.

[0122] 10 shows an example of the cache information 204. Each node 101 holds its own unique cache information 204. The cache information 204 includes a data dirty queue 900, a code dirty queue 901, a clean queue 902, a free queue 903, and an intermediate dirty queue 904. The dirty queues 900, 901, and 904 are used to store data that has not yet been stored in the drive 113. The data on the reflective cache 181 is shown.

[0123] A cell in the queue indicates an entry, and the information of the entry corresponds to the information in the cache bitmap table 905, and stores information selected from the cache bitmap table 905. The arrows in the queue represent pointers connecting the entries. The black circle is the starting point.

[0124] The data dirty queue 900 indicates write data (stripe) from a host that is stored in the drive 113 of its own system. The write data belongs to one of the site stripe types. The data dirty queue 900 includes a queue for each site stripe type to which the node 101 belongs as a data node.

[0125] The code dirty queue 901 indicates a stripe for generating redundant code on the cache 181 that has not been reflected in the drive 113. The stripe and the redundant code generated from the stripe are dirty data.

[0126] The code dirty queue 901 includes queues for stripes received from other nodes for redundant code generation. Since the node 101 belongs to multiple protection layers, queues of stripe types of different protection layers are prepared. In the example of Fig. 10, queues of site stripe type and geo stripe type are shown. A dirty queue for each pair of stripe type and data location (node) is used.

[0127] Each queue indicates a list of data that belongs to the corresponding stripe type and is stored in the physical area of ​​the corresponding node. The queue for "SITE STRIPETYPE#0,0" is a queue for data that belongs to the site stripe with site stripe type number 0 and is stored in the node with node number 0.

[0128] The intermediate dirty queue 904 indicates intermediate code in the cache 181 that has not been reflected in the drive 113. The intermediate code is data generated from a new stripe and an old stripe. For example, it is an xor of the new stripe and the old stripe. The intermediate code is differential data between the new stripe and the old stripe, and the node 101 can use the intermediate code to update the redundant code of the old stripe stored in the drive 113 to the redundant code of the new stripe. The method of using the intermediate code will be described in detail later.

[0129] The configuration of the intermediate dirty queue 904 is the same as the queue for redundant code in the dirty queue 901. That is, in this example, a queue is used for each pair of stripe type and data location (node). Since the node 101 belongs to multiple protection layers, stripe type queues for different protection layers are prepared. In the example of Fig. 10, site stripe type and geo stripe type queues are shown.

[0130] The clean queue 902 indicates data in the cache 181 that has been reflected in the drive 113. The free queue 903 indicates an area of ​​the cache 181 that is not being used.

[0131] The cache bitmap table 905 includes a logical address of data, a cache address (location in memory), a size, a dirty bitmap, and a staging bitmap. For example, one entry indicates information of one slot of a predetermined size in the cache 181.

[0132] The logical address corresponds to the logical address of the stripe described with reference to Fig. 8. The logical address of the stripe transferred from the other node 101 may be, for example, a site number, a node number, The bitmap includes the number, LDEV number, LBA, and offset. The dirty bitmap indicates which part of the area is dirty. The staging bitmap indicates which part of the area has been staged on the cache 181. For example, one bit corresponds to one block of the drive 113.

[0133] Figure 11 shows a mapping image of the site protection layer (layer number 2). Basically, the mapping images of the node protection layer (layer number 1) and the geo protection layer (layer number 3) are similar. In the following, the number of cycles of the stripe type is represented as c, the number of redundant codes (number of parities) as p, and the number of stripes (number of data) as d.

[0134] In the example of FIG. 11, the number of cycles is 5, the number of redundant codes is 1, and the number of stripes is 3. Specifically, in one site stripe type, one redundant code is generated from a maximum of three stripes and stored in a node in the site stripe type. As described later, a redundant code is generated from any number of stripes up to three. When multiple redundant codes are generated, they are distributed and stored in different redundant code nodes.

[0135] Table 621 shows stripe type data nodes and redundant code nodes. The columns correspond to nodes with node numbers 0 to 8. The capacity of each of the columns 622 showing the physical storage areas of the nodes with node numbers 0 to 8 is indicated by the height of the column.

[0136] In table 621, the numbers in the cells indicate stripe type numbers. Cells in the D region indicate the stripe type numbers to which data nodes belong. Cells in the Q region indicate the stripe type numbers to which redundant code nodes belong.

[0137] The cells in the S area indicate the stripe type number to which the spare node belongs and the type of data stored (stripe / redundancy code). A spare node is a node that temporarily stores the data of a node when a failure occurs in the node. This allows redundancy to be restored in the event of a node failure.

[0138] The stripe type number of the write data is determined by the stripe number of the write data and the node number of the node that receives and stores the write data. Specifically, the node 101 determines the stripe number by (logical address value of the write data ÷ stripe size). In this example, the logical address is a logical address in the pool volume. A logical address in the virtual volume may also be used. Furthermore, the node 101 calculates the row number of the write data by (stripe number modc).

[0139] The node 101 refers to the site static mapping table 211 of layer number 2 and determines the stripe type number from the node number of its own device and the calculated row number. For example, the node 101 selects an entry including its own node number as a data node in the site static mapping table 211, and determines the site stripe type number of the entry in the order indicated by the row number as the site stripe type number of the write data.

[0140] Furthermore, the node 101 determines the redundant code node of the write stripe type to which the stripe belongs by referring to the site static mapping table 211 of layer number 2. This point will be explained again in the explanation of the write processing to be described later.

[0141] 11, for example, the stripe with row number 0 in the nodes with node numbers 0, 5, and 7 belongs to stripe type number 0. The stripe with row number 4 in the nodes with node numbers 1, 3, and 8 belongs to stripe type number 13.

[0142] Furthermore, the node number of a redundant code node belonging to a stripe type with stripe type number 0 is 1, and the node number of a redundant code node belonging to a stripe type with stripe type number 13 is 4. Some nodes store redundant code for multiple stripe types.

[0143] In the example of FIG. 11, the distribution of stripes in the D area is uniform. The number of data nodes of a stripe type may be changed depending on the storage capacity of the node. Also, when the total number of nodes is small or a fractional number occurs, the number of redundant codes of some stripe types may be reduced. Different stripe types may be made redundant using different algorithms.

[0144] A redundant code node in a stripe type is selected from a node different from the data node of that stripe type. Data writes from data nodes are concentrated in the redundant code node. Therefore, the redundant code node is selected so that the redundant codes are arranged as evenly as possible. This equalizes the lifespan of the nodes 101. This is particularly effective when the drives 113 are SSDs. If the lifespans are uneven between nodes, the arrangement of the redundant codes Q may be changed to equalize them.

[0145] A spare node is a temporary storage destination for recovering redundancy when a node failure occurs. A spare node that stores redundant code is selected from among nodes different from data nodes of the same stripe type. In the example of FIG. 10, a failure occurs in node number 6. A spare node associated with the stripe type number of a stripe or redundant code temporarily stores the corresponding stripe or redundant code.

[0146] For example, node number 0 stores the stripe with stripe type number 2 that was stored by node number 6. Node number 7 stores the redundant code Q with stripe type number 3 that was stored by node number 6. Data recovery is performed by the node that stores the data or a different node. Data (stripes and redundant code) stored in a spare node is returned to one node from the spare node when the node is recovered or when a node is added.

[0147] In the above example, the stripe type is determined by the address within the pool volume, regardless of the LDEV number of the pool volume. Data at the same address within the same volume in different pool volumes belongs to the same stripe type. The address area of ​​one pool volume is classified into multiple stripe types. As will be described later, the redundant code node selects any number of stripes from stripes within the same stripe type, regardless of the address within the volume of the stripe, and generates redundant code from the selected stripes.

[0148] Fig. 12A shows state transitions of a node 101 in the distributed storage system. Fig. 12B shows state transitions of a site 102 in the distributed storage system. Basically, state transitions are similar in each protection layer.

[0149] The normal state indicates the initial state and the normal state during operation. The state transitions to the rebuilding state if a drive failure occurs. During the rebuild, the node 101 can accept application I / O by collection read / write.

[0150] In a blocked state, the node 101 is down and I / O cannot be performed. However, the drive 113 may not be broken. In that case, the data is resynced to the blocked node 101, which reflects only the data that was newly written after the blockage occurred. It is possible to recover and change the blocked state to a normal state.

[0151] 13 shows an example of a logical configuration of a virtual provisioning layer in one node 101 of a distributed storage system. Virtual volumes 1301A and 1301B are virtual storage areas recognized by a host (the same node or another node), and are the volumes that are the target when a read or write command is issued from the host.

[0152] The pool 1306 is configured from one or more pool volumes. In the example of Fig. 13, the pool 1306 includes pool volumes 1303A to 1303F. The pool 1306 may also include pool volumes of other nodes. The pool volumes 1303A to 1303F are configured from the storage areas of the drives 113. Specifically, the processor 119 configures logical pool volumes by managing the correspondence between the logical addresses of the pool volumes and the physical addresses of the drives 113. This will be described in detail later.

[0153] A storage administrator can create multiple virtual volumes on the pool 1306 by issuing an instruction to the processor 119 via an input / output device. The processor 119 allocates a real storage area from the pool 1306 only to a storage area in a virtual volume for which a write command has been issued.

[0154] 13, virtual volume 1301A includes virtual pages 1302A, 1302B, and 1302C, to which logical pages 1304A, 1304E, and 1304C are respectively allocated. Virtual volume 1301B includes virtual pages 1302D and 1302E, to which logical pages 1304D and 1304F are respectively allocated.

[0155] Logical pages are dynamically allocated to virtual pages. For example, when a write command is issued for the first time to a virtual page 1302A of a virtual volume 1301A, the processor 119 associates it with an unused area (logical page 1304A) of a pool volume 1303A (association 1305A). For the next read / write command to the same page, the processor 119 also executes I / O processing for the logical page 1304A of the pool volume 1303A based on the association 1305A.

[0156] The above operation makes it appear to the host that the node 101 is executing I / O processing (access processing) to the virtual volume. Using the virtual volume, the area of ​​the pool volume is allocated only to the area to be used, allowing efficient use of limited storage area. When all data of a logical page allocated to a virtual page is erased, the processor 119 cancels the association between the virtual page and the logical page, and manages the logical page as an unused page. This allows the limited storage area to be used more efficiently.

[0157] Pool 1306 is made up of multiple tiers 115, 116, and 117. In this example, pool 1306 has three tiers: SSD tier 115 (TIER 1), SAS tier 116 (TIER 2), and SATA tier 117 (TIER 3). SSD tier 115 has the highest performance, and SATA tier 117 has the lowest performance. Pool volumes are classified into tiers 115, 116, and 117, and each pool volume belongs to one of the tiers. Pool volume 1303A belongs to tier 115, pool volumes 1303B and 1303C belong to tier 116, and pool volumes 1303D and 1303E belong to tier 117.

[0158] Each virtual page has its own characteristics for I / O processing from the host. For example, there are virtual pages with high I / O frequency (access frequency) and virtual pages with low I / O frequency. This is called access locality. Virtual pages with high I / O frequency are placed in the upper tier, i.e., A virtual page is allocated to a logical page in a higher tier, thereby improving the performance of the entire system. Note that when a virtual page is allocated to a logical page in any tier, the virtual page is also expressed as being allocated to a tier or to a pool volume.

[0159] For example, suppose that pool 1306 contains SSD tier 115 capable of processing 100 IOPS and SAS tier 116 capable of processing 10 IOPS, and that there are virtual pages 1302A with 20 IOPS characteristics and 1302C with 50 IOPS characteristics. Suppose page 1302A is assigned to SSD tier 115, and page 1302C is assigned to SAS tier 116. Since SAS tier 116 can only deliver a maximum performance of 10 IOPS, node 101 as a whole can only deliver a performance of 10 + 20 = 30 IOPS. This state is called a bottleneck state.

[0160] If the allocation of virtual page 1302C can be promoted from the SAS tier 116 to the SSD tier 115, node 101 can achieve an overall performance of 50+20=70 IOPS. As described above, by allocating virtual pages with high I / O frequency to higher tiers, the performance of the entire system can be improved.

[0161] The above promotion copies the data of logical page 1304C to unused logical page 1304B, and changes the association (1305C) between virtual page 1302C and logical page 1304C to the association (1305B) between virtual page 1302C and logical page 1304B. Demotion of a page can also be executed in a similar manner.

[0162] The graph 271 represents the distribution of the I / O frequency (I / O load) of pages. The processor 119 can create load distribution data showing this graph 271 from the page load distribution table 217. The distribution line 1309 is a line that represents the number of I / Os for each page when the pages are arranged in order of the most frequent I / O frequency. In other words, pages with a high I / O count are located on the left side, and pages with a low I / O frequency are located on the right side. The tier allocation thresholds 1308A and 1308B are thresholds that determine which tier a page with a certain I / O frequency is to be allocated to.

[0163] As mentioned above, by allocating pages with high I / O frequency to higher tiers, the performance of the entire system can be improved. Therefore, virtual pages can be allocated from higher tiers in descending order of I / O frequency. After starting up the storage system, when the page load distribution 271 has not yet been created, the initial values ​​of the tier allocation thresholds 1308A and 1308B may be, for example, 0.

[0164] For example, the processor 119 allocates pages belonging to the page range 1310A with the highest I / O frequency from the intersection of the tier allocation threshold 1308A and the distribution line 1309 to the SSD tier 115. The processor 119 allocates pages belonging to the range 210B from the intersection of the tier allocation threshold 1308A and the distribution line 1309 to the intersection of the tier allocation threshold 1308B and the distribution line 1309 to the SAS tier 116. The processor 119 allocates pages from the intersection of the tier allocation threshold 1308B and the distribution line 1309 to the page with the lowest I / O frequency to the SATA tier 117.

[0165] The storage administrator may specify the values ​​of the tier allocation thresholds 1308A, 1308B, or the processor 119 may calculate the values ​​of the tier allocation thresholds 1308A, 1308B. For example, the processor 119 may determine the tier allocation threshold that defines the tier based on the I / O frequency distribution of the virtual page, the capacity of the tier, and the drive performance of the tier. The drive performance of the tier is defined in advance, for example, by the amount of IO data per unit time in the tier.

[0166] Fig. 14 shows an example of page mapping of a plurality of nodes in a distributed storage system. In Fig. 14, the distributed storage system provides virtual volumes 1301A to 1301C. Node 101A provides virtual volume 1301A. Node 101B provides virtual volumes 1301A and 1301B. Node 101N provides virtual volume 1301C.

[0167] A node 101 (any of the nodes 101A to 101N) can hold two types of volumes. One is a pool volume 1303A made up of a storage area of ​​the drive 113 of its own system. The data stored in the pool volume 1303A is arranged in the drive 113 of its own system.

[0168] The other is a volume 1303C that performs straight mapping on the pool volume 1303B of another node 101. The volume 1303C is managed as a pool volume. The node 101 can perform I / O processing of the pool volume 1303B of the other system via the pool volume 1303C.

[0169] This function is known as a storage external connection function. The node 101 converts the access address of the volume 1303C into the address of the pool volume 1303B of the other system, and transmits a command to the node 101 of the other system. The node 101 holds an address mapping table (not shown) between the pool volume 1303C of its own system and the pool volume 1303B of the other system.

[0170] The processor 119 maps virtual pages with a large amount of host access in its own system to the Own pool volume 1303A, and maps virtual pages with a large amount of host access in the other system to the pool volume 1303B of the other system. This shortens the response time to the host. The data of the virtual pages allocated to the pool volume 1303B of the other system is stored in the drive 113 of the other system.

[0171] Each node 101 selects the number of pool volumes of other systems to be mapped and the virtual pages to be allocated to the pool volumes of other systems based on the network performance and the drive performance of its own system in each hierarchical layer, and allocates logical pages so that the network does not become a bottleneck. Details of this allocation will be described later with reference to Figures 23, 24A, and 24B.

[0172] The distributed storage system may manage storage capacity in the entire system, and may increase or decrease the number of pool volumes in each node 101 according to the page usage capacity of the virtual volume. The node 101 may use the pool volume 1303A as a virtual volume by straight mapping. This can reduce memory usage of the mapping table, and improve performance and availability.

[0173] 15 shows a flowchart of a read process in a distributed storage system. The processor 119 refers to the page mapping table 215 to determine whether or not the access destination virtual page for the specified address of the received read command is unallocated to a pool volume (S501). The specified address is specified by, for example, a virtual volume number and a logical address. The LBA is represented by a start LBA and a block length.

[0174] If the virtual page is unallocated (S501: Y), the processor 119 determines whether or not exclusion is necessary (S506). The processor 119 refers to the virtual volume management table 218, and determines that exclusion is unnecessary if the own node is the only owner node of the virtual volume.

[0175] If exclusion is necessary (S506: Y), the processor 119 acquires exclusion (S507) and again determines whether the virtual page is unallocated to a pool volume (S508). In one example of an exclusion method, the processor 119 uses a hash function to identify a representative node that is uniquely determined from the read address, requests arbitration from the representative node, and the representative node performs the arbitration.

[0176] If the virtual page has already been allocated (S508: N), the processor 119 releases the exclusion (S512) and proceeds to step S502. If the virtual page has not yet been allocated to a logical page (S508: Y), the processor 119 returns zero data (S509) and determines whether exclusion is necessary (S510) in the same manner as in the determination in step S506. If exclusion is necessary (S510: Y), the exclusion has already been acquired, so the processor 119 releases the exclusion (S511).

[0177] If the virtual page has been allocated in step S501 (S501: N) and is allocated to a pool volume (S502: Y), the processor 119 secures a cache area for its own system, reads data from the pool volume, and returns the read data (S504). The processor 119 refers to the pool volume management table 219 and external connection management information (not shown) to determine whether the virtual page has been allocated to a pool volume for its own system.

[0178] When a virtual page is allocated to a pool volume 1303B of another node 101 via a pool volume 1303C of the own system, the virtual page is determined to be allocated to a pool volume of the other system.

[0179] When reserving a cache area, the processor 119 refers to the cache information 204 to identify the cache area associated with the target logical address. If no corresponding cache area exists, the processor 119 reserves a new area from the free queue 903. If the free queue 903 is empty, the processor 119 reserves a new area from the clean queue 902. If the clean queue 902 is empty, the processor 119 destages an area in the dirty queue 900, 901, or 904, changing it to a free area.

[0180] If the access destination virtual page is allocated to a pool volume of another system (S502: N), the processor 119 transfers a read command to the other node 101 (S505). The processor 119 does not cache the read data in its own system. In other words, if the allocation destination of the virtual page is another node, the processor 119 does not cache the read data in its own system memory 118 (read-through), and the other node 101 caches the read data.

[0181] Fig. 16 shows a flowchart of synchronous write processing. This processing is executed when a write command is issued from a host (for example, an application program). In addition to storing the write data in the pool volume of its own system, this processing transfers the write data to another node 101 in order to generate a site redundancy code (inter-node redundancy code) and a geo redundancy code (inter-site redundancy code).

[0182] The processor 119 of the node 101 that received the write command judges whether the page is unallocated or not (S601). Specifically, the processor 119 refers to the page mapping table 215, and searches for the corresponding pool volume number and LBA from the specified address (virtual volume number and LBA) of the write command. The processor 119 judges whether the virtual page is unallocated or not based on the presence or absence of corresponding address information.

[0183] In the system of this example, multiple applications are running, and at least one node in the system is running each application. Here, it is considered that a data read request is often issued to the node that originally received the write command for that data. Therefore, in this application, when a node receives a write request, the data of the write request is preferentially stored in the storage area of ​​the node. This increases the probability that a read request can be read from the node, making it possible to respond to the read request quickly.

[0184] However, when the performance of the drives 113 connected to the node 101 is low compared to the performance of the network 103, there are cases where the throughput of the system can be improved by distributing data among a large number of nodes. Taking the above into consideration, the allocated storage area may be changed using a technique such as round robin depending on the performance of the network 103 and the performance of the drives 113 connected to the node 101. Furthermore, the above-mentioned allocation policy is not only based on an index of performance, but also on the use of an index such as lifespan when flash is used as the drive 113, to improve the efficiency of cost-effectiveness.

[0185] If the virtual page is unallocated (S601: Y), the processor 119 executes a process to allocate the virtual page to a pool volume. The processor 119 first determines whether or not exclusivity is required for updating the page mapping table 215 (S611). The reason for obtaining exclusivity is to prevent areas of multiple different pool volumes from being allocated to the virtual page when the virtual page is allocated simultaneously in other nodes 101.

[0186] The processor 119 refers to the virtual volume management table 218, and if the owner node includes any node other than the own node, it determines that exclusion is necessary, and if the owner node is only the own node, it determines that exclusion is unnecessary. If it is determined that exclusion is necessary (S611: Y), the processor 119 acquires exclusion (S612). The method of acquiring exclusion is the same as the method shown in the read processing described in FIG. 16.

[0187] Next, the processor 119 again determines whether the virtual page is unallocated (S613). This is because after determining whether the virtual page has been allocated in step S601, and before obtaining the exclusive right in step S612, there is a possibility that the exclusive right has been obtained by another node.

[0188] If the page is unallocated (S613: Y), the processor 119 determines the pool volume to which the virtual page is to be allocated (S614). The processor 119 first checks whether or not there is a free page in the pool volume of its own system.

[0189] Specifically, the target capacity and usage capacity of the local area capacity table 802 are referenced to determine whether the usage capacity is less than the target capacity in the entry of the stripe type of write data. If the usage capacity is less than the target capacity, the processor 119 allocates the virtual page to a pool volume of its own system. For example, the node 101 holds local area hierarchical management information (not shown) and selects a pool volume in the highest hierarchical level that includes free pages.

[0190] If there is no free area in the own system, the processor 119 locally mounts a pool volume of another system (another node) and allocates a page to that area. After determining the pool volume, the processor 119 allocates a virtual page to the pool volume (S615). Specifically, the processor 119 updates the correspondence in the page mapping table 215.

[0191] This step allows the node receiving the write request to In such a case, or if the performance of the drive 113 of the node is insufficient, the storage area of ​​another node is used to prevent performance degradation of the node that received the write request, thereby improving capacity efficiency and maintaining performance of the entire system.

[0192] Next, processor 119 determines whether or not exclusion is necessary (S616). This determination is the same as step S611. If exclusion is necessary (S616: Y), processor 119 releases the acquired exclusion (S618). If exclusion is not necessary (S616: N), processor 119 proceeds to step S602.

[0193] The processor 119 refers to the page mapping table 215 to determine whether the logical address (virtual page) in the virtual volume of the write command is assigned to the local pool volume (step 602).

[0194] If it is not assigned to the pool volume of the own system (S602: N), the processor 119 transfers the write command to the other node 101 (S603). The other node 101 executes the write process according to this flowchart. To maintain data coherency, the processor 119 does not cache the write data in the own system.

[0195] If the virtual page is assigned to the local pool volume (S602: Y), the processor 119 starts the write process for each protection layer (S604 to S610). For example, assume that the distributed storage system is configured with three protection layers. These layers are, for example, a node protection layer, a site protection layer, and a geo protection layer. The processor 119 repeats the process three times in total for the three layers. In this example, the node protection layer is set to synchronous write.

[0196] The processor 119 determines whether the layer is a synchronous write target (S604). Specifically, the processor 119 makes this determination by referencing the Sync / Async field in the virtual volume management table 218 that corresponds to the virtual volume that is the write target.

[0197] If it is not the target of synchronous write (S604: N), the processor 119 records "incomplete" in the status field of the area of ​​the data mapping table 701 without transferring the write data (stripe) to the other node 101. The status field indicates the status of each protection layer. Data in the cache 181 whose status field indicates "incomplete" is maintained until it is transferred.

[0198] The processor 119 determines whether all protected layers have been completed (S608), and if so, ends this process. If not completed (S608: N), the processor 119 repeats the process of the next protected layer from step S604. If the protected layer is a synchronous write target (S604: Y), the processor 119 secures cache in the cache area 181 of its own system (S605). The method is the same as that described with reference to FIG. 15.

[0199] Next, the processor 119 determines whether to transfer the intermediate code (S606). The intermediate code represents the update difference between old data (latest data up to now) and new data (data to be written this time). For example, in the case of redundant data equivalent to RAID5, the intermediate code is the xor value of the old data and the new data. Additionally, when erasure coding is used, the processor 119 may generate multiple xor results multiplied by matrix coefficients.

[0200] Several criteria can be used to determine whether or not intermediate code transfer is necessary. For example, the processor 119 determines that intermediate code transfer is necessary when the remaining amount of the redundant code area in the destination node 101 is less than a threshold value. This makes it possible to secure the necessary redundant code in the destination node. The processor 119 acquires information on the local area capacity of the destination node 101 from the destination node 101.

[0201] The processor 119 may generate the intermediate code when the response reduction effect at the time of a cache hit in the own system is small. For example, when a write mode is set in the own system, when a specific low latency drive is used in the own system, when the own system is in a load state higher than a threshold, or when the inter-node communication distance is longer than a threshold, the processor 119 transfers the intermediate code.

[0202] Alternatively, the processor 119 transfers the intermediate code when the write life of the drive 113 is sufficient. Note that in the write mode, the processor 119 returns a completion response to the host after destaging the write data from the cache 181 to the drive 113.

[0203] If it is determined that the intermediate code needs to be transferred (S606: Y), the processor 119 generates an intermediate code from the stripe (write data) on the cache 181 and the old stripe read from the drive 133 (S609), and writes the intermediate code to the cache 181 of the target node (destination node) (S610).

[0204] The processor 119 identifies the target node (destination node) of the intermediate code by the following method. The processor 119 calculates the line number (the value of the vertical axis of the D area in FIG. 11) by the following formula. The method of calculating the line number is the same as the method of calculating the line number of the stripe with reference to FIG. 11. (address value / stripe size)modc

[0205] The processor 119 references the static mapping table of the protection layer from the calculated row number and its own node number, and determines the stripe type number (the number in the cell in FIG. 11).

[0206] The processor 119 refers to the static mapping table of the protection layer and determines the destination node 101 from the stripe type number. The processor 119 transfers the intermediate code together with its own address information (site number, node number, LDEV number, LBA, TL (Transfer Length)) and an identifier indicating that it is an intermediate code to the destination of the destination node 101. The LDEV number is an identifier of the pool volume.

[0207] The processor 119, for example, refers to the static mapping table 211 of layer number 2 and determines the redundancy code node that will ultimately store the site redundancy code Q as the destination node.

[0208] The processor 119, for example, references the static mapping table 212A of layer number 3 to determine the destination site (the site that stores the geo redundant code R). For example, the representative node 101 of the site is set in advance, and the processor 119 transfers the intermediate code to the representative node 101 together with the associated data.

[0209] The representative node 101 calculates a hash value from the source address information using a hash function. The representative node 101 refers to the consistent hashing table 212B, and determines the destination node 101 from the calculated hash value. This node 101 is the final storage node (redundancy code node) of the geo redundancy R.

[0210] The method of transferring data via the representative node 101 has problems in that it requires two data transfers, access concentration on the representative node 101, and reduced availability due to failure of the representative node 101. Therefore, multiple representative nodes 101 may be prepared and the destination representative node 101 may be selected in a round robin manner.

[0211] The processor 119 may directly determine the other site node that stores the geo redundancy code R, instead of the representative node 101. Specifically, the source node 101 holds the consistent hashing table 212B of the destination site in advance, and the processor 119 determines the destination node 101 according to the table.

[0212] When each node 101 holds the consistent hashing table 212B of the other site, synchronization of the consistent hashing table 212B between the sites becomes an overhead. Therefore, the distributed storage system may perform periodic updates without performing close synchronization by exclusive updates. In that case, the destination node 101 that received the intermediate code from the other site may refer to the consistent hashing table 212B of its own site to determine whether the destination is the correct destination, and if the transfer destination is incorrect, the received data may be transferred to the correct node 101.

[0213] In the case where the intermediate code and dirty data of the same source address exist in the destination node 101, the processor 119 of the destination node 101 calculates the xor of the intermediate code and the dirty data, and updates the data in the cache 181. The processor 119 of the destination node 101 connects the cache information related to the intermediate code to the intermediate dirty queue 904. The destination node 101 may also calculate the xor of intermediate codes from different sources that are the source of the same redundant code, and update the data in the cache 181.

[0214] If it is determined in step S606 that the intermediate code is not to be transferred (S606: N), the processor 119 writes the write data to the cache 181 of the target node (destination) (S607). In this example, the write data is basically stored preferentially in the node that received the access. As described above, by transferring data to a target node (destination) other than the write destination, redundancy is ensured in the cache. Furthermore, by separately generating redundant data between nodes, the storage capacity for redundant code is reduced while maintaining redundancy, improving efficiency.

[0215] The method of determining the destination node 101 and the method of transferring data are the same as those in step S610. The source node 101 transfers the write data to the destination node together with its own address information (site number, node number, LDEV number, LBA, TL) of the transfer data and an identifier indicating that it is normal data. In the destination node 101, the processor 119 connects the cache information corresponding to the write data to the dirty queue 901 of the corresponding redundant code.

[0216] For the purpose of reducing the write traffic, a conventional erasure coding method may be adopted when writing write data to a pool volume of another system instead of to a pool volume of the own system. The conventional erasure coding method splits the write data into stripes, generates redundant data from the split data, and distributes and stores the split data and redundant data across multiple nodes.

[0217] By including information on the encoding method in the redundant code, it may be possible to determine which redundant code generation method is being used. The application of the conventional erasure coding method may be limited to only data that does not cause the network to become a bottleneck due to reading from other systems.

[0218] FIG. 17 shows a flowchart of the asynchronous write process. This process is not related to the host I / O. This is executed synchronously, and transfers data that has not yet been transferred to another system in a protection layer for which Async is specified. Steps S702 to S708 in Fig. 17 are the same as steps S605 to S608 in Fig. 16. Only the differences will be explained here. In each node 101, the processor 119 refers to the page mapping table 215, and executes this process for all registered virtual volumes.

[0219] The processor 119 judges whether the target virtual page is the target of asynchronous write (S701). Specifically, the processor 119 checks the state of the area on the pool volume corresponding to the virtual page in the data mapping table 701. If the state is "incomplete" in the protection layer, the processor 119 judges that the virtual page is the target of asynchronous write (S701: Y) and proceeds to step S702.

[0220] When the processing of all virtual pages is completed (S709: Y), the processor 119 ends this flow. The processor 119 may execute the asynchronous write processing periodically or continuously. The processor 119 may dynamically change the execution frequency of this processing and the data transfer speed according to the amount of pages in the "uncompleted" state.

[0221] 18 shows a flowchart of the destage process. This process is executed asynchronously with host I / O when dirty data exists in the cache 181, that is, when data that has not been reflected in the drive 113 exists. Since the generation of redundant data is basically completed within each node (redundant data is generated within a node from data from different senders), the amount of communication between nodes for generating redundant data can be reduced. In addition, since the destinations of the redundant data are distributed among many nodes by the static mapping table 211, the destage process can be efficiently distributed.

[0222] There are two types of dirty data in the cache 181. One is write data stored in the drive 113 of the local system. The other is data transferred from the other node 101 for the purpose of generating redundant data. Here, the data transferred from the other node includes intermediate code.

[0223] The dirty data is managed in a data dirty queue 900, a code dirty queue 901, and an intermediate dirty queue 904. The flowchart in Figure 18 shows the destaging of the dirty data managed in the data dirty queue 900 and the code dirty queue 901.

[0224] When this process starts, the processor 119 finds the target dirty data by referring to the data dirty queue 900 and the code dirty queue 901. The processor 119 determines whether the target data is write data to be stored in its own drive 113 (S801). If the target data is indicated by the data dirty queue 900, the target data is write data.

[0225] If the target data is write data (S801: Y), the processor 119 writes the write data to the drive 113 of its own system (S808). The data is stored in a log-structured format. When storing the write data in the drive 113 in a log-structured format, the processor 119 records the correspondence between the logical address in the pool volume and the physical address in the drive 113, and the state of the data, in a data mapping table 701, as shown in Fig. 8.

[0226] Furthermore, the processor 119 records the correspondence between the logical addresses in the pool volume and the physical addresses in the drive 113 in the reverse mapping table 703. If there is no free space in the drive 113, the processor 119 may execute a capacity exhaustion management process, which will be described with reference to FIG.

[0227] The processor 119 determines whether or not all the dirty data has been processed (S806). If the processing of all the dirty data has been completed (S806: Y), the processor 119 ends this flow.

[0228] If the target data is not write data, that is, if the target data is a stripe for generating redundancy code (S801: N), the processor 119 finds dirty data of the same stripe type (S802).

[0229] Specifically, the processor 119 obtains multiple stripes including the target data transferred from different nodes 101 from the queue of the target data in the code dirty queue 901. The processor 119 obtains as many X stripes as possible according to a user-specified data protection policy (XDYP: maximum number of Data X and number of redundant data Y). User specification of the data protection policy will be described later with reference to FIG. 27.

[0230] Specifically, the processor 119 selects as many stripes as possible without exceeding the number of data node IDs indicated by the site static mapping table 211 or the geo static mapping table 212A. This provides redundancy that satisfies user specifications as much as possible. The source nodes of the selected stripes are all different. If the queue of target data indicates stripes from all data nodes that belong to the stripe type, the processor 119 selects stripes from all data nodes. The logical address at the source does not matter when selecting a stripe.

[0231] In this way, the number of stripes that are elements for generating redundant code is not fixed but is undefined. In addition, the set of logical addresses of stripes that are elements for generating redundant code is also undefined. This allows redundant code to be efficiently generated only from transferred stripes. If there are no stripes of the same stripe type from other nodes 101 in the code dirty queue 901, the processor 119 may store the single target data in the drive 113 as a redundant code.

[0232] Furthermore, when write data is transferred to the node where the redundant code is generated during synchronous write processing, if a new synchronous write processing occurs at a time when the corresponding write data has not been destaged to the drive of the source node, the write data will be overwritten in the cache, which may make it impossible to restore the data.

[0233] For this reason, the redundant data storage destination node must use only data that has been destaged by the source node to generate redundant data. To achieve this, the source node may notify the redundant data storage destination node that it has destaged, and only when that notification is received may the redundant data storage destination node be subject to destaging. Also, the above may be configured to transfer data to the redundant data storage destination node at the timing of destaging at the source node. Also, data in the cache may be prevented from being overwritten when it is updated (for example, saved in a log buffer format).

[0234] The processor 119 may find dirty data from queues of the same stripe type in the intermediate dirty queue 904. The processor 119 updates the redundancy code by calculating the xor of the corresponding redundancy code stored in the drive 113 and the intermediate code. If the updated redundancy code is generated only from stripes of a node 101 other than the source node 101 of the target data, the processor 119 calculates the xor of the target data and the updated redundancy code. Generate new redundant code.

[0235] The processor 119 may select stripes for generating redundant codes so that the ratio of old data (old stripes) is as large as possible. If the redundant code can be generated using only old stripes, the processor 119 selects stripes in this manner. By increasing the ratio of old data in the generation of redundant codes, the time when the redundant codes become invalid data can be accelerated, and the free space in the redundant code storage area can be efficiently increased.

[0236] The processor 119 calculates a redundant code from the selected stripe and writes it to the drive 113 (S803). The write to the drive 113 is basically an append in a log-structured format, similar to step S808. This omits the reading of old data, and realizes high-speed and efficient generation of a redundant code and drive write processing.

[0237] The processor 119 records the correspondence between the physical area in which the calculated redundancy code is stored and the page of the pool volume in a redundancy code mapping table 702, instead of in a data mapping table 701. The processor 119 further records the correspondence between the logical address in the pool volume and the physical address in the drive 113 in an inverse mapping table 703. Since the redundancy code is generated from multiple stripes, the mapping table has multiple references to one physical address.

[0238] After writing the redundant code to the drive 113, the processor 119 notifies the source node 101 (S805). The source node 101 changes the state of the target layer of the target data in the data mapping table 701 to "completed". The state field is referenced to determine whether or not the data is to be retransferred in the event of a node failure. When all dirty data has been processed (S806: Y), the processor 119 ends this flow.

[0239] In addition, when using coding with two or more redundant codes, such as erasure coding, if the redundant codes are generated independently from different combinations of data at multiple nodes that generate each redundant code, data restoration may become difficult (e.g., MDS properties may be lost, the amount of calculation required for restoration may increase, etc.).

[0240] Therefore, after the first redundant code is generated in the node that generates the redundant code, the node that generates the second and subsequent redundant codes is identified by referring to the static mapping table 211, and the node that generates the second and subsequent redundant codes is notified of the set of addresses of the data for which the redundant code was generated in the node that generated the first redundant code to the node that generates the second and subsequent redundant codes.

[0241] The node that generates the second and subsequent redundant codes can maintain the MDS property and enable data recovery by generating the second and subsequent redundant codes with the address pair of the notified data. Another possible method is to have the node that generates the first redundant code generate the second and subsequent redundant codes and transfer the redundant codes to the corresponding nodes.

[0242] In destaging the intermediate code, the processor 119 generates new redundant code from the old redundant code stored in the drive 113 and the intermediate code, and overwrites the old redundant code in the drive 113. Because of the overwrite, the mapping table does not change. Updating the redundant code with the intermediate code requires reading the old data, but it is possible to reduce the amount of local area usage in the redundant code node.

[0243] If there are multiple intermediate codes for one redundant code in the intermediate dirty queue 904, the processor 119 calculates the xor of all the intermediate codes to generate a new intermediate code, The redundant code is updated by the new intermediate code. The intermediate code corresponding to the same redundant code includes data of different generations with the same logical address and intermediate codes of different nodes 101.

[0244] For example, assume that the old redundant code is AxorB. Examples of intermediate codes corresponding to the same redundant code are intermediate code AxorA', intermediate code BxorB', and intermediate code A'xorA''. Here, A'' is the latest data and A' is the oldest data. Data B is new data and data B' is old data.

[0245] The processor 119 can use the redundant code mapping table 702 to know the physical address of the redundant code of the intermediate code selected from the intermediate dirty queue 904. Furthermore, the processor 119 can use the reverse mapping table 703 to identify the logical address of the intermediate code corresponding to the redundant code.

[0246] A specific example of updating the redundant code is shown below. In the following, RAID6 (Galois coefficients: A1 to A3) using Reed-Solomon codes is taken as an example.

[0247] (1) Code Dirty Queue 901 The processor 119 selects dirty data X1 to X3 from the dirty queue 901, and calculates the redundant code P1 or P2 using the following formula. P1=X1xorX2xorX3 P2=(X1*A1)xor(X2*A2)xor(X3*A3) The redundant codes P1 and P2 are each written to a new area in the storage device of the own system.

[0248] (2) Intermediate dirty queue 904 The processor 119 extracts new intermediate dirty data M1, M2 corresponding to the old redundant data P1' or P2' that has been written to the drive 113 of its own system from the intermediate dirty queue 904. The number of intermediate dirty data is not limited to 2. The processor 119 calculates the new intermediate code MP1 or MP2 using the following formula. MP1=M1xorM2 MP2 = (M1*A1) x or (M2*A2)

[0249] The processor 119 calculates the new redundant code P1 or P2 using the following formula. P1=P1'xorMP1 P2=P2'xorMP2 The new redundant codes P1 and P2 are overwritten in the old areas (P1' and P2').

[0250] As described above, the redundancy code node 101 dynamically selects a stripe from the stripes in one stripe type and generates a redundancy code from the selected stripe. This allows the redundancy code to be efficiently generated from the transferred stripe without reading the existing redundancy code.

[0251] In this example, the dynamic selection of stripes is a selection in which at least one of the combination of stripes and the number of stripes to be selected is indefinite. In the above example, stripes are selected independently of both the number of stripes and the address combination, but one of them may be fixed. Here, the address in the address combination is an address specified by the node, volume, and intra-volume address.

[0252] The log-structured method does not have to be applied to the drive write of the redundant code. In other words, the node 101 writes a new redundant code generated from the same address combination as the old redundant code. The old redundant code may be replaced with the new redundant code without being added to the cull area. In a configuration in which the log-structured method is not adopted, the redundant code of the address combination different from all existing redundant codes is added to the local area.

[0253] The above example generates redundancy codes only from stripes within a predefined stripe type. Alternatively, the system may generate redundancy codes from any combination of stripes without defining a stripe type.

[0254] FIG. 19 shows a flowchart of the capacity exhaustion management process. This process attempts to erase data when the amount of data on the drive 113 exceeds a set target amount. This allows necessary data to be stored in a limited area. The types of data that are erased are write data (stripes) and redundant code. This process may be performed asynchronously with host I / O. The relationship between usage and target amount is shown in the local area amount table 802.

[0255] 19 applies to data erasure in the redundant code area and data stripe area, and does not apply to data erasure in the spare area. When the local area capacity table 802 is used for each tier, this process is executed for each tier.

[0256] The processor 119 refers to the local area amount table 802 and checks whether the usage amount of the selected target data type exceeds the target amount (S901). If the usage amount of the target data type exceeds the target amount (S901: Y), the processor 119 determines whether the target data type is a redundant code type (S902).

[0257] In this example, the data types are classified into redundant code types, write data types (stripe types), and data types on the spare area, as shown in the local area amount table 802. Furthermore, the redundant code types are classified into node redundant code, site redundant code, and geo redundant code types, and the write data types are classified into site stripe types.

[0258] If the data type that has exceeded the usage amount is any of the redundant code types (S902: Y), the processor 119 refers to the invalid list 801B and the log-structured mapping table 213 to search for a redundant code of that redundant code type (S907). An invalid redundant code is a redundant code in which all stripes of the calculation source are invalid. All stripes of the calculation source are old data that have been updated, and the redundant code can be erased.

[0259] If there is an invalid redundant code of the target redundant code type (S907: Y), the processor 119 releases the area (S908). The area release deletes the relationship between the physical address of the target area and the logical address of the pool volume in the redundant code mapping table 702, deletes the target area from the invalid list 801B and reconnects it to the free list 801C, and reduces the area usage of the corresponding redundant code type in the local area amount table 802.

[0260] If there is no invalid redundant code of the target redundant code type (S907: N), the processor 119 executes a redundant code merging process (S909). This process can reduce the amount of redundant code used.

[0261] For example, if there exists redundant code P1=X'xorY'xorZ (' represents invalid data), redundant code P2=JxorKxorL', and J, K, and Z are stripes that exist on different nodes, processor 119 can calculate a new redundant code P3=JxorKxorZ using P1xorP2xorX'xorY'xorL'.

[0262] The processor 119 refers to the log-structured mapping table 213 to acquire the logical address and generation information of the stripes that constitute the redundant code. The processor 119 acquires X′, Y′, and L′ from the other nodes 101.

[0263] The processor 119 can reduce the amount of data used by the redundant codes by releasing the areas for the redundant codes P1 and P2 and writing a new redundant code P3 to the drive 113. The redundant codes may be preferentially selected so that the amount of data used by the redundant codes can be reduced as much as possible.

[0264] After the merge process, the processor 119 rechecks whether the usage amount of the target redundant code type exceeds the target amount (S910). If the usage amount exceeds the target amount (S910: Y), the processor 119 executes a rebalance process (S906). As described below, the rebalance process adjusts the page usage amount between pool volumes. For example, data is moved to a pool volume of another tier or a pool volume of another node 101 (another system pool volume). After executing the rebalance, the processor 119 proceeds to step S901. If the usage amount of the target redundant code type does not exceed the target amount (S910: N), the processor 119 proceeds to step S901.

[0265] If the target data type is not a redundant code type, that is, if it is any stripe type (S902: N), the processor 119 judges whether there is erasable write data (stripes) in the target stripe type (S903). Erasable stripes are old stripes that have been updated and are invalid stripes. The processor 119 refers to the invalid list 801B and the log-structured mapping table 213 to search for invalid stripes of the stripe type.

[0266] If there is an erasable stripe (S903: Y), the processor 119 performs a redundant code cleanup process (S904). This process performs cleanup of the redundant code corresponding to the stripe to be erased. Cleanup of both site redundant code and geo redundant code is performed. Specifically, the following steps are performed in each protection layer.

[0267] (1) The processor 119 inquires of the redundant code node 101 of the stripe to be erased whether there is a redundant code including the stripe to be erased. The target stripe is specified by, for example, the site number, node number, LDEV number, and LBA.

[0268] (2) If there is a redundant code including the stripe to be erased in the query destination redundant code node 101, the processor 119 transmits the stripe to be erased to the redundant code node 101. If there is no redundant code, the process ends.

[0269] (3) The redundancy code node 101 generates a new redundancy code by erasing the stripe to be erased from the current redundancy code using the received stripe to be erased. For example, the redundancy code node 101 calculates the xor of the stripe to be erased and the old redundancy code to generate a new redundancy code. The redundancy code node 101 overwrites the old redundancy code stored in the drive 113 with the new redundancy code.

[0270] The update of the redundancy code accompanying the above stripe deletion prevents the deletion of the stripe from which the redundancy code was generated from reducing the redundancy of other stripes of the redundancy code.

[0271] When a redundant code node erases a redundant code, it may inquire whether the stripe corresponding to the redundant code node is the latest version. The redundancy code is specified by a logical address indicated by the pointer 703. If the corresponding stripe is the latest version, the redundancy code node regenerates a new redundancy code for the stripe.

[0272] Next, the processor 119 releases the target area (S905). This is similar to step S908. After that, the processor 119 returns to step S901.

[0273] If the usage of the spare area exceeds the target amount, for example, the processor 119 erases the stripes in the flowchart of Fig. 19, then erases the redundant code, and then executes rebalancing. The order of erasing the stripes and the redundant code may be reversed. If the usage falls below the target amount in any step, the subsequent steps are not necessary.

[0274] Fig. 20 shows the concept of capacity exhaustion management processing. This diagram shows the cleanup processing of redundant code. Node 101A transfers stripe 781 to be written to node 101B (T212). Nodes 101C and 101D similarly transfer stripes 782 and 783 to node 101B. The transferred stripes 781 to 783 are represented by Z, D, and J, respectively.

[0275] Here, when the drive 113 of node 101A becomes exhausted, that is, when the usage amount of the drive 113 of node 101A exceeds a threshold, node 101A attempts to erase old data. Let the old stripe be X''. X' and X'' represent past data (invalid data), and X represents current data.

[0276] Redundancy code generated only from past stripes is no longer meaningful to keep and can be erased. However, redundancy code generated from a stripe set that includes the current stripe cannot be erased. Also, if such redundancy code exists, the past stripes used to generate that redundancy code cannot be erased from the drive, because the stripe would become unrecoverable.

[0277] Therefore, before deleting a past stripe, a node sends the stripe to a node that stores the redundant code of that stripe for cleanup. For example, in FIG. 20, a redundant code X''xorCxorH exists in node 101B. Before deleting past stripe X'', node 101A sends past stripe X'' to node 101B (T202).

[0278] The node 101B calculates CxorH from X''xorCxorHxorX'' using the past stripe X'' and the redundancy code X''xorCxorH. After that, the node 101A erases the past stripe X'' from the drive 113.

[0279] 21 shows a flowchart of the save rebuild process. When an abnormality occurs in the distributed storage system, this process is executed by each node 101 that should respond to the abnormality. The processor 119 of each node 101 can detect the occurrence of an abnormality by referring to the status control table for each protection layer, specifically, the drive status management table 221, the node status management table 222, and the site status management table 223. As described above, information about an abnormality detected by any node 101 is shared within the system.

[0280] 21, the node 101 determines whether an abnormal resource (drive, node, site, etc.) is blocked (S211). There are three types of resource states: "normal", "blocked", and "warning". The node 101 can determine the state of the abnormal resource by referring to the state management table for each protection layer.

[0281] 11, when a failure occurs in any resource such as a node or a site, a node (spare node) that rebuilds the data held by the resource is set in advance. Each node 101 holds information indicating the resource for which the node is the spare node and the data to be rebuilt, and when the processor 119 detects a blocked state of the resource corresponding to the node, it rebuilds the necessary data.

[0282] If the status in the status management table is “blocked”, the processor 119 determines that the status of the abnormal resource is blocked (S211: Y) and executes a priority rebuild (S212). The priority rebuild executes rebuilds in order starting from stripe type data with the lowest redundancy in the protection layer.

[0283] Rebuild recovers lost data from remaining stripes and redundant data. The node 101 refers to the static mapping tables 210-212 of each protection layer to learn the stripe type and its redundancy level that will be reduced due to the loss of data stored in the error resource.

[0284] The nodes 101 notify each other of the processes they are executing and the progress of those processes, and wait for the completion of priority rebuilds with lower redundancy by the other nodes 101. For example, a node 101 waits for the other nodes 101 to complete a rebuild of a stripe type with redundancy 0 before starting a rebuild of a stripe type with redundancy 1. This makes it possible to prevent the rebuild time of a stripe type with redundancy 0 from being extended by a rebuild of a stripe type with redundancy 1.

[0285] It is generally known that data can be restored from the loss of any number of data redundancies by using erasure coding techniques with MDS (Maximum Distance Separable) properties.

[0286] Basically, the spare node that holds the rebuilt data in its own storage device reads the redundant code and stripes and rebuilds the data. If the spare node is under heavy load, other nodes may rebuild the data and transfer it to the spare node.

[0287] Also, if the data of the failed node is unnecessary, for example if there is no owner of the virtual volume, it is possible to only change the redundancy code without rebuilding it on the spare node. For example, the spare node writes zero data, and the redundancy code node generates a new redundancy code using the stripes other than the lost stripe of the old redundancy code and the zero data.

[0288] The redundancy codes of the upper protection layer that have been lost due to the blocked resource are regenerated. For example, if a failure occurs in a drive of a certain node, the node 101 regenerates the site redundancy code and the geo redundancy code within the node 101. The node 101 requests other nodes 101 to transfer stripes required for generating the site redundancy code and the geo redundancy code, respectively. The node 101 can identify the node that holds the stripes from the redundancy code mapping table 702 and the inverse mapping table 703.

[0289] The site redundancy code and the geo-redundancy code may be made redundant within a node. Although the overhead (processor processing time, storage capacity, flash media life consumption, etc.) increases due to redundancy, communication between nodes in the event of a drive failure is not required. In addition, after performing a priority rebuild, the node updates the registered node of the corresponding stripe type in the static mapping tables 210 to 212 with the spare node.

[0290] In addition, the old data (the data on which the new data was written) is used for redundancy. When a long code is generated, it is necessary to regenerate the redundant code by dirtying only new data among the multiple data corresponding to the redundant code at the parity storage node.

[0291] Each node 101 checks whether the redundancy of all stripe types in the protection layer has been restored (S213). The nodes 101 notify each other of the completion of data recovery. When the redundancy of all stripe types in the protection layer has been restored, the process proceeds to step S214. If the process is not completed in all layers (S214: N), the distributed storage system executes again from step S211 for the higher protection layer.

[0292] When the processing of all layers is completed (S214: Y), the distributed storage system reviews the owner of the virtual volume (S215). Specifically, when a node 101 is blocked, another predetermined node 101 takes over the virtual volume that the node 101 had.

[0293] If it is determined in step S211 that the node is not blocked (S211: N), that is, if the state is "warning", the node 101 determines whether data evacuation is necessary (S216). The necessity is determined based on the degree of risk of data loss occurring in the distributed storage system.

[0294] It is generally known that a drive in a "warning" state has a higher probability of failure compared to a "normal" drive. However, even if the drive is in a "warning" state, there are cases where the drive will not fail. Therefore, there is a trade-off between the increased load on the storage system and avoiding the risk of data loss due to the evacuation process.

[0295] For example, if the system redundancy is 2 and two or more drives are in a "warning" state, priority can be given to evacuating stripe-type data with more stripes in the warning state, which is efficient and reduces the amount of data moved for evacuation. Here, system redundancy is the minimum redundancy number for the entire system.

[0296] In one example, when N or more resources are in the "warning" state, the node 101 determines in step S216 that evacuation is necessary. N is an integer that is preset based on the system redundancy. When it is determined that evacuation is necessary (S216: Y), the node 101 executes priority evacuation (S217).

[0297] Priority evacuation copies low-redundancy data stored in a resource that is in a warning state to a predefined spare area. The data is evacuated to the same destination as in rebuild. In the destination data area (spare area), the evacuated data may be overwritten every time a warning occurs, like an LRU cache.

[0298] In the above example, the execution priority is determined based on the redundancy of the stripe type, but the node 101 may determine the execution priority based on the redundancy of the stripe and the redundancy code. The stripe and the redundancy code belong to multiple protection layers, and the total redundancy of the stripes and the redundancy code is the redundancy of the data. This makes it possible to increase the system redundancy as the rebuild / evacuation process progresses.

[0299] As described above, in the event that a node (site) is blocked, the owner of the virtual volume is allocated in advance so that processing can be continued at another node (site). For example, different nodes within a site and nodes at other sites are each set as the owner of the same virtual volume. It is determined.

[0300] To speed up rebuilding and evacuation, rebuilding and evacuation processes may be performed across protection layers. For example, when a drive fails and a rebuild process is performed, in addition to performing the rebuild process within the node, the data on the drive is simultaneously restored using redundant code between nodes. This allows data to be read from more drives at the same time, and rebuilding can be performed at high speed. The degree to which recovery is performed across protection layers may be adjusted depending on factors such as the network load and allowable load.

[0301] FIG. 22 shows a flowchart of the data resync process. This process is executed as a recovery process or copy-back process when a power failure occurs. The copy-back process is a process of copying data from the spare area after a rebuild or resource exchange to a new resource. After the execution of this process is completed, the resource status becomes normal.

[0302] The processor 119 of the node 101 executing this process judges whether the process to be executed is a restoration process (S221). Specifically, the processor 119 judges whether the node itself is a new node or has recovered from a failure such as a power outage. If the node has recovered from the failure, the processor 119 judges that the process is a restoration process (S221: Y).

[0303] More specifically, processor 119 holds a correspondence table between an identifier that is uniquely determined for each node, such as the MAC address of a LAN controller, and a node number as shared information within the distributed storage system, and refers to the correspondence table to determine whether or not its own node is registered in the storage system.

[0304] If it is a restoration process (S221: Y), the processor 119 checks the area that needs to be restored. A specific method of checking the area that needs to be restored is to refer to the state of the data mapping table 701 of the other node 101 for the redundant code, and to obtain stripes of the redundant code in an unreflected state from the other node 101. If the redundant code has been rebuilt in a spare area, the processor 119 obtains the redundant code.

[0305] For write data (stripes), the other node 101 manages the difference written after the occurrence of a failure in a bitmap. The processor 119 recovers the data by copying back only the difference from the spare area. The processor 119 may also refer to the inverse mapping table 703 of its own system to identify the last update time, and request valid data written after the last update time from the other node 101. In this way, the processor 119 determines the write data (stripes) and redundant code to be recovered, and executes area recovery processing (S225).

[0306] If the process to be executed is not a restoration process (S221: N), the processor 119 executes a copy-back process. The processor 119 copies back the rebuilt write data (stripes) and redundant code to the spare area. The processor 119 executes this process for each protection layer hierarchy. For higher layers, only the copying of redundant code is executed. When the process is completed for all layers (S227: Y), this flow ends.

[0307] 23 shows a flowchart of the relocation process. This process optimizes the page allocation of the distributed storage system. This process is executed by each associated node 101 when adding new resources to the distributed storage system, when removing resources, when the capacity of some pool volumes is exhausted, at regular intervals when reviewing changes in load, etc.

[0308] When this process starts, the processor 119 calculates the overall threshold value of the pool based on the page load distribution table 217 indicating the total I / O load of each virtual page (S231). The total I / O load of a virtual page is the total load due to host accesses in all owner nodes of the virtual page. On the other hand, the I / O load due to host accesses to a virtual page in each owner node is called the local load. The I / O load of a virtual page is expressed, for example, by I / O frequency.

[0309] The overall threshold can be calculated in the same way as the tier allocation threshold explained in Fig. 13. Each overall threshold indicates the boundary page I / O frequency between tiers. The capacity and I / O performance of each tier in a pool are determined from the capacity and I / O performance of all pool volumes in each tier. Pool volume tiers, capacity and I / O performance are managed by management information (not shown).

[0310] Next, the processor 119 calculates the local system threshold for each tier based on the page load distribution table 217 indicating the total I / O load of each virtual page and the page load distribution table 217 indicating the local system load of the local node (S232). The local system threshold indicates the boundary I / O frequency of the virtual page that allocates the data to the local node in the virtual page in the tier determined by the overall threshold.

[0311] Figures 24A and 24B each show an example of a method for determining the own threshold. The graphs in Figures 24A and 24B can be read in the same way as graph 271 in Figure 13. The vertical axis shows the page I / O load indicated by the page I / O frequency, and the horizontal axis shows the virtual pages arranged in descending order of their own I / O load.

[0312] 24A and 24B respectively show a total I / O load line 241 and a local I / O load line 242 in one tier. As described above, the virtual pages allocated to each tier are determined by the total I / O load of the virtual pages and an overall threshold.

[0313] 24A and 24B each show an I / O load distribution of a virtual page allocated to one virtual in a virtual page owned by the own node 101. The virtual page owned by the own node 101 may include a virtual page allocated to a pool volume of another system in addition to a virtual page allocated to a pool volume of the own system.

[0314] 24A and 24B each show the local system threshold 246. A virtual page with a local I / O load higher than the local system threshold 246 is allocated to a local system pool volume. Data of the virtual page currently allocated to a remote system pool volume is moved to the local system drive 113.

[0315] A virtual page with an internal I / O load equal to or less than the internal threshold 246 is allocated to an internal pool volume or an external pool volume. Specifically, the processor 119 determines that a virtual page currently allocated to an external pool volume is to be allocated to the external pool volume as is. The processor 119 determines whether or not to move (rebalance) data of a virtual page currently allocated to an internal pool volume to the other node 101, depending on the free capacity of the internal pool volume. Details will be described later.

[0316] 24A and 24B respectively show a capacity limit 243, a drive performance limit 244, and an allowable network limit 245. The processor 119 determines an internal threshold value 246 so that the virtual pages to be allocated to the internal pool volume are within these limit values.

[0317] In this example, the processor 119 determines the page I at the intersection of the minimum of the capacity limit 243, the drive performance limit 244, and the allowable network limit 245 with the local I / O load line 242. The / O load is determined as the local system threshold 246. In Fig. 24A, the drive performance limit 244 is the minimum value, and in Fig. 24B, the allowable network limit 245 is the minimum value.

[0318] The capacity limit 243 indicates the capacity limit that can be allocated to the local system. The capacity limit 243 is determined by a predetermined formula based on the local pool volume capacity and page size of the node 101. The drive performance limit 244 is determined so that the size of all virtual pages allocated to the local pool volume is within the capacity of the local pool volume. The local pool volume capacity is the capacity of the pool volume formed from the local drive 113.

[0319] The drive performance limit 244 is determined by a predetermined formula based on the access performance of the local pool volume and the total I / O load line 241. The access performance of the pool volume is indicated, for example, by the amount of I / O per unit time. The drive performance limit 244 is determined so that the sum of all I / O loads of virtual pages allocated to the local pool volume is within the access performance of the local pool volume. The hatched area in Fig. 24A indicates the sum of all I / O loads of virtual pages allocated to the local pool volume.

[0320] On the other hand, in Fig. 24B, the hatched area indicates the sum of the other system I / O loads, that is, (total I / O load - own system I / O load). The allowable network limit 245 is determined by a predetermined formula based on the sum of the other system I / O loads and the own system network performance. The network performance is indicated by, for example, the I / O amount per unit time.

[0321] When a virtual page is allocated to a local pool volume, the node 101 receives access to the virtual page from another system via the network. Therefore, the processor 119 determines the network limit 245 so that the I / O load of the other system is within the network performance of the local system.

[0322] As described above, by determining the local system threshold based on the drive performance and network performance, it is possible to suppress the occurrence of bottlenecks in data transfer in the host I / O. In particular, by using the drive performance limit 244, it is possible to effectively suppress the occurrence of bottlenecks on the network caused by data placed in other nodes. Note that while the capacity limit 243 is essential, the drive performance limit 244 and the allowable network limit 245 do not have to be used.

[0323] Next, the processor 119 reviews the pool volume configuration in the pool (S233). In determining the local system threshold in step S232, the processor 119 calculates the total capacity and total I / O load of virtual pages (local system virtual pages) to be allocated to the local system pool volumes in each tier.

[0324] The processor 119 determines the number of pool volumes 1303C to be mapped to the other system pool volumes 1303B based on these values ​​and the capacity and performance of the local system drives 113 in each tier. If the capacity or performance of the local system drives 113 is insufficient for the total capacity of the local system virtual pages or the total I / O load, the processor 119 increases the number of pool volumes 1303C.

[0325] Next, the processor 119 sequentially selects virtual pages of the virtual volume of which the own node 101 is the owner, and repeatedly executes the following steps.

[0326] First, the processor 119 judges whether it is necessary to move the data of the virtual page from the other system pool volume to the own system pool volume (S234). Specifically, the processor determines the tier of the virtual volume from the overall threshold, and further determines the tier of the virtual volume from the own system threshold. The processor 119 determines whether to allocate a page to a local pool volume. As described above, the processor 119 determines that a virtual page with an I / O load greater than the local threshold should be allocated to a local pool volume. The processor 119 determines that a virtual page with an I / O load equal to or less than the local threshold does not need to be allocated to a local volume.

[0327] If it is determined that the virtual page is to be allocated to a self-system pool volume and the virtual page is currently allocated to an other-system pool volume, the processor 119 determines that the data of the virtual page needs to be moved from the other-system pool volume to the self-system pool volume.

[0328] If it is determined that the virtual page does not need to be allocated to the local pool volume, or if the virtual page is currently allocated to the local pool volume, the processor 119 determines that the data of the virtual page does not need to be moved to the local pool volume.

[0329] If it is determined that data migration is necessary (S234: Y), the processor 119 migrates the data of the virtual page to the local pool volume (local drive 113) (S235). This migration includes the necessary tier migration of the virtual page.

[0330] The specific procedure includes the following steps: Step 1 stages the data in the local cache 181. Step 2 changes the pool volume area corresponding to the virtual page in the page mapping table 215 to the local pool volume.

[0331] Step 3 destages the data to the local pool volume. Step 4 releases the cache area. Step 5 clears (e.g., writes zero data to) the page area of ​​the originally allocated other pool volume to make it free. That is, this step connects the area to the free list 801C of the local area control table 214, and reduces the usage and available amount in the local area amount table 802.

[0332] In this way, each node 101 uses its own threshold to determine which virtual page to move to its own pool volume, and when the virtual page is owned by multiple nodes 101, one node is determined to hold the data of the virtual page.

[0333] For example, when the node 101 currently holding the data of the virtual page and another node 101 each determine that the virtual page should be allocated to its own pool volume, the data is moved to the other node 101. Therefore, the node 101 that is different from the node 101 holding the data of the virtual page and that last determined that the current virtual page should be allocated to its own pool volume holds the data of the virtual page.

[0334] If it is determined that the data of the virtual page does not need to be moved to the local pool volume (S234: N), the processor 119 determines whether tier migration is necessary (S236). If it is determined that the virtual page needs to be allocated to the local pool volume, is currently allocated to the local pool volume, and further, the current tier is different from the tier determined from the overall threshold, the processor 119 determines that tier migration is necessary.

[0335] If it is determined that hierarchical movement is necessary (S236: Y), the processor 119 executes hierarchical movement (S237). The specific method of hierarchical movement is basically realized in the same manner as in step S235.

[0336] If it is determined that hierarchical movement is not necessary (S236: N), the processor 119 It is then determined whether or not rebalancing of the virtual page is necessary (S238). In this example, rebalancing of the virtual page involves moving the data of the virtual page from the current pool volume to the other pool volume.

[0337] If the processor 119 determines that the virtual page does not need to be allocated to a local pool volume and that the local pool volume to which the virtual page is currently allocated is depleted, it determines that rebalancing is necessary to allocate the virtual page to an other system pool volume.

[0338] The processor 119 refers to the local area capacity table 802 of the tier and determines whether the area of ​​the entry of the virtual page is depleted (insufficient). For example, if the value obtained by subtracting the available capacity from the target capacity is less than a threshold, it is determined that the area is depleted.

[0339] If it is determined that rebalancing is necessary (S238: Y), the processor 119 moves the data of the virtual page from the local pool volume (local node) to the other system pool volume (other node) (S239). A specific method for moving the page for rebalancing is basically realized in the same manner as in step S235.

[0340] The processor 119 inquires of the other nodes 101 or acquires information on the local area amount table 802 from the other nodes, and selects another node 101 having an area that is not depleted and that stores the data of the virtual page.

[0341] The determination of whether a node 101 has an area that is not depleted is based on the local area amount table 802 of the tier in the node 101. The destination node 101 is selected, for example, from among the owner node of the virtual page and the nodes belonging to the stripe type of the virtual page.

[0342] If an unprocessed virtual page remains (S240: N), the processor 119 returns to step S234. When the processing of all virtual pages is completed (S240: Y), the processor 119 ends this process.

[0343] 25A shows a flowchart of a configuration change process. This process is executed when changing the configuration of the distributed storage system. For example, each node executes the process when adding a new resource to the distributed storage system.

[0344] When this process starts, the processor 119 changes the static mapping table of the protection layer (S251). For example, when a node is added, each node 101 of the site protection layer increases the number of stripe types and changes the data nodes and redundant code nodes of each of the multiple stripe types. For example, one node 101 determines a new node configuration of the stripe type, and each node 101 updates the static mapping table accordingly.

[0345] The node 101 changes some of the stripe nodes corresponding to some of the stripe types in the current mapping table 211 to new nodes to be added, and selects a plurality of the nodes to be included in the new stripe type.

[0346] FIG. 25B shows an example of adding a stripe type and rearranging stripes when a node is added. Nodes 101A to 101D are existing nodes, and node 101E is an added node. The rectangles in each node indicate the data position (address) of the stripe, and the numbers in the rectangles indicate the stripe type numbers. Stripe types 1 to 5 are existing stripes. It is a stripe type, and stripe type 6 is an additional stripe type.

[0347] Before the addition, the stripe address of node 101E does not belong to any stripe type, and the rectangle is empty. The stripe type to which some of the stripe addresses of nodes 101A, 101C, and 101D, which are part of the existing nodes, belong has been changed to stripe type 6. Some of the stripe addresses of the added node 101E are assigned to stripe types 2, 3, and 4, which were changed in the existing nodes.

[0348] In this way, by distributing stripes of one stripe type to different nodes, the tolerance to node failures is increased. The redundant code nodes are re-determined so that the usage of the site redundant code Q is as uniform as possible between the added node and the existing nodes.

[0349] Although the above example describes the addition of a node, the configuration change process can be executed in the same manner when adding drives or sites.

[0350] Next, each node 101 recalculates the target capacity in the local area capacity table 802 (S252). For example, as shown in the local area capacity table 802 in Fig. 9, the recalculation of the target capacity determines each site stripe type, the redundant code of each protection layer, and the target capacity of the spare area. The target capacity of the redundant code of each protection layer is determined, for example, from the following formula in accordance with the data protection policy (XDYP: maximum number of Data X, number of redundant codes Y) specified by the user (explained in Fig. 27).

[0351] Target capacity =Total capacity × Max (Y ÷ number of resources, Y ÷ (X+Y)) (However, the number of resources must be greater than Y.)

[0352] The total capacity is the total capacity of the local area of ​​node 101, Max(A,B) is the maximum value of A and B, and the number of resources is the number of resources in the protection layer. In the node protection layer, the number of resources is the number of drives in the node, and in the site protection layer, the number of resources is the number of nodes in the site.

[0353] For example, the target amount of spare area is a fixed value, and the target amount of each site stripe type is an equal division of the remaining total capacity.

[0354] Next, rebalancing of the redundant codes is performed (S253). This involves performing a process of replacing the redundant codes with respect to the difference between the static mapping tables of the protection layer before and after the change. Specifically, the difference data (intermediate code) is sent to the redundant code node, and the redundant code node updates the old redundant code with the intermediate data. Note that instead of rebalancing the redundant codes, the static mapping table of the previous protection layer may be recorded to maintain the correspondence of the redundant codes.

[0355] Finally, the processor 119 executes page rebalancing and reallocation (S254). This process executes page reallocation for newly added nodes and drives. The specific method is as described with reference to FIG. 23. If the set targets for the redundant code and spare area cannot be achieved, the target amounts may be gradually reduced by known methods such as feedback control. This configuration makes it possible to control the data to be allocated to each node constituting the system while taking into account the performance of the entire system.

[0356] FIG. 26 shows an example of a command line management I / F. An application program 2601, an API 2603, and a storage device realized by software are connected to the same node 101. The message device 2602 is operational.

[0357] The application program 2601 specifies a virtual page in a virtual volume to be allocated to a local logical page to the storage device 2602 via the API 2603. The application program 2601 specifies a virtual page by, for example, a virtual volume number, an LBA, and a data length. This makes it possible to specify the virtual page on a page-by-page basis.

[0358] The storage device 2602 determines the node of the logical page allocated to the specified virtual page by referring to the page mapping table 215. If a logical page of a pool volume of another node is allocated to the specified virtual page and the corresponding data is stored in a drive of the other node, the storage device 2602 reads the corresponding data from the other node, allocates the specified virtual page to a logical page of a pool volume of its own system, and stores the data in its own system drive. Also, if a page is not allocated to the storage area specified by the above-mentioned API 2603, the data is stored in the own system drive when a new page is allocated in response to a write request.

[0359] With this configuration, the logical pages that the application program 2601 will next use in its own system can be allocated in advance in its own system, realizing the optimal page allocation for the application.

[0360] The node 101 may receive, from a user via a user interface, a designation of a virtual page in a virtual volume to be allocated to a local logical page (a local storage device). As described above, a virtual page is indicated by a virtual volume identifier and a logical address in the virtual volume. Furthermore, the node 101 may receive an instruction to allocate a virtual page to a logical page of another node.

[0361] 27 shows an example of a management I / F of the GUI of the distributed storage system. The GUI 2701 is an I / F for a user to implement various settings of the distributed storage system. The node 101 accepts various settings from the user via an input / output device.

[0362] The GUI 2701 accepts resource designation (2702A-C) for each protection layer, enabling hierarchical setting. For example, when a site 2702A is designated, the GUI 2701 accepts the selection of each node (2702B) of the designated site. When a node is designated, the GUI 2701 accepts settings for a volume (2702C) within the designated node.

[0363] The following describes the items that are set commonly between sites, nodes, and volumes. Network performance is information about the network bandwidth. If AUTO is specified, each node 101 automatically determines the network bandwidth from the results of measuring the network bandwidth. If specified by the user, each node uses the specified network bandwidth when determining page placement.

[0364] The failure threshold indicates the number of errors that occur before the resource is determined to be blocked when a communication error or the like occurs to the resource. Takeover specifies a resource to be used as a takeover destination when a failure occurs in the resource. Multiple takeover destinations can be selected. If the user does not specify a takeover destination, the storage system may automatically select one.

[0365] There is a protection policy that can be specified for each protection layer. The data protection policy (XDYP: maximum number of data X, number of redundant codes Y) can be specified for each protection layer. If the number of nodes is less than X+Y, or if the storage capacity of the resources is different, the storage The system will use values ​​close to these in real configurations.

[0366] The settings that can be specified for each virtual volume include synchronous / asynchronous information. For each virtual volume, it is possible to specify whether to copy synchronously or asynchronously for each protection layer. It is also possible to specify copy disablement for each protection layer.

[0367] For example, the geo-protection layer copy can be disabled. In that case, rebuilding the virtual volume is not possible in the event of a site failure, and rebuilding in the event of a site failure is skipped. As described above, it is possible to operate in a way that important data is copied asynchronously between sites, and even more important data is copied synchronously.

[0368] The cache mode can be selected from "write" and "write back". In write mode, write data is stored in the cache and at the same time reflected on the drive, and the host (application program) is notified that the write is complete. In write back mode, write data is stored in the cache and then the host (application program) is notified that the write is complete.

[0369] The node on which the virtual volume is mounted is set by specifying the node to be used. This setting is reflected in the virtual volume management table 218.

[0370] Fig. 28 shows an example of a hardware configuration of a distributed storage system. The difference from the example of the configuration shown in Fig. 1 is that a backend switch 2801 is shared between multiple nodes 101. The drive 113 shared via the backend switch 2801 can be accessed by each node 101 sharing the backend switch 2801 without going through other nodes, and is a local drive managed by each node 101. In this way, one drive 113 can be included in multiple nodes 101 via the backend switch 2801.

[0371] In the case of a shared backend configuration, the shared range may be defined as a domain, and data protection may be multidimensional within and between domains. Also, a domain may be defined as a relatively wide bandwidth section according to the transfer bandwidth.

[0372] <Embodiment 2> Fig. 29 shows a method for improving the efficiency of transfer between nodes for redundancy. In the above-mentioned method, the transfer amount increases in proportion to the redundancy with respect to the amount of writes to the node. For example, in the example of Fig. 1, in order to recover data when two nodes fail, write data is transferred from one node to the cache memories 181 of the two nodes.

[0373] For example, write data DATA1 (1501A) written to node 101A is transferred to the cache memory 181 of node 101B and node 101D. That is, in this example, network transfer occurs in an amount twice as large as the amount of data written to the node. A method for reducing the amount of data transferred for generating redundant code in other nodes will be described below.

[0374] 29 shows an example of protecting data with a 2D2P redundant configuration in four nodes, nodes 101A to 101D. In other words, this system has redundancy that allows all data to be restored when two nodes fail.

[0375] For example, the node 101A divides the received write data having a long data length into two blocks (d1 and d2 blocks) 2901 and 2902, and further generates two parities (p and q parities) 2903 and 2904 as intra-node redundancy codes. A data block is a superordinate word that includes a data unit. The p parity 2901 and the q parity 2902 are primary redundancy codes (Class 1 Code). Next, the node 101A distributes and copies the write data and parity to the caches (buffers) of the nodes 101B to 101D. A combination of one or more data blocks is a data block.

[0376] In this example, one write data block (d2 block) 2902 and two parities (p, q parities) 2903, 2904 are distributed and copied to three nodes 101B to 101D. When copying is complete, the necessary redundancy is obtained (data recovery is possible when two nodes fail), and synchronous write processing is completed.

[0377] Similarly, each of the nodes 101B to 101D divides the received write data into two blocks (blocks d1 and d2), and further generates p and q parities. Each of the nodes 101B to 101D distributes and copies one write data block (d2 data block) and two parities (p and q parities) to the caches (buffers) of the other three nodes. Each node stores the data blocks (write data or parity) from each of the other three nodes in its cache.

[0378] Each of the nodes 101A to 101D asynchronously generates a secondary redundancy code (x1, y1 parity) from the data blocks (write data or parity, respectively) aggregated from the other three nodes, writes it to the local drive, and releases the cache. This redundancy code (x1, y1 parity) is called Class 2 Code. Class 2 Code corresponds to the redundancy code described in FIG. 1.

[0379] For example, node 101C receives p parity 2903 from node 101A, p parity 2905 from node 101B, and q parity 2906 from node 101D. From these, node 101C generates x1 parity 2908 and y1 parity 2909, writes them to the local drive, and releases the cache.

[0380] Furthermore, each of the nodes 101A to 101D writes the write data (d1+d2) to the local drive and releases the cache. For example, the node 101A writes the d1 block 2901 and the d2 block 2902 to the local drive and releases the cache.

[0381] In the example of Figure 1, in order to enable data recovery when two nodes fail, the write data (d1+d2) is transferred to the other two nodes. In contrast, in this example, a portion of the write data (d2) and a primary redundancy code (p, q parity) generated from the write data are transferred to the other node. This makes it possible to efficiently transfer data between nodes while maintaining the required redundancy. In addition, all stripe data (d1+d2) is stored on the local drive.

[0382] 29 shows an example of a 2D2P redundant configuration, but the method of this example can be applied to any mDnP configuration (m and n are natural numbers). Write data (mD) is stored in the local drive, and data with redundancy reduced by 1 (redundancy is n-1) is transferred to other nodes.

[0383] For example, in a 3D2P configuration (d1, d2, d3, p, q), write data (d1+d2+d3) is stored in a local drive, and data blocks d2, d3, p, and q are transferred to different nodes, respectively. The set of transferred data blocks is not limited to this, and for example, data blocks d1, d2, d3, and p may be transferred to other nodes.

[0384] By combining the method of this example with the method described in embodiment 1, which dynamically selects stripes from stripes in one stripe type, generates redundant code from the selected stripes, and stores the information about them as metadata (e.g., log-structured mapping table 213), it is possible to reduce the amount of read-modify-write and network transfer, and to achieve high performance in write processing. Also, the method of this example can be applied to a system having multiple protection layers described above.

[0385] Furthermore, when the data length of the received write data is short (for example, random write), the data transfer for redundancy has little impact on the network bandwidth. Therefore, the redundancy process of this example may be executed only when the data length is greater than the threshold (sequential write). When the data length is equal to or less than the threshold, for example, the method shown in FIG. 1 is applied.

[0386] This can improve the utilization of processor processing and network bandwidth. In this case, the system may add information indicating whether the Class 2 Code generation method is applied to metadata (e.g., log-structured mapping table 213) and switch data processing according to the information. Also, the Class 1 code may be written to a local drive as intra-node parity to make parity generation processing more efficient.

[0387] Fig. 30 shows a data restoration method in the method for improving the efficiency of transfer between nodes for redundancy described with reference to Fig. 29. Fig. 30 shows an example in which nodes 101A and 101B fail and write data is restored.

[0388] The nodes 101C and 101D each restore the Class 1 code from the Class 2 code, and further restore the user data of the nodes 101A and 101B from the Class 1 code.

[0389] Specifically, the node 101C restores the p parity of the nodes 101A and 101B from the q parity of the node 101D acquired from the node 101D and the local x1 and y1 parities. The node 101D generates the q parity of the node 101D (if parity is stored locally, it may be used instead) from the user data of the node 101D (local user data).

[0390] The node 101D restores the q parity of the nodes 101A and 101B from the q parity of the node 101C acquired from the node 101C and the local x1 and y1 parity. The node 101C generates the q parity of the node 101C from the write data of the node 101C.

[0391] Furthermore, node 101C restores user data d1 and d2 of node 101A from the q parity of node 101A acquired from node 101D and the restored p parity of node 101A. Node 101D restores user data d1 and d2 of node 101B from the p parity of node 101B acquired from node 101C and the restored q parity of node 101B. As described above, the write data can be recovered by the two-stage restoration process.

[0392] <Embodiment 3> (Log structure (drive) + parity generation (drive) offload method) FIG. 31 shows a hardware configuration example of a distributed storage system. The main difference from the configuration example shown in FIG. 3 is that the back-end ports of the computer nodes 101 connected by the network 104 are connected to multiple flash drives 3105 via a virtual or physical network 103. One site has one or multiple computer nodes 101. has been installed.

[0393] The computer node 101 can communicate with each of the flash drives 3105 via the network 103 without going through other computer nodes, and can use them as local drives. One flash drive 3105 communicates with only one computer node 101.

[0394] The back-end network 103 may interconnect multiple computer nodes 101, and communication between the computer nodes 101 connected to the back-end network 103 is performed using the back-end network 103. Communication between nodes not connected by the back-end network 103 uses, for example, an external network 104.

[0395] A flash drive 3105, an example of a storage drive, is composed of an I / F 3101 for connecting to a computer node 101, a buffer memory 3102 for temporarily storing data, an internal processor 3103 for controlling the flash drive 3105, and multiple flash memories 3104 for storing data.

[0396] (overview) Figure 32 shows an overview of this example. In this example, parity generation processing and data storage processing in a log-structured format are performed on a flash drive. This allows the computer node to perform write processing without being aware of redundant code generation and the log-structured format, thereby shortening the time required for write processing.

[0397] The computer node 101, for example, uses the static mapping table (for example, the site static mapping table 211) described in embodiment 1 to determine the drives for storing the write data and the redundant code. The drives are determined in place of the computer nodes of embodiment 1. For example, the two D drive 3219, P1 drive 3220, and P2 drive 3221 shown in Fig. 32 correspond to one stripe type data drive and redundant code drive.

[0398] For example, the computer node 101 selects an entry in the static mapping table based on the access destination of the write data from the host (for example, a volume identifier and an address within the volume), and determines the multiple drives indicated by the entry as drives for storing the write data and redundant code. If an inter-site protection layer exists, the computer node 101 transfers the write data to the computer node 101 at the other site. The host program is executed, for example, within the computer node 101.

[0399] For example, when writing write data to a drive, the computer node 101 writes (dual-writes) the data to one drive (D drive) 3219 that stores the write data, and one drive (P1 drive) 3220 that stores the main parity. At this time, the computer node 101 writes to the D drive 3219 using a normal write command (D_WRITE) (3210), and writes the data to the medium (LBA area) 3204 via the data buffer 3202 of the D drive 3219.

[0400] The computer node 101 issues a parity write command (P_WRITE) to the P1 drive 3220, and writes the data together with the storage destination information of the data stored in the D drive 3219 (3211). After writing the data to the parity generation buffer 3203, the P1 drive 3220 generates P1 parity 3207 inside the drive, and writes the P1 parity 3207 to the medium 3204.

[0401] As described for the stripe type redundancy code generation in the first embodiment, the P1 drive 3220 dynamically combines the data blocks written to the parity generation buffer 3203 to generate the P1 parity 3227. The P1 drive 3220 writes information on the storage destination of the data for which the P1 parity 3207 has been generated, as metadata 3209, to the metadata storage area 3205.

[0402] For example, when the parity number is 2, the computer node 101 writes (triple writes) data to the D drive 3219, the P1 drive 3220, and also to a drive (P2 drive) 3221 that stores the sub-parity (P2 parity) that is the second and subsequent parities. The P2 drive 3221, like the P1 drive 3220, stores data in the parity generation buffer 3203 and dynamically combines the data blocks written to the parity generation buffer 3203 to generate P2 parity 3227.

[0403] When generating P2 parity, the data block combination of the parity generated by the P1 drive 3220 and the P2 drive 3221 must be the same. After the P1 drive 3220 generates the P1 parity, the P1 drive 3220 notifies the P2 drive 3221 of the combination of data blocks for which the P1 parity was generated (3215) via the computer node 101 (P_GET, P_PUSH). After that, the P2 drive 3221 generates P2 parity with the notified combination of data blocks.

[0404] When reading the latest data, the computer node 101 reads the latest data 3206 from the D drive 3219 with a normal read command (D_READ) (3212). Also, the computer node 101 reads the old data 3208 from the D drive 3219 with a read command (OLD_D_READ) for reading the old data 3208 (3213).

[0405] To secure an area for writing in log-structured format, the computer node 101 monitors the usage (free space) of the drives 3219 to 3221, and performs garbage collection processing as necessary. The capacity management job 3201 of the computer node 101 issues a command (STAT_GET) to acquire the drive usage (free space) after writing is completed or periodically, and monitors and detects the drive usage (drive free space) (3214). If the usage is larger than the threshold (free space is smaller than the threshold) and depletion of the drive free space is detected, the computer node 101 performs garbage collection processing.

[0406] The garbage collection process issues a command (SEARCH) to the P2 drive 3221 to search for the parity to be deleted (3218), and obtains from the drive 3221 storage location information for the parity to be deleted and information on the data that constitutes the parity to be deleted. Next, it is determined from the parity configuration data information whether the data that constitutes the parity is the latest data, and the latest data is transferred to the P1 drive 3220 and redirted. The parity configuration data information indicates information on each data block used in generating the parity. After all of the latest data that constitutes the parity has been redirted, a command (INVALID) is issued to delete the parity and invalidate the old data (3217), and the old data is deleted.

[0407] (Data management structure within the drive) 33 shows a table structure managed by the drive 3105 for controlling the storage system. The flash memory 3104 stores a logical-physical conversion table 3301, which is information relating to the log structure, a log conversion table 3302, a parity data conversion table 3307, which is information relating to data protection, a data parity conversion table 3308, and an address identifier free queue 3309.

[0408] The logical-physical conversion table 3301 is a table showing the logical addresses that the drive 3105 provides to the computer node 101. 33 shows the correspondence between a physical address 3302 and a physical address 3303 of data stored in a physical storage area.

[0409] The log conversion table 3304 indicates the correspondence between an address identifier 3305 for uniquely identifying data and log information 3306 that stores logical-physical conversion information. Each time data is written, the drive 3105 assigns an address identifier to the updated logical-physical conversion information as log information and manages it. Information on data constituting parity held by other drives is held by the address identifier.

[0410] As a result, even if the physical address of data stored in the drive 3105 is changed due to garbage collection processing or wear leveling processing performed by the drive, there is no need to notify other drives of the changed physical address, thereby reducing communication overhead between drives.

[0411] The parity-data conversion table 3307 indicates the correspondence between the address (LBA, data length) of the physical storage area storing the parity of the drive itself and the address (drive number, LBA, data length, address identifier) ​​of the physical storage area of ​​the data of the other drive that generated the parity.

[0412] Since parity is generated by calculation based on multiple data, one parity corresponds to the logical address of the data storage destination on multiple other drives. Also, since data is stored in log structure format, the data of the logical address may also include the address of the old data. For this reason, an address identifier is stored at the same time so that the storage destination of the data for which the parity was generated can be uniquely identified.

[0413] The data-parity conversion table 3308 is a reverse conversion table of the parity data conversion table described above. It shows the correspondence between the address (LBA, drive number) of the physical storage area storing the data of the other drive and the address of the physical storage area storing the parity of the own drive.

[0414] When a failure occurs in another drive and data needs to be restored, the drive 3105 identifies the address of the physical storage area storing the parity required for data recovery on the other drive using the data-parity conversion table 3308. Also, the parity-data conversion table 3307 can identify the address of the physical storage area storing the data of the other drive required for data recovery.

[0415] The address identifier free queue 3309 is a queue used when executing write processes in parallel, which will be described later, and stores unused address identifiers. When writing data, the computer node 101 acquires (dequeues) an address identifier from the top of the address identifier free queue 3309, and issues a data write process to the drive 3105 together with the address identifier.

[0416] The drive 3105 stores the log information in the log conversion table 3304 with the specified address identifier. In addition, the computer node 101 registers (enqueues) the invalidated address identifier at the end of the address identifier free queue 3309 when the old data is invalidated.

[0417] (I / F list) FIG. 34 shows a communication interface between the computer node 101 and the flash drive 3105. A D_WRITE command 3401 is sent to the drive of the D drive 3219. The number, LBA, and data transfer length are used as arguments, and writing is performed to the D drive 3219. After that, the address identifier, which is metadata of the log structure, is output.

[0418] The address identifier is an immutable identifier associated with data stored in the drive. Specifically, the address identifier is an identifier that is unique within the drive and is assigned to mapping information between logical addresses and physical addresses within the drive.

[0419] The P_WRITE command 3402 writes data to the drive using the drive number of the P1 drive 3220 or P2 drive 3221 storing the parity, the data transfer length, and data storage information as arguments. The data storage information consists of the drive number of the D drive, the LBA, and an address identifier.

[0420] A D_READ command 3403 takes the drive number, LBA, and data transfer length as arguments and reads the latest data from the D drive 3219. An OLD_D_READ command 3404 is a command that reads old data from the D drive 3219, taking the drive number, address identifier, and data transfer length as arguments.

[0421] The P_GET command 3405 takes the drive number of the D drive as an argument, and outputs unnotified parity configuration data information to the P2 drive 3221 using the parity generated by asynchronous destage processing from the P1 drive 3220 specified by the argument. The parity configuration data information consists of the drive number, LBA, and address identifier of the D drive for each data block used in generating the parity.

[0422] A P_PUSH command 3406 takes the drive number of the P1 drive 3220 and the parity configuration data information as arguments, and notifies the parity configuration data information to the P2 drive 3221. The parity configuration data information consists of the drive number of the D drive, an LBA, and an address identifier.

[0423] The STAT_GET command 3407 takes a drive number as an argument and outputs information about the usage of the drive specified by the argument. The STAT_GET command 3407 is used to monitor the capacity depletion of the drive. The INVALID command 3408 takes the drive number and address identifier of the D drive 3219 as arguments during garbage collection processing, and invalidates old data that is no longer necessary.

[0424] During garbage collection processing, the SEARCH command 3409 requests the P1 drive 3220 to search for parity to be deleted, and outputs information on the parity to be deleted and parity configuration data information of the parity to be deleted as the search results. The parity information to be deleted consists of the drive number and LBA of the P1 drive 3220, and the parity configuration data information to be deleted consists of the drive number of the D drive, the LBA, an address identifier, and information on whether it is the latest data or not.

[0425] The above commands allow communication between the computer node 101 and the drive 3105 to realize processing.

[0426] (Lead processing) (Read the latest data) 35 shows a flowchart of the process in which the computer node 101 reads the latest data from the D drive 3219. This process is executed when a read command is received from the host (S3501).

[0427] First, the processor 119 of the computer node 101 that has received a read command from the host checks whether the data exists in the cache (S3502). If the data exists in the cache (S3502: Y), the processor 119 returns the data in the cache to the host (S3510).

[0428] If the data does not exist in the cache (S3502: N), the processor 119 secures the cache (S3503) and then issues a D_READ command to the D drive 3219 (S3504).

[0429] When the D drive 3219 receives the D_READ command (S3505), it refers to the logical-physical conversion table 3301 to obtain the physical address where the data is stored (S3506). Next, the D drive 3219 reads the data from the flash memory (medium) 3104 (S3507) and returns the result to the computer node 101 (S3508). When the computer node 101 receives the result of the D_READ from the D drive 3219 (S3509), it returns the result to the host (S3510).

[0430] (Read old data) Fig. 36 shows the process of reading old data. In the process of reading old data, first, the computer node 101 issues an OLD_D_READ command to the D drive 3219 (S3601). When the D drive 3219 receives the OLD_D_READ command (S3602), it obtains the physical address storing the old data corresponding to the specified address identifier from the log conversion table 3304 (S3603).

[0431] Next, the D drive 3219 reads the old data from the flash memory (medium) 3104 (S3604) and returns the result to the computer node 101 (S3605). The computer node 101 receives the result of OLD_D_READ from the D drive 3219 (S3606).

[0432] (Light processing) Fig. 37 shows a flowchart of the process in which the computer node 101 writes data to the D drive 3219. The write process includes two processes. One process is a synchronous write process until the write result is returned to the host. The other process is an asynchronous write process in which parity is generated from data accumulated in the parity generation buffer in the drive and stored on the media.

[0433] First, the synchronous write process will be described. This process is executed when a write command is received from the host. This process stores write data in the D drive 3219, and writes the data together with an address identifier to the drives (P1 drive 3220 and P2 drive 3221) that generate parity.

[0434] When the processor 119 of the computer node 101 receives a write command from the host (S3701), it issues a D_WRITE command to the D drive 3219 (S3702). The D_WRITE command includes write data. When the D drive 3219 receives the D_WRITE command (S3703), it writes the write data to the flash memory (medium) 3104 in a log structure format (S3704), and further, the D drive 3219 updates the metadata (logical-physical conversion table 3301 and log conversion table 3304) (S3705). The D drive 3219 returns the address identifier of the data storage destination to the computer node 101 (S3706).

[0435] When the computer node 101 receives the result of D_WRITE from the D drive 3219 ( S3707), and a P_WRITE command is issued to the P1 drive 3220 together with information on storing data in the D drive 3219 (S3708).

[0436] When the P1 drive 3220 receives the P_WRITE command (S3709), it stores the write data in the parity generation buffer 3203 of the drive (S3710) and returns the result to the computer node 101 (S3711).

[0437] When the computer node 101 receives the result of the P_WRITE command from the P1 drive 3220 (S3712), it issues a P_WRITE command to the P2 drive 3221 together with the data storage information for the D drive 3219 (S3713).

[0438] When the P2 drive 3221 receives the P_WRITE command (S3714), it writes the write data to the parity generation buffer 3203 (S3715) and returns the result to the computer node 101 (S3716). When the computer node 101 receives the result of the P_WRITE command from the P2 drive 3221 (S3717), it returns the result to the host (S3718).

[0439] When a predetermined amount of data is accumulated in the parity generation buffer 3203 of the P1 drive 3220 as a result of repeatedly executing the above-mentioned synchronous write process, or when a predetermined time has elapsed, the P1 drive 3220 performs an asynchronous write process internally (S3719).

[0440] First, the P1 drive 3220 dynamically selects a data block from the data stored in the parity generation buffer 3203 and generates P1 parity (S3720). Next, it updates the metadata (parity-data conversion table 3307 and data parity conversion table 3308) (S3721), and writes the P1 parity to the flash memory (medium) 3104 (S3722).

[0441] Next, the computer node 101 uses a P_GET command to obtain the parity configuration data information of the P1 parity from the P1 drive 3220 (S3723, S3724). The computer node 101 notifies the P2 drive 3221 of the parity configuration data information obtained from the P1 drive 3220 using a P_PUSH command (S3725).

[0442] When the P2 drive 3221 receives a P_PUSH command from the computer node 101, it generates P2 parity based on the received parity configuration data information (S3726), updates the metadata (P2 parity parity-data conversion table 3307 and data-parity conversion table 3308) (S3727), and writes the P2 parity to the flash memory (medium) 3104 (S3728).

[0443] Fig. 38 shows a processing flow when data write processing is performed in parallel to each drive in synchronous write processing. The difference from Fig. 37 is that the computer node 101 issues a write command to the drives 3220 and 3221 that generate parity by specifying the address identifier to be used for the drives 3219 to 3221, without waiting for a response from the D drive 3219.

[0444] Furthermore, writing to the D drive 3219 uses a D_WRITE2 command 3805 for specifying an address identifier and writing, rather than the D_WRITE command 3401. The D_WRITE2 command 3805 is a command for writing to the D drive 3219, using the drive number, LBA, data transfer length, and address identifier of the D drive 3219 as arguments.

[0445] When the computer node 101 receives a write command from the host (S3701), it acquires an address identifier from the top of the address identifier free queue 3309 (S3801) and updates the top pointer of the address identifier free queue 3309 (S3802). Next, the computer node 101 issues a D_WRITE2 command to the D drive 3219, using the acquired address identifier as an argument (S3803).

[0446] The computer node 101 designates the acquired address identifier as data storage information to the P1 drive 3220 and the P2 drive 3221, and issues a P_WRITE command (S3708, S3713).

[0447] The D drive 3219 uses the specified address identifier to store the log information in the log conversion table 3304. The P1 drive 3220 and the P2 drive 3221 each execute processing similar to that in Fig. 37, and then return the results to the computer node 101 (S3703 to S3706, S3709 to S3711, S3714 to S3716).

[0448] The computer node 101 waits (S3804) until it receives the results from all the drives 3219 to 3221. When it has received the results from all the drives 3219 to 3221, the computer node 101 returns the results to the host (S3718).

[0449] 37, the P1 drive 3220 and the P2 drive 3221 generate parity asynchronously and store it in the flash memory (medium) 3104. As described above, by performing write processing in parallel in each drive, the response time to the host can be reduced.

[0450] (Garbage collection process) FIG. 39 shows a flowchart of garbage collection processing. This processing erases unnecessary data when the amount of data stored in the drive exceeds a preset target capacity (threshold). This allows necessary data to be stored in a limited area. The types of data that are erased are write data and parity. This processing may be executed synchronously with host I / O, or may be executed asynchronously with host I / O.

[0451] The computer node 101 checks whether the usage amount of the D drive 3219 exceeds the target amount (S3901). Specifically, the computer node 101 determines whether the usage amount exceeds the target capacity based on the monitoring result of the capacity management job 3201. The monitoring result of the capacity management job 3201 may be managed by the local area amount table 802.

[0452] If the drive usage exceeds the target capacity (S3901: Y), the computer node 101 starts garbage collection processing. In the garbage collection processing, the computer node 101 issues a SEARCH command to search for the P1 parity to be deleted to the P1 drive 3220 that stores the P1 parity generated from the data of the D drive 3219 where the capacity exhaustion was detected.

[0453] When the P1 drive 3220 receives the SEARCH command, it refers to the parity-data conversion table 3307 and searches for P1 parity that has the drive number specified by the argument as parity configuration data information. When the target P1 parity is found, the P1 drive 3220 then refers to the data-parity conversion table 3308 to check whether the data in the search result is old data.

[0454] If the result of the check is that the data is old data, the P1 drive 3220 deletes the old data from the P1 parity. Next, the P1 drive 3220 checks whether all data constituting the P1 parity is new or old by referring to the data-parity conversion table 3308, and returns the result (the parity to be deleted and information on data constituting the parity to be deleted) to the computer node 101 (S3902).

[0455] Next, the computer node 101 checks the old / new information of each data constituting the P1 parity from the returned data information constituting the parity to be deleted, and determines whether the P1 parity to be deleted can be deleted immediately (S3903). If all data constituting the P1 parity is old data (S3903: Y), the computer node 101 deletes the P1 parity (S3906), and further deletes the data constituting the P1 parity from the data storage destination D drive 3219 using the INVALID command (S3907).

[0456] When write processing is parallelized, upon receiving the result of an INVALID command, the computer node 101 registers (enqueues) the INVALID address identifier at the end of the address identifier free queue 3309. The computer node 101 also instructs the P2 drive 3221 to delete the P2 parity that is composed of the same data combination.

[0457] Next, if the data that makes up the P1 parity includes the latest data (S3903:N), the computer node 101 reads the latest data from the D drive 3219 using the D_READ command, and writes it together with the data storage information to the P1 drive 3220 and P2 drive 3221 using the P_WRITE command (S3905, S3908).

[0458] After writing, the computer node 101 deletes the old P1 parity and old P2 parity from the P1 drive 3220 and P2 drive 3221 (S3906, S3909), and also deletes the old data from the D drive 3219 using an INVALID command (S3907). The above processes are repeated to delete the parity and data.

[0459] 37, the P1 drive 3220 generates new P1 parity, updates metadata, and stores the new P1 parity in the flash memory (medium) 3104. Similarly, the P2 drive 3221 generates new P2 parity, updates metadata, and stores the new P2 parity in the flash memory (medium) 3104, by asynchronous write processing.

[0460] <Embodiment 4> (Log structure (drive) + parity generation (controller) offload method) Fig. 40 shows an example of the hardware configuration of a distributed storage system. The difference from embodiment 3 is that a parity generation processing unit is implemented inside the computer node 101. The parity generation processing unit can be implemented by hardware or software. The storage system is configured to include a plurality of computer nodes 101, and each computer node 101 includes a parity generation processing unit 4006 internally having a function of generating parity.

[0461] Furthermore, each computer node 101 is connected to a host computer 4001 via a front-end network 4002, the computer nodes 101 are connected to each other via an internal network 4003, and the computer nodes 101 and the drive 3105 are connected to each other via a back-end network 4004. Multiple computer nodes 101 can access one drive 3105.

[0462] (overview) Fig. 41 shows an overview of this example. The difference from the third embodiment is that the parity generation process is Therefore, when the number of parities is 2 or more, there is no need to notify the P2 drive 3221 of the parity configuration data information of the P1 parity, and the processing load of the computer node 101 and the drives 3219 to 3221 can be reduced, and the write processing time can be shortened.

[0463] Specifically, in the write process, data received from the host is stored in the parity generation buffer 4101 in the computer node 101, and the parity generation buffer 4101 requests the parity generation processing unit 4006 to perform parity generation processing (4101). Next, the parity generation processing unit 4006 generates parity, and writes the generated parity to the drive that stores the parity (4102).

[0464] Also, the difference in garbage collection processing from the third embodiment is that when the latest data is included in the data constituting the parity to be deleted, the latest data is read from the D drive 3219, and then the data is transferred to the parity generation processing unit 4006 to generate new parity. The read processing is the same as in the third embodiment.

[0465] (I / F list) 42 shows a communication interface between the computer node 101 and the drives 3219 to 3221. A P_WRITE2 command 4201 is provided instead of the P_WRITE command 3402 in the third embodiment.

[0466] The P_WRITE2 command 4201 takes the drive number, LBA, data transfer length, and an array of parity configuration data information as arguments, and writes parity to the drive. The parity configuration data information consists of the drive number, LBA, and address identifier. In other words, the P_WRITE2 command 4201 writes multiple data storage destinations to the drive together with the parity as parity configuration data information.

[0467] (Light processing) (Synchronous write processing) The write processing of this example includes synchronous write processing and asynchronous write processing, as in embodiment 3. Figure 43 shows a flowchart of the synchronous write processing of this example. First, when a write command is received from the host (S4301), the computer node 101 issues a D_WRITE command to the D drive 3219 (S4302).

[0468] When the D drive 3219 receives the D_WRITE command (S4303), it writes the data to the flash memory (medium) 3104 (S4304), updates the metadata (logical-physical conversion table 3301 and log conversion table 3304) (S4305), and returns the result (address identifier) ​​to the computer node 101 (S4306).

[0469] Next, when the computer node 101 receives the result from the D drive 3219 (S4307), it stores the data in the parity generation buffer 4101 within the computer node 101 (S4308) and returns the result to the host (S4309).

[0470] In addition, the synchronous write process may be performed in parallel by writing data to the D drive 3219 and storing the data 4101 in the parity generation buffer by using the address identifier free queue 3309 and the D_WRITE2 command 3805 as described in Figure 38.

[0471] (Asynchronous write processing) 44 shows a flowchart of the asynchronous write processing in this example. As a result of repeatedly executing the synchronous write processing, when a predetermined amount of data is accumulated in the parity generation buffer 4101, or when a predetermined time has elapsed, the computer node 101 performs the asynchronous write processing (S4401).

[0472] The main processing unit 4405 of the computer node 101 transfers data for which parity is to be generated from the data stored in the parity generation buffer 4101 to the parity generation processing unit 4006 (S4402). The main processing unit 4405 is realized, for example, by the processor 119 that operates according to a program. When the parity generation processing unit 4006 receives the data (S4403), it stores the received data in its internal buffer (S4404).

[0473] Next, the parity generation processing unit 4006 generates P1 parity and P2 parity from the received data (S4405), and transfers the generated parity to the main processing unit 4405 (S4406).

[0474] When the main processing unit 4405 receives the P1 parity and P2 parity from the parity generation processing unit 4006 (S4407), it writes them to the P1 drive 3220 and P2 drive 3221 together with the data information constituting the parity using a P_WRITE command (S4408).

[0475] When the P1 drive 3220 receives the P_WRITE command (S4409), it writes the parity to the flash memory (medium) 3104 (S4410), updates the metadata (parity-data conversion table 3307 and data-parity conversion table 3308) (S4411), and returns the result to the computer node 101 (S4412).

[0476] On the other hand, the P2 drive 3221 also performs the same processing as the P1 drive 3220, and returns the results to the computer node 101 (S4413 to S4416). Upon receiving the results from the P1 drive 3220 and the P2 drive 3221, the main processing unit 4405 ends the processing (S4417).

[0477] (Garbage collection process) 45 is a flowchart showing the garbage collection process of this embodiment. Steps S4201 to S4204 and S4207 correspond to steps S3901 to S3904 and S3907.

[0478] The main difference with the third embodiment is that the latest data among the data constituting the parity to be deleted is stored in the parity generation buffer 4101 of the computer node 101 (S4501). This makes it unnecessary to rewrite the data to the drive as in the third embodiment, and improves the processing performance of the garbage collection. Also, steps S4501 and S4206 are executed for the P1 parity storage drive and the P2 parity storage drive.

[0479] When a predetermined amount of data has accumulated in the parity generation buffer 4101 or a predetermined time has elapsed, the computer node 101 performs the asynchronous write processing described in Figure 44, generates new parity, and then writes the parity to the drive.

[0480] In the above example, the correspondence between the redundant code and data addressing is managed in each node. In another example, data protection technology may be implemented by preparing two types of virtual spaces and dynamically changing the correspondence between the virtual spaces. Specifically, the system prepares a first virtual space provided to a higher-level logical device, and a second virtual space that is statically associated with the storage addresses of the redundant code and data on the physical storage area. The system prepares the first virtual space, By dynamically associating the space with a second virtual space, it is possible to generate redundant code data from data from multiple nodes.

[0481] In this case, the system shares write destination pointers and the like between multiple nodes that make up the stripe type. The write destination pointer is a pointer that indicates the current position of the write, assuming that data is incrementally added to the second virtual space in log format between multiple nodes.

[0482] Furthermore, the system controls the correspondence between the first virtual space and the second virtual space so that the write destination pointers match, i.e., so that multiple data consisting of each data from multiple nodes and their corresponding redundant codes are written in correspondence with a specified area in the second virtual space.

[0483] The data protection technology and data placement technology disclosed herein dynamically generate redundant codes from a set of data units (data blocks) transferred from multiple different nodes on the cache. In other words, as a result of arbitrarily selecting the same stripe type data from among the data managed in the code dirty queue 901 (S802 in FIG. 18), the logical addresses of the data blocks constituting the inter-node redundant code combination generated by one node are not fixed to one combination, but allow two or more combinations.

[0484] On the other hand, in the present disclosure, as shown in FIG. 8, each data block and its source address are associated and managed, thereby allowing redundancy code generation with dynamic combinations of logical addresses. Furthermore, the number of data used for redundancy code generation is not limited to a specific value and can be changed dynamically. With the above configuration, data protection with low overhead is realized, network bottlenecks are avoided, and data placement that allows high-speed local access is realized. Furthermore, if the drive is an SSD, the amount of writes can be reduced and a longer life can be achieved.

[0485] The data protection and data placement technologies disclosed herein enable both data placement suitable for local reads and data protection, and can avoid bottlenecks caused by networks. Furthermore, since management information for data stored in local storage devices can be maintained within the local system, virtual volume and pool volume information can be limited to sharing among a small number of nodes, reducing the amount of shared information. This allows for high scalability that is not dependent on the number of nodes. Furthermore, high scalability can reduce the network costs required for system construction.

[0486] In addition, the above-mentioned functions in the distributed storage system can be implemented independently. For example, only one of the redundant code generation function, the relocation function, and the function of accepting the designation of the page allocation node may not be implemented in the distributed storage system. The node configuration is not limited to the above-mentioned computer configuration. The above-mentioned node protection layer may be omitted. Furthermore, only one of the site protection layer or the site protection layer may be implemented.

[0487] The present invention is not limited to the above-mentioned embodiment, and includes various modified examples. For example, the drive 113 shown in FIG. 3 does not need to be present in the housing of the computer node 101, and each processor only needs to recognize it as a storage device of its own system and as a management target. The above-mentioned embodiment has been described in detail to easily explain the present invention, and is not necessarily limited to those having all of the described configurations. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0488] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD, or a recording medium such as an IC card or SD card.

[0489] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other.

[0490] In addition to the features set forth in the claims, further features of the present disclosure are set forth below. (1-1) The storage system includes one or more computers and a plurality of storage drives. the one or more computers determine a data drive for storing a write data block and a first redundancy code drive for storing a redundancy code of the write data block; the one or more computers transmit the write data block to the data drive and the first redundant code drive, the data drive stores the write data block in a storage medium; The first redundancy code drive generates a redundancy code using a plurality of write data blocks received from the one or more computers, and stores the redundancy code in a storage medium. (1-2) The first redundant code drive includes: determining a stripe type to which each of the received write data blocks belongs based on a write destination of each of the received write data blocks; A redundancy code is generated from a plurality of write data blocks included in the same stripe type. (1-3) The first redundant code drive includes: further receiving storage destination information of the write data block from the one or more computers; The relationship between the storage destination of the redundant code and the storage destination of the write data block is managed. (1-4) The one or more computers further transmit the write data block together with storage destination information of the write data block to a second redundant code drive; The second redundancy code drive acquires configuration information indicating information of data blocks used in generating a redundancy code in the first redundancy code drive, and generates a redundancy code using data blocks selected according to the configuration information. (1-5) The storage system includes a computer and a plurality of storage drives. the computer determines a data drive for storing a write data block and a redundancy code drive for storing a redundancy code of the write data block; The computer transmits the write data block to the data drive; the data drive stores the write data block in a storage medium; The computer generates a redundancy code using the write data; the computer transmits the redundancy code and configuration information indicating information of the data block used in generating the redundancy code to the redundancy code drive; the redundancy code drive stores the redundancy code in a storage medium; The redundancy code drive manages the relationship between the storage destination of the redundancy code and the storage destination of the write data block. (2-1) 1. A distributed storage system, comprising: A plurality of nodes communicating over a network; The distributed storage system further includes a plurality of storage devices; A first node group including at least three nodes is predefined; each node in the first node group transmits data to be stored in a storage device that it manages to other nodes that belong to the first node group; a first node of the first node group receives data from two or more other nodes of the first node group; The first node generates a redundancy code using a combination of data received from the two or more other nodes; the first node stores the generated redundancy code in a storage device different from a storage device that stores the data from which the redundancy code is generated; A distributed storage system, wherein at least two of the redundant codes generated by the first node have different data combinations of logical addresses of the constituent data. (2-2) 2-1, a distributed storage system according to the present invention, A distributed storage system, wherein each node of the first node group generates intra-node redundancy code from data stored in the storage device that it manages. (2-3) 2-1, a distributed storage system according to the present invention, The first node is Including cache, Temporarily storing data received from the two or more other nodes in the cache; selecting data from the data temporarily stored in the cache; A distributed storage system that generates a redundancy code from the selected data. (2-4) 2-1, a distributed storage system according to the present invention, A distributed storage system in which the first node manages each of the redundant codes in association with logical address information in each of the source nodes of the data used to generate each of the redundant codes. (2-5) 2-1, a distributed storage system according to the present invention, A distributed storage system, wherein the number of data for which each of the redundancy codes is generated is indefinite. (2-6) 2. The distributed storage system according to claim 1, A second node group and a third node group including at least three or more nodes are predefined among the plurality of nodes, The second node belonging to the second node group is generating a second level redundancy code using data received from the nodes belonging to the first group of nodes and the nodes belonging to the third group of nodes; The second level redundancy code is stored in a storage device managed by the second node. (2-7) 2-1, a distributed storage system according to the present invention, After the area for storing the redundant code reaches a threshold, the first node Selecting a first redundant code and a second redundant code stored in the area; merging the first redundant code and the second redundant code to generate a third redundant code for data transmitted only from different nodes; The distributed storage system erases the first redundancy code and the second redundancy code, and stores the third redundancy code in the area. (2-8) 2-1, a distributed storage system according to the present invention, a second node belonging to the first node group that has transmitted the first data to the first node transmits the first data to the first node before erasing the first data from a storage device managed by the second node; The first node uses the first data to update a first redundancy code generated using the first data. (2-9) 2-1, a distributed storage system according to the present invention, A second node that belongs to the first node group and has transmitted first data to the first node, Generate intermediate data using update data of the first data and the first data, and transmit the intermediate data to the first node; The first node uses the intermediate data to update a redundancy code generated using the first data. (2-10) 2-2, a distributed storage system according to the present invention, The first node is Dividing data to be stored in the managed storage device to generate intra-node redundancy code; Transmitting at least a portion of the divided data and the intra-node redundancy code to other nodes of the first node group; A distributed storage system, wherein the combination of data used for the redundancy code generated by the first node includes intra-node redundancy codes transmitted from other nodes. (2-11) 2-1, a distributed storage system according to the present invention, A distributed storage system, wherein a node that generates a redundant code using data stored by a plurality of nodes belonging to the first node group is distributed within the first node group. (2-12) A data control method executed in one node in a distributed storage system including a plurality of nodes communicating via a network, comprising: The distributed storage system further includes a plurality of storage devices; A first node group including at least three nodes is predefined; The data control method includes: Transmitting data to be stored in a storage device that the node manages to another node that belongs to the first node group; generating a redundancy code using a combination of data received from two or more other nodes belonging to the first node group; storing the generated redundancy code in a storage device different from a storage device that stores the data from which the redundancy code is generated; A method in which at least two of the redundancy codes to be generated have different combinations of logical addresses of the constituent data. (2-13) 2-12. The method according to claim 2-12, The method further comprising generating intra-node redundancy code from data stored in the managing storage device. (2-14) 2-12. The method according to claim 2-12, Temporarily storing the data received from the two or more other nodes in a cache; selecting data from the data temporarily stored in the cache; generating a redundancy code from the selected data. (2-15) 2-12. The method according to claim 2-12, A method for managing each of the redundant codes in association with logical address information in each of the nodes that transmitted data used to generate the redundant codes. (3-1) 1. A distributed storage system, comprising: a plurality of nodes communicating over a network; Each of the plurality of nodes has a storage device for storing data; A first node of the plurality of nodes: receiving a plurality of user data together with a write request from a host device; generating a first redundancy code based on the received plurality of user data; transmitting the generated first redundant code and a plurality of user data on which the first redundant code was generated to a plurality of other nodes different from each other among the plurality of nodes, and storing the received plurality of user data in a storage device of the first node; A second node of the plurality of nodes: receiving user data and a first redundant code from a plurality of other nodes different from each other among the plurality of nodes; generating a second redundant code based on the user data received from the plurality of other nodes and the first redundant code; A distributed storage system, characterized in that the second redundancy code is stored in a storage device of the second node. (3-2) 3-1, a distributed storage system according to the present invention, A distributed storage system, wherein each of the nodes has a function as the first node and the second node. (3-3) 3-2, a distributed storage system according to the present invention, A distributed storage system, characterized in that the second node does not store the received user data and / or the first redundancy code in the storage device of the node. (3-4) 3-2, a distributed storage system according to the present invention, A distributed storage system characterized in that the number of user data and first redundant codes that the first node transmits to other nodes is less than the number of user data received and first redundant codes generated. (3-5) 3-2, a distributed storage system according to the present invention, A distributed storage system, characterized in that the first redundancy code is generated multiple times from the same multiple pieces of user data. (3-6) 3-2, a distributed storage system according to the present invention, A distributed storage system, characterized in that the second redundancy code is generated multiple times from the same user data and / or the first redundancy code. (3-7) A data processing system in a distributed storage system, comprising a plurality of nodes communicating with each other via a network, each of the plurality of nodes having a storage device for storing data. A method of analyzing a A first node of the plurality of nodes: receiving a plurality of user data together with a write request from a host device; generating a first redundancy code based on the received plurality of user data; Transmitting the generated first redundant code and a plurality of user data on which the first redundant code was generated to a plurality of other nodes that are different from each other among the plurality of nodes; storing the received user data in a storage device of the first node; A second node of the plurality of nodes: receiving user data and / or a first redundant code from a plurality of other nodes different from each other among the plurality of nodes; generating a second redundant code based on the user data and / or the first redundant code received from the plurality of other nodes; A data processing method comprising storing the second redundancy code in a storage device of the second node. (4-1) A distributed storage system including a plurality of nodes, A storage device for storing data; A processor for processing data input and output to the storage device, The processor, managing a logical address provided to an I / O request source in association with a physical address of a storage location in the storage device; By associating multiple physical addresses with one logical address, when data is updated, it is possible to selectively read out the updated data and the data before the update. Distributed storage system. (4-2) 4-1, a distributed storage system according to the present invention, When updating data related to the logical address, the processor stores the update data related to the update in a physical address different from a physical address of the data before the update, and adds an association between the logical address and the physical address in which the update data is stored. Distributed storage system. (4-3) 4-2, a distributed storage system according to the present invention, the association between the one logical address and each of the plurality of physical addresses is managed by an identifier; By using the identifier, the data before the update can be read. Distributed storage system. (4-4) 4-3, a distributed storage system according to the present invention, If the read request does not include an identifier, the processor reads the latest data of the plurality of data items at the logical address. Distributed storage system. (4-5) 4-1, a distributed storage system according to the present invention, The storage device of the node stores data and parity generated based on data stored in a plurality of different nodes; When the data is updated, the updated data is transmitted to a node that stores the data before the update and to a node that stores parity related to the data; a processor of a node that stores the data before the update and a processor of a node that stores the parity related to the data update the data and the parity asynchronously. Distributed storage system. (4-6) 4-5. A distributed storage system according to claim 4-5, a processor of a node that stores the data before the update and a processor of a node that stores the parity related to the data, the processor sequentially and asynchronously updates the data and the parity. Distributed storage system. (4-7) 4-5. A distributed storage system according to claim 4-5, a processor of a node that stores the data before the update and a processor of a node that stores the parity related to the data update the data and the parity in parallel; Distributed storage system. (4-8) 4-5. A distributed storage system according to claim 4-5, A plurality of parities are generated based on data stored in a plurality of different nodes, and the plurality of parities are stored in the different nodes. Distributed storage system. (4-9) 4-5. A distributed storage system according to claim 4-5, a processor of a node that has received update data for updating the stored data stores the update data in a physical address different from a physical address of the data before the update, and adds a correspondence between the logical address and the physical address at which the update data is stored; a processor of a node that receives update data for updating the stored parity generates an update parity based on the update data and the pre-update parity, stores the update parity in a physical address different from the physical address of the pre-update parity, and adds a correspondence between the logical address and the physical address in which the update parity is stored; Distributed storage system. (4-10) 4-5. A distributed storage system according to claim 4-5, deleting the parity when it is confirmed that the latest data in which each logical address can be placed is not included in the plurality of data using the parity for data recovery; Distributed storage system. (4-11) 4-5. A distributed storage system according to claim 4-5, asynchronously with the writing of the update data, by checking whether each of the plurality of data from which the parity is generated is the latest data in the logical address, it is determined whether the parity needs to be updated, and the parity is updated; Distributed storage system. (4-12) A method for managing data in a distributed storage system including a plurality of nodes, comprising: managing a logical address provided to an I / O request source in association with a physical address of a storage location in the storage device; By associating multiple physical addresses with one logical address, when data is updated, it is possible to selectively read out the updated data and the data before the update. method.

Claims

1. 1. A distributed storage system, comprising: It includes multiple sites communicating over a network, Each of the plurality of sites comprises: A plurality of nodes; a storage device for storing data; a first node at a first site of the plurality of sites; receiving first user data together with a write request from a higher-level device, and storing the first user data in a storage device at the first site; transmitting first data for storing redundant data of the first user data at another site to a second site of the plurality of sites; a third node of a third site of the plurality of sites transmits, to the second site, second data for storing, in another site, redundant data of the second user data received from the higher-level device; a second node at the second site generating parity based on the first data and the second data and storing the parity in a storage device at the second site; A first node at the first site receives a read request for the first user data from the host device, reads the first user data from a storage device at the first site, and transmits the first user data to the host device. A distributed storage system comprising:

2. The distributed storage system according to claim 1, site status management information storing a status of each of the plurality of sites; a second node at the second site; Detecting a failure of the first site by referring to the site status management information; When the failure is detected, the first user data stored in the first site is rebuilt using the parity stored in the second site. A distributed storage system comprising:

3. The distributed storage system of claim 1, The first node at the first site comprises: Transmitting the first data to the second site in synchronization with the reception of the write request and the first user data. A distributed storage system comprising:

4. The distributed storage system of claim 1, When the write request indicates that data is to be updated, a first node at the first site stores the received first user data in a storage area of ​​a storage device at the first site, the storage area being different from the storage area of ​​the data before the update, so that the data before the update can be read; The first data is transmitted to the second site asynchronously with respect to the write request and the reception of the first user data. A distributed storage system comprising:

5. A data processing method in a distributed storage system, comprising: The distributed storage system includes a plurality of sites communicating over a network; Each of the plurality of sites includes a plurality of nodes and a storage device for storing data; The data processing method includes: a first node at a first site of the plurality of sites; receiving first user data together with a write request from a higher-level device, and storing the received first user data in a storage device at the first site; transmitting first data for storing redundant data of the first user data at another site to a second site of the plurality of sites; a third node in a third site among the plurality of sites transmits second data to the second site for storing redundant data of the second user data received from the higher-level device in another site; a second node in a second site among the plurality of sites generates parity based on the first data and the second data and stores the parity in a storage device in the second site; A first node at the first site receives a read request for the first user data from the host device, reads the first user data from a storage device at the first site, and transmits the first user data to the host device.

23. A data processing method comprising:

Citation Information

Patent Citations

  • Information system and data transfer method

    JP2008134988A

  • Storage of parity information for data recovery

    JP2008511064A

  • Storage device, storage method and program

    JP2013037611A

  • Distributed hosting of web content using partial replication

    US7546342B2