Storage device and protocol conversion method thereof

Abstract

To enable protocol conversion in a controller interface of a storage device.SOLUTION: A storage device includes a plurality of storage controllers. Each of the storage controllers includes a controller interface for connecting between the storage controllers. The controller interface includes one or more logical ports corresponding to each storage controller of a connection destination. When converting a first request of a first protocol used in the storage controller into a second request of a second protocol used in an inter-storage controller network, the controller interface stores identification information of the first request and identification information of the second request in a send queue of the logical port.SELECTED DRAWING: Figure 23

Description

[Technical field]

[0001] The present invention relates to a storage device in which a plurality of storage controllers communicate over a network. [Background technology]

[0002] A storage device with a cluster configuration that includes multiple storage nodes can achieve high data access performance through parallel access by operating each storage node in parallel, and can also achieve high availability through data redundancy. A large-scale storage device can be configured by interconnecting multiple storage nodes via a network. For example, Patent Document 1 discloses that a storage device can achieve high functionality and high performance by connecting multiple storage controllers with an interface that uses a unique protocol based on PCIe (Peripheral Component Interconnect Express).

[0003] In the following description and drawings, the internal network of a storage device to which multiple storage nodes are connected is referred to as a storage internal network. The storage internal network may also be referred to simply as an internal network, and the storage nodes may also be referred to simply as nodes.

[0004] A storage node generally has a storage controller and a randomly accessible nonvolatile recording medium. This recording medium is, for example, a drive box equipped with a large number of nonvolatile semiconductor memory drives or hard disk drives. The storage controller has a front-end interface for connecting a higher-level device (such as a host system), a back-end interface for connecting the drive box, and a cache memory for temporarily storing user data that the higher-level device reads and writes to the drive box.

[0005] Furthermore, the storage controller has a control memory that stores control data handled within the storage controller, and a processor for controlling data transfer of user data and control data. In a storage device to which multiple storage nodes are connected, the multiple storage nodes transmit and receive user data and control data between the nodes via a storage internal network.

[0006] Also, for example, Ethernet is known as a network standard suitable for connecting computer nodes including storage nodes, and for example, RoCE (RDMA over Ethernet) is known as a protocol that enables data transfer by RDMA (Remote Direct Memory Access) over Ethernet. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2019-204463 A Summary of the Invention [Problem to be solved by the invention]

[0008] For example, by connecting storage controllers with Ethernet-based RoCE, which has a faster transfer speed than PCIe, it is possible to improve the performance of the storage device. However, in conventional storage devices such as the storage device of Patent Document 1, high functionality and high performance are achieved by applying a PCIe-based proprietary protocol to the connection between storage controllers. Therefore, a general RoCE interface cannot realize all storage-oriented functions, and it is difficult to replace the conventional PCIe-based controller interface.

[0009] An object of one embodiment of the present invention is to provide a method for converting between packets of two protocols suitable for a storage controller interface, such as PCIe and RoCE, and a storage device using the same. [Means for solving the problem]

[0010] One aspect of the present invention is a storage device including a plurality of storage controllers, each of the plurality of storage controllers including a controller interface for connecting between the storage controllers, the controller interface including one or more logical ports corresponding to each of the storage controllers to which the controller interface is connected. When the controller interface converts a first request of a first protocol used in the storage controller into a second request of a second protocol used in a network between the storage controllers, the controller interface stores identification information of the first request and identification information of the second request in a transmission queue of the logical port. Effect of the Invention

[0011] According to one aspect of the present invention, the controller interface of the storage device can perform appropriate protocol conversion. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating a storage device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a storage node. [Diagram 3] FIG. 3 is a diagram illustrating the logical connection between the storage controllers. [Figure 4] FIG. 4 is a diagram illustrating the configuration of the edge interface. [Diagram 5] FIG. 5 is a diagram for explaining the address-QP number conversion table. [Figure 6] FIG. 6 is a diagram illustrating the requester ID-QP number conversion table. [Figure 7] FIG. 7 is a diagram illustrating a processing flow of the WQE converter. [Figure 8] FIG. 8 is a diagram illustrating the RDMA Write sequence. [Figure 9] FIG. 9 is a diagram illustrating an RDMA Write request packet. [Figure 10] FIG. 10 is a diagram for explaining an ACK packet. [Figure 11] FIG. 11 is a diagram illustrating the RDMA Read sequence. [Figure 12] FIG. 12 is a diagram illustrating an RDMA Read request packet. [Figure 13] FIG. 13 is a diagram illustrating an RDMA Read response packet. [Figure 14] FIG. 14 is a diagram illustrating an RDMA Read response packet. [Figure 15] FIG. 15 is a diagram for explaining the format of the BTH. [Figure 16] FIG. 16 is a diagram for explaining the format of RETH. [Figure 17] FIG. 17 is a diagram illustrating an operation of converting a PCIe Write request into an RDMA Write request. [Figure 18] FIG. 18 is a diagram illustrating the contents of the WQE for an RDMA Write request. [Figure 19] FIG. 19 is a diagram for explaining the write operation to the retransmission buffer. [Figure 20] FIG. 20 is a diagram for explaining the read operation from the retransmission buffer. [Figure 21] FIG. 21 is a diagram illustrating the format of the PCIe request header. [Figure 22] FIG. 22 is a diagram illustrating the conversion process between the PCIe Write request header and the RDMA Write request header. [Figure 23]FIG. 23 is a diagram illustrating an operation of converting a PCIe read request into an RDMA read request. [Figure 24] FIG. 24 is a diagram illustrating the contents of the WQE for an RDMA Read request. [Diagram 25] FIG. 25 is a diagram illustrating the PSN-PCIeTag conversion table. [Figure 26] FIG. 26 is a diagram illustrating the conversion process between a PCIe Read request header and an RDMA Read request header. [Figure 27] FIG. 27 is a diagram illustrating the format of the PCIe completion header. [Figure 28] FIG. 28 is a diagram illustrating a conversion process between a PCIe completion header and a header of an RDMA Read response. [Figure 29] FIG. 29 is a diagram illustrating the conversion process between the header of an RDMA Read response and the PCIe completion header. [Diagram 30] FIG. 30 is a diagram illustrating the configuration of an edge interface in the storage device of the second embodiment. [Diagram 31] FIG. 31 is a diagram illustrating an operation of converting a PCIe Write request into an RDMA Write request in the storage device of the second embodiment. [Diagram 32] FIG. 32 is a diagram illustrating the contents of the RDMA Write WQE for control data transfer in the storage device of the second embodiment. [Diagram 33] FIG. 33 is a diagram illustrating the requester ID-data transfer type management table in the storage device of the second embodiment. [Diagram 34] FIG. 34 is a diagram illustrating a processing flow of the WQE converter in the storage device of the second embodiment. [Diagram 35] FIG. 35 is a diagram illustrating the configuration of an edge interface in the storage device of the third embodiment. [Diagram 36]FIG. 36 is a diagram illustrating a write operation to the retransmission buffer in the storage device of the third embodiment. [Figure 37] FIG. 37 is a diagram illustrating a PCIe Write request received by the WQE converter in the storage device of the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, several embodiments of the present invention will be described with reference to the drawings. In all the embodiments, components with the same reference numerals are substantially the same. Furthermore, since the processing executed by the processor is performed using storage resources (e.g., memory) and communication interface devices (e.g., communication ports) as appropriate, the subject of the processing may be the processor. The processor may have dedicated hardware in addition to a CPU (Central Processing Unit). EXAMPLES

[0014] A storage device according to a first embodiment will be described with reference to Fig. 1 to Fig. 29. Fig. 1 is a diagram for explaining a hardware configuration of a storage device according to a first embodiment. A storage device 100 according to a first embodiment has a plurality of storage nodes including storage nodes 101 and 111 interconnected by an internal network. Although two storage nodes are illustrated in Fig. 1, the number of storage nodes connected to the internal network is arbitrary. This internal network is referred to as a storage internal network in this specification.

[0015] In the storage device 100 according to the first embodiment, a network constructed of switches and links conforming to the Ethernet (registered trademark) standard is used for the storage internal network. The storage nodes 101 and 111 are interconnected via Ethernet switches 120 and 130. The number of Ethernet switches is arbitrary, and switches and links conforming to a protocol other than Ethernet may be used.

[0016] The storage node 101 has two storage controllers 102 and 103 therein. The storage controller 102 has an edge interface (EIF) 104 for connecting to the Ethernet switches 120 and 130. Similarly, the storage controller 103 has an edge interface (EIF) 105 for connecting to the Ethernet switches 120 and 130. The edge interface is a controller interface. The storage node 111 has a similar configuration to the storage node 101. Note that the number of storage controllers in the storage node is arbitrary. Different storage controllers may have different configurations.

[0017] 2 is a diagram illustrating a hardware configuration of a storage node according to Example 1. A storage node 101 includes storage controllers 102 and 103, and a drive box 230 having a plurality of hard disk drives (HDDs) or a plurality of solid state drives (SSDs).

[0018] The storage controller 102 has a processor (MP) 200, a memory 202, a front-end interface (FE) 204, and a back-end interface (BE) 205. Similarly, the storage controller 103 has a processor (MP) 210, a memory 212, a front-end interface (FE) 214, and a back-end interface (BE) 215. Each of the processors 200 and 210 has multiple processor cores (not shown) therein.

[0019] A host system (not shown) that accesses the storage device 100 is connected to the storage device 100 via front-end interfaces 204, 214. The host system and the front-end interfaces 204, 214 are connected by a transmission line such as a Fibre Channel cable or an Ethernet cable.

[0020] Alternatively, the host system and the front-end interfaces 204, 214 may be connected via a storage area network that is configured with multiple transmission lines and multiple switches. The front-end interfaces 204, 214 convert the data transfer protocol between the host system and the storage node 101 and the data transfer protocol within the storage controllers 102, 103.

[0021] The drive box 230 is connected to the storage controllers 102, 103 via back-end interfaces 205, 215. The back-end interfaces 205, 215 convert the data transfer protocol in the storage controllers 102, 103 and the data transfer protocol between the storage controllers 102, 103 and the drive box 230. If the drive in the drive box is a PCIe-connected NVMe SSD, the back-end interfaces 205, 215 are PCIe switches that do not perform protocol conversion.

[0022] The processors 200, 210 control data transfer between the host systems connected via the front-end interfaces 204, 214 and the drive boxes 230 connected via the back-end interfaces 205, 215. Furthermore, the processors 200, 210 control data transfer between the storage nodes.

[0023] The memories 202 and 212 are main memories for the processors 200 and 210, respectively, and store programs (storage control programs, etc.) executed by the processors 200 and 210, and management tables referenced by the processors 200 and 210. The memories 202 and 212 are also used as cache memories for the storage controllers 102 and 103, respectively.

[0024] Furthermore, the storage controllers 102 and 103 include NTBs (Non-Transparent Bridges) 203 and 213 connected to the processors 200 and 210 via PCIe, respectively, and edge interfaces (EIFs) 104 and 105.

[0025] The NTB 203 and the NTB 213 are connected via a non-transparent link 220. The processors 200 and 210 can communicate with each other via the non-transparent link 220. In this manner, within the storage node 101, the two controllers 102 and 103 form a dual controller.

[0026] Each of the edge interfaces 104, 105 has one or more physical ports for connecting Ethernet links. Hereinafter, these physical ports are referred to as Ethernet ports. In the storage device 100 according to this embodiment, the edge interface 104 has one or more Ethernet ports 207, and the edge interface 105 has one or more Ethernet ports 217. The processors 200, 210 are connected to the Ethernet switches 120, 130 in FIG. 1 via the Ethernet ports 207, 217, respectively. As a result, the storage controllers belonging to different storage nodes are able to communicate with each other.

[0027] 3 is a diagram for explaining logical connection by Ethernet of a storage internal network according to the first embodiment. In the storage device according to the first embodiment, RoCE is used for data transfer between storage controllers. RoCE is a protocol that enables data transfer by RDMA on Ethernet. Edge interfaces 104 and 105 connecting the storage controllers are capable of RDMA data transfer by RoCE.

[0028] 3, the edge interface 104 of the storage controller 102 includes a PCIe-EIF logic unit 302 and a PCIe-RoCE conversion unit 301. The PCIe-EIF logic unit 302 realizes a data transfer function for storage of the edge interface. When applying PCIe to an internal network, a storage internal network is configured by connecting the PCIe-EIF logic unit 302 to a PCIe switch.

[0029] The PCIe-RoCE converter 301 converts between PCIe packets transmitted and received by the PCIe-EIF logic unit 302 and RoCE packets transferred on the internal network (Ethernet). The edge interface 325 of the storage controller 320 includes a PCIe-EIF logic unit 322 and a PCIe-RoCE converter 321. Similarly, the edge interface 335 of the storage controller 330 includes a PCIe-EIF logic unit 332 and a PCIe-RoCE converter 331.

[0030] In RoCE, data transfer is performed using a Queue Pair (QP), which is a logical communication port (logical port). In particular, in the RoCE RC (Reliable Connection) service, a QP is prepared for each storage controller that is a communication partner. In the storage device according to the first embodiment, the RoCE RC service is used. Therefore, the edge interface of each storage controller has at least an individual QP for each storage controller to which it is connected.

[0031] For example, the edge interface 104 of the storage controller 102 includes a QP 303 for connecting to the storage controller 320. In addition, the edge interface 104 of the storage controller 102 includes a QP 304 for connecting to the storage controller 330.

[0032] Similarly, the edge interface 325 of the storage controller 320 includes a QP 323 for connecting to the storage controller 102. In addition, the edge interface 325 of the storage controller 320 includes a QP 324 for connecting to the storage controller 330.

[0033] Similarly, the edge interface 335 of the storage controller 330 has a QP 334 for connecting to the storage controller 102. Furthermore, the edge interface 335 of the storage controller 330 has a QP 333 for connecting to the storage controller 320. Note that bidirectional communication is possible between the two QPs.

[0034] 4 is a diagram illustrating the configuration of the edge interface 104 according to the embodiment 1. As described above, the edge interface 104 includes the PCIe-EIF logic unit 302 and the PCIe-RoCE conversion unit 301.

[0035] The PCIe-EIF logic unit 302 includes an interface unit 433 for connecting to the processor 200 in the storage controller, a control unit 431, a DMA (Direct Memory Access) 434, and an internal bus 432 that connects them.

[0036] The PCIe-RoCE conversion unit 301 includes a WQE converter 401, a retransmission buffer 402, sorters 403, 413, 423, arbiters 404, 411, 421, a plurality of QPs 405, Ethernet frame builders 412, 422, and Ethernet header removers 414, 424. Of the plurality of QPs 405, one QP is indicated by the reference symbol 405 as an example.

[0037] The WQE converter 401 converts a PCIe request received from the PCIe-EIF logic unit 302 into a RoCE WQE (Work Queue Element), and transmits it to the sorter 403. The WQE includes a data transfer instruction of RDMA to be processed by the QP. When a PCIe completion is received from the PCIe-EIF logic unit 302, the WQE converter 401 transmits the PCIe completion to the sorter 403 as is without converting it. Also, when the WQE converter 401 receives a PCIe Write request, it stores the transmission data included in the payload in the retransmission buffer 402. Details of the processing of the WQE converter 401 will be described later.

[0038] The sorter 403 distributes the WQE received from the WQE converter 401 to one of the QPs 405, QP_1 to QP_2n-2, according to the QP number specified in the WQE. When a PCIe completion is received, the sorter 403 distributes the PCIe completion to one of the QPs 405, QP_1 to QP_2n-2, according to the requester ID in the header.

[0039] The QP 405 converts the WQE or PCIe completion into a RoCE transport layer packet and transmits it to the arbiters 411, 421. The QP 405 also controls retransmission in the RoCE transport layer. When retransmitting data, the QP 405 reads the data to be retransmitted from the retransmission buffer 402. When the number of connection destination controllers per port of the edge interface 104 is a maximum of n-1, the PCIe-RoCE conversion unit 301 has QPs for two ports. Therefore, the PCIe-RoCE conversion unit 301 has a total of 2n-2 QPs.

[0040] The arbiters 411 and 421 transmit the RoCE transport layer packets received from the QP 405 to the Ethernet frame builders 412 and 422 .

[0041] The Ethernet frame builders 412, 422 add an Ethernet header, an IP (Internet Protocol) header, a UDP (User Datagram Protocol) header, and an FCS (Frame Check Sequence) to the RoCE transport layer packets received from the arbiters 411, 421, assemble an Ethernet frame, and transmit it from the Ethernet port to the internal network.

[0042] The Ethernet header removers 414, 424 remove the Ethernet header, IP header, UDP header and FCS from the Ethernet frames received from the internal network via the Ethernet ports, and send the resulting RoCE transport layer packets to the sorters 413, 423.

[0043] The sorters 413, 423 sort the RoCE transport layer packets received from the Ethernet header removers 414, 424 into any one of the QPs 405 from QP_1 to QP_2n-2 according to the QP number in the header.

[0044] The QP 405 converts the RoCE transport layer packets received from the sorters 413 and 423 into PCIe packets and sends them to the arbiter 404.

[0045] The arbiter 404 transmits the PCIe packet received from the QP 405 to the PCIe-EIF logic unit 302 .

[0046] Next, the operation of the WQE converter 401 will be described with reference to Fig. 5 to Fig. 7. Fig. 5 shows an address-QP number conversion table 500 included in the WQE converter 401. The address-QP number conversion table 500 stores a destination address range 501, a destination QP number 502, and a source QP number 503 of a PCIe request.

[0047] Here, the destination QP number 502 is the QP number of another controller which is the destination of the RoCE packet, and the source QP number 503 is the QP number of the own controller which is the source of the RoCE packet. In the storage device of the first embodiment, the destination address range 501 of the PCIe request sent by the PCIe-EIF logic unit and the destination QP number 502 of the RoCE packet are in one-to-one correspondence.

[0048] In addition, in the RoCE RC service, there is a one-to-one correspondence between the destination QP number 502 and the source QP number 503. Therefore, the WQE converter 401 can identify the source QP number from the destination address range of the PCIe request by referring to this table.

[0049] When the sorter 403 is realized by a PCIe switch, the address-QP number conversion table 500 stores a QP address 504 that is the destination of the WQE. The WQE converter 401 identifies the address 504 of the QP that is the destination of the WQE from the destination address of the source PCIe request of the WQE. Then, the WQE converter 401 transmits the PCIe Write with the WQE stored in its payload to the identified QP address 504.

[0050] 6 shows a requester ID-QP number conversion table 600 included in the WQE converter 401. The requester ID-QP number conversion table 600 stores a destination requester ID 601 and a source QP number 602 of a PCIe completion packet. The WQE converter 401 can identify the source QP number from the destination requester ID of the PCIe completion packet by referring to this table.

[0051] When the sorter 403 is realized by a PCIe switch, the WQE converter 401 does not need to include the requester ID-QP number conversion table 600 because the PCIe switch can route PCIe packets based on the requester ID.

[0052] FIG. 7 is a flowchart illustrating an example of processing executed by the WQE converter 401.

[0053] In step 701, the WQE converter 401 determines whether the packet received from the PCIe-EIF logic unit 302 is a PCIe request or a PCIe completion. If the received packet is a PCIe request (701:N), the process proceeds to step 702. If the received packet is a PCIe completion (701:Y), the process proceeds to step 705.

[0054] In step 702, the WQE converter 401 converts the PCIe request into a WQE. The contents of this conversion process will be described later. If the PCIe request is a PCIe Write, the WQE converter 401 transmits the transmission data included in the payload to the retransmission buffer 402.

[0055] In step 703, the WQE converter 401 identifies the source QP number by referring to the address-QP number conversion table 500. When the sorter 403 is realized by a PCIe switch, the WQE converter 401 identifies the address of the destination QP by referring to the address-QP number conversion table 500.

[0056] In step 704, the WQE converter 401 transmits the WQE via the sorter 403 to the QP 405 with the specified number.

[0057] In step 705, the WQE converter 401 identifies the source QP number by referring to the requester ID-QP number conversion table 600. Note that this step can be omitted if the sorter 403 is realized by a PCIe switch.

[0058] In step 706, the WQE converter 401 sends the PCIe completion via the sorter 403 to the QP 405 with the specified number.

[0059] Next, the RoCE RDMA Write operation and its packet format will be described with reference to Fig. 8 to Fig. 10. Fig. 8 is a diagram showing an example sequence of an RDMA Write operation. RDMA Write is an operation for writing data in the memory space of the requester to the memory space of the responder. The requester transmits an RDMA Write request 801 with the write data stored in the payload to the responder. The responder transmits an ACK (Acknowledgement) 802 to the requester to notify the requester that it has completed receiving the write data.

[0060] 9 is a diagram showing the packet format of an RDMA Write request. The RDMA Write request packet includes a Base Transport Header (BTH) 901, an RDMA Extended Transport Header (RETH) 902, a payload 903, and an Invariant Cyclic Redundancy Check (ICRC) 904.

[0061] 10 is a diagram showing the packet format of an ACK packet. The ACK packet includes a BTH 1001, an AETH (ACK Extended Transport Header) 1002, and an ICRC 1003.

[0062] Next, the RoCE RDMA Read operation and its packet format will be described with reference to Figures 11 to 14. Figure 11 is a diagram showing an example of the sequence of the RDMA Read operation. The RDMA Read is an operation to read data in the memory space of the responder to the memory space of the requester.

[0063] The requester transmits an RDMA Read request 1101 that stores read destination information to the responder. The responder transmits an RDMA Read response that stores read data in the payload to the requester. If the read data size exceeds the maximum payload size, the responder divides the read data and transmits multiple RDMA Read responses 1102, 1103, and 1104 consisting of First, Middle, and Last.

[0064] 12 is a diagram showing the packet format of an RDMA Read request. The RDMA Read request packet includes a BTH 1201, a RETH 1202, and an ICRC 1203.

[0065] 13 is a diagram showing the packet format of an RDMA Read Response (First / Last / Only). The RDMA Read Response (First / Last / Only) packet includes a BTH 1301, an AETH 1302, a payload 1303, and an ICRC 1304. The RDMA Read Response (Only) packet is used when the response is a single packet.

[0066] 14 is a diagram showing the packet format of an RDMA Read Response (Middle). The RDMA Read Response (Middle) packet includes a BTH 1401, a payload 1402, and an ICRC 1403.

[0067] 15 is a diagram for explaining the format of the BTH used in the storage device according to the embodiment 1. In the BTH format 1500, the data size of the BTH is 8B as a result of omitting fields that are not used in the storage device according to the embodiment 1 from the BTH (12B length) of the RoCE standard.

[0068] OpCode (Operation Code) 1501 is a field that stores the RoCE operation code.

[0069] Destination QP 1502 indicates the QP number of the packet transmission destination. The QP number in the RoCE standard is 3B long, but in the storage device according to the first embodiment, the total number of QPs may be small, so that the length is shortened to 2B.

[0070] 1stBE (First Byte Enable) 1504 and LastBE (Last Byte Enable) 1503 store the values ​​of the 1st Byte Enable and Last Byte Enable fields of the source PCIe packet header as is. As will be described later, in the internal network of the storage device according to the first embodiment, addresses are specified in 4B units like PCIe, so these fields, which are not in the RoCE standard, are necessary. The meaning of the fields is the same as Byte Enable of PCIe.

[0071] A(ACKREQ) 1505 indicates an ACK Request bit. When a requester sends an RDMA Write request with A set to 1, the responder that receives it schedules the transmission of an ACK. ACKREQ is valid only for RDMA Write requests.

[0072] E (Error Poisoned) 1506 is a field that does not exist in the RoCE standard, and stores the value of the Error Poisoned field of the source PCIe packet header as is.

[0073] CS (Completion Status) 1507 is a field that is not included in the RoCE standard, and stores the value of the Completion Status field of the source PCIe completion header as is. This field is valid only when the RoCE packet is an RDMA Read response.

[0074] A PSN (Packet Sequence Number) 1508 is a packet sequence number assigned to each RoCE transmission packet.

[0075] Other fields of the BTH are not used in the storage device according to the first embodiment or are the same as the RoCE standard, so that the description will be omitted. As described above, by storing PCIe information in a field that is not included in the RoCE standard, conversion between PCIe and RoCE becomes easy.

[0076] 16 is a diagram for explaining the format of the RETH used in the storage device according to the embodiment 1. In the RETH format 1600, fields that are not used in the storage device according to the embodiment 1 are omitted from the RoCE standard RETH (16B length), and as a result, the data size of the RETH is 12B.

[0077] In the RoCE standard, remote addresses 1601 and 1602 are specified in 1B units using a 64-bit value from 0 bit to 63 bit, but in the storage device according to the first embodiment, they are specified in 4B units using a 62-bit value from 2 bit to 63 bit, similar to PCIe. Therefore, the unused lower 2 bits are a reserved field (R: Reserved). When an address of 4B or less needs to be specified, it is specified in the Byte Enable field of BTH.

[0078] The data transfer length is specified in the DMA length 1603 field. In the RoCE standard, the data transfer length is specified in 1B units in the DMA length 1603, but in the storage device according to the first embodiment, the data transfer length is specified in 4B (double word) units like PCIe.

[0079] The AETH used in the storage device according to the first embodiment has the same content as the RoCE standard format, and therefore a description thereof will be omitted.

[0080] 17 is a diagram for explaining an operation in which the PCIe-RoCE converter converts a PCIe Write request into an RDMA Write request in the storage device according to the first embodiment. As an example, a case will be described in which the processor 200 transmits data in the memory 202 to the memory 1702 connected to the processor 1701. The processor 200 and the processor 1701 are connected via the EIF 104 and the EIF 1703. An Ethernet switch and an Ethernet link connecting between the EIF 104 and the EIF 1703 are omitted.

[0081] First, the processor 200 transmits a PCIe write request 1707 to the PCIe-EIF logic unit 302 to instruct the PCIe-EIF logic unit 302 to write data from the memory 202 to the memory 1702 .

[0082] The PCIe-EIF logic unit 302 transmits a PCIe Write request 1708, the payload of which includes the transmission data in the memory 202, to the PCIe-RoCE converter 301. The PCIe-RoCE converter 301 converts the received PCIe Write request into a WQE and posts it to a transmission queue 1706. After processing the WQE, the PCIe-RoCE converter 301 transmits an RDMA Write request 1709 including the transmission data to the internal network (Ethernet). The data transfer operation in the PCIe-RoCE converter 301 via the retransmission buffer 402 in this case will be described later with reference to Figs. 19 and 20. Here, the PCIe-RoCE converter 301 operates as a RoCE requester.

[0083] The EIF 1703 includes a PCIe-EIF logic unit 1704 and a PCIe-RoCE converter 1705. The PCIe-RoCE converter 1705 converts an RDMA Write request 1709 received via an internal network (Ethernet) into a PCIe Write request 1710 and transmits the request to the PCIe-EIF logic unit 1704. Here, the PCIe-RoCE converter 1705 operates as a RoCE responder. The PCIe-RoCE converter 1705 that transmitted the PCIe Write request 1710 transmits an ACK 1712 to the PCIe-RoCE converter 301, which is the requester.

[0084] The PCIe-EIF logic unit 1704 transmits the received PCIe write request 1710 to the processor 1701 as a PCIe write request 1711. The processor 1701 that receives the PCIe write request 1711 writes to the memory 1702 the data stored in the payload.

[0085] 18 is a diagram illustrating the contents of a WQE for an RDMA Write request converted from a PCIe Write request by the WQE converter of the PCIe-RoCE conversion unit 301 in the storage device according to the first embodiment. The WQE 1800 includes an SSN (Send Sequence Number) 1801, a requester ID and a Tag 1802 of the source PCIe Write request, a BTH 1803 and a RETH 1804 of the converted RDMA Write request, and a retransmission buffer address 1805 of the transmission data storage destination. This allows the source PCIe Write request and the converted RDMA Write request to be associated with each other on a one-to-one basis.

[0086] 19 and 20 are diagrams for explaining a data transfer operation related to the retransmission buffer 402. In FIG. 19, the DMA 434 of the PCIe-EIF logic unit 302 transmits a PCIe read request 1901 to the processor 200, and reads out transmission data from the memory 202 by receiving a PCIe completion 1902 in response to the PCIe read request. The PCIe-EIF logic unit 302 transmits the transmission data to the PCIe-RoCE conversion unit 301 by a PCIe write request 1903. The PCIe-RoCE conversion unit 301 performs a conversion process to be described later on the header of the PCIe write request 1903, and then stores it as a WQE in the transmission queue 1706 of the QP. The PCIe-RoCE conversion unit 301 also stores the contents of the payload of the PCIe write request 1903 in the retransmission buffer 402 by a PCIe write request 1904.

[0087] 20, when processing a WQE in a transmission queue 1706 of a QP, the PCIe-RoCE converter 301 refers to a buffer address 1805 stored in the WQE. Then, the PCIe-RoCE converter 301 transmits a PCIe read request 2001 to the retransmission buffer 402, and reads out the transmission data by receiving a PCIe completion 2002 in response to the request. Then, the PCIe-RoCE converter 301 transmits an RDMA write request 1709, the payload of which includes the transmission data, to the internal network (Ethernet). When retransmitting an RDMA write request, the PCIe-RoCE converter 301 also reads out the transmission data from the retransmission buffer 402 and retransmits the request.

[0088] Next, the conversion process from a PCIe Write request to a WQE will be described with reference to Figures 21 and 22. Figure 21 is a diagram for explaining the header format of a PCIe request. The header format 2100 of a PCIe request is unchanged from the PCIe standard format, but here, the fields related to the conversion process from a PCIe Write request to a WQE will be described.

[0089] Length 2101 stores the data size in DW (Double Word, 4B) units stored in the payload of the PCIe Write request.

[0090] Type 2102 stores the PCIe request type, such as Memory Write or Memory Read.

[0091] The 1stBE 2103 stores the Byte Enable value of the first data of a 4B unit.

[0092] LastBE 2104 stores the Byte Enable value of the last data of the 4B unit.

[0093] The requester ID 2106 stores identification information of the PCIe device that has sent the PCIe request. The requester ID 2106 and the tag 2105 together form identification information of the PCIe request.

[0094] Addresses 2107 and 2108 store the destination address of the PCIe request.

[0095] The other fields are not used in the conversion to an RDMA Write request, so their explanation is omitted.

[0096] 22 is a diagram explaining a conversion process between a header 2201 of a PCIe Write request and a header 2202 of an RDMA Write request when the PCIe-RoCE converter on the requester side receives a PCIe Write request. The conversion from a PCIe request header to an RDMA request header in the PCIe-RoCE converter on the requester side will be described below, but the conversion from an RDMA request header to a PCIe request header in the PCIe-RoCE converter on the responder side is similar except for the direction.

[0097] If the Type field of the PCIe request header is Memory Write, the OpCode field of the RoCE BTH is set to RDMA Write Only (2203). In the storage device of the first embodiment, a PCIe Write request is converted to an RDMA Write request on a one-to-one basis. Here, it is assumed that the maximum payload size of PCIe and RoCE is the same.

[0098] The values ​​of the 1st Byte Enable and Last Byte Enable fields of the PCIe request header are stored as is in the 1st Byte Enable and Last Byte Enable fields provided in the BTH of RoCE (2204, 2205).

[0099] The Error Poisoned field of the PCIe request header is stored as is in the Error Poisoned field provided in the BTH of RoCE (2206).

[0100] The PCIe-RoCE converter on the requester side saves the requester ID and Tag of the source PCIe request header in the WQE in the transmission queue of the QP (2207). The WQE also saves the retransmission buffer address, which is the storage destination of the transmission data included in the payload. The PCIe-RoCE converter on the responder side does not save the requester ID and Tag of the PCIe request header.

[0101] The value of the PCIe Address field (63:2) of the PCIe request header is stored as is in the Remote Address field (63:2) of the RETH of RoCE (2208, 2209).

[0102] The value of the Length field of the PCIe request header (10 bits long) is stored as is in the DMA Length field of the RETH of RoCE (2210). The DMA Length field of the RETH (31:10) is set to 0.

[0103] 23 is a diagram for explaining an operation in which the PCIe-RoCE converter converts a PCIe Read request into an RDMA Read request in the storage device according to the first embodiment. As an example, a case will be described in which the processor 200 receives data in a memory 1702 connected to the processor 1701 into the memory 202. The processor 200 and the processor 1701 are connected via an EIF 104 and an EIF 1703. An Ethernet switch and an Ethernet link connecting between the EIF 104 and the EIF 1703 are omitted.

[0104] First, the processor 200 instructs the PCIe-EIF logic unit 302 to read data from the memory 1702 to the memory 202 by transmitting a PCIe write request 2301 with a data read instruction from the memory 1702 stored in the payload.

[0105] The PCIe-EIF logic unit 302 transmits a PCIe read request 2302 instructing to read data from the memory 1702 to the PCIe-RoCE converter 301. Then, the PCIe-RoCE converter 301 transmits an RDMA read request 2303 obtained by converting the received PCIe read request 2302 to the internal network (Ethernet). Here, the PCIe-RoCE converter 301 operates as a RoCE requester.

[0106] The PCIe-RoCE converter 1705 of the EIF 1703 converts the RDMA read request 2303 received via the internal network (Ethernet) into a PCIe read request 2304 and transmits it to the PCIe-EIF logic unit 1704. Here, the PCIe-RoCE converter 1705 operates as a RoCE responder.

[0107] The PCIe-EIF logic unit 1704 transmits the received PCIe read request 2304 to the processor 1701 as a PCIe read request 2305. The processor 1701 that receives the PCIe read request 2305 reads data from the memory 1702 at the specified address.

[0108] Next, the processor 1701 transmits a PCIe completion 2306 storing the read data to the PCIe-EIF logic unit 1704 of the EIF 1703. The PCIe-EIF logic unit 1704 transmits the received PCIe completion 2306 as a PCIe completion 2307 to the PCIe-RoCE conversion unit 1705.

[0109] The PCIe-RoCE conversion unit 1705 converts the PCIe completion 2307 received from the PCIe-EIF logic unit 1704 into an RDMA Read response 2308, and transmits it to the internal network (Ethernet).

[0110] The PCIe-RoCE conversion unit 301 of the EIF 104 converts the RDMA Read response 2308 received via the internal network (Ethernet) into a PCIe completion 2309 and transmits it to the PCIe-EIF logic unit 302 .

[0111] The PCIe-EIF logic unit 302 converts the received PCIe completion 2309 into a PCIe write request 2310 and transmits it to the processor 200 .

[0112] The processor 200 writes the data included in the payload of the PCIe Write request received from the PCIe-EIF logic unit 302 to the memory 202 .

[0113] 24 is a diagram illustrating the contents of a WQE for an RDMA Read request converted from a PCIe Read request by the WQE converter of the PCIe-RoCE conversion unit 301 in the storage device according to the first embodiment. The WQE 2400 includes an SSN (Send Sequence Number) 2401, a requester ID and a Tag 2402 of the source PCIe Read request, and a BTH 2403 and a RETH 2404 of the converted RDMA Read request. This allows the source PCIe Read request and the converted RDMA Read request to be in one-to-one correspondence.

[0114] 25 is a diagram for explaining a PSN-PCIeTag conversion table provided in the PCIe-RoCE conversion unit on the responder side. The PSN-PCIeTag conversion table 2500 stores a QP number 2501 and a PSN 2502 included in the BTH of the RDMA Read request received by the PCIe-RoCE conversion unit on the responder side. Furthermore, the PSN-PCIeTag conversion table 2500 stores a requester ID and a Tag 2503 included in the header of the PCIe Read request converted from the received RDMA Read request. The PCIe-RoCE conversion unit can convert the PCIe completion received from the PCIe-EIF logic unit into an RDMA Read response corresponding to the original RDMA Read request by referring to the PSN-PCIeTag conversion table 2500.

[0115] 26 is a diagram explaining a conversion process between a PCIe read request header 2601 and an RDMA read request header 2602 when the requester-side PCIe-RoCE converter receives a PCIe read request. The conversion from a PCIe request header to an RDMA request header in the requester-side PCIe-RoCE converter will be described below, but the conversion from an RDMA request header to a PCIe request header in the responder-side PCIe-RoCE converter is similar except for the direction.

[0116] If the Type field of the PCIe request header is Memory Read, the OpCode field of RoCE BTH is set to RDMA Read (2603).

[0117] The values ​​of the 1st Byte Enable and Last Byte Enable fields of the PCIe request header are stored as is in the 1st Byte Enable and Last Byte Enable fields provided in the BTH of RoCE (2604, 2605).

[0118] The Error Poisoned field of the PCIe request header is stored as is in the Error Poisoned field provided in the BTH of RoCE (2606).

[0119] The PCIe-RoCE converter on the requester side stores the requester ID and Tag of the source PCIe request header in the WQE in the transmission queue of the QP. The PCIe-RoCE converter on the responder side stores the requester ID and Tag included in the PCIe request header and the QP number and PSN included in the RDMA Read request header in the PSN-PCIe Tag conversion table 2500 in FIG. 25 (2607).

[0120] The value of the PCIe Address field (63:2) of the PCIe request header is stored as is in the Remote Address field (63:2) of the RETH of RoCE (2608, 2609).

[0121] The value of the Length field of the PCIe request header (10 bits long) is stored as is in the DMA Length field of the RETH of RoCE (2610). The DMA Length field of the RETH (31:10) is set to 0.

[0122] 27 is a diagram for explaining the header format of a PCIe completion. The header format 2700 of the PCIe completion is unchanged from the format of the PCIe standard, but here, fields related to the conversion process from a PCIe completion to an RDMA Read response will be explained.

[0123] Length 2701 stores the data size in DW (Double Word) units stored in the payload of the PCIe completion.

[0124] Type 2702 stores a type indicating a PCIe completion.

[0125] Byte Count 2703 stores the number of remaining bytes, including the current packet, with respect to the read data size specified by the PCIe Read request.

[0126] Completion status (CplCts) 2704 stores the status of the result of executing PCIe Read on the responder side.

[0127] Lower Address 2706 stores the start address of the data included in the PCIe completion.

[0128] Completer ID 2705 stores identification information of the PCIe device that transmitted the PCIe completion.

[0129] The requester ID 2708 stores identification information of the PCIe device that transmitted the PCIe read request corresponding to the PCIe completion. The requester ID 2708 and the tag 2707 together form identification information of the PCIe read request corresponding to the PCIe completion.

[0130] FIG. 28 is a diagram illustrating the conversion contents of a header 2801 of a PCIe completion 2307 received by the responder side PCIe-RoCE converter 1705 and a header 2802 of a converted RDMA Read response 2308.

[0131] If the Type field of the PCIe completion header is Completion, the OpCode field of the BTH of the RDMA Read Response is set to RDMA Read Response (2803). However, as shown in Fig. 11, the OpCode of the RDMA Read Response must be changed to RDMA Read Response First (1102), Middle (1103), Last (1104), or RDMA Read Response Only depending on the read data size or the position of the read data included in the response relative to the entire data.

[0132] The Length field of the PCIe completion header is not converted because there is no corresponding RDMA Read response header field (2804).

[0133] The value of the CplSts field of the PCIe completion header is stored as is in the CS field provided in the BTH of the RDMA Read response (2805).

[0134] The Byte Count field of the PCIe completion header is not converted because there is no corresponding RDMA Read response header field (2806).

[0135] The requester ID and Tag of the PCIe completion header are converted to the QP number and PSN in the BTH of the RDMA Read response by referring to the PSN-PCIe Tag conversion table 2500 in FIG. 25 (2807).

[0136] The Lower Address field of the PCIe completion header is not converted because there is no corresponding RDMA Read response header field (2808).

[0137] In addition, even if the read data size is equal to or smaller than the maximum payload size, the processor may divide the read data into multiple packets and transmit the PCIe completion. In this case, the PCIe-RoCE converter collects the multiple read data that have been divided and transmitted into the maximum payload size unit of the RDMA Read response, and converts the PCIe completion into an RDMA Read response.

[0138] FIG. 29 is a diagram illustrating the contents of conversion between a header 2901 of an RDMA Read response 2308 received by the requester side PCIe-RoCE converter 301 and a header 2902 of a PCIe completion 2309 as a conversion destination.

[0139] If the OpCode field of the RDMA Read response header is an RDMA Read response (First, Middle, Last, Only), the type field of the PCIe completion header is set to Completion (2903).

[0140] The setting value of the Length field (2904) of the PCIe completion header changes depending on the type of RDMA Read response as follows: Here, it is assumed that the maximum payload size of PCIe and RoCE is the same.

[0141] If the OpCode field is RDMA Read Response Only, the Length field of the PCIe completion header is set to the value of the DMA Length field by referencing the RETH stored in the WQE corresponding to the PSN of the RDMA Read response header. If the OpCode field is RDMA Read Response First or Middle, the Length field is set to 1 / 4 of the maximum payload size (in bytes). If the OpCode field is RDMA Read Response Last, the Length field of the PCIe completion header is set to the value of the remainder when the DMA Length field value is divided by 1 / 4 of the maximum payload size by referencing the RETH stored in the WQE corresponding to the PSN of the RDMA Read response header.

[0142] The CplSts field of the PCIe completion header is set to the value of the CS field provided in the BTH of the RDMA Read response (2905).

[0143] The Byte Count field (2906) of the PCIe completion header references the RETH stored in the WQE corresponding to the PSN of the RDMA Read response header, and sets the value obtained by subtracting the received data size (number of received responses x maximum payload size, not including the current packet) from four times the value of the DMA Length field. In other words, the Byte Count field sets the number of remaining bytes, including the current packet, for the read data size specified by the Read request.

[0144] The requester ID and Tag fields of the PCIe completion header store the values ​​that were saved by referencing the WQE corresponding to the PSN in the RDMA Read response header (2907).

[0145] The Lower Address field (2908) of the PCIe completion header refers to the WQE corresponding to the PSN of the RDMA Read response header, and sets a value converted from the saved 1st Byte Enable field of BTH, the Remote Address field of RETH, and the received data size (not including the current packet). If the maximum payload size of PCIe and RoCE is a multiple of 128B, the Lower Address field (2908) sets the lower 7 bits of the sum of the value of the Remote Address field and the following offset. Here, if the 1st Byte Enable is '1111b', the offset is 1 if the 1st Byte Enable is '1110b', the offset is 2 if the 1st Byte Enable is '1100b', and the offset is 3 if the 1st Byte Enable is '1000b'.

[0146] The above-described conversion operation between a PCIe read request and an RDMA read request in the storage device according to the first embodiment will be summarized. First, the requester side PCIe-RoCE converter 301 receives a PCIe request 2302 from the PCIe-EIF logic unit 302 and stores the requester ID and Tag of the received PCIe request 2302 in the WQE of the transmission queue 1706. Furthermore, the PCIe-RoCE converter 301 also stores the PSN of an RDMA read request 2303 obtained by converting the received PCIe request 2302 in the WQE. Then, the PCIe-RoCE converter 301 receives an RDMA read response 2308 from the responder side PCIe-RoCE converter 1705. Then, the PCIe-RoCE converter 301 can convert the RDMA Read response 2308 into a PCIe completion 2309 by referring to the PSN of the RDMA Read response 2308 and the PSN, requester ID, and Tag stored in the WQE.

[0147] As described above, the storage device according to the first embodiment can convert a PCIe request transmitted by a PCIe-EIF logical unit that realizes a storage-oriented function into a RoCE RDMA request on a one-to-one basis. As a result, the storage device according to the first embodiment can realize the storage-oriented data transfer function provided in the PCIe-EIF logical unit even when Ethernet is applied to the internal network.

[0148] For example, as described with reference to FIG. 17 or 23, when an edge interface, which is a controller interface, converts a PCIe request used within a storage controller into an RDMA request used in a network between storage controllers, it includes identification information of the PCIe request and identification information of the RDMA request in a WQE and stores it in a transmission queue of a QP.

[0149] The identification information of the PCIe request may include a requester ID and a Tag, for example, as described with reference to FIG. 18 or 24. The requester ID and the Tag are the identification information of the sending device and the packet identification number of the request, respectively. Furthermore, the identification information of the RDMA request may include a PSN. The PSN is the packet identification number of the RDMA request and is included in the BTH.

[0150] For example, as described with reference to Fig. 5, the edge interface may store information on the correspondence between the destination address of a PCIe request and identification information of a QP, such as DestQPN and SrcQPN. Furthermore, the edge interface may store information on the correspondence between the identification information of a QP and address information in PCIe of the QP, such as a QP address.

[0151] For example, as described with reference to Fig. 15, 27, or 28, when the edge interface of the storage controller that has received the RDMA Read request converts the PCIe Completion of the PCIe into an RDMA Read Response, the edge interface stores the status of the read execution result in the header of the RDMA Read Response and transmits it. The Completion Status is the status of the read execution result.

[0152] For example, as described with reference to FIG. 25, when an edge interface receives an RDMA read request and converts the request into a PCIe request, the edge interface may store information on the correspondence between identification information of the QP that received the RDMA read request, identification information of the RDMA read request, and identification information of the converted PCIe request.

[0153] Although the first embodiment describes conversion between PCIe and RoCE, the features of the present disclosure can be applied to conversion between different protocols. This is the same for the other embodiments described below. EXAMPLES

[0154] The storage device according to the second embodiment will be described with reference to Fig. 30 to Fig. 34. Other configurations of the storage device according to the second embodiment are similar to those of the storage device according to the first embodiment, and therefore descriptions thereof will be omitted.

[0155] Data transmitted between storage controllers includes user data and control data. To improve storage performance, throughput performance is required for user data transfer and transfer time is required for control data transfer. In the storage device of the second embodiment, a conversion method between a PCIe Write request and a RoCE RDMA Write request that is particularly suitable for a short control data transfer time is described.

[0156] 30 is a diagram for explaining the configuration of a PCIe-RoCE conversion unit 3000 according to the second embodiment. The PCIe-RoCE conversion unit 3000 includes a WQE converter 3001, a retransmission buffer 3002, sorters 3003, 3013, and 3023, arbiters 3004, 3011, and 3021, multiple QPs 3005, multiple QPs 3006, Ethernet frame builders 3012 and 3022, and Ethernet header removers 3014 and 3024. Multiple QPs 3005 are surrounded by dashed lines, and one of the QPs is indicated by a symbol as an example. Multiple QPs 3006 are surrounded by dashed lines, and one QP is indicated by a symbol as an example.

[0157] The WQE converter 3001 converts a PCIe request received from the PCIe-EIF logic unit 302 into a RoCE WQE (Work Queue Element), and transmits it to the sorter 3003. When a PCIe completion is received from the PCIe-EIF logic unit 302, the WQE converter 3001 transmits the PCIe completion to the sorter 3003 without converting it. When the WQE converter 3001 receives a PCIe write request for user data transfer, the WQE converter 3001 stores the user data included in the payload in the retransmission buffer 3002. When the WQE converter 3001 receives a PCIe write request for control data transfer, the WQE converter 3001 stores the transmission data included in the payload in a WQE and transmits it to the sorter 3003. Details of the processing of the WQE converter 3001 will be described later.

[0158] If the WQE received from the WQE converter 3001 is for user data transfer, the sorter 3003 sorts the WQE to one of QPs 3005, QP1_1 to QP1_2n-2, in accordance with the QP number specified in the WQE. Also, if the WQE received from the WQE converter 3001 is for control data transfer, the sorter 3003 sorts the WQE to one of QPs 3006, QP0_1 to QP0_2n-2, in accordance with the QP number specified in the WQE.

[0159] When a PCIe completion is received, the sorter 3003 distributes the PCIe completion to any one of QPs 3005, QP1_1 to QP1_2n-2, or any one of QPs 3006, QP0_1 to QP0_2n-2, according to the requester ID in the header.

[0160] The QPs 3005 and 3006 convert the WQE or PCIe completion into a RoCE transport layer packet and transmit it to the arbiters 3011 and 3021. The QPs 3005 and 3006 also handle retransmission control in the RoCE transport layer. When retransmitting user data, the QP 3005 reads the data to be retransmitted from the retransmission buffer 3002. When retransmitting control data, the QP 3006 obtains the data to be retransmitted from the WQE stored in the transmit queue 1706. When the number of connected controllers is a maximum of n-1, the PCIe-RoCE conversion unit 3000 has a total of 4n-4 QPs, including 2n-2 QPs for user data transfer and 2n-2 QPs for control data transfer.

[0161] The arbiters 3011 and 3021 transmit the RoCE transport layer packets received from the QPs 3005 and 3006 to the Ethernet frame builders 3012 and 3022 .

[0162] The Ethernet frame builders 3012 and 3022 add an Ethernet header, an IP header, a UDP header and an FCS to the RoCE transport layer packets received from the arbiters 3011 and 3021, assemble an Ethernet frame, and transmit the frame from the Ethernet port to the internal network.

[0163] The Ethernet header removers 3014 and 3024 remove the Ethernet header, IP header, UDP header and FCS from the Ethernet frames received from the internal network via the Ethernet ports, and send the resulting RoCE transport layer packets to the sorters 3013 and 3023.

[0164] The sorters 3013 and 3023 sort the RoCE transport layer packets received from the Ethernet header removers 3014 and 3024 to any one of the QPs 3005, QP1_1 to QP1_2n-2, or any one of the QPs 3006, QP0_1 to QP0_2n-2, according to the QP number in the header.

[0165] The QPs 3005 and 3006 convert the RoCE transport layer packets received from the sorters 3013 and 3023 into PCIe packets and send them to the arbiter 3004.

[0166] The arbiter 3004 transmits the PCIe packets received from the QPs 3005 and 3006 to the PCIe-EIF logic unit 302 .

[0167] 31 is a diagram for explaining an operation in which a PCIe-RoCE converter converts a PCIe Write request for control data transfer into an RDMA Write request in the storage device according to the second embodiment. As an example, a case in which the processor 200 transmits data in the memory 202 to the memory 1702 connected to the processor 1701 will be described. The processor 200 and the processor 1701 are connected via two EIFs, EIF 104 and EIF 3108. An Ethernet switch and an Ethernet link connecting between the EIF 104 and EIF 3108 are omitted.

[0168] First, the processor 200 transmits a PCIe write request 3101 for control data transfer to the PCIe-EIF logic unit 302, thereby instructing the PCIe-EIF logic unit 302 to write data from the memory 202 to the memory 1702. The payload of this PCIe write request 3101 includes the transmission data.

[0169] The PCIe-EIF logic unit 302 transmits a PCIe Write request 3102, in which the transmission data in the memory 202 is stored in the payload, to the PCIe-RoCE conversion unit 3000. The PCIe-RoCE conversion unit 3000 converts the received PCIe Write request into a WQE and posts it to a transmission queue 3103. After processing the WQE, the PCIe-RoCE conversion unit 3000 transmits an RDMA Write request 3104 including the transmission data to the internal network (Ethernet). Here, the PCIe-RoCE conversion unit 3000 operates as a RoCE requester. In addition, in the control data transfer, the PCIe-RoCE conversion unit 3000 stores the transmission data in the WQE in the transmission queue 3103 without storing it in the retransmission buffer 3002.

[0170] The EIF 3108 includes a PCIe-EIF logic unit 1704 and a PCIe-RoCE converter 3109. The PCIe-RoCE converter 3109 converts an RDMA Write request 3104 received via an internal network (Ethernet) into a PCIe Write request 3105 and transmits the request to the PCIe-EIF logic unit 1704. Here, the PCIe-RoCE converter 3109 operates as a RoCE responder. The PCIe-RoCE converter 1705 that transmitted the PCIe Write request 3105 transmits an ACK 3107 to the PCIe-RoCE converter 3000, which is the requester.

[0171] The PCIe-EIF logic unit 1704 transmits the received PCIe write request 3105 to the processor 1701 as a PCIe write request 3106. The processor 1701 that receives the PCIe write request 3106 writes to the memory 1702 the data stored in the payload.

[0172] 32 is a diagram illustrating the contents of a WQE for an RDMA Write request converted from a PCIe Write request for control data transfer by a WQE converter 3001 of a PCIe-RoCE conversion unit 3000 in a storage device according to a second embodiment. The WQE 3200 includes a send sequence number (SSN) 3201, a requester ID and a tag 3202 of the source PCIe Write request, a BTH 3203 and a RETH 3204 of the destination RDMA Write request, and a payload 3205 of the source PCIe Write request. The payload 3205 includes transmission data.

[0173] 33 illustrates a requester ID-data transfer type management table 3300 included in the WQE converter 3001 of the PCIe-RoCE conversion unit 3000 in the storage apparatus according to the second embodiment. The requester ID-data transfer type management table 3300 stores a source requester ID 3301 and a data transfer type 3302 of a PCIe request received by the WQE converter 3001.

[0174] The data transfer type 3302 is information indicating whether the transfer is control data or user data. In the storage apparatus according to the second embodiment, the control data and the user data are transmitted by different PCIe devices having different requester IDs. The WQE converter 3001 can determine whether the received PCIe request is for a control data transfer or a user data transfer by referring to this table.

[0175] FIG. 34 is a diagram illustrating a flowchart of an example of processing executed by the WQE converter 3001 of the PCIe-RoCE conversion unit 3000 in the storage apparatus according to the second embodiment.

[0176] In step 3401, the WQE converter 3001 determines whether the packet received from the PCIe-EIF logic unit 302 is a PCIe request or a PCIe completion. If the received packet is a PCIe request, the process proceeds to step 3402, and if the packet is a PCIe completion, the process proceeds to step 3407.

[0177] In step 3402, the WQE converter 3001 refers to the requester ID-data transfer type management table 3300 and determines whether the received PCIe request is for a control data transfer or a user data transfer according to the requester ID of the received PCIe request. If the received PCIe request is for a control data transfer, the process proceeds to step 3403, and if it is for a user data transfer, the process proceeds to step 3404.

[0178] In step 3403, the WQE converter 3001 converts the PCIe request into a WQE for control data transfer.

[0179] In step 3404, the WQE converter 3001 converts the PCIe request into a WQE for user data transfer. If the PCIe request is a PCIe Write request, the WQE converter 3001 stores the transmission data included in the payload in the retransmission buffer 3002.

[0180] In step 3405, the WQE converter 3001 refers to the address-QP number conversion table 500 to identify the source QP number.

[0181] In step 3406 , the WQE converter 3001 transmits the WQE via the sorter 3003 to the QPs 3005 and 3006 with the specified numbers.

[0182] In step 3407, the WQE converter 3001 refers to the requester ID-QP number conversion table 600 to identify the source QP number.

[0183] In step 3408, the WQE converter 3001 sends the PCIe completion to the QPs 3005 and 3006 with the specified numbers via the sorter 3003.

[0184] In the storage device according to the second embodiment, the operation of the PCIe-RoCE converter 3000 for converting a PCIe read request for control data transfer and a PCIe read request for user data transfer into an RDMA read request is similar to that of the storage device according to the first embodiment, and therefore a description thereof will be omitted.

[0185] As described above, when transmitting control data, the storage device according to the second embodiment stores the transmission data 3205 in the WQE 3200. Therefore, when the QP 3006 processes the WQE in the transmission queue 3103, it is not necessary to read the transmission data from the retransmission buffer 3002. Therefore, the storage device according to the second embodiment can reduce the time required to read from the retransmission buffer 3002 in transmitting the control data transfer.

[0186] For example, as described with reference to Figures 33 and 34, the edge interface may determine whether to store the transmission data in a buffer or in a transmission queue based on a requester ID of a PCIe request. The requester ID is identification information indicating a source device of a PCIe request. Also, as described with reference to Figure 30, for example, the edge interface may include a QP that reads the transmission data from the buffer and a QP that stores the transmission data in a transmission queue together with identification information of an RDMA request. EXAMPLES

[0187] The storage device according to the third embodiment will be described with reference to Fig. 35 to Fig. 37. Other configurations of the storage device according to the third embodiment are similar to those of the storage device according to the first embodiment, and therefore descriptions thereof will be omitted.

[0188] In order to achieve high reliability, the storage device stores user data in the memories of two controllers. In this case, in the storage device of the first embodiment, the PCIe-EIF logic unit 302 transmits a total of two PCIe Write requests, each of which stores the same transmission data in the payload, to the two controllers. As a result, the PCIe-RoCE conversion unit 301 stores the transmission data included in the two PCIe Write requests in the retransmission buffer 402. In other words, a retransmission buffer capacity for two pieces of transmission data is required. In the storage device according to the third embodiment, a method of conversion between a PCIe Write request and a RoCE RDMA Write request, which is particularly suitable for reducing the capacity of the retransmission buffer, will be described.

[0189] 35 is a diagram for explaining the configuration of an edge interface 3500 according to the third embodiment. The edge interface 3500 is composed of a PCIe-EIF logic unit 3502 and a PCIe-RoCE conversion unit 3501. The edge interface 3500 differs from the edge interface 104 of the first embodiment in the operation of storing transmission data in the retransmission buffer 402. A DMA 3503 stores transmission data to other nodes in the retransmission buffer 402 without going through a WQE converter 3504 (3505).

[0190] 36 is a diagram for explaining the operation of writing transmission data to the retransmission buffer 402 in the storage device of the third embodiment. In FIG. 36, the DMA 3503 of the PCIe-EIF logic unit 3502 transmits a PCIe read request 3601 to the processor 200, and reads the transmission data from the memory 202 by receiving a PCIe completion 3602 in response to the PCIe read request. The PCIe-EIF logic unit 3502 transmits the transmission data to the PCIe-RoCE conversion unit 3501 by a PCIe write request 3603. The PCIe-RoCE conversion unit 3501 stores the transmission data included in the payload of the PCIe write request in the retransmission buffer 402.

[0191] FIG. 37 is a diagram illustrating a PCIe Write request packet 3700 transmitted to the WQE converter 3504 when the DMA 3503 transfers user data in the storage device according to the third embodiment.

[0192] The PCIe Write request packet 3700 includes a PCIe request header 3701 and a payload 3702. The payload 3702 includes a buffer address 3703 of the retransmission buffer 402 that stores the transmission data, and a data size 3704 of the transmission data.

[0193] The WQE converter converts the PCIe Write request packet 3700 into a WQE 1800 for transferring user data, based on a buffer address 3703 stored in a payload 3702 and a data size 3704 of the transmission data.

[0194] As described above, in the storage device according to the third embodiment, when transmitting a PCIe Write request to another controller, first, the DMA 3503 of the PCIe-EIF logic unit 3502 transmits retransmission data to the retransmission buffer 402. Then, the DMA 3503 transmits a PCIe Write request 3700 storing a retransmission buffer address 3703 of the transmission data storage destination and a transmission data size 3704 to the PCIe-RoCE conversion unit 3501.

[0195] When transmitting the same user data to two controllers, the DMA 3503 transmits two PCIe Write requests 3700 that do not include transmission data. As a result, even when transmitting user data to two controllers, the retransmission buffer capacity is consumed for only one piece of user data, so that the capacity of the retransmission buffer can be reduced.

[0196] 36 and 37, the edge interface may include an interface logic unit including a DMA for transmitting and receiving PCIe packets, a converter for converting packets between PCIe and RoCE, and a buffer. The DMA may store transmission data in the buffer and transmit a PCIe request that stores a buffer address for storing the transmission data and a transmission data size to the converter.

[0197] The present invention is not limited to the above-mentioned embodiment, but includes various modified examples. For example, the above-mentioned embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0198] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in a memory or a recording medium of a recording device such as a hard disk drive or solid state drive.

[0199] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0200] 100 storage device, 120, 130 Ethernet switch, 101, 111 storage node, 102, 103, 320, 330 storage controller, 104, 105, 325, 335, 1703, 3108, 3500 edge interface, 200, 210, 1701 processor, 202, 212, 1702 memory, 203, 213 NTB, 204, 214 front-end interface, 205, 215 back-end interface, 207, 217 Ethernet port, 220 NTB link, 230 drive box, 303, 304, 323, 324, 333, 334, 405, 3005, 3006 QP, 301, 321, 331, 1705, 3000, 3109 PCIe-RoCE converter, 302, 322, 332, 1704 PCIe-EIF logic unit, 401, 3001, 3504 WQE converter, 402, 3002 Retransmission buffer, 403, 413, 423, 3003, 3013, 3023 Sorter, 404, 411, 421, 3004, 3011, 3021 Arbiter, 412, 422, 3012, 3022 Ethernet frame builder, 414, 424, 3014, 3024 Ethernet header remover, 431 Control unit, 432 Internal bus, 433 Interface unit, 434, 3503 DMA, 1706, 3103 Transmit queue

Claims

1. A storage device including a plurality of storage controllers, Each of the plurality of storage controllers includes a controller interface for connecting the storage controllers, The controller interface includes one or more logical ports corresponding to each connected storage controller, and a buffer; The controller interface includes: When converting a first request of a first protocol used in the storage controller into a second request of a second protocol used in a network between storage controllers, storing identification information of the first request and identification information of the second request in a transmission queue of the logical port; The storage device determines whether to store the transmission data in the buffer or in the transmission queue based on identification information of a device that has sent the first request.

2. 2. The storage device according to claim 1, the identification information of the first request includes identification information of a source device of the first request and packet identification information of the first request; The storage device, wherein the identification information of the second request includes packet identification information of the second request.

3. 2. The storage device according to claim 1, The storage device, wherein the controller interface stores information on a correspondence relationship between destination information of the first request and identification information of the logical port.

4. The storage device according to claim 1, The storage device, wherein the controller interface includes a logical port that reads the transmission data from the buffer, and a logical port that stores the transmission data together with identification information of the second request in a transmission queue.

5. The storage device according to claim 1, A storage device, wherein when the controller interface of a storage controller receives the second request, which is a read request, converts a first response of the first protocol, which is the result of the read execution, into a second response of the second protocol, the controller interface stores the status of the result of the read execution in a header of the second response and transmits it.

6. The storage device according to claim 3, The storage device, wherein the controller interface further stores information on a correspondence relationship between identification information of the logical port and address information of the logical port in the first protocol.

7. The storage device of claim 1, A storage device, wherein the controller interface of a storage controller that receives the second request, which is a read request, stores information on the correspondence between the identification information of the logical port that received the second request, the identification information of the second request, and the identification information of the third request when converting the second request into a third request of the first protocol.

8. The storage device of claim 1, The controller interface includes: an interface logic unit including a DMA for transmitting and receiving packets of the first protocol; a conversion unit that converts packets between the first protocol and the second protocol; a buffer; The storage device, wherein the DMA stores transmission data in the buffer and transmits to the conversion unit a request of the first protocol that stores a buffer address of a storage destination of the transmission data and a size of the transmission data.

9. A protocol conversion method executed by a storage device, comprising: The storage device includes a plurality of storage controllers; Each of the plurality of storage controllers includes a controller interface for connecting the storage controllers, The controller interface includes one or more logical ports corresponding to each connected storage controller, and a buffer; The protocol conversion method includes: When converting a first request of a first protocol used in the storage controller into a second request of a second protocol used in a network between storage controllers, storing identification information of the first request and identification information of the second request in a transmission queue of the logical port; A protocol conversion method for selecting a logical port that reads transmission data from the buffer or a logical port that stores the transmission data together with the identification information of the second request in a transmission queue based on the identification information of the device that sent the first request.

10. The protocol conversion method according to claim 9, the identification information of the first request includes identification information of a source device of the first request and packet identification information of the first request; The protocol conversion method, wherein the identification information of the second request includes packet identification information of the second request.

11. The protocol conversion method according to claim 9, a protocol conversion method for selecting the logical port by referring to a correspondence relationship between destination information of the first request stored in the controller interface and identification information of the logical port.

12. The protocol conversion method according to claim 9, the first request is a read request; A protocol conversion method, when converting a first response of the first protocol, which is the result of executing the read, into a second response of the second protocol, storing a status of the result of executing the read in a header of the second response and transmitting it.

13. The protocol conversion method according to claim 11, a protocol conversion method for transmitting identification information of the first request to an address of the logical port selected by referring to a correspondence relationship between identification information of the logical port and address information of the logical port in the first protocol.

Images