Controller and host

Abstract

To provide a controller that realizes indirect communication between a host and a storage device.SOLUTION: In a controller according to an embodiment, a connection unit receives first data and an I / O command from a first node and transmits second data, which is a calculation processing result on the first data, and the I / O command to a second node. A memory stores the first data. A virtual register table stores a virtual register number identified based on a calculation option representing the calculation processing, in association with a virtual address of third data used to process a calculation command in accordance with the calculation option and a data size of the third data. A memory management unit writes the first data to the memory and updates the virtual register table. A calculation processing unit refers to the virtual register table and performs the calculation processing on the first data. A relay table stores multiple pairs of transmission source information and destination information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION An embodiment of the present invention relates to a controller and a host. [Background technology]

[0002] NVMe-oF TM The standard (Non-Volatile Memory express over Fabric) is NVMe TM This standard (Non-Patent Document 1) describes the implementation of a message-based transport model (message-only transport model or message / memory-based transport model). There are two types of NVMe transport protocols used in the NVMe-oF standard: RDMA (Remote Direct Memory Access) (Non-Patent Document 2) and TCP (Transmission Control Protocol) (Non-Patent Document 3).

[0003] In the NVMe transport model, messages are defined as information units called capsules. There are two types of capsules: command capsules used for commands and response capsules used for responses. Command capsules contain a Submission Queue Entry field and a Data field. If no data is attached to a command, the Data field of the command capsule is omitted. Response capsules contain a Completion Queue Entry field and a Data field. If no data is attached to a response, the Data field of the response capsule is omitted.

[0004] The NVMe transport protocol is a direct two-way communication protocol between a host and a storage device. The NVMe transport protocol cannot control indirect communication that occurs via nodes that relay messages between the host and the storage device. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US Patent Application Publication No. 2023 / 216655 [Patent Document 2] U.S. Patent No. 11,032,259 [Patent Document 3] US Patent Application Publication No. 2017 / 255583 [Non-patent literature]

[0006] [Non-Patent Document 1] NVM Express Base Specification, revision 2.0c, https: / / nvmexpress.org / wp-content / uploads / NVM-Express-Base-Specification-2.0c-2022.10.04-Ratified.pdf [Non-patent document 2] NVM Express RDMA Transport Specification 1.0b,https: / / nvmexpress.org / wp-content / uploads / NVM-Express-RDMA-Transport-Specification-1.0b-2022.10.04-Ratified.pdf [Non-patent document 3] NVM Express TCP Transport Specification 1.0c, https: / / nvmexpress.org / wp-content / uploads / NVM-Express-TCP-Transport-Specification-1.0c-2022.10.03-Ratified.pdf Summary of the Invention [Problem to be solved by the invention]

[0007] An object of the present invention is to provide a controller and a host that realize indirect communication between the host and a storage device. [Means for solving the problem]

[0008] A controller according to an embodiment includes a connection unit, a memory, a virtual register table, a memory management unit, a calculation processing unit, and a relay table. The connection unit is connectable to a first node and a second node using the NVMe transport protocol, receives first data and an I / O command from the first node, and transmits second data, which is a result of calculation processing on the first data, and the I / O command to the second node. The memory stores the first data. The virtual register table associates a virtual register number associated with the first data and specified based on a calculation option representing the calculation processing with a virtual address of third data used to process a calculation command according to the calculation option and the data size of the third data. The memory management unit writes the first data to the memory and updates the virtual register table. The calculation processing unit references the virtual register table and performs calculation processing on the first data. The relay table stores multiple pairs of source information and destination information. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a storage system according to a first embodiment. [Figure 2] FIG. 3 is a diagram for explaining an example of a virtual register table according to the first embodiment; [Figure 3] FIG. 2 is a diagram illustrating an example of indirect communication between a host and a storage device according to the first embodiment. [Figure 4] FIG. 3 is a diagram for explaining an example of a relay table according to the first embodiment. [Figure 5] 10 is a flowchart illustrating an example of a relay process of a host according to the first embodiment. [Figure 6]10 is a flowchart illustrating an example of a relay process of a node or a storage device according to the first embodiment. [Figure 7] 10 is a flowchart illustrating another example of the relay process of the node or storage device according to the first embodiment. [Figure 8] FIG. 2 is a diagram for explaining an example of an instruction set of a secure computation instruction according to the first embodiment. [Figure 9] FIG. 2 is a diagram for explaining an example of a virtual register number according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a configuration of a host according to a second embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of indirect communication between a host and a storage device according to the second embodiment. [Figure 12] FIG. 11 is a diagram illustrating an example of indirect communication between a host and a storage device according to the third embodiment. [Figure 13] FIG. 11 is a diagram for explaining an example of a relay table according to the third embodiment. [Figure 14] FIG. 13 is a diagram illustrating an example of a configuration of a host according to a fourth embodiment. [Figure 15] FIG. 13 is a diagram illustrating an example of a node according to the fourth embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of a network according to a fourth embodiment. [Figure 17] FIG. 10 is a diagram illustrating an example of a network according to a fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The following describes embodiments with reference to the drawings. The following description exemplifies devices and methods embodying the technical concepts of the embodiments. The technical concepts of the embodiments are not limited to the structures, shapes, arrangements, materials, etc. of the components described below. Modifications that can be easily conceived by those skilled in the art are naturally within the scope of the disclosure. For clarity of explanation, the drawings may schematically depict elements with different sizes, thicknesses, planar dimensions, shapes, etc. compared to the actual elements. Elements with different dimensional relationships or ratios may be included in multiple drawings. Corresponding elements may be designated by the same reference numerals in multiple drawings, and redundant description may be omitted. Some elements may be designated by multiple names, but these names are merely examples and do not exclude the use of other names for these elements. Elements without a plural name may also be designated by other names. "Connection" may include not only direct connection but also connection via other elements. Unless the number of elements is specified as being plural, the element may be a singular element or multiple elements.

[0011] (First embodiment) 1 is a diagram illustrating an example of a storage system 2 according to the first embodiment. The storage system 2 includes a host 52, a storage device 54, an upstream node 24, a node 20, a downstream node 26, and a network 22. The storage system 2 is also referred to as an information processing system.

[0012] The storage device 54 includes a storage medium and a storage controller. An example of the storage medium is a NAND flash memory. The storage device 54 may be a solid state drive (SSD). The host 52 is an information processing device serving as an external device that accesses the storage device 54. The node 20 is a device that realizes indirect communication between the host 52 and the storage device 54. The node 20 includes an accelerator 10. The accelerator 10 operates to improve the processing speed of the host 52. The accelerator 10 is also referred to as a controller. The accelerator 10 includes a network interface (network I / F) 30, a main memory 32, a virtual register table 34, a page table 36, a memory management unit 38, a calculation processing unit 40, a relay table 42, a relay table management unit 44, and a storage interface (storage I / F) 46. The node 20 may include a local storage device 14 and the storage I / F 46.

[0013] The node 20 is connected to an upstream node 24 and a downstream node 26 via a network 22. The upstream node 24 includes at least one node 20. When the upstream node 24 includes multiple nodes 20, the multiple nodes 20 are connected via the network 22. The node 20 may be connected to a host 52 via the network 22. The host 52 includes at least a relay table 42, a relay table management unit 44, and a network I / F 30, which are components of the accelerator 10.

[0014] The downstream node 26 includes at least one node 20. When the downstream node 26 includes multiple nodes 20, the multiple nodes 20 are connected via a network 22. The node 20 may be connected to a storage device 54 via the network 22. The storage device 54 includes at least a storage medium, a relay table 42, a relay table management unit 44, and a network I / F 30, which are components of the accelerator 10. The storage medium of the storage device 54 may realize the function of the local storage device 14.

[0015] The host 52 sends a computing storage Input / Output command (hereinafter referred to as an I / O command) to the upstream node 24, node 20, or storage device 54. The command is accompanied by host data. The host data is accompanied by metadata. When the application of the host 52 causes the upstream node 24, node 20, or storage device 54 to perform computational processing on the host data, the application includes computation options in the metadata. The computation options indicate the content of the computational processing of the host data by the upstream node 24, node 20, or storage device 54. The accelerator 10 performs computational processing on the host data in accordance with the computation options and replaces the host data with the processing result (computation result).

[0016] The network I / F 30 receives I / O commands from the host 52, the upstream node 24, or the downstream node 26. The I / O commands include read commands for reading host data from the storage device 54 and write commands for writing host data to the storage device 54. The host data accompanying the I / O command (read command) includes read data read from the storage device 54 based on the I / O command. The host data accompanying the I / O command (write command) includes write data written to the storage device 54 based on the I / O command. The host data accompanying the I / O command is specified by a logical address used to access the storage device 54 (to read data from the storage device 54 and to write data to the storage device 54).

[0017] When the host data is read data, the network I / F 30 receives the read data from the storage device 54, the upstream node 24, or the downstream node 26. When the host data is write data, the network I / F 30 transmits the write data to the storage device 54, the upstream node 24, or the downstream node 26.

[0018] The network I / F 30 includes a TCP processing unit 48. The TCP processing unit 48 adds a TCP / IP header (described later) to a capsule to be transmitted to the network 22.

[0019] The main memory 32 stores host data (including read data and write data) associated with I / O commands. The main memory 32 can be accessed faster than the local storage device 14. The main memory 32 may be realized by a non-volatile memory (not shown) such as a DRAM provided within the node 20.

[0020] Data used to process a computation command according to computation options included in metadata accompanying host data accompanying an I / O command is held in a virtual register. The virtual register table 34 is a table for managing virtual registers. The virtual register table 34 is stored in a non-volatile memory provided within the node 20.

[0021] 2 is a diagram for explaining an example of the virtual register table 34 according to the first embodiment. The data structure of the virtual register table 34 (virtual registers managed in the virtual register table 34) will be explained.

[0022] The virtual register table 34 includes virtual register numbers 1 to N. reg , virtual address a[1]~a[N reg ] and data size s[1]~s[N reg ]. Virtual register numbers 1 to N reg is determined (calculated) based on the calculation options. Virtual address a[1]~a[N reg ] is represented by the page number and page offset assigned to the page where the data used to process the calculation command according to the calculation option is stored. reg ] is in bytes. Virtual register number 1 to N is specified based on the calculation option. reg Details will be given later.

[0023] The virtual register table 34 stores virtual register numbers in association with virtual addresses and data sizes. In other words, one virtual register is referenced using the virtual register number assigned to that virtual register, and is represented by a pair of a virtual address and a data size.

[0024] The calculation options include a content identifier and a data size (bytes). The content identifier is represented by a set of type, key ID, and data ID.

[0025] As an example of the structure of a computation option, consider a computation option that can be used with Torus Fully Homomorphic Encryption (TFHE), a secure computation technology. The type is a TFHE data type, the key ID is a key number, and the data ID is a TFHE data identifier. The type is represented by a value between 0 and 4, the key ID is represented by a value equal to or greater than 0, and the data ID is represented by a value equal to or greater than 0. Note that a torus in TFHE is a mathematical structure called an algebraic torus or a circular group, and is a multiplicative group T={z∈C:|z|=1}× defined by a set of points {z∈C:|z|=1} on a unit circle on a complex plane C and a binary operation "×". TFHE uses a lattice cryptography called Torus Learning with Errors (TLWE). TFHE ciphertexts are called TLWE samples and are represented as vectors in a torus. In this embodiment, the torus is scaled and encoded as a 32-bit integer value.

[0026] The virtual register number in the virtual register table 34 is calculated (specified) from the content identifier included in such a calculation option.

[0027] Returning to the explanation of FIG. 1, the page table 36 is a table for managing, for each page number, whether the storage destination of data in that page is the main memory 32 or a storage. The page table 36 is saved in a non-volatile memory provided in the node 20. The page table 36 may store, in association with each page number, a flag indicating the storage destination of the data and the actual address of the storage destination. When the data is stored in a storage, the storage may be the local storage device 14 or the storage device 54. The local storage device 14 also includes a storage medium and a storage controller. An example of the storage medium is a NAND flash memory. The local storage device 14 may be an SSD.

[0028] The memory management unit 38 refers to the page table 36 according to the operation mode (described later) of the accelerator 10, stores the host data accompanying the I / O command in the main memory 32, and updates the virtual address in the virtual register table 34.

[0029] The calculation processing unit 40 refers to the virtual register table 34, processes a calculation command (a calculation command using the host data) according to a calculation option included in the metadata accompanying the host data accompanying the I / O command, and encrypts the host data.

[0030] When relaying a capsule received from a node to another node, the relay table management unit 44 references the relay table 42 to generate information to be added to the IP header and TCP header (details will be described later). The relay table 42 is stored in a non-volatile memory provided within the node 20.

[0031] The local storage device 14 is a storage used for paging.

[0032] The accelerator 10 may be configured by having one or more processing circuits (processors) execute the processing contents of the memory management unit 38, the calculation processing unit 40, the relay table management unit 44, and the TCP processing unit 48. The processing by a processing circuit (processor) may be realized by a CPU (Central Processing Unit) executing firmware, or may be realized by hardware. Furthermore, some of the processing by the processing circuit may be realized by a CPU executing firmware, and other processing may be realized by hardware. The hardware is realized by at least one of a register, an adder, a multiplier, and other arithmetic units. The register is realized by, for example, a logic circuit such as a flip-flop. The adder, multiplier, and other arithmetic units are realized by, for example, a logic circuit.

[0033] FIG. 3 is a diagram illustrating an example of indirect communication between a host 52 and a storage device 54. As shown in FIG.

[0034] A series circuit of two nodes 20-1 and 20-2 is connected between a host 52 and a storage device 54. The host 52 includes an accelerator 10. Each of the nodes 20-1 and 20-2 includes the accelerator 10 and a local storage device 14. The storage device 54 includes the accelerator 10 and storage (e.g., a solid state drive: SSD). The storage device 54 including the accelerator 10 is also referred to as a computing storage device (CSD) that processes calculation instructions.

[0035] The IP (Internet Protocol) address of the host 52 is assumed to be A0. The IP address of the node 20-1 is assumed to be A1. The IP address of the node 20-2 is assumed to be A2. The IP address of the storage device 54 is assumed to be A3. In the network 22, the Transmission Control Protocol (TCP) defined in RFC 9293 is assumed to be used as the NVMe transport protocol. Since TCP operates on the Internet Protocol (IP) and also controls the header contents of IP datagrams in this embodiment, in this embodiment, TCP messages and IP datagrams are collectively referred to as TCP / IP messages.

[0036] A TCP / IP message is represented in the format of TCP / IP [header] {payload}. The header is a combination of an IP header and a TCP header. Figure 3 shows only four fields: the source address (src_addr) and destination address (dst_addr) of the IP header, and the source port number (src_port) and destination port number (dst_port) of the TCP header. The contents of the payload are application data. When TCP is used as the NVMe transport, the application data is a capsule (command capsule or response capsule).

[0037] 3 illustrates a case where secure computation is performed as an example of computation processing by each node 20. One example of secure computation is Compute on Write (CoW) processing using an NVMe Write Command. A command capsule is transmitted from the host 52 to the storage device 54 via the serial circuit of the nodes 20-1 and 20-2. A response capsule is transmitted from the storage device 54 to the host 52 via the serial circuit of the nodes 20-2 and 20-1.

[0038] The command capsule includes an NVMe write command and write data. The response capsule includes an NVMe write response. When node 20-1 receives a command capsule including an NVMe write command and write data from the host 52, it determines whether the write data satisfies a predetermined condition. When node 20-2 receives a command capsule including an NVMe write command and write data from node 20-1, it determines whether the write data satisfies a predetermined condition. Note that nodes 20-1 and 20-2 store the write data in the main memory 32 as a virtual register. If there is insufficient free space in the main memory, nodes 20-1 and 20-2 store the write data in the local storage device 14 or the storage device 54. If the write data satisfies the predetermined condition, nodes 20-1 and 20-2 perform secure computation using the write data. Node 20-1 generates a command capsule including encrypted write data, which is the result of the secure computation, and sends the command capsule to node 20-2. The node 20-2 generates a command capsule including the updated write data, which is the result of the secure computation, and transmits the command capsule to the storage device .

[0039] The header contents of the TCP / IP message sent from the host 52 to the node 20-1 are src_addr=A0, dst_addr=A1, src_port=P1, dst_port=nvmeof-relay-port, which is a port number common to all nodes predefined in a network using NVMe-oF.

[0040] The header contents of the TCP / IP message transmitted from the node 20-1 to the node 20-2 are src_addr=A1, dst_addr=A2, src_port=P2, and dst_port=nvmeof-relay-port.

[0041] The header contents of the TCP / IP message sent from the node 20-2 to the storage device 54 are src_addr=A2, dst_addr=A3, src_port=P3, and dst_port=nvmeof-relay-port.

[0042] The header contents of the TCP / IP message sent from the storage device 54 to the node 20-2 are src_addr=A3, dst_addr=A2, src_port=nvmeof-relay-port, dst_port=P3.

[0043] The header contents of the TCP / IP message transmitted from the node 20-2 to the node 20-1 are src_addr=A2, dst_addr=A1, src_port=nvmeof-relay-port, dst_port=P2.

[0044] The header contents of the TCP / IP message sent from the node 20-1 to the host 52 are src_addr=A1, dst_addr=A0, src_port=nvmeof-relay-port, dst_port=P1.

[0045] FIG. 4 is a diagram for explaining an example of the relay table 42 according to the first embodiment.

[0046] The relay table 42 is used in creating TCP / IP messages. In the first embodiment, the relay tables 42 implemented in the host 52, the nodes 20-1 and 20-2, and the storage device 54 store the same information.

[0047] The relay table 42 includes a pair of a previous hop designation section and a next hop designation section. The previous hop designation section has three fields: a source address, a source port number, and a destination port number. The next hop designation section has three fields: a destination address, a source port number, and a destination port number.

[0048] The first record in the relay table 42 has a previous hop designation part of A0:P1:nvmeof-relay-port and a next hop designation part of A1:P1:nvmeof-relay-port. ":" represents a field separator.

[0049] Before transmitting a TCP / IP message containing a capsule in its payload, the relay table management unit 44 in each accelerator 10 of the host 52, nodes 20-1 and 20-2, and storage device 54 refers to the relay table 42 to search for a record that has the IP address of its own node as the source address in the previous hop designation section, the source port number of its own node as the source port number in the previous hop designation section, and nvmeof-relay-port as the destination port number in the previous hop designation section. If the search results in a record that matches the conditions, the relay table management unit 44 assigns the destination address, source port number, and destination port number in the next hop designation section of the record that matches the conditions to the destination address, source port number, and destination port number in the header, respectively.

[0050] When the host 52 sends a TCP / IP message including a command capsule, the first record in the relay table 42 satisfies the above conditions, so the relay table management unit 44 uses the first record to generate a TCP / IP header.

[0051] When node 20-1 sends a TCP / IP message including a command capsule, the second record from the top of relay table 42 meets the above condition. Therefore, relay table management unit 44 generates a TCP / IP header using the second record from the top.

[0052] When node 20-2 sends a TCP / IP message including a command capsule, the third record from the top of relay table 42 meets the above condition, so relay table management unit 44 uses the third record from the top to generate a TCP / IP header.

[0053] When the storage device 54 sends a TCP / IP message including a response capsule, the fourth record from the top of the relay table 42 meets the above condition. Therefore, the relay table management unit 44 generates a TCP / IP header using the fourth record from the top.

[0054] When node 20-2 sends a TCP / IP message including a response capsule, the fifth record from the top of the relay table 42 meets the above condition, so the relay table management unit 44 generates a TCP / IP header using the fifth record from the top.

[0055] When node 20-1 transmits a TCP / IP message including a response capsule, the last record in the relay table 42 satisfies the above condition, so the relay table management unit 44 generates a TCP / IP header using the last record.

[0056] The contents of the header determined by each node using the above method may be passed to the TCP processing unit 48 of each node as a socket parameter when each node creates a TCP socket. The relay table management unit 44 and the network I / F 30 cooperate to form a connection unit that can be connected to the upstream node 24 or the downstream node 26 using the NVMe transport protocol. The connection unit is also referred to as a connection interface. In FIG. 3, the upstream node 24 is the host 52, node 20-1, or node 20-2, and the downstream node 26 is the storage device 54, node 20-2, or node 20-1. The connection unit or connection interface receives first data and an I / O command from the upstream node and transmits second data, which is the result of computation on the first data, and an I / O command to the two downstream nodes.

[0057] FIG. 5 is a flowchart illustrating an example of relay processing by the accelerator 10 included in the host 52 according to the first embodiment.

[0058] The relay table management unit 44 selects a source port number by the function SelectSrcPort( ) and sets the source port number to the variable sport (step S401).

[0059] The relay table management unit 44 refers to the relay table 42 using the function RelayTableLookUpHost(own node address, sport) and searches for a record that has the IP address of the own node as the source address of the previous hop designation part, the variable sport as the source port number of the previous hop designation part, and nvmeof-relay-port as the destination port number of the previous hop designation part. If a record that matches the conditions is found, the relay table management unit 44 sets the destination address of the next hop designation part of the record that matches the conditions to the variable daddr and sets the destination port number of the next hop designation part to the variable dport (step S402).

[0060] The relay table management unit 44 sends the NVMe command by a TCP / IP message having a TCP / IP header with the source address = its own node address, the destination address = variable daddr, the source port number = variable sport, and the destination port number = variable dport (step S403). The variable dport is nvmeof-relay-port.

[0061] FIG. 6 is a flowchart illustrating an example of relay processing of the accelerator 10 included in the node 20 or the storage device 54 according to the first embodiment.

[0062] The relay table management unit 44 sets the source port number of the received TCP / IP message to the variable r_src_port, and sets the destination port number of the received TCP / IP message to the variable r_dst_port (step S501).

[0063] The relay table management unit 44 refers to the relay table 42 using the function RelayTableLookUpNonHost(local node address, r_src_port, r_dst_port) to search for a record that has the IP address of the local node as the source address of the previous hop designation part, the variable r_src_port as the source port number of the previous hop designation part, and the variable r_dst_port as the destination port number of the previous hop designation part. If a record that matches the conditions is found, the relay table management unit 44 sets the destination address of the next hop designation part of the record that matches the conditions to the variable daddr, the source port number of the next hop designation part to the variable sport, and the destination port number of the next hop designation part to the variable dport (step S502).

[0064] The relay table management unit 44 determines whether the variable daddr is 0 (step S503). If the variable daddr is 0 (Yes in step S503), the local node address is the final address, and the relay process ends. If the variable daddr is not 0 (No in step S503), the relay table management unit 44 sends the NVMe command using a TCP / IP message having a TCP / IP header with source address = local node address, destination address = variable daddr, source port number = variable sport, and destination port number = variable dport (step S504). The variable dport is nvmeof-relay-port.

[0065] 7 is a flowchart illustrating an example of relay processing of the accelerator 10 included in the node 20 or the storage device 54 according to the first embodiment when the operating mode is the CoW mode. The CoW mode is an operating mode for processing a calculation command using host data (write data) accompanying a write command from the host 52.

[0066] The network I / F 30 receives a write command (I / O command) from the host 52 (step S601). The write command includes write data and a logical address used to access the write data.

[0067] When the process of step S601 ends, the memory management unit 38 stores the write data included in the write command in variable D (step S602).

[0068] The memory management unit 38 determines whether or not a calculation option is included in the metadata added to the write data (step S603).

[0069] If the metadata includes a calculation option (Yes in step S603), the memory management unit 38 calculates a virtual register number based on the calculation option (content identifier) ​​and stores the virtual register number in the variable num (step S604).

[0070] The memory management unit 38 copies the variable D (write data) to a free area in the main memory 32 (step S605).

[0071] The memory management unit 38 sets the virtual address indicating the memory area of ​​the main memory 32 to which the variable D has been copied as the virtual address corresponding to the variable num (the virtual address referenced by the virtual register number stored in the variable num) in the virtual register table 34. In other words, the memory management unit 38 sets the leading virtual address of the copy destination of the variable D in the virtual address field (reg[num].addr) of the variable num-th virtual register (step S606).

[0072] The memory management unit 38 sets the data size of the variable D as the data size corresponding to the variable num (the data size referenced by the virtual register number stored in the variable num) in the virtual register table 34. In other words, the memory management unit 38 sets the byte length of the variable D in the data size field (reg[num].size) of the variable num-th virtual register (step S607).

[0073] The calculation processing unit 40 executes the program stored in the program register by referring to the virtual register table 34 (step S608). The program register is part of the virtual register. The virtual register is a register defined in the virtual address space. Executing the program in step S608 corresponds to processing a calculation command using write data. The write data (TLWE sample (described later)) that is the target of calculation (the target when processing the calculation command) is read from the CoW register (described later).

[0074] When the processing of step S608 is executed, the processing result (i.e., the processing result of the calculation instruction using the write data) data is stored in the virtual address set in the virtual address field (reg[num].addr) of the variable num-th virtual register.

[0075] The memory management unit 38 refers to the virtual register table 34, reads data of the byte length (number of bytes) of the data size set in the data size field (reg[num].size) of the virtual register from the virtual address set in the virtual address field of the variable num-th virtual register, and copies the data to variable D (step S609).

[0076] When the process of step S609 is executed, the network I / F 30 transmits a write command including the variable D as write data (a write command for the variable D) to the next hop node (S610).

[0077] The end of the program is a Return command (Return num) that takes the variable num as an argument.

[0078] If it is determined in step S603 that the metadata added to the write data does not include a calculation option (No in step S603), the process proceeds to step S610.

[0079] 6 and 7, node 20-1 can process a calculation instruction based on a write command sent from host 52 and transmit the processing result as write data to storage device 54 via next hop node 20-2. Node 20-1 can store the processing result of the calculation instruction using the write data as write data in main memory 32.

[0080] Similarly, node 20-2 can process a calculation instruction based on a write command transmitted from node 20-1 and transmit the processing result as write data to the storage device 54. Node 20-2 can store the processing result of the calculation instruction using the write data as write data in the main memory 32. The write data stored in the main memory 32 can be used when processing the calculation instruction. The nodes 20-1 and 20-2 can perform calculations based on read commands in the same way as write commands, and store the results of the calculations in the main memory 32 as read data.

[0081] Next, an example of an instruction set of secure computation instructions used by the accelerator 10 will be shown.

[0082] FIG. 8 is a diagram for explaining an example of an instruction set of a secure computation instruction according to the first embodiment.

[0083] An example of an instruction set of secure computation instructions includes a Return instruction, a Move instruction, a Push instruction, a Pop instruction, a Gate Bootstrap instruction, an Add instruction, a Sub instruction, an IntMult instruction, a PubKS (Public Functional Key Switching) instruction, a PrvKS (Private Functional Key Switching) instruction, a Vertical Packing instruction, and a Circuit Bootstrap instruction.

[0084] The ciphertext register number represents a virtual register number for referencing a ciphertext register, and the LUT register number represents a virtual register number for referencing a LUT (Look Up Table) register.

[0085] The Return instruction corresponds to command type 0. The argument of the Return instruction is the ciphertext register number num. The Return instruction causes the value of the ciphertext register referenced by the ciphertext register number num to be sent to an adjacent node. If the ciphertext register is a CoR register, the value of the ciphertext register is sent to an upstream adjacent node. If the ciphertext register is a CoW register, the value of the ciphertext register is sent to a downstream adjacent node. After the value of the ciphertext register is sent, a stack pointer for managing the reference position of the stack area included in the virtual address space is set to 0.

[0086] The Move instruction corresponds to command type 1. Argument 1 of the Move instruction is the ciphertext register number num1, and argument 2 is num2. The Move instruction copies the value of the ciphertext register referenced by the ciphertext register number num1 to the ciphertext register referenced by the ciphertext register number num2.

[0087] The PUSH instruction corresponds to command type 2. The argument of the PUSH instruction is the ciphertext register number num. The PUSH instruction copies the value of the ciphertext register referenced by the ciphertext register number num to the top of the stack area included in the virtual address space, and decrements the stack pointer (subtracts 1 from the value of the stack pointer).

[0088] The Pop instruction corresponds to command type 3. The argument of the Pop instruction is the ciphertext register number num. The Pop instruction copies the top value of the stack area included in the virtual address space to the ciphertext register referenced by the ciphertext register number num, and increments the stack pointer (adds 1 to the value of the stack pointer).

[0089] The Gate Bootstrap (GBS) instruction corresponds to command type 4. Argument 1 of the Gate Bootstrap instruction is the LUT register number num1, and argument 2 is the ciphertext register number num2. The Gate Bootstrap instruction executes GBS or PBS (Programmable Bootstrapping) on ​​the value of the ciphertext register referenced by the ciphertext register number num2, using the LUT register referenced by the LUT register number num1. For example, if the LUT register number num1 = 0, GBS is executed, and if the LUT register number num1 > 0, PBS is executed. The execution result (output value) of the GBS or PBS is copied to the ciphertext register referenced by the ciphertext register number num2. For example, if the value of the LUT register referenced by the LUT register number num1 is the LUT for the function f(x), and the value of the ciphertext register referenced by the ciphertext register number num2 before the Gate Bootstrap instruction is executed is the TLWE sample for x, then the value of the ciphertext register referenced by the ciphertext register number num2 after the Gate Bootstrap instruction is executed will be the TLWE sample for f(x). The CMux function is used in the Blind Rotate process executed in the Gate Bootstrap command.

[0090] The Add instruction corresponds to command type 5. Argument 1 of the Add instruction is the ciphertext register number num1, and argument 2 is num2. The Add instruction adds the value of the ciphertext register referenced by the ciphertext register number num1 and the value of the ciphertext register referenced by the ciphertext register number num2 for each vector component, and copies the addition result (calculation result) to the ciphertext register referenced by the ciphertext register number num1.

[0091] The Sub instruction corresponds to command type 6. Argument 1 of the Sub instruction is the ciphertext register number num1, and argument 2 is num2. The Sub instruction subtracts the value of the ciphertext register referenced by the ciphertext register number num2 from the value of the ciphertext register referenced by the ciphertext register number num1 for each vector component, and copies the result of the subtraction (calculation result) to the ciphertext register referenced by the ciphertext register number num1.

[0092] The IntMult instruction corresponds to command type 7. Argument 1 of the IntMult instruction is the ciphertext register number num, and argument 2 is the integer value val. The IntMult instruction multiplies the value of the ciphertext register referenced by the ciphertext register number num by the integer value val for each vector component, and copies the multiplication result (calculation result) to the ciphertext register referenced by the ciphertext register number num.

[0093] The PubKS instruction corresponds to command type 8. Argument 1 of the PubKS instruction is the ciphertext register number num1, argument 2 is num2, and argument 3 is the key switching key number num3. The key switching key number in the PubKS instruction is a virtual register number for referencing the PubKSK (PubKS Key) register. The PubKS instruction executes public function based key switching on the value of the ciphertext register referenced by the ciphertext register number num1 (i.e., the ciphertext) using the key switching key stored in the PubKSK register referenced by the key switching key number num3, and the ciphertext after applying the public function based key switching is stored in the ciphertext register referenced by the ciphertext register number num2. An example of a function in the PubKS instruction is the identity function (f(x)=x).

[0094] The PrvKS instruction corresponds to command type 9. Argument 1 of the PrvKS instruction is a ciphertext register number num1, argument 2 is num2, and argument 3 is a key switching key number num3. The key switching key number in the PrvKS instruction is a virtual register number for referencing the PrvKSK (PrvKS Key) register. The PrvKS instruction executes private function type key switching on the value of the ciphertext register referenced by the ciphertext register number num1 (i.e., the ciphertext) using the key switching key stored in the PrvKSK register referenced by the key switching key number num3, and the ciphertext after applying the private function type key switching is stored in the ciphertext register referenced by the ciphertext register number num2. The PrvKSK register referenced by the key switching key number num3 stores k+1 key switching keys for public functional key switching as one key switching key for private functional key switching. Specifically, the key switching key stored in the PrvKSK register is obtained by encrypting the function (f_u(x)=-Ku·x if u≦k, otherwise f_u(x)=1·x if u=k+1) for x=k_i / 2j (1≦i≦n+1, 1≦j≦t) for k+1 TLWE (or TRLWE) samples, where · is the symbol representing the multiplication of an integer and a torus. When k=1, two samples are counted as one key switching key (PrvKSK) for private function-based key switching.

[0095] When the first embodiment is applied to a multi-party computation system (key switching multi-party computation), there may be multiple PubKSK registers and PrvKSK registers per ciphertext register. Therefore, argument 3 of the PubKS instruction and argument 3 of the PrvKS instruction specify the key switching key numbers for referencing the PubKSK register and PrvKSK register to be used.

[0096] The Vertical Packing instruction corresponds to command type 10. The Vertical Packing instruction is an instruction to execute the VP (Vertical Packing) algorithm. Argument 1 of the Vertical Packing instruction is the LUT register number num1, argument 2 is the ciphertext register number num2, and argument 3 is num3. num1 is the virtual register number containing s LUTs used to calculate each output bit of an arbitrary d-bit input, s-bit output function used in VP. num2 is the virtual register number of the ciphertext register containing d TRGSW samples. num3 is the virtual register number of the ciphertext register containing s TLWE samples. The Vertical Packing instruction executes s Blind Rotate operations, within which the CMux function is used. Each of the s outputs from the s Blind Rotate operations is converted into a TLWE sample by the Sample Extract operation.

[0097] The Circuit Bootstrap instruction corresponds to command type 11. The Circuit Bootstrap instruction is an instruction to execute Circuit Bootstrapping (CBS). Argument 1 of the Circuit Bootstrap instruction is the LUT register number num1, argument 2 is the ciphertext register number num2, argument 3 is num3, argument 4 is the key switching key number num4, and argument 5 is num5. num1 is the virtual register number containing the LUT used in CBS. num2 is the virtual register number of the ciphertext register containing s TLWE samples. num3 is the virtual register number of the ciphertext register containing s TRGSW samples. num4 is the virtual register number of PubKSK. num5 is the virtual register number of PrvKSK_NTT. The CMux function is used in the Blind Rotate process executed in the Circuit Bootstrap instruction. The host 52 writes in the calculation options the content of the calculation process that uses the instruction set of the secure calculation instruction shown in Fig. 8. This enables the accelerator 10 to execute the Circuit Bootstrap instruction and transmit the ciphertext after the execution of the Circuit Bootstrap instruction to the adjacent node as write data.

[0098] FIG. 9 is a diagram illustrating an example of virtual register numbers according to the first embodiment.

[0099] The virtual registers in this embodiment include a program register, an LUT register, a BK register, a BKNTT register, a PubKSK register, a PrvKSK register, a PrvKSKNTT register, a TLWE ciphertext register, and a TRGSW ciphertext register. The entities of these registers exist in the main memory 32.

[0100] The type of the program register is 0 (PRG), the key ID is 0, the data ID is 0, and the virtual register number is 0. The program register stores a program (a sequence of calculation instructions).

[0101] The type of the LUT register is 1 (LUT), the key ID is 0, the data ID is x, and the virtual register number is 1+x. The LUT register stores the test vector of the TFHE. An example of the test vector (LUT) stored in the LUT register is the coefficient for a predetermined function (polynomial).

[0102] The type of the BK register is 2 (KEY), the key ID is k, the data ID is y (=0), and the virtual register number is 1+N LUT+5k+y. The BK register stores the bootstrapping key of the TFHE. The bootstrapping key stored in the BK register is used in Gate Bootstrapping (GBS) of the TFHE. The bootstrapping key may also be used in Programmable Bootstrapping (PBS). PBS is a bootstrapping method that outputs a TLWE sample obtained by homomorphically evaluating an input TLWE sample (ciphertext) using a predetermined function, after reducing the noise to the noise level of a fresh sample.

[0103] The type of the BKNTT register is 2 (KEY), the key ID is k, the data ID is y (=1), and the virtual register number is 1+N LUT +5k+y. The BKNTT register stores the TFHE bootstrapping key that has undergone number-theoretic transformation processing.

[0104] The type of the PubKSK register is 2 (KEY), the key ID is k, the data ID is y (=2), and the virtual register number is 1+N LUT +5k+y. The PubKSK register stores the key switching key of the TFHE. Specifically, the PubKSK register stores the key switching key used in public function key switching. The key switching key stored in the PubKSK register is usually used in post-processing of the above-mentioned GBS or PBS (i.e., bootstrapping process).

[0105] The type of the PrvKSK register is 2 (KEY), the key ID is k, the data ID is y (=3), and the virtual register number is 1+N LUT +5k+y. The PrvKSK register stores the key switching key of the TFHE. Specifically, the PrvKSK register stores the key switching key used in private function based key switching. The key switching key stored in the PrvKSK register is usually used in post-processing of the above-mentioned GBS or PBS (i.e., bootstrapping process).

[0106] The type of the PrvKSKNTT register is 2 (KEY), the key ID is k, the data ID is y (=4), and the virtual register number is 1+N LUT +5k+y. The PrvKSKNTT register stores the PrvKSK that has undergone number-theoretic transformation.

[0107] The TLWE ciphertext register type is 3 (TLWE-CoR) or 4 (TLWE-CoW), the key ID is k, the data ID is z, and the virtual register number is 1+N LUT +5N key +k(N TLWE +N TRGSW ) + z. The TLWE ciphertext register stores the TLWE sample. The TLWE ciphertext register includes two types of registers: TLWE-CoR (CoR register) and TLWE-CoW (CoW register).

[0108] The TRGSW ciphertext register type is 5 (TRGSW-CoR) or 6 (TRGSW-CoW), the key ID is k, the data ID is z, and the virtual register number is 1+N LUT +5N key +k(N TLWE +N TRGSW )+N TLWE +z. The TRGSW ciphertext register stores the TRGSW sample. The TRGSW ciphertext register includes two types of registers: TRGSW-CoR (CoR register) and TRGSW-CoW (CoW register).

[0109] In Figure 9, when the type of a virtual register is 0, the KeyID is 0, and the DataID is 0, the virtual register number (=0) is calculated from the type, KeyID, and DataID (i.e., the content identifier), indicating that the virtual register is a program register.

[0110] If the type of a virtual register is 1, the KeyID is 0, and the DataID is x, the virtual register number (=1+x) is calculated from the type, KeyID, and DataID (i.e., the content identifier), indicating that the virtual register is a LUT register.

[0111] If the type of the virtual register is 2, the KeyID is k, and the DataID is y, the virtual register number (= 1 + N LUT +5k+y) is calculated, indicating that the virtual register is the BK register, the BKNTT register, the PubKSK register, the PrvKSK register, or the PrvKSKNTT register. If y=0, the virtual register is the BK register, if y=1, the virtual register is the BKNTT register, if y=2, the virtual register is the PubKSK register, if y=3, the virtual register is the PrvKSK register, and if y=4, the virtual register is the PrvKSKNTT register.

[0112] If the type of the virtual register is 3 or 4, the Key ID is k, and the Data ID is z, the virtual register number (= 1 + N LUT +5N key +k(N TLWE +N TRGSW ) + z) is calculated, indicating that the virtual register is a TLWE ciphertext register. If the type is 3, it indicates that the virtual register is a TLWE-CoR (CoR register). If the type is 4, it indicates that the virtual register is a TLWE-CoW (CoW register).

[0113] If the type of the virtual register is 5 or 6, the Key ID is k, and the Data ID is z, the virtual register number (= 1 + N LUT +5N key +k(N TLWE +N TRGSW )+NTLWE +z) is calculated, indicating that the virtual register is a TRGSW ciphertext register. If the type is 5, it indicates that this virtual register is a TRGSW-CoR (CoR register). If the type is 6, it indicates that this virtual register is a TRGSW-CoW (CoW register).

[0114] x is 0 or greater and N LUT It is assumed that the integer is less than (0≦x <N LUT ) y is assumed to be an integer greater than or equal to 0 and less than or equal to 4 (0≦y≦4). k is assumed to be an integer greater than or equal to 0 and less than or equal to N key It is assumed that the integer is less than (0≦k <N key ) z is 0 or greater and N TLWE It is assumed that the integer is less than (0≦z <N TLWE ).

[0115] N LUT is the maximum number of LUT registers. key is the maximum number of BK, BKNTT, PubKSK, PrvKSK, and PrvKSKNTT registers. NTLWE is the total number of TLWE ciphertext registers per BK or BKNTT register. NTRGSW is the total number of TRGSW ciphertext registers per BK or BKNTT register.

[0116] The management of the relay table 42 will now be described.

[0117] First, we will discuss the construction of the relay table 42. There are three main construction methods to choose from. The present invention can be properly implemented using any of these construction methods.

[0118] The first method is to create one relay table during the design phase of the storage system 2 and install it in each accelerator 10 of the host 52, node 20, and storage device 54. With this method, the configuration of each device, namely, the host 52, node 20, and storage device 54, cannot be dynamically changed, but this is not a problem if the scale of the storage system 2 is small and the administrator of the storage system 2 can configure the information in the pre-hop designation section and post-hop designation section. The relay table 42 can be installed in each accelerator 10 via a vendor-specific function of the NVMe transport protocol. Alternatively, the administrator may install the relay table 42 in each accelerator 10 by directly operating each accelerator 10 via a control interface such as a universal asynchronous receiver / transmitter (UART).

[0119] In the second method, the accelerator 10 constructs the relay table 42 itself. The relay table management unit 44 of the accelerator 10 employing this method has a neighbor discovery function and detects the hosts 52, nodes 20, and storage devices 54 in the storage system 2. This neighbor discovery function may be an existing communication protocol such as SSDP (Simple Service Discovery Protocol) extended to add information indicating compatibility with this embodiment, or a dedicated communication protocol may be used. The relay table management unit 44 collects IP addresses and port numbers at which nodes compatible with this embodiment await connections, thereby collecting information corresponding to the source address and source port number in the previous hop designation field of the relay table 42 shown in FIG. 4, as well as information corresponding to the destination address and destination port number in the next hop designation field. For each piece of collected information, the relay table management unit 44 creates one entry in the relay table 42 by setting the port number used by the accelerator 10 to await connections from other nodes as the destination port number in the previous hop designation field and the port number used by the accelerator 10 to connect to other nodes as the source port number in the next hop designation field. By carrying out these steps for each piece of collected information, the relay table 42 is constructed.

[0120] The third method is a method in which the construction of a relay path and the construction of an entry in the routing control table at each node are performed simultaneously. The relay table management unit 44 of the accelerator 10 employing this method sequentially constructs a relay path based on instructions from the host 52 issuing the NVMe command. Assume that node 20-1 shown in FIG. 3 receives a request from the host 52 to establish a relay path from node 20-1 to node 20-2. The relay table management unit 44 creates an entry by setting the IP address and port number of the host 52 notified by the host 52 as the source address and source port number in the previous hop designation section, respectively, and setting the IP address and port number of node 20-2 notified by the host 52 as the destination address and destination port number in the next hop designation section, respectively. The destination port number in the previous hop designation section may be a port number previously determined between the node 20-1 and the host 52 or a port number notified to the host 52 in response to the establishment request. If there is a port number previously determined between the node 20-1 and the accelerator 10, this may be used as the source port number in the next hop designation section. If there is no previously determined port number, a port number determined by the node 20-1 itself (node ​​20-1) may be used as the source port number in the next hop designation section. Although it has been assumed that the host 52 instructs each node on the route to establish a relay route, this is not limiting. For example, the host 52 may send a relay route establishment instruction including information on all nodes on the route to a certain node 20, and the host 52 may forward the instruction to the next node 20 in a bucket brigade style, thereby establishing a route to the end point of the relay route. In this case, if the relay table management unit 44 of each node waits for a response from the node to which the message was sent before building an entry in the relay table 42, it is possible to avoid a situation in which an error occurs during the route establishment process and an indefinite entry remains.

[0121] In the third construction method, when the establishment request is forwarded to the end-point storage device 54 in a bucket brigade manner, the relay table management unit 44 may analyze and process information about the bucket brigade at the level of the application program being executed, or may utilize an improved version of an existing communication protocol. Examples of the latter include processing using the SRv6 Header defined in RFC8754, or processing using the Network Service Header defined in RFC8300 based on the concept of Service Function Chaining defined in RFC7665.

[0122] Next, the handling of the source port number in the relay table 42 will be explained in more detail.

[0123] In the explanations so far (particularly the first and second methods for constructing the relay table 42), the value of the source port number is determined when an entry in the relay table 42 is constructed. However, in actual TCP communication, the source port number can be dynamically determined when a TCP connection is established. In this case, the source port number used by the message source node for TCP communication is unknown until a request is received. The same is true for the source port number when communication is initiated from the node itself to the message destination node. If this behavior were permitted, an entry in the relay table 42 could not be constructed until communication begins. On the other hand, there is a demand to know in advance only the IP addresses of the message source node and the message destination node in order to prevent connection requests from unknown nodes. To meet this demand, an entry can be created in which the source port number in the previous hop designation section and the source port number in the next hop designation section are set to arbitrary values, and the source port number portion of the entry can be updated when communication begins. The start of communication refers to the time when a relay route is established. More precisely, there are two possibilities for the start of communication: when a TCP connection is established or when an NVMe / TCP connection is established, but either timing is acceptable as long as an appropriate response is made when an error occurs. Rather than updating the entry itself that allows any source port, a new entry that holds all the information derived from that entry may be added.

[0124] In the first and second construction methods for the relay table 42 described above, all nodes in the storage system 2 share a single relay table 42. On the other hand, in the third construction method, a different relay table 42 is constructed for each node depending on the adjacent nodes (message source node and message destination node) to which it is connected. As described above, in this embodiment, all nodes may share a single relay table 42, or each node may have a different relay table 42. In other words, in the first construction method, the host 52 may instruct each node to maintain a different relay table 42. In the second construction method, there is no problem even if the information collected by the relay table management unit 44 differs for each node. In this state, a process may be added in which each node exchanges the relay tables 42 that it has constructed, thereby constructing a single relay table 42 as a whole.

[0125] According to the first embodiment, each node 20 can perform calculation processing on data based on an I / O command sent from an upstream node 24, and transmit the processing results to the downstream node 26. This allows at least one node 20 to be connected between the host 52 and the storage device 54, and for each of the at least one node 20 to perform calculation processing on data, and for the calculation processing to be transmitted to the storage device 54 while being relayed between the nodes 20.

[0126] (Second embodiment) In the second embodiment, when multiple nodes 20 are connected between a host 52 and a storage device 54, the host 52 controls whether to send an I / O command to the storage device 54 via one of the multiple nodes 20 or to send the I / O command directly to the storage device 54. FIG. 10 is a diagram illustrating an example of the configuration of a host 52 according to the second embodiment. The host 52 is connected to at least one storage device 54 and at least one node 20. Although not shown in FIG. 10, the host 52 may be connected to the storage device 54 and the node 20 via a network. In the second embodiment, the node 20 has a computation function for host data, but the storage device 54 does not have a computation function. The host 52 executes an application program. The application program generates host data. The host 52 issues an I / O command for writing host data to the storage device 54. The I / O command is accompanied by host data. The host 52 may write the host data to the storage device 54 via the node 20. In this case, the host data is computed by the node 20, and the processing result is transmitted from the node 20 as host data. When the application program causes node 20 to perform computational processing on host data, it generates computation options that indicate the content of the processing and includes the computation options in the metadata accompanying the host data. When the application program does not cause node 20 to perform computational processing on host data, it does not generate computation options. Therefore, in this case, the computation options are not included in the metadata accompanying the host data.

[0127] The host 52 includes a computing middleware 62, an NVMe host control unit 64, and an NVMe transport I / F 66. The computing middleware 62 receives a host data write request or a host data read request from an application program.

[0128] The computation middleware 62 includes a computation option extraction unit 68 and a direct / indirect communication determination unit 70. The computation option extraction unit 68 extracts a computation option from a read request or a write request with a computation option passed from an application program. The direct / indirect communication determination unit 70 determines whether to transmit computation data to the storage device 54 (direct communication) or to transmit the computation data to the storage device 54 via a node (indirect communication) depending on the content of the computation option extracted by the computation option extraction unit 68. The direct / indirect communication determination unit 70 outputs transport control information including a determination result indicating either direct communication or indirect communication. An example of a computation option is secure computation.

[0129] The NVMe host controller 64 receives the computation data, the computation options, and the transport control information from the computation middleware 62 and generates an NVMe command including the computation data or the computation options and the transport control information. If the storage device 54 and the node 20 support the function of including the computation options in the metadata, the NVMe host controller 64 includes the computation options in the metadata and generates one NVMe command including the metadata. If the storage device 54 and the node 20 do not support the function of including the computation options in the metadata, the NVMe host controller 64 generates an NVMe command including the computation data and an NVMe command including the computation options. The processing contents of the NVMe host controller 64 may be configured to be executed by one or more processing circuits (processors). The processing by the processing circuit (processor) may be realized by a CPU (Central Processing Unit) executing firmware or may be realized by hardware. Furthermore, some of the processing by the processing circuit may be realized by a CPU executing firmware and other processing may be realized by hardware. The hardware is realized by at least one of a register, an adder, a multiplier, and other arithmetic units. Registers are implemented, for example, by logic circuits such as flip-flops, etc. Adders, multipliers, and other arithmetic units are implemented, for example, by logic circuits.

[0130] The NVMe transport I / F 66 receives an NVMe command and transport control information from the NVMe host control unit 64. If the determination result included in the transport control information indicates direct communication, the NVMe transport I / F 66 sends the NVMe command directly to the storage device 54 using transport protocol port number 4420 (NVM Express over Fabrics storage access) as the destination port number. 4420 is specified as the port number of the storage device in the NVMe transport protocol. If the determination result indicates indirect communication, the NVMe transport I / F 66 sends the NVMe command to one of one or more nodes using a fixed transport protocol port number other than 4420 that can be used in the NVMe transport protocol as the destination port number. The port number relay-port-num represents the port number of the node 20. The NVMe transport I / F 66 is connected to the storage device 54 and the node 20 via Ethernet or the like. The NVMe transport I / F 66 sends the NVMe command as a command capsule as shown in Figure 3.

[0131] The transport control information may further include a source port number. For example, the host 52 may have a first sending port for sending messages to the storage device 54 and a second sending port for sending messages to the node 20-1. The NVMe transport I / F 66 may select one of the one or more nodes using the source port number included in the transport control information. If the source port number indicates the first sending port, the NVMe transport I / F 66 may perform direct communication, and if the source port number indicates the second sending port, the NVMe transport I / F 66 may perform indirect communication. The computational middleware 62 and the NVMe host control unit 64 may be implemented as software by the CPU of the host 52. The NVMe transport I / F 66 includes hardware components such as a PCIe controller and an Ethernet controller. Components of the NVMe transport I / F 66 other than the hardware components may be implemented as software by the CPU of the host 52.

[0132] FIG. 11 is a diagram illustrating an example of indirect communication between the host 52 and the storage device 54 according to the second embodiment.

[0133] The path from the host 52 to the storage device 54 includes two types of paths: an indirect path from the host 52 to the node 20-1 to the node 20-2 to the storage device 54, and a direct path from the host 52 to the storage device 54.

[0134] The method for transferring TCP / IP messages on an indirect path is the same as the method for transferring messages using the relay table 42 in the network configuration shown in Fig. 3. A command capsule is sent from the host 52 to the storage device 54 via zero or more nodes 20, and in response, a response capsule is sent from the storage device 54 to the host 52 via zero or more relay nodes 20. Going via zero relay nodes means a direct path. The host data of the TCP / IP message is a capsule (command capsule or response capsule).

[0135] The TCP / IP message transfer method over the direct path is the same as when using TCP transport in regular NVMe-oF.

[0136] The direct / indirect communication determination unit 70 notifies the NVMe host control unit 64 whether to use a direct path or an indirect path based on the destination port number. If the destination port number is 4420, the NVMe host control unit 64 uses the direct path. If the destination port number is relay-port-num, the NVMe host control unit 64 uses the indirect path.

[0137] The direct / indirect communication determination unit 70 may notify the NVMe host control unit 64 to use a direct path if there is no calculation option, since calculation processing by the node is not required, and to use an indirect path if there is a calculation option.

[0138] According to the second embodiment, when at least one node 20 is located between the host 52 and the storage device 54, the host 52 controls whether to send a capsule indirectly to the storage device 54 via the at least one node 20 or to send a capsule directly to the storage device 54.

[0139] (Third embodiment) FIG. 12 is a diagram illustrating an example of an indirect communication network configuration between a host 52 and a storage device 54 according to the third embodiment.

[0140] Similar to the network configuration shown in Fig. 11, the network configuration shown in Fig. 12 also has two types of paths: indirect paths and direct paths. Unlike the second embodiment, the indirect paths from the host 52 to the storage device 54 include two types of indirect paths: a first indirect path of (host 52)-(node ​​20-1)-(node ​​20-2)-(storage device 54) and a second indirect path of (host 52)-(node ​​20-3)-(node ​​20-2)-(storage device 54).

[0141] A command capsule is sent from the host 52 to the storage device 54 via zero or more nodes 20, and in response, a response capsule is sent from the storage device 54 to the host 52 via zero or more nodes. Going via zero nodes means a direct path. The host data of the TCP / IP message is a capsule (command capsule or response capsule).

[0142] The method for transferring TCP / IP messages on an indirect path is the same as the method for transferring messages using the relay table 42 in the network configuration shown in Fig. 3. However, the relay table 42 is different from that shown in Fig. 4.

[0143] The TCP / IP message transfer method over the direct path is the same as when using TCP transport in regular NVMe-oF.

[0144] The direct / indirect communication determination unit 70 notifies the NVMe host control unit 64 whether to use a direct path or an indirect path based on the destination port number. If the destination port number is 4420, the NVMe host control unit 64 uses the direct path. If the destination port number is relay-port-num, the NVMe host control unit 64 uses the indirect path.

[0145] The direct / indirect communication determination unit 70 may select the direct path if the calculation option is empty, and may select the indirect path if the calculation option is present. The direct / indirect communication determination unit 70 may notify the NVMe host control unit 64 whether to use indirect path 1 or indirect path 2 based on the source port number.

[0146] FIG. 13 is a diagram illustrating an example of a relay table 42A according to the third embodiment.

[0147] The first six entries of relay table 42A are the same as all six entries of relay table 42. The header contents of a TCP / IP message using the first six entries of relay table 42A are the same as those shown in Fig. 3. The relay control algorithm using relay table 42A is the same as those shown in Figs. 5 and 6.

[0148] The header contents of the TCP / IP message sent from the host 52 to the node 20-1 are src_addr=A0, dst_addr=A1, src_port=P1, dst_port=nvmeof-relay-port. The nvmeof-relay-port is a common port number predefined in systems using NVMe-oF. The source port number from which the host 52 sends the message to the node 20-1 is P1.

[0149] The header contents of the TCP / IP message transmitted from the node 20-1 to the node 20-2 are src_addr=A1, dst_addr=A2, src_port=P2, and dst_port=nvmeof-relay-port.

[0150] When node 20-2 receives a TCP / IP message from node 20-1 whose TCP / IP header contents are src_addr=A1, dst_addr=A2, src_port=P2, dst_port=nvmeof-relay-port, the header contents of the TCP / IP message sent from node 20-2 to storage device 54 will be src_addr=A2, dst_addr=A3, src_port=P3, dst_port=nvmeof-relay-port.

[0151] When the storage device 54 receives a TCP / IP message from the node 20-2 whose TCP / IP header contents are src_addr=A2, dst_addr=A3, src_port=P3, dst_port=nvmeof-relay-port, the header contents of the TCP / IP message sent from the storage device 54 to the node 20-2 will be src_addr=A3, dst_addr=A2, src_port=nvmeof-relay-port, dst_port=P3.

[0152] When node 20-2 receives a TCP / IP message from storage device 54 with the TCP / IP header contents src_addr=A3, dst_addr=A2, src_port=nvmeof-relay-port, dst_port=P3, the header contents of the TCP / IP message sent from node 20-2 to node 20-1 will be src_addr=A2, dst_addr=A1, src_port=nvmeof-relay-port, dst_port=P2.

[0153] The header contents of the TCP / IP message sent from the node 20-1 to the host 52 are src_addr=A1, dst_addr=A0, src_port=nvmeof-relay-port, dst_port=P1.

[0154] The header contents of the TCP / IP message sent from the host 52 to the node 20-3 are src_addr=A0, dst_addr=A4, src_port=P4, dst_port=nvmeof-relay-port. The source port number used by the host 52 to send the message to the node 20-3 is P4.

[0155] The header contents of the TCP / IP message transmitted from the node 20-3 to the node 20-2 are src_addr=A4, dst_addr=A2, src_port=P5, dst_port=nvmeof-relay-port.

[0156] When node 20-2 receives a TCP / IP message from node 20-3 with the TCP / IP header contents src_addr=A4, dst_addr=A2, src_port=P5, dst_port=nvmeof-relay-port, the header contents of the TCP / IP message sent from node 20-2 to storage device 54 will be src_addr=A2, dst_addr=A3, src_port=P6, dst_port=nvmeof-relay-port.

[0157] When the storage device 54 receives a TCP / IP message from the node 20-2 with the TCP / IP header contents src_addr=A2, dst_addr=A3, src_port=P6, dst_port=nvmeof-relay-port, the header contents of the TCP / IP message sent from the storage device 54 to the node 20-2 will be src_addr=A3, dst_addr=A2, src_port=nvmeof-relay-port, dst_port=P6.

[0158] When a TCP / IP message with the TCP / IP header contents of src_addr=A3, dst_addr=A2, src_port=nvmeof-relay-port, dst_port=P6 is received from storage device 54, the header contents of the TCP / IP message sent from node 20-2 to node 20-3 will be src_addr=A2, dst_addr=A4, src_port=nvmeof-relay-port, dst_port=P5.

[0159] When node 20-3 receives a TCP / IP message from node 20-2 with the TCP / IP header contents src_addr=A2, dst_addr=A4, src_port=nvmeof-relay-port, dst_port=P5, the header contents of the TCP / IP message sent from node 20-3 to host 52 will be src_addr=A4, dst_addr=A0, src_port=nvmeof-relay-port, dst_port=P4.

[0160] According to the third embodiment, when at least one node 20 is disposed between the host 52 and the storage device 54, the host 52 controls whether to send the capsule indirectly to the storage device 54 via the at least one node 20 or to send the capsule directly to the storage device 54.

[0161] (Fourth embodiment) The above embodiments relate to examples in which the node 20 exists as a single node independent of the host 52 and the storage device 54. Below, an example in which the node 20 is built into the host 52 and the storage device 54 will be described.

[0162] FIG. 14 is a diagram illustrating an example of the configuration of a host 52A according to the fourth embodiment.

[0163] The host 52A is the host 52 shown in FIG. 10 integrated with a node. The node inside the host 52A is called the node 20A. The NVMe transport I / F 66 is connected to the storage device 54 and the node 20A. The NVMe transport I / F 66 is physically connected to the node 20A via an internal bus (for example, a PCIe bus) of the host 52. The NVMe transport I / F 66 is connected to the storage device 54 via Ethernet or the like.

[0164] Although one NVMe transport I / F 66 is illustrated, multiple physical interfaces may be provided for each physical medium. Although only one node 20A is illustrated in the host 52A, multiple nodes 20A may be integrated into the host 52A. The node 20A may be directly connected to the storage device 54, or may be indirectly connected to the storage device 54 via at least one node 20 outside the host 52A.

[0165] The relay table of the accelerator 10 included in the NVMe host control unit 64 is similar to the relay table 42 shown in FIG. 4. However, in the fourth embodiment, an IP address is also assigned to the node 20A connected to the internal bus, and TCP / IP communication is performed within the host 52. For example, TCP / IP communication is realized by communicating in a form such as TCP / IP over PCIe. In this case, the assigned IP address may be any address as long as it can ensure reachability and does not interfere with other communications.

[0166] On the other hand, the relay table 42 may be modified to distinguish and control the connection to the NVMe transport I / F 66 as an internal bus. Specifically, by changing the combination of IP address and port number to an ID (an identifier such as domain:bus:device.function) used to identify a PCIe device, the connected device can be uniquely identified. However, if node 20A is treated as a PCIe device, the transfer of host data corresponding to an NVMe command must be performed by DMA. Therefore, if the previous hop node or next hop node in the relay table 42 is a PCIe device, the transmission and reception of NVMe commands and the DMA transfer of data are performed over the PCIe bus rather than NVMe / TCP. This function can be realized by extending the memory management unit 38 so that transmission and reception are performed over PCIe rather than TCP / IP and so that the transmission and reception are performed over an interface connected to a local bus rather than a network I / F.

[0167] FIG. 15 is a diagram illustrating an example of the node 20A according to the fourth embodiment.

[0168] Node 20A includes a local I / F 74 in addition to the configuration of node 20 shown in FIG. 1. The local I / F 74 is connected to the NVMe transport I / F 66 via an internal bus such as PCIe. The memory management unit 38 has a function of selecting a command transmission / reception method and an interface (network I / F 30 and local I / F 74) to be used for transmission / reception according to information in the previous hop designation field and next hop designation field of the relay table 42.

[0169] When notifying a node 20 or storage device 54 that exists outside the host 52A of information about the built-in node 20A, the IP address and port number used by the host 52A to communicate with the outside may be added to the message in addition to the identifier of the device inside the host 52A. When multiple nodes 20A are built into the host 52A, different IP addresses and port numbers may be used for notification for each node.

[0170] As described in the third embodiment, when a branch point exists on the path and the node of the PCIe device is the branch point, the PCIe device (e.g., host 52A) that transmits the NVMe command to the node needs to transmit the NVMe command so that the path beyond the branch point can be uniquely identified. As an identifier for identifying the path, any identifier on the PCIe bus of the transmission PCIe device, any identifier on the PCIe bus of the relay, or any identifier on the NVMe bus of the relay (e.g., a namespace identifier) ​​can be used. Any identifier can be used as long as it is agreed upon between the transmitting PCIe device and the relay when constructing the relay table 42.

[0171] FIG. 16 is a diagram illustrating an example of a network in the case where a Namespace identifier is used as an identifier for specifying a route in the fourth embodiment.

[0172] The host 52A includes a node 20A as a PCIe device. The node 20A is connected to the storage device 54, node 201, and node 203 using NVMe / TCP. The symbols Px, Py, and Pz attached to the edges of each node are the source port numbers used when the node 20A sends an NVMe / TCP packet, the symbol Pr is the receiving port number used by the node 20 or the storage device 54, and the number 4420 is the receiving port number used when the storage device 54 receives an NVMe command without going through the node 20. The node 20A includes three Namespaces (identifiers are NS0, NS1, and NS2, respectively). Although the node 20A is illustrated as being divided into three, it is not actually divided. The sending port number of the Namespace with the identifier NS0 is Px, the sending port number of the Namespace with the identifier NS1 is Py, and the sending port number of the Namespace with the identifier NS2 is Pz.

[0173] Figure 16 also shows a portion of the relay table 42 used by node 20A when relaying an NVMe command. The PID (Host) in each row represents the ID on the PCIe device of the host 52 that initially sends the NVMe command. Since there is no information corresponding to the source port on the PCIe device, the src_port "-" indicates that it is unused. The destination port holds one of the Namespace identifiers NS0 to NS2 that receives the NVMe command. For example, the entry in the first row of the relay table 42 indicates that when an NVMe command is received in the Namespace with the identifier NS0 and the corresponding data is processed, a message is sent from port Px to port 4420 of the device with IP address = A3 using NVMe / TCP.

[0174] In this way, when the node 20A is integrated into the host 52A as a PCIe device, it is possible to express the branching of the route using some kind of identifier. Note that although the case where the relay of the PCIe device is the branching point has been described, the same method can be applied to cases where the branching point is located after the relay of the PCIe device.

[0175] According to the fourth embodiment, a host 52A is provided that integrates the node 20A.

[0176] (Fifth embodiment) FIG. 17 is a diagram illustrating an example of a network according to the fifth embodiment.

[0177] The storage device 54A includes a storage 84, a storage controller 82, and a node 20B. The storage is a storage medium. An example of the storage is a NAND flash memory. The host 52 is connected to the node 20 and the node 20B inside the storage device 54A via Ethernet or the like. The node 20B inside the storage device 54A is connected to the node 20. The connection method between the node 20B and the node 20 may be the same as the connection method (Ethernet or the like) between the host 52 and the storage device 54A, or may be a different method. The node 20B inside the storage device 54A may be connected to a node of another storage device. The connection method between the storage devices may be the same as the connection method (Ethernet or the like) between the host 52 and the storage device 54A, or may be a different method. Note that the storage controller 82 and the storage 84 may be omitted if a local storage device of the node 20B is used as the storage medium and storage controller of the storage device 54A.

[0178] The changes to the relay table 42 according to the fifth embodiment are the same as those according to the fourth embodiment. However, since the part of the storage device 54A that is connected to the outside is the node 20B, if an IP address and a port number are added when notifying the outside of the node 20B, an IP address and a port number that can be used by the node 20B may be added. The combination of the IP address and the port number used for the calculation process of the node 20B may be different from the combination of the IP address and the port number that is assigned to the node 20B for communicating with the outside.

[0179] According to the fifth embodiment, a storage device 54A is provided that integrates the node 20B.

[0180] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0181] 10...accelerator, 20...node, 22...network, 30...network I / F, 32...main memory, 34...virtual register table, 36...page table, 38...memory management unit, 40...calculation processing unit, 42...relay table, 44...relay table management unit, 46...storage I / F

Claims

1. A connection unit that can connect to a first node and a second node using an NVMe transport protocol, receives first data and an I / O command from the first node, and transmits second data, which is a result of a computation process on the first data, and the I / O command to the second node; a memory for storing the first data; a virtual register table that stores virtual register numbers that are associated with the first data and that are specified based on a calculation option that represents the calculation process, in association with a virtual address of third data that is used to process a calculation instruction according to the calculation option and a data size of the third data; a memory management unit that writes the first data to the memory and updates the virtual register table; a calculation processing unit that refers to the virtual register table and performs calculation processing on the first data; a relay table that stores a plurality of pairs of source information and destination information;

2. the first node is a host that transmits the I / O command and the first data accompanied by the computing option; The controller according to claim 1 , wherein the second node is a storage device that receives the I / O command and the second data accompanied by the computing option and stores the second data.

3. the first node is connectable to a host that transmits the I / O command and the first data accompanied by the calculation option, and includes a first controller that executes the same processing content as the controller; 2. The controller of claim 1, wherein the second node is connectable to a storage device that receives the I / O command and the second data accompanied by the calculation option and stores the second data, and includes a second controller that performs the same processing as the controller.

4. The controller according to claim 1 , wherein the connection unit determines a destination of the second data by referring to the relay table.

5. the source information includes a source address, a source port number, and a destination port number; the destination information includes a destination address, a source port number, and a destination port number; The controller according to claim 4 , wherein the connection unit determines the destination of the second data based on destination information in a record including the source information including a source port number and a destination port number included in the I / O command.

6. The controller according to claim 1 , wherein the virtual address includes a page number and a page offset assigned to a page in which the second data is stored.

7. A host capable of connecting to a storage device or node using an NVMe transport protocol, The host executes an application program, generates data, and, when causing the node to perform a calculation process on the data, generates a calculation option that represents the calculation process, and transmits a write command, the data, and the calculation option to the node. When the application program is executed, if the node does not perform a calculation process on the data, the write command and the data are sent to the storage device; The storage device receives the write command and the data and stores the data; the node receives the write command, the data, and the calculation option, performs a calculation process on the data based on the calculation option, sends the write command to the storage device, and sends a result of the calculation process as the data to the storage device; host.

8. the node is one node among a plurality of nodes, The host If the calculation option is not generated, the destination port number of the data is set to a first port number representing the storage device; 8. The host according to claim 7, wherein, when the calculation option is generated, the destination port number of the data is set to a second port number common to the plurality of nodes.

Images