Packet processing device, packet processing method, and program

Abstract

This packet processing device comprises: an information acquisition unit that acquires, from a database, a pseudo virtual address shared with another packet processing device; a network adapter that receives a packet including the pseudo virtual address; and an information generation unit that secures, in a memory, a memory area in which data in the packet is written. The information generation unit generates a table in which a physical address of the memory area is associated with the pseudo virtual address. The network adapter refers to the pseudo virtual address in the packet and the table and writes the data in the memory area.

Description

Packet processing device, packet processing method, and program

[0001] The present disclosure relates to a packet processing device, a packet processing method, and a program.

[0002] InfiniBand is known as an I / O architecture that achieves high-speed, low-latency communication between terminals (Non-Patent Documents 1 and 2). InfiniBand supports a unique communication method called Remote Direct Memory Access (RDMA). In RDMA, the Host Channel Adapter (HCA) hardware of the terminal executes protocol processing when sending and receiving packets, bypassing the copying of packet data in kernel space in memory and transferring packet data directly between the HCA and user space in memory. This enables high-speed, low-latency packet communication.

[0003] RDMA is classified into two types: one-sided RDMA and two-sided RDMA. In the former, RDMA Read / Write operations are performed, while in the latter, send / recv operations are performed. In one-sided RDMA, the CPU of the responder terminal, i.e., the sending terminal where data is read from memory in RDMA Read or the receiving terminal where data is written to memory in RDMA Write, is not involved in the communication process at all. Therefore, one-sided RDMA can achieve higher performance than two-sided RDMA (Non-Patent Document 3).

[0004] “InfiniBand Architecture Specification Volume 1”, 2020, InfiniBand Trade Association. “InfiniBand Network Architecture”, MindShare, Inc. C. Mitchell, et al., “Using One-Sided RDMA Reads to Build a Fast, CPU-Efficient Key-Value Store”, 2013, USENIX Annual Technical Conference. RDMA Multicast”, ANCS '21, December 13-16, 2021, Layfette, IN, USAWenxue Li, et al., “Gleam: An RDMA-accelerated Multicast Protocol for Datacenter Networks”. “InfiniBand Architecture Specification Volume 1, ANNEX A17: RoCEv2 (IP ROUTABLE RoCE)”, 2020, InfiniBand Trade Association.

[0005] One-sided RDMA packets contain values ​​unique to each terminal, such as the virtual address (VA) that points to the beginning of the memory region (MR) on the receiving terminal. This value cannot be made the same across multiple terminals. This is because memory usage differs from terminal to terminal, and the VA of the unused memory region that can be secured at the start of communication will inevitably differ from terminal to terminal. For this reason, one-sided RDMA packets cannot be multicast, and are intended to be used for one-to-one communication. Currently, InfiniBand supports multicast communication only in unreliable two-sided RDMA, and not in one-sided RDMA.

[0006] Therefore, research is being conducted to realize multicast in one-sided RDMA (Non-Patent Documents 4 and 5). Non-Patent Documents 4 and 5 consider a method of rewriting terminal-specific data in a packet in a leaf switch or router. Non-Patent Document 4 also mentions a method of adding hardware (specifically, a CPU core, an Application Specific Integrated Circuit (ASIC), or a Field Programmable Gate Array (FPGA)) before a Network Interface Card (NIC) and rewriting terminal-specific data in a packet in this hardware.

[0007] However, when rewriting terminal-specific data in packets in a leaf switch or router on a network, there is a problem that general-purpose switches cannot be used. Furthermore, when using an optical switch as a switch, it is difficult to execute complex processes such as rewriting data in packets. Furthermore, when rewriting data in packets by adding a CPU core, as in Non-Patent Document 4, the overhead of the rewriting process may affect communication quality (e.g., delay and throughput).

[0008] The present disclosure has been made in light of such circumstances, and aims to realize multicasting without rewriting data in packets on a network.

[0009] One aspect of the present disclosure is a packet processing device comprising: an information acquisition unit that acquires a pseudo-virtual address common to other packet processing devices from a database; a network adapter that receives a packet including the pseudo-virtual address; and an information generation unit that reserves a memory area in memory into which data in the packet is written, wherein the information generation unit generates a table in which the pseudo-virtual address is associated with a physical address of the memory area, and the network adapter refers to the pseudo-virtual address in the packet and the table and writes the data to the memory area.

[0010] One aspect of the present disclosure is a packet processing method performed by a packet processing device, which includes obtaining a pseudo-virtual address common to other packet processing devices from a database, receiving a packet including the pseudo-virtual address, reserving a memory area in memory into which data in the packet is written, generating a table in which the pseudo-virtual address is associated with a physical address of the memory area, referencing the pseudo-virtual address in the packet and the table, and writing the data to the memory area.

[0011] One aspect of the present disclosure is a program that causes a computer to function as the packet processing device.

[0012] According to the present disclosure, multicasting can be achieved without rewriting data in packets on a network.

[0013] FIG. 1 is a diagram illustrating an example of a communication system according to an embodiment. FIG. 2 is a diagram illustrating an example of a message and an opcode. FIG. 3 is a functional block diagram of a transmitting terminal and a receiving terminal. FIG. 4 is a diagram illustrating an example of a Global Routing Header (GRH) of a packet. FIG. 5 is a diagram illustrating an example of a Base Transport Header (BTH) of a packet. FIG. 6 is a diagram illustrating an example of distribution information stored in a database. FIG. 7 is a diagram illustrating an example of an RDMA Extended Transport Header (RETH) of a packet. FIG. 8 is a sequence diagram illustrating an example of a schematic operation of a communication system. FIG. 9 is a flowchart illustrating an example of a schematic operation of a transmitting terminal. FIG. 10 is a flowchart illustrating an example of a schematic operation of a receiving terminal. FIG. 11 is a diagram illustrating an example of intra-host processing of pre-reception processing at a receiving terminal. FIG. 12 is a flowchart illustrating an example of pre-reception processing at a receiving terminal. FIG. 13 is a diagram illustrating an example of message reception processing at a receiving terminal. FIG. 14 is a diagram illustrating an example of message reception processing (first half) at a receiving terminal. FIG. 15 is a flowchart illustrating an example of message reception processing (first half) at a receiving terminal. FIG. 16 is a flowchart illustrating an example of message reception processing (second half) at a receiving terminal. Fig. 17 is a diagram showing an example of conversion from a packet to a transaction layer packet (TLP). Fig. 18 is a diagram showing an example of TLP destination address control. Fig. 19 is a flowchart showing an example of RDMA Write First packet processing. Fig. 20 is a flowchart showing details of RDMA Write First packet processing. Fig. 21 is a flowchart showing an example of RDMA Write Middle packet processing. Fig. 22 is a flowchart showing details of RDMA Write Middle packet processing. Fig. 23 is a flowchart showing an example of RDMA Write Last packet processing. Fig. 24 is a flowchart showing details of RDMA Write Last packet processing. Fig. 25 is a flowchart showing an example of RDMA Write Last with Immediate Data packet processing.Fig. 26 is a flowchart showing details of RDMA Write Last with Immediate Data packet processing. Fig. 27 is a flowchart showing an example of RDMA Write-Only packet processing. Fig. 28 is a flowchart showing details of RDMA Write-Only packet processing. Fig. 29 is a flowchart showing an example of RDMA Write-Only with Immediate Data packet processing. Fig. 30 is a flowchart showing details of RDMA Write-Only with Immediate Data packet processing. Fig. 31 is a flowchart showing an example of reception termination processing by a receiving terminal. Fig. 32 is a diagram showing an example of the hardware configuration of a packet processing device.

[0014] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0015] (1) Configuration of the Communication System The configuration of the communication system of this embodiment will be described with reference to FIG. 1. In the communication system of this embodiment, a transmitting terminal 10 and multiple receiving terminals 20 (packet processing devices) are connected via a network N. A database DB is also connected to the network N. The database DB of this embodiment is equipped with an access authentication function that authenticates access from the transmitting terminal 10 and the receiving terminal 20, but the authentication format is not important. The transmitting terminal 10 transmits packets to the receiving terminal 20 via the network N. The receiving terminal 20 receives packets from the transmitting terminal 10 via the network N. The transmitting terminal 10 and the receiving terminal 20 also acquire distribution information from the database DB via the network N. If the database DB authenticates the access, the transmitting terminal 10 and the receiving terminal 20 can acquire the distribution information.

[0016] For ease of explanation, the names transmitting terminal 10 and receiving terminal 20 are used, but the transmitting terminal 10 can also receive packets from the receiving terminal 20, and the receiving terminal 20 can also transmit packets to the transmitting terminal 10.

[0017] The communication system of this embodiment uses InfiniBand, which supports RDMA, as a communication protocol. However, this embodiment assumes that some modifications are made to the InfiniBand protocol specifications and the network interface operation specifications, as will be described later.

[0018] The sending terminal 10 and the receiving terminal 20 are computers equipped with a memory controller (MC) 300, a memory 400, a central processing unit (CPU) 500, etc. (See Figure 3). Network N is an electrical network consisting of switches and routers capable of forwarding IP multicast packets. Alternatively, network N is a mixed network consisting of an electrical network consisting of the switches and routers installed at bases near the sending terminal 10 and the receiving terminal 20, and an optical network connecting these bases via optical transmission. The switches, routers, and optical transmission devices that make up network N are general-purpose devices (commercial devices), but are not limited to this. Note that packets sent and received via network N are assumed to be InfiniBand packets encapsulated in User Datagram Protocol (UDP) / Internet Protocol (IP) using the RoCEv2 protocol (Non-Patent Document 6). Multicast packet duplication and routing are assumed to be performed at this IP layer.

[0019] With reference to FIG. 2 , a description will be given of messages transmitted and received using one-sided RDMA and opcodes assigned to packets constituting the messages. In this embodiment, the format and type of opcodes of packets transmitted and received between the transmitting terminal 10 and the receiving terminal 20 are not particularly limited, but those used in existing InfiniBand are used, but are not limited to this. Messages in this embodiment are written from the memory 400 of the transmitting terminal 10 to the memory 400 of the receiving terminal 20 via the network adapter 100 provided in the transmitting terminal 10 and the network adapter 100 provided in the receiving terminal 20. Furthermore, since this embodiment targets one-to-multiple data transmission using RDMA, RDMA Write, which uses opcodes from existing InfiniBand, is used, but is not limited to this. In RDMA Write, the transmitting terminal 10 uses a gather buffer list to sequentially transmit data to be transmitted stored in discontinuous physical areas in the memory 400 to the network N. The receiving terminal 20 secures a continuous physical area (Memory Region, MR) in the memory 400, and writes data received from the network N to the MR without using the CPU 500. These operations are also the same as those of existing InfiniBand network adapters.

[0020] In this embodiment, data is transmitted and received in message units, similar to the existing one-sided RDMA of InfiniBand. Messages can be composed of multiple packets or a single packet. As shown in FIG. 2, in message A composed of multiple packets, the first packet is assigned an opcode of RDMA Write First, the middle packets are assigned an opcode of RDMA Write Middle, and the last packet is assigned an opcode of RDMA Write Last. In message C composed of a single packet, this packet is assigned an opcode of RDMA Write-Only. When explicitly notifying the end of a message, the last packet of multiple packets may be assigned an opcode of RDMA Write Last with Immediate as shown in message B, or a single packet may be assigned an opcode of RDMA Write-Only with Immediate as shown in message D. These opcodes are also reused from existing InfiniBand.

[0021] Here, a virtual address (VA) pointing to the beginning of a memory area (MR) secured in memory 400 of the receiving terminal 20 and a remote key (R_Key) for authenticating access rights to the MR from the transmitting terminal 10 are stored in the first packet constituting the message transmitted from the transmitting terminal 10 (only the VA is shown in the figure). Specifically, the VA and R_Key are stored in the RDMA Extended Transport Header (RETH) of the first packet.

[0022] The VA and R_Key are values ​​that are unique to each receiving terminal 20 and cannot be the same across multiple receiving terminals 20. This is because the usage status of the memory 400 differs for each receiving terminal 20, and therefore the VA of the unused memory area (MR) that can be secured at the start of communication also differs for each receiving terminal 20. Furthermore, the value of the R_Key is also determined according to the secured MR, and therefore differs for each receiving terminal 20. For this reason, in this embodiment, the VA and R_Key portions of the RETH store pseudo VA and pseudo R_Key values ​​rather than values ​​that are unique to each receiving terminal 20 (see FIG. 7).

[0023] The configurations of the transmitting terminal 10 and the receiving terminal 20 of this embodiment will be described with reference to Figure 3. The transmitting terminal 10 and the receiving terminal 20 have the same basic configuration, but the information storage unit 141 and the information generation unit 411, which will be described later, are used only in the receiving terminal 20. In other words, the transmitting terminal 10 does not need to include the information storage unit 141 and the information generation unit 411. Therefore, although the configuration of the receiving terminal 20 will be mainly described below, the description of the transmitting terminal 10 may also be read as an explanation of the transmitting terminal 10.

[0024] The receiving terminal 20 includes a network adapter 100, a root complex 200, an MC 300, a memory 400, and a CPU 500. The network adapter 100 and the root complex 200 are connected by a Peripheral Component Interconnect Express (PCIe) bus. The memory controller 300 and the memory 400 are connected by a memory bus.

[0025] The network adapter 100 is hardware for connecting the receiving terminal 20 to the network N. The network adapter 100 is, for example, a Host Channel Adapter (HCA). The HCA is a Network Interface Card (NIC) that executes the above-mentioned RoCEv2 and InfiniBand protocol processing. The network adapter 100 includes a port 110, a multicast group table 120, an SQ logic 130, and an RQ logic 140.

[0026] The port 110 includes a first input buffer 111. Packets transmitted from the transmitting terminal 10 and arriving from the network N are input to the first input buffer 111. As shown in Fig. 4, the Global Routing Header (GRH) of an InfiniBand packet includes a Global Identifier (GID) (Destination GID (DGID) in the figure) that identifies the multicast group.

[0027] The multicast group table 120 is a table that associates a GID that identifies a multicast group with a Queue Pair (QP) list that is a member of the multicast group. A QP is a pair of a Send Queue (SQ) and a Receive Queue (RQ), and is controlled by the SQ logic 130 and the RQ logic 140. That is, the multicast group table 120 identifies the member QP to which the packet is to be forwarded and the RQ logic 140 corresponding to that RQ from the GID of the packet input to the first input buffer 111. Furthermore, entries may be added to or deleted from the multicast group table 120 by the information generator 411. The contents of the entries and their addition / deletion will be described later in the description of the information generator 411.

[0028] The SQ logic 130 and the RQ logic 140 are logics that perform packet transmission and reception processing. The SQ logic 130 is control logic for transmitting packets to the transmitting terminal 10. The RQ logic 140 is control logic for receiving packets from the transmitting terminal 10. However, since packets are not transmitted from the receiving terminal 20 to the transmitting terminal 10 in the multicast communication of this embodiment, the RQ logic 140 will be described in more detail below.

[0029] The RQ logic 140 includes an information holding unit 141, a second input buffer 142, and an output buffer 143. The RQ logic 140 also includes a hardware register (not shown). The RQ logic 140 converts a packet input to the second input buffer 142 into a PCIe memory write request transaction layer packet (memory write request TLP, hereinafter simply referred to as TLP). Furthermore, the RQ logic 140 transfers the TLP to the root complex 200 via the output buffer 143.

[0030] The information holding unit 141 acquires and holds a pseudo VA, a pseudo R_Key, a message length, and a pseudo page table, which will be described later, from the information generating unit 411 .

[0031] Packets input to the first input buffer 111 are transferred to the second input buffer 142. The second input buffer 142 is provided to absorb the difference in speed between this transfer speed and the processing speed at which the RQ logic 140 converts this packet into a TLP. The packets to be transferred are determined based on the multicast group table 120, as described above.

[0032] The RQ logic 140 converts the packet input to the second input buffer 142 into a PCIe memory write request TLP based on the value of the opcode (OpCode in FIG. 5) included in the Base Transport Header (BTH). In the case of multicast, the DestQP of the BTH shown in FIG. 5 includes a fixed value (0xFFFFFF). In addition, in this embodiment, the Packet Sequence Number (PSN) of the BTH is not used. In other words, this embodiment only targets the Unreliable transport type in InfiniBand.

[0033] The converted TLP is input to the output buffer 143 and then transferred to the root complex 200 (described later) via the PCIe bus. The output buffer 143 is provided to absorb the speed difference between the input speed of the TLP to itself and the output speed of the TLP to the PCIe bus.

[0034] The payload (data) of the TLP transferred to the root complex 200 is written to the memory area (MR) secured in the user space 420 of the memory 400 via the MC 300. Here, the RQ logic 140 refers to the pseudo VA and pseudo page table stored in the information storage unit 141, and can grasp the first PA of the MR into which the payload of the packet is written, and can set the destination of the TLP accordingly.

[0035] In addition, the RQ logic 140 confirms that the pseudo R_Key contained in the RDMA Extended Transport Header (RETH) of the packet shown in Fig. 7 is the same as the pseudo R_Key that is stored in the information storage unit 141, and then starts the process of converting the packet received from the network N into TLP. That is, the RQ logic 140 starts the process of converting into TLP only when the authentication by the pseudo R_Key is successful, and if the authentication is unsuccessful, discards the packet that arrives at the second input buffer.

[0036] The root complex 200 is the component located at the highest position in the I / O hierarchy of the PCIe bus. The network adapter 100, memory 400, and CPU 500 are connected via this root complex 200 (and MC 300). The root complex 200 includes a third input buffer 210. A TLP is input to the third input buffer 210 from the RQ logic 140. The TLP is transferred via the MC 300 to an MR reserved in a user space 420 of the memory 400. At this time, the TLP is transferred to the user space 420 by direct memory access (DMA) without the intervention of the CPU 500.

[0037] The memory 400 may be called a main memory or a host memory, and is, for example, a dynamic random access memory (DRAM). The memory 400 includes a kernel space 410 that is under the control of the OS, and a user space 420 that is used by user application programs. The OS kernel or the device driver of the network adapter 100 runs in the kernel space 410. The kernel space 410 includes an information generation unit 411 as software. The user space 420 also includes an information acquisition unit 421 as software.

[0038] The information acquisition unit 421 acquires distribution information from the database DB via the network N and the network adapter 100. The distribution information is stored in the database DB. As shown in FIG. 6 , the distribution information is a table that stores a content ID for identifying the content, an IP multicast group address, a pseudo VA used in place of a VA, a pseudo R_Key used in place of an R_Key, and a message length when the content is distributed. Note that the content is data to be distributed by multicast, such as a movie or live video, but is not limited to video data. In this embodiment, the content is transmitted from the transmitting terminal 10 to the receiving terminal 20 as one or more messages as described above.

[0039] In this embodiment, a packet transmitted from the transmitting terminal 10 includes a pseudo VA and pseudo R_Key acquired by the information acquisition unit 421 instead of a VA and R_Key. Specifically, as shown in Fig. 7, the pseudo VA and pseudo R_Key are included in the RDMA Extended Transport Header (RETH) (Extended Transport Header in the figure) of the packet. The VA is a virtual address of a memory region (MR) in the user space 420 to which the packet is written. The R_Key is authentication data that indicates that the transmitting terminal 10 has access rights to the MR.

[0040] The information generation unit 411 secures an MR in the user space 420, corresponding to the message length acquired by the information acquisition unit 421 from the database DB as distribution information. All messages, except for the last message that stores the last remaining data of the content, have the same message length, and the MR is reused each time reception of a message is completed. When the MR is secured, the VA (hereinafter also referred to as VA_orig to distinguish it from the pseudo VA) and R_Key (hereinafter also referred to as R_Key_orig to distinguish it from the pseudo R_Key) of the MR are determined. The information generation unit 411 acquires the physical address (PA) of the MR corresponding to the VA_orig via the OS. Furthermore, the information generation unit 411 acquires the pseudo VA from the information acquisition unit 421 and generates a pseudo page table that associates the pseudo VA with the PA. The information generation unit 411 also acquires the pseudo R_Key from the information acquisition unit 421. In this embodiment, the R_Key_orig is not used. Instead of using R_Key_orig, authentication is required for access from the transmitting terminal 10 to the database DB that stores the pseudo R_Key. This allows the receiving terminal 20 to confirm that the transmitting terminal 10 that sent the packet containing the pseudo R_Key has the right to access the MR of the receiving terminal 20.

[0041] The information generation unit 411 writes the pseudo VA, pseudo R_Key, message length, and pseudo page table to the information storage unit 141. Furthermore, the information generation unit 411 may add or delete entries to the multicast group table 120. The addition or deletion of an entry may be the addition or deletion of a GID corresponding to the content, or may be the addition or deletion of a QPN belonging to this GID (see FIG. 12 ). When the software of the network adapter 100 joins a multicast group to receive packets, the information generation unit 411 adds its own QPN, which is the number of its own QP (own QP) secured at the start of communication, to the GID entry. Furthermore, if the GID corresponding to the multicast group to be joined does not exist in the multicast group table 120, the information generation unit 411 adds an entry for that GID and further adds its own QPN as a member. Note that the operation of this software may be adapted from the operation of an existing InfiniBand HCA.

[0042] As described above, the receiving terminal 20 (packet processing device) of this embodiment includes an information acquisition unit 421 that acquires a pseudo virtual address (pseudo VA) that is common to other receiving terminals 20 (packet processing devices) from the database DB, a network adapter 100 that receives packets that include the pseudo virtual address, and an information generation unit 411 that reserves a memory area (MR) into which data in the packet is written in memory 400. The information generation unit 411 generates a table (pseudo page table) in which the pseudo virtual address is associated with the physical address (PA) of the memory area, and the network adapter 100 references the pseudo virtual address in the packet and the table to write data to the memory area.

[0043] Furthermore, in the receiving terminal 20 of this embodiment, the network adapter 100 writes data to the memory area without going through the kernel space 410 of the memory 400. Furthermore, when the database DB authenticates the access from the information acquisition unit 421, the information acquisition unit 421 acquires from the database DB a pseudo remote key (pseudo R_Key) that is common to other receiving terminals 20 (packet processing devices). Furthermore, the network adapter 100 writes data to the memory area only when authentication using the pseudo remote key is successful.

[0044] Furthermore, in the receiving terminal 20 of this embodiment, the network adapter 100 includes an information storage unit 141 in which pseudo virtual addresses and tables are written, and an RQ logic 140 that converts packets into transaction layer packets. The RQ logic 140 references the table in the information storage unit 141 and writes the data in the transaction layer packet into a memory area. To explain the data flow in more detail, the RQ logic 140 outputs the data in the transaction layer packet to be transferred to the memory area to an output buffer 143. The data output to the output buffer 143 is written to the user space 420 of the memory 400 via the PCIe bus, the root complex 200, and the memory controller 300. In this way, the RQ logic 140 sends the transaction layer packet onto the PCIe bus that connects the network adapter 100 and the root complex 200 connected to the memory 400. The information acquisition unit 421 is included in the user space 420 of the memory 400 , and the information generation unit 411 is included in the kernel space 410 of the memory 400 .

[0045] (2) General Operation of the Communication System Before describing the general operation of the communication system of this embodiment, the premise of this embodiment will be described. First, a connection is not established between the transmitting terminal 10 and the receiving terminal 20, and Unreliable Datagram (UD) type is used as the transport type. However, current InfiniBand does not support UD type one-sided RDMA. Therefore, this embodiment uses a protocol and network adapter based on the current InfiniBand protocol and a network adapter, and modifies the RQ logic 140 corresponding to the UD type QP to process one-sided RDMA packets without discarding them as error packets.

[0046] Since this embodiment uses UD-type communication, the existing RDMA Write packet format is utilized, but the transmitting terminal 10 does not set a value in the PSN field of the BTH header, and the receiving terminal 20 also ignores the PSN field. Also, since a connection is not established between the transmitting terminal 10 and the receiving terminal 20, information regarding the message length for each message cannot be exchanged. For this reason, the receiving terminal 20, assuming that the message length of all messages is the same, secures an MR of a size corresponding to that message length before starting communication, and repeatedly uses the secured MR until reception of all messages is complete.

[0047] The operation of the communication system of this embodiment will be described below. As shown in Figures 8 and 9, the transmitting terminal 10 first obtains distribution information from the database DB. That is, the transmitting terminal 10 obtains the IP multicast group address, pseudo VA, pseudo R_Key, and message length corresponding to the content (S11). Then, the transmitting terminal 10 starts message transmission processing (S12). The transmitting terminal 10 starts transmission processing asynchronously with the timing of the receiving terminal 20's reception start processing.

[0048] If the message for which transmission processing has been completed is the final message (YES in S13), the transmitting terminal 10 ends the series of operations. The method for determining whether a message is the final message may be any existing method, but is not particularly limited to this. If the message for which transmission processing has been completed is not the final message (NO in S13), the transmitting terminal 10 repeats steps from S12.

[0049] 8 and 10, the receiving terminal 20 performs pre-reception processing (S21). Specifically, the receiving terminal 20 acquires distribution information from the database DB. That is, the receiving terminal 20 acquires the IP multicast group address, pseudo VA, pseudo R_Key, and message length corresponding to the content (S31).

[0050] Furthermore, the receiving terminal 20 performs in-host processing (S32 to S36) as pre-reception processing (S21) and joins the multicast group (S37). These steps will be described in detail later. Note that in-host processing refers to processing within the memory 400.

[0051] Next, the receiving terminal 20 performs message receiving processing (S22). If the message for which the receiving processing has been completed is the final message (YES in S23), the receiving terminal 20 proceeds to reception termination processing (S24). If the message for which the receiving processing has been completed is not the final message (NO in S23), the receiving terminal 20 repeats the steps from S22. As described above, there are no particular limitations on the method for determining the final message. It is assumed that the same MR into which messages are written continues to be reused.

[0052] (3) Detailed Operation of the Communication System (3.1) Processing before Reception Start Detailed operation of the communication system, specifically, detailed operation of the receiving terminal 20 will be described. First, the processing before reception start (S21) will be described with reference to Figures 11 and 12. Note that S31 shown in Figure 12 has been explained as an outline of the operation of the receiving terminal 20, so it will be omitted (see Figure 8).

[0053] The receiving terminal 20 reserves a memory area (MR) in the user space 420 with a number of bytes equal to the message length (Length) acquired in S31 (S32). Next, the receiving terminal 20 writes the pseudo VA and pseudo R_Key acquired in S31 to the information storage unit 141 of the RQ logic 140 (S33). Furthermore, the receiving terminal 20 generates the pseudo page table described above from the pseudo VA and the PA corresponding to the VA determined when the MR was reserved (S34), and writes this pseudo page table to the information storage unit 141 of the RQ logic 140 (S35).

[0054] Finally, the receiving terminal 20 adds an entry to the multicast group table 120 (S36). As described above, adding an entry involves adding a QPN including the own RQ to the GID for receiving packets. If the GID corresponding to the multicast group you want to join does not exist in the multicast group table 120, you can add an entry for that GID and then add a QPN including the own RQ. This allows the receiving terminal 20 to join the multicast group (S37).

[0055] (3.2) Message Reception Processing (3.2.1) Packet Transfer Processing from Port to RQ Logic With reference to Figures 13 to 15, the packet transfer processing from the first input buffer 111 of the port 110 to the second input buffer 142 of the RQ logic 140, part of the message reception processing (S22), will be described. Figure 13 is a diagram showing the flow of packets in the network adapter 100. Figure 14 is a diagram showing the flow of packets from the first input buffer 111 to the second input buffer 142 of the RQ logic 140. The specific flow of packets shown in Figure 14 will be described with reference to Figure 15.

[0056] 15, first, the receiving terminal 20 references the GRH of the packet input to the first input buffer 111 (see FIG. 4) and extracts the packet's destination GID (DGID) (S41). Next, the receiving terminal 20 searches the multicast group table 120 using the GID as a key and extracts the QPN to which the packet should be forwarded (S42). Finally, the receiving terminal 20 forwards the packet input to the first input buffer 111 to the second input buffer 142 of each RQ logic 140 corresponding to the extracted QPN (S43).

[0057] (3.2.2) Packet transfer process from RQ logic to memory With reference to Figures 13, 16 to 30, we will explain the process of the message reception process (S22) in which the RQ logic 140 transfers a packet to the memory 400 via the second input buffer 142 and the output buffer 143.

[0058] As shown in Figure 16, when a packet is input to the second input buffer 142, the receiving terminal 20 references the BTH of the packet (see Figure 5) and classifies the packet based on the value of its opcode (S52). The opcodes are the above-mentioned RDMA Write First, RDMA Write Middle, RDMA Write Last, RDMA Write Last with Immediate Data, RDMA Write-Only, and RDMA Write-Only with Immediate Data. The packet input to the second input buffer is processed by the RQ logic 140 for each opcode (S53 to S58). Note that after S53 or S54, the process returns to S51, but after S55 to S58, S22 is considered to be complete and the process proceeds to S23 (see Figure 10).

[0059] As shown in FIG. 17, the RQ logic 140 divides the payload of a packet input to the second input buffer 142 (Data Payload in the figure) into portions corresponding to the maximum payload length of a TLP (Max_Payload in the figure), adds a memory write request header, and issues (generates) a memory write request TLP (hereinafter simply referred to as TLP) to write it to the output buffer 143. The TLP input to the output buffer 143 is transferred to the root complex 200 via the PCIe bus. As shown in FIG. 18, the root complex 200 sequentially writes each TLP to an MR allocated in the user space 420 of the memory 400 via the memory controller 300. The packet processing in the RQ logic 140 will be described in more detail below for each opcode value. It is assumed that the MRs are allocated in physically contiguous areas, i.e., areas consisting of contiguous physical addresses, within the user space 420 of the memory 400. It is assumed that Max_Payload is an integer multiple of the word length of the memory 400.

[0060] (3.2.2.1) RDMA Write First Packet With reference to FIGS. 16 and 19, the processing of a packet having an RDMA Write First opcode classified in S52 (hereinafter also referred to as an RDMA Write First packet) will be described (S53). First, the RQ logic 140 reads the pseudo VA of the MR from the information storage unit 141 (S110). Next, the RQ logic 140 converts the pseudo VA into the PA of the MR based on the pseudo page table that the information storage unit 141 acquires from the information generation unit 411 (S120). Furthermore, the RQ logic 140 converts the RDMA Write First packet into a TLP and sends this TLP to the output buffer 143 (S130). Note that the TLP sent to the output buffer 143 is further sent onto the PCIe bus and written to the user space 420 of the memory 400. Between the output buffer 143 and the memory 400 there are a root complex 200 and an MC 300, but the functions of these have been described above and will not be described here.

[0061] 20, the step S130 performed by the RQ logic 140 will be described in more detail. First, as a preliminary step of the RDMA Write First packet processing, 0 is stored in Addr_offset, 0 is stored in Payload_offset, and the payload length of the RDMA Write First packet is stored in Remain (S131). Addr_offset is the difference between the destination address (PA) of the MR into which the payload of the first TLP is written and the destination address of the MR into which the payload of the nth TLP is written. Payload_offset is the difference between the beginning of the payload of the RDMA Write First packet (the beginning of the portion that will be the payload of the first TLP) and the beginning of the portion that will be the payload of the nth TLP. Remain is the payload length of the portion of the RDMA Write First packet that has not yet been processed.

[0062] Compare Remain with the maximum payload length of the TLP (Max_Payload in the figure), and if Remain > Max_Payload (YES in S132), issue a TLP with PA + Addr_offset as the destination address of the TLP, and with a payload length of Max_Payload from a position offset by Payload_offset with respect to the beginning of the RDMA Write First packet (S133). Then, the sum of Addr_offset and Max_Payload / word length of memory 400 is stored in Addr_offset. The sum of Payload_offset and Max_Payload is stored in Payload_offset. The value obtained by subtracting Max_Payload from Remain is stored in Remain (S134). Then, return to S132.

[0063] In S132, if Remain>Max_Payload is not satisfied (NO in S132), PA+Addr_offset is set as the destination address of the TLP, and the remaining payload length of the RDMA Write First packet is set as the payload (S135).Then, the sum of Addr_offset and Remain / word length of memory 400 is stored in NextPA, which indicates the destination address of the next packet (S136).

[0064] 19, NextPA indicating the destination address of the next packet is recorded in the hardware register (S140). Note that the function of the hardware register may be performed by the information holding unit 141 or may be implemented separately.

[0065] (3.2.2.2) RDMA Write Middle Packet With reference to FIGS. 16 and 21, the processing of a packet having an RDMA Write Middle opcode classified in S52 (hereinafter also referred to as an RDMA Write Middle packet) will be described (S54). First, the RQ logic 140 reads the NextPA described in (3.2.2.1) from the hardware register (S210). Next, the RQ logic 140 converts the RDMA Write Middle packet into a TLP and sends this TLP to the output buffer 143 (S220). The TLP sent to the output buffer 143 is further sent onto the PCIe bus and written to the user space 420 of the memory 400. As with the description of the RDMA Write First packet, the root complex 200 and MC 300 interposed between the output buffer 143 and the memory 400 will not be described here.

[0066] S220 will be described in more detail with reference to Figure 22. First, as a preliminary step in the RDMA Write Middle packet processing, NextPA is stored in Addr_offset, 0 is stored in Payload_offset, and the payload length of the RDMA Write Middle packet is stored in Remain (S221). For details about Addr_offset, Payload_offset, and Remain, please refer to the description of the RDMA Write First packet.

[0067] Compare Remain with the maximum payload length of the TLP (Max_Payload in the figure), and if Remain>Max_Payload (YES in S222), issue a TLP with PA+Addr_offset as the destination address of the TLP, and with a payload length of Max_Payload from a position offset by Payload_offset with respect to the beginning of the RDMA Write Middle packet (S223). Then, the sum of Addr_offset and Max_Payload / word length of memory 400 is stored in Addr_offset. The sum of Payload_offset and Max_Payload is stored in Payload_offset. The value obtained by subtracting Max_Payload from Remain is stored in Remain (S224). Then, return to S222.

[0068] In S222, if Remain>Max_Payload is not satisfied (NO in S222), issue a TLP with PA+Addr_offset as the destination address of the TLP and the remaining payload length of the RDMA Write Middle packet as the payload (S225).Then, store the sum of Addr_offset and Remain / word length of memory 400 in NextPA, which indicates the destination address of the next packet (S226).

[0069] 21, NextPA indicating the destination address of the next packet is recorded in the hardware register (S230). Note that the function of the hardware register may be performed by the information holding unit 141 or may be implemented separately.

[0070] (3.2.2.3) RDMA Write Last Packet With reference to FIGS. 16 and 23, the processing of a packet having an RDMA Write Last opcode classified in S52 (hereinafter also referred to as an RDMA Write Last packet) will be described (S55). First, the RQ logic 140 reads the NextPA described in (3.2.2.2) from the hardware register (S310). Furthermore, the RQ logic 140 converts the RDMA Write Last packet into a TLP and sends this TLP to the output buffer 143 (S320). Note that the TLP sent to the output buffer 143 is further sent onto the PCIe bus and written to the user space 420 of the memory 400. As with the description of the RDMA Write First packet, the root complex 200 and the MC 300 interposed between the output buffer 143 and the memory 400 will not be described here.

[0071] S320 will be described in more detail with reference to Figure 24. First, as a preliminary step to processing the RDMA Write Last packet, NextPA is stored in Addr_offset, 0 is stored in Payload_offset, and the payload length of the RDMA Write Middle packet is stored in Remain (S221). For details about Addr_offset, Payload_offset, and Remain, please refer to the description of the RDMA Write First packet.

[0072] The Remain is compared with the maximum payload length of the TLP (Max_Payload in the figure), and if Remain > Max_Payload (YES in S322), PA + Addr_offset is used as the destination address of the TLP, and a TLP with a payload length of Max_Payload is issued from a position offset by Payload_offset from the beginning of the RDMA Write Last packet (S323). After that, the sum of Addr_offset and Max_Payload / word length of memory 400 is stored in Addr_offset. The sum of Payload_offset and Max_Payload is stored in Payload_offset. The value obtained by subtracting Max_Payload from Remain is stored in Remain (S324). After this, the process returns to S322.

[0073] In S322, if Remain>Max_Payload is not satisfied (NO in S322), a TLP is issued with PA+Addr_offset as the TLP destination address and the remaining payload length of the RDMA Write Last packet as the payload (S325).

[0074] The RDMA Write Last packet is the last packet in a message (see FIG. 2), so completion of the processing of the RDMA Write Last packet (completion of S325) means completion of S22 (see FIG. 10).

[0075] (3.2.2.4) RDMA Write Last with Immediate Data Packet With reference to FIGS. 16 and 25, the processing of a packet having an RDMA Write Last with Immediate Data opcode classified in S52 (hereinafter also referred to as an RDMA Write Last with Immediate Data packet) will be described (S56). First, the RQ logic 140 reads the NextPA described in (3.2.2.2) from the hardware register (S410). Furthermore, the RQ logic 140 converts the RDMA Write Last with Immediate Data packet into a TLP and sends this TLP to the output buffer 143 (S420). Note that the TLP sent to the output buffer 143 is further sent onto the PCIe bus and written to the user space 420 of the memory 400. As with the description of the RDMA Write First packet, the root complex 200 and the MC 300 interposed between the output buffer 143 and the memory 400 will not be described here.

[0076] S420 will be described in more detail with reference to Figure 26. S420 is largely the same as S320 described above. Specifically, S421 to S425 shown in Figure 26 correspond to S321 to S325 shown in Figure 24. Therefore, for S421 to S425, please refer to the description of S321 to S325.

[0077] After S425, a Completion Queue Entry (CQE) containing the Immediate Data in the RDMA Write Last with Immediate Data packet is generated and written to the Completion Queue (CQ) (S426). Note that the CQ is a queue for notifying the completion of packet processing, and the writing of the CQE means that the packet processing has been completed successfully.

[0078] The RDMA Write Last with Immediate Data packet is the last packet in the message (see FIG. 2), so the completion of the processing of the RDMA Write Last with Immediate Data packet means the completion of S22 (see FIG. 10).

[0079] (3.2.2.5) RDMA Write-Only Packet With reference to FIGS. 16 and 27, the processing of a packet having an RDMA Write-Only opcode classified in S52 (hereinafter also referred to as an RDMA Write-Only packet) will be described (S57). First, the RQ logic 140 reads a pseudo VA from the information storage unit 141 (S510). Next, the RQ logic 140 converts the pseudo VA into a PA of an MR based on the pseudo page table that the information storage unit 141 acquires from the information generation unit 411 (S520). Furthermore, the RQ logic 140 converts the RDMA Write-Only packet into a TLP and sends this TLP to the output buffer 143 (S530). Note that the TLP sent to the output buffer 143 is further sent onto the PCIe bus and written to the user space 420 of the memory 400. As with the RDMA Write First packet, the explanation of the root complex 200 and MC 300 that are interposed between the output buffer 143 and the memory 400 will be omitted.

[0080] Figure 28 is a flowchart that explains S530 in more detail, but most of it is the same as Figure 24. Specifically, S531 to S535 shown in Figure 28 correspond to S331 to S335 shown in Figure 24. Therefore, for S531 to S535, please refer to the explanation of S331 to S335.

[0081] The RDMA Write-Only packet is the last packet in the message (see FIG. 2), so the completion of the processing of the RDMA Write-Only packet (completion of S525) means the completion of S22 (see FIG. 10).

[0082] (3.2.2.6) RDMA Write-Only with Immediate Data Packet With reference to FIGS. 16 and 29, the processing of a packet having an RDMA Write-Only with Immediate Data opcode classified in S52 (hereinafter also referred to as an RDMA Write-Only with Immediate Data packet) will be described (S58). First, the RQ logic 140 reads a pseudo VA from the information storage unit 141 (S610). Next, the RQ logic 140 converts the pseudo VA into a PA of an MR based on the pseudo page table that the information storage unit 141 acquires from the information generation unit 411 (S620). Furthermore, the RQ logic 140 converts the RDMA Write-Only with Immediate Data packet into a TLP and sends this TLP to the output buffer 143 (S630). Note that the TLP sent to the output buffer 143 is further sent onto the PCIe bus and written to the user space 420 of the memory 400. As with the RDMA Write First packet, the explanation of the root complex 200 and MC 300 that are interposed between the output buffer 143 and the memory 400 will be omitted.

[0083] Figure 30 is a flowchart that explains S630 in more detail, but most of it is the same as Figure 26. Specifically, S631 to S636 shown in Figure 30 correspond to S421 to S426 shown in Figure 26. Therefore, for S631 to S636, please refer to the explanation of S421 to S426.

[0084] The RDMA Write-Only with Immediate Data packet is the last packet in the message (see FIG. 2), so the completion of processing the RDMA Write-Only with Immediate Data packet means the completion of S22 (see FIG. 10).

[0085] (3.3) Reception End Processing As described above, after the message reception processing (S22), if the received message is the last message (YES in S23), the process proceeds to reception end processing (S24) (see FIG. 10). Note that the method for determining whether the message is the last message is not specified.

[0086] S24 will be described in detail with reference to Figure 31. First, the MR secured in the user space 420 of the memory 400 is erased (S61). Next, the pseudo VA, pseudo R_Key, and message length written to the information storage unit 141 are deleted (S62). Furthermore, the pseudo page table written to the information storage unit 141 is deleted (S63). Furthermore, the entry added to the multicast group table 120 is deleted (S64). Finally, the IP multicast group is left (S65). Note that S61 to S65 correspond to S32, S33, and S35 to S37 shown in Figure 12, respectively.

[0087] (4) Effects The receiving terminal 20 (packet processing device) of this embodiment can replace VA with pseudo VA, thereby making it possible to set the virtual address (VA) value in packets sent to multiple receiving terminals 20 that are the targets of multicast to the same pseudo VA value, thereby enabling multicast by the sending terminal 10.

[0088] This eliminates the need to use a special switch capable of rewriting destination addresses as a switch for forwarding packets to multiple receiving terminals 20 in the network N, and allows the use of a general-purpose switch. It is also possible to use an optical switch, which makes it difficult to rewrite data within packets. Furthermore, there is no need to prepare additional hardware other than the network adapter 100. As described above, the receiving terminal 20 of this embodiment can significantly reduce the limitations and costs associated with building a network compared to conventional techniques.

[0089] Furthermore, by replacing the R_Key with a pseudo R_Key, the receiving terminal 20 of this embodiment can set the remote key (R_Key) value in packets sent to multiple receiving terminals 20 that are the targets of multicast to the same pseudo R_Key value. This makes the multicast described above possible. Furthermore, because the pseudo R_Key can only be obtained from a database DB that requires access authentication, it is possible to confirm that the transmitting terminal 10 that sent the packet containing the pseudo R_Key has access rights to the MR of the receiving terminal 20, and secure communication can be achieved even when the pseudo R_Key is used instead of the R_Key.

[0090] The receiving terminal 20 (packet processing device) described above can be, for example, a general-purpose computer system including a central processing unit (CPU) 901, memory 902, storage 903, communication device 904, input device 905, and output device 906, as shown in Figure 32. In this computer system, the receiving terminal 20 (packet processing device) is realized by the CPU 901 executing a predetermined program loaded onto memory 902. This program can be recorded on a computer-readable non-transitory recording medium such as a magnetic disk, optical disk, or semiconductor memory, or can be distributed via a network.

[0091] 10 Transmitting terminal 20 Receiving terminal (packet processing device) 100 Network adapter 110 Port 111 First input buffer 120 Multicast group table 130 SQ logic 140 RQ logic 141 Information holding unit 142 Second input buffer 143 Output buffer 200 Route complex 210 Third input buffer 300 MC 400 Memory 410 Kernel space 411 Information generation unit 420 User space 421 Information acquisition unit 500 CPU A to D Message DB Database N Network

Claims

A packet processing device, an information acquisition unit that acquires a pseudo virtual address common to other packet processing devices from a database; a network adapter that receives packets containing the pseudo-virtual address; an information generating unit that reserves a memory area in a memory into which the data in the packet is written; Equipped with the information generating unit generates a table in which the pseudo virtual addresses are associated with physical addresses of the memory areas; the network adapter refers to the pseudo virtual address in the packet and the table, and writes the data to the memory area; Packet processing device. 　 the network adapter writes the data to the memory area without going through a kernel space of the memory; The packet processing device according to claim 1 . 　 the information acquisition unit, when the database authenticates the access from the information acquisition unit, acquires from the database a pseudo remote key that is common to the other packet processing device; the network adapter writes the data to the memory area if authentication using the pseudo remote key is successful; The packet processing device according to claim 1 . 　 The network adapter an information storage unit to which the pseudo virtual addresses and the table are written; RQ logic that converts the packets into transaction layer packets; Equipped with the RQ logic refers to the table in the information storage unit and writes the data in the transaction layer packet to the memory area; The packet processing device according to claim 1 . 　 the RQ logic transmits the transaction layer packet onto a PCIe bus connecting the network adapter and a root complex connected to the memory; The packet processing device according to claim 4 . 　 the information acquisition unit is included in a user space of the memory; The information generation unit is included in a kernel space of the memory. The packet processing device according to claim 1 . 　 A packet processing method performed by a packet processing device, Obtaining a pseudo virtual address common to other packet processing devices from the database; receiving a packet including the pseudo virtual address; allocating a memory area in a memory into which the data in the packet is written; generating a table in which the pseudo-virtual addresses are associated with physical addresses of the memory regions; writing the data to the memory area by referring to the pseudo virtual address in the packet and the table; Packet processing method. 　 A program that causes a computer to function as the packet processing device according to any one of claims 1 to 6.

Images