Processing system, processing device, processing method, and program

Abstract

A local terminal L transmits, to a processing device 1, an RDMA transmission packet in which processing data to be transferred to a memory of an accelerator of both a first remote terminal R1 and a second remote terminal R2 is set. The processing device 1 acquires a QPN from the first remote terminal R1, converts a destination QP of a BTH of the RDMA transmission packet into the QPN acquired from the first remote terminal R1, and transmits the converted RDMA transmission packet to the first remote terminal R1. When establishing a connection, the processing device 1 acquires a QPN from the second remote terminal R2, converts a destination QP of a BTH of the RDMA transmission packet into the QPN acquired from the second remote terminal R2, and transmits the converted RDMA transmission packet P2 to the second remote terminal R2. The first remote terminal R1 and the second remote terminal R2 each receive the converted RDMA transmission packets and transfer processing data to the memories of the accelerators.

Description

[Technical Field]

[0001] The present invention relates to a processing system, a processing device, a processing method, and a program. [Background technology]

[0002] In recent years, as content quality has improved, communication methods that distribute the same data to multiple devices, such as streaming distribution, video conferencing, and online games, have become more widespread. These communication methods require high-speed, low-latency processing of large amounts of data, from reception to processing.

[0003] There is a communication method that transfers data directly to the accelerator's memory without going through the CPU. Accelerators are hardware specialized for specific calculations, such as GPUs (Graphics Processing Units) and TPUs (Tensor Processing Units). This communication method directly connects the network and computing, achieving high speed and low latency for data reception and calculations.

[0004] RDMA is known as a protocol that can transfer data directly to the accelerator's memory (Non-Patent Document 1). The SEND operation method of the RDMA protocol connects a local terminal and a remote terminal via P2P (Peer to Peer) in the RC (Reliable Connection) service type, enabling high-speed communication between memories. The local terminal creates an SQ (Send Queue) for the remote terminal that is the destination of the SEND operation, and transfers data without going through the operating systems of both computers. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] InfiniBand Architecture Specification Volume 1 Release 1.4, April 7, 2020 Summary of the Invention [Problem to be solved by the invention]

[0006] However, when using RDMA to communicate between memories from a local terminal to multiple remote terminals, the local terminal may be burdened. The local terminal creates an SQ for each remote terminal, which causes a processing burden on the local terminal. In addition, since each SQ is sent from the local terminal to each remote terminal, the transmission traffic on the local terminal becomes enormous.

[0007] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique capable of reducing the burden on a local terminal that transfers data to a plurality of remote terminals. [Means for solving the problem]

[0008] A processing system according to one embodiment of the present invention includes a local terminal, a first remote terminal, a second remote terminal, and a processing device. The local terminal transmits an RDMA transmit packet to the processing device, the RDMA transmit packet containing processing data to be transferred to memories of accelerators in the first remote terminal and the second remote terminal. The processing device includes a local-side controller that establishes a connection with the local terminal and receives the RDMA transmit packet from the local terminal, a first remote-side controller that establishes a connection with the first remote terminal, a second remote-side controller that establishes a connection with the second remote terminal, and a duplicator that inputs the RDMA transmit packet to the first remote-side controller and the second remote-side controller. When establishing a connection, the first remote-side controller acquires a QPN from the first remote terminal, converts a Destination QP in a Base Transport Header (BTH) of the RDMA transmit packet to the QPN acquired from the first remote terminal, and transmits the converted RDMA transmit packet to the first remote terminal. The second remote-side control unit acquires a QPN from the second remote terminal when establishing a connection, converts a BTH Destination QP of the RDMA transmit packet to the QPN acquired from the second remote terminal, and transmits the converted RDMA transmit packet to the second remote terminal. The first remote terminal receives the converted RDMA transmit packet and transfers the processing data to a memory of an accelerator. The second remote terminal receives the converted RDMA transmit packet and transfers the processing data to a memory of an accelerator.

[0009] A processing device according to one embodiment of the present invention includes a local-side controller that establishes a connection with a local terminal and receives, from the local terminal, an RDMA transmit packet containing processing data to be transferred to memories of accelerators of a first remote terminal and a second remote terminal, a first remote-side controller that establishes a connection with the first remote terminal, a second remote-side controller that establishes a connection with the second remote terminal, and a duplicator that inputs the RDMA transmit packet to the first remote-side controller and the second remote-side controller. When establishing the connection, the first remote-side controller acquires a QPN from the first remote terminal, converts a Destination QP of a BTH of the RDMA transmit packet to the QPN acquired from the first remote terminal, and transmits the converted RDMA transmit packet to the first remote terminal. When a connection is established, the second remote side control unit obtains a QPN from the second remote terminal, converts the BTH Destination QP of the RDMA transmission packet to the QPN obtained from the second remote terminal, and transmits the converted RDMA transmission packet to the second remote terminal.

[0010] In one aspect of the processing method of the present invention, a local terminal transmits to a processing device an RDMA transmit packet in which processing data to be transferred to memories of accelerators of a first remote terminal and a second remote terminal is set, the processing device establishes a connection with the local terminal and receives the RDMA transmit packet from the local terminal, a first remote side control unit of the processing device establishes a connection with the first remote terminal, a second remote side control unit of the processing device establishes a connection with the second remote terminal, the processing device inputs the RDMA transmit packet to the first remote side control unit and the second remote side control unit, the first remote side control unit of the processing device acquires a QPN from the first remote terminal when establishing the connection and converts a Destination QP of a Base Transport Header (BTH) of the RDMA transmit packet to the QPN acquired from the first remote terminal, and transmits the converted RDMA transmit packet to the first remote terminal, the second remote side control unit of the processing device acquires a QPN from the second remote terminal when establishing the connection and converts a Destination QP of a Base Transport Header (BTH) of the RDMA transmit packet to the QPN acquired from the first remote terminal, The QP is converted into a QPN obtained from the second remote terminal, and the converted RDMA transmission packet is sent to the second remote terminal; the first remote terminal receives the converted RDMA transmission packet and transfers the processed data to a memory of an accelerator; and the second remote terminal receives the converted RDMA transmission packet and transfers the processed data to a memory of an accelerator.

[0011] One aspect of the present invention is a program that causes a computer to function as the processing device. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a technique that can reduce the burden on a local terminal that transfers data to a plurality of remote terminals. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating the system configuration of a processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating a process of transmitting a general RDMA transmission packet via P2P. [Figure 3] FIG. 3 is a diagram illustrating functional blocks of a processing device according to an embodiment of the present invention. [Figure 4] FIG. 4 is a diagram illustrating an example of the data structure and data of a conversion table in the processing device. [Figure 5] FIG. 5 is a diagram illustrating an example of the data structure and data of a history table in the processing device. [Figure 6] FIG. 6 is a sequence diagram (part 1) illustrating a process for establishing a connection in a processing system according to an embodiment of the present invention. [Figure 7] FIG. 7 is a sequence diagram (part 2) illustrating the process of establishing a connection in the processing system according to the embodiment of the present invention. [Figure 8] FIG. 8 is a sequence diagram illustrating a process for transferring processing data in the processing system according to the embodiment of the present invention. [Figure 9] FIG. 9 is a diagram illustrating an RDMA transmit packet transmitted by a local terminal and an RDMA transmit packet transmitted to a first remote terminal. [Figure 10] FIG. 10 is a flowchart illustrating the establishment process by the establishment unit of the processing device. [Figure 11] FIG. 11 is a diagram for explaining the settings in the establishment process. [Figure 12] FIG. 12 is a flowchart illustrating the conversion process performed by the conversion unit of the processing device. [Figure 13] FIG. 13 is a diagram for explaining settings in the conversion process. [Figure 14] FIG. 14 is a diagram illustrating updates in the conversion process. [Figure 15]FIG. 15 is a diagram illustrating the hardware configuration of a computer used in the processing device. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are designated by the same reference numerals and the description thereof will be omitted.

[0015] In the embodiments of the present invention, the following abbreviations are used: RDMA Remote Direct Memory Access QP Queue Pair SQ Send Queue RQ Receive Queue CQ Completion Queue CM Communication Management BTH Base Transport Header RC Reliable Connection QPN QP Number PSN Packet Sequence Number WQE Work Queue Element CQE Completion Queue Element RC Reliable Connection REQ Connect Request REP Connect Reply RTU Ready To Use

[0016] (Processing System) A processing system 5 according to an embodiment of the present invention will be described with reference to FIG. 1. The processing system 5 includes a processing device 1, a local terminal L, a first remote terminal R1, and a second remote terminal R2. When the first remote terminal R1 and the second remote terminal R2 are not particularly distinguished from each other, they may be referred to as remote terminals R. In the embodiment of the present invention, a case will be described in which processing data is transferred from the local terminal L to two remote terminals R, but this is not limiting. The number of remote terminals R may be two or more.

[0017] In an embodiment of the present invention, an RDMA transmit packet P is transmitted by a local terminal L using the SEND operation method (RC) of the RDMA protocol. Processing data to be transferred to a memory of an accelerator of a remote terminal R is set in the RDMA transmit packet P. The RDMA transmit packet P1 is transmitted from a processing device 1 to a first remote terminal R1. The RDMA transmit packet P1 is generated by the processing device 1 converting the header of the RDMA transmit packet P. The RDMA transmit packet P2 is transmitted from the processing device 1 to a second remote terminal R2. The RDMA transmit packet P2 is generated by the processing device 1 converting the header of the RDMA transmit packet P.

[0018] In the processing system 5, the local terminal L transmits an RDMA transmit packet P to the processing device 1, in which processing data to be transferred to the memory of each accelerator of a first remote terminal R1 and a second remote terminal R2 is set. The processing device 1 converts the header of the received RDMA transmit packet P to generate RDMA transmit packets P1 / P2. The processing device 1 transmits the converted RDMA transmit packets P1 / P2 to each of the first remote terminal R1 and the second remote terminal R2. Each of the first remote terminal R1 and the second remote terminal R2 receives the RDMA transmit packet P1 / P2 from the processing device 1 and transfers the processing data to the memory of the accelerator. Here, the replica unit 20 of the processing device 1 is implemented in a computer that is physically or virtually different from the computers of the local terminal L, the first remote terminal R1, and the second remote terminal R2.

[0019] In such a processing system 5, the processing device 1 generates an RDMA transmit packet Pn corresponding to each of multiple remote terminals R from an RDMA transmit packet P received from the local terminal L, and transfers the processing data to each of the multiple remote terminals R. The local terminal L only needs to generate one RDMA transmit packet P regardless of the number of destination remote terminals R, which reduces the processing load compared to creating an RDMA transmit packet for each remote terminal R. Furthermore, because the processing device 1 generates and transmits multiple packets corresponding to each remote terminal R, the local terminal L only needs to transmit one RDMA transmit packet P regardless of the number of destination remote terminals R, which reduces the amount of data to be sent.

[0020] A typical process for transmitting an RDMA transmission packet via P2P will be described with reference to Fig. 2. A local terminal L holds an SQ, and a remote terminal R holds an RQ. The SQ of the local terminal L and the RQ of the remote terminal R form a QP.

[0021] Before transmitting an RDMA transmission packet, a connection is established between the local terminal L and the remote terminal R. When the connection is established, the local terminal L sets the Local QPN and Starting PSN values of the SQ in the CM header of the REQ and notifies the remote terminal R. The remote terminal R sets the Local QPN and Starting PSN values of the RQ in the CM header of the REP and notifies the local terminal L. The Local QPN identifies the QP in the local terminal L or the remote terminal R. The PSN identifies the transmitted and received bytes in the processing data specified in the byte stream in the local terminal L or the remote terminal R.

[0022] Once the connection is established, the processing data is transferred. This section explains data transfer using the SEND operation method (RC) of the RDMA protocol. The local terminal L loads a WQE, which specifies the address of the memory area where the processing data is stored, into an SQ. The remote terminal R loads a WQE, which specifies the address of the memory area where the processing data will be stored, into an RQ.

[0023] The local terminal L transmits an RDMA transmission packet with processing data set in the payload to the remote terminal R. The QPN of the RQ obtained from the remote terminal R when the connection was established is set in the Destination QP field of the BTH of the RDMA transmission packet that is sent first after the connection is established. The value of the Starting PSN obtained from the remote terminal R when the connection was established is set in the PSN field.

[0024] The remote terminal R receives the RDMA transmission packet, and if it successfully receives the processed data, it loads the CQE into the CQ and sends an ACK packet to the local terminal L. The QPN of the SQ is set in the Destination QP field of the BTH of the ACK packet. The value of the Starting PSN sent to the remote terminal R when the connection was established is set in the PSN field.

[0025] When the local terminal L receives an ACK packet from the remote terminal R, it places the CQE in the CQ. At this time, the WQE is released from the SQ.

[0026] After the connection is established, the PSN set in the second and subsequent RDMA transmission packets is set to a value incremented from the Starting PSN.

[0027] (Processing device) A processing device 1 according to an embodiment of the present invention will be described with reference to FIGS.

[0028] The processing device 1 includes a local-side control unit 10, a duplication unit 20, a first remote-side control unit 30, and a second remote-side control unit 40. The first remote-side control unit 30 and the second remote-side control unit 40 have similar functions, although the remote terminals to which the data is transferred are different. The processing device 1 includes the same number of remote-side control units as the number of remote terminals R to which the processing data is transferred.

[0029] In the embodiment of the present invention, a case will be described in which one computer implements each of the processing units of the local-side control unit 10, the duplication unit 20, the first remote-side control unit 30, and the second remote-side control unit 40, but this is not limiting. Each processing unit may be distributed and implemented on multiple computers.

[0030] As shown in FIG. 1, in the processing system 5, the local terminal L has an SQ. The first remote terminal R1 and the second remote terminal R2 each have an RQ. The local side control unit 10 functions as a pseudo RQ for the SQ of the local terminal L. The first remote side control unit 30 functions as a pseudo SQ for the RQ of the first remote terminal R1. The second remote side control unit 40 functions as a pseudo SQ for the RQ of the second remote terminal R2. The duplicating unit 20 inputs the RDMA transmission packet P received by the local side control unit 10 to the first remote side control unit 30 and the second remote side control unit 40, respectively.

[0031] The local-side control unit 10 establishes a connection with the local terminal L and receives an RDMA transmission packet P from the local terminal L.

[0032] The duplicating unit 20 duplicates an RDMA transmission packet P received from a local terminal L and inputs the duplicated packet to a first remote-side control unit 30 and a second remote-side control unit 40 .

[0033] The first remote side control unit 30 generates an RDMA transmit packet P1 by converting the header of the input RDMA transmit packet P, and transmits the RDMA transmit packet P1 to the first remote terminal R1. As shown in FIG. 3, the first remote side control unit 30 includes data in a conversion table 31 and a history table 32, and functions of an establishment unit 33 and a conversion unit 34. Each piece of data is stored in a storage device such as a memory 902 or a storage 903. Each function is implemented in a CPU 901.

[0034] 4(a), the conversion table 31 has the following data items: Local dQPN, IP address, MAC address, dQPN, Local PSN, and Remote PSN. Before the first remote-side control unit 30 establishes a connection with the first remote terminal R1, each item in the conversion table 31 is set to a NULL value.

[0035] The local dQPN is the QPN of the opposite end of the local terminal L. The local dQPN is the Destination QP included in the BTH of the RDMA transmission packet P transmitted by the local terminal L. The local dQPN is set when the local terminal L and the local-side control unit 10 first receive the RDMA transmission packet P after establishing a connection.

[0036] The IP address is the IP address of the first remote terminal R1. When a connection is established, the source IP address included in the REP received from the first remote terminal R1 is set as the IP address.

[0037] The MAC address is the MAC address of the first remote terminal R1. When a connection is established, the source MAC address included in the REP received from the first remote terminal R1 is set as the MAC address.

[0038] The dQPN is the QPN of the first remote terminal R1. When a connection is established, the Local QPN included in the REP received from the first remote terminal R1 is set as the dQPN.

[0039] The Local PSN is the PSN of the RDMA transmission packet P transmitted from the local terminal L. When the RDMA transmission packet P is received for the first time after the connection is established, the PSN included in the BTH of that RDMA transmission packet P is set as the Local PSN. After that, the value of the Local PSN is incremented by one each time an RDMA transmission packet P is received. Generally, the value of the Local PSN in the conversion table 31 matches the PSN included in the BTH of the RDMA transmission packet P transmitted from the local terminal L.

[0040] The Remote PSN is the PSN of the RDMA transmission packet P1 to be transferred to the first remote terminal R1. When a connection is established, the Starting PSN included in the REP received from the first remote terminal R1 is set to the Remote PSN. After that, the value of the Remote PSN is incremented by one each time an RDMA transmission packet P is received from the local terminal L.

[0041] History table 32 is data on the history of the values of Local PSN and Remote PSN in conversion table 31. As shown in FIG. 5(a), history table 32 includes Local PSN and Remote PSN. When a value is registered in conversion table 31, the Local PSN and Remote PSN at the time of registration are set in the first row. When the value in conversion table 31 is updated, specifically, every time an RDMA transmission packet P is received from local terminal L, the updated Local PSN and Remote PSN are set in a new row. When the first remote-side control unit 30 detects packet loss in RDMA transmission packet P1, it refers to history table 32 when identifying the RDMA transmission packet P for which a retransmission process is to be requested.

[0042] The establishment unit 33 establishes a connection with the first remote terminal R1. When establishing a connection, the establishment unit 33 acquires a QPN and a Starting PSN from the first remote terminal R1. The establishment unit 33 sets the acquired QPN as the dQPN in the conversion table 31. The establishment unit 33 sets the Starting PSN as the Remote PSN in the conversion table 31 and as the Remote PSN in the first row of the history table 32. The establishment unit 33 sets the Source IP address and Source MAC address of the first remote terminal R1 as the IP address and MAC address in the conversion table 31.

[0043] The conversion unit 34 converts the BTH Destination QP and PSN of the RDMA transmit packet P input from the duplicating unit 20. The conversion unit 34 converts the Source IP address and Source MAC address into the IP address and MAC address of the first remote side control unit 30. The conversion unit 34 converts the Destination IP address and Destination MAC address into the IP address and MAC address registered in the conversion table 31, specifically, the IP address and MAC address of the first remote terminal R1. The conversion unit 34 transmits the converted RDMA transmit packet P1 to the first remote terminal R1.

[0044] First, the conversion of the Destination QP will be described. The conversion unit 34 converts the BTH Destination QP of the RDMA transmission packet P input from the duplicating unit 20 into the QPN acquired from the first remote terminal R1. At this time, the conversion unit 34 sets the value of the BTH Destination QP of the RDMA transmission packet P input from the duplicating unit 20 to the value of the dQPN in the conversion table 31.

[0045] Next, a description will be given of the PSN conversion performed by the conversion unit 34. The PSN conversion method differs between the first RDMA transmission packet P received first after the connection is established and the RDMA transmission packets P received thereafter.

[0046] When a first RDMA transmission packet P is input for the first time after the connection is established, the conversion unit 34 sets the PSN of the BTH of the first RDMA transmission packet P to the Local PSN of the conversion table 31 and to the Local PSN in the first row of the history table 32. The conversion unit 34 converts the value of the PSN of the BTH of the first RDMA transmission packet P to the value of the Remote PSN of the conversion table 31, specifically, the value of the Starting PSN acquired from the first remote terminal R1. At this time, the conversion unit 34 sets the Destination QP of the BTH of the RDMA transmission packet P to the Local dQPN of the conversion table 31.

[0047] When a second RDMA transmission packet P is input after a connection is established and a first RDMA transmission packet P is input, the conversion unit 34 increments each of the Local PSN and Remote PSN in the conversion table 31 and sets the incremented values as the Local PSN and Remote PSN in the second row of the history table 32. The conversion unit 34 converts the PSN value of the BTH of the second RDMA transmission packet P to the Remote PSN value in the conversion table 31, specifically, to a PSN value obtained by incrementing the Starting PSN acquired from the first remote terminal R1.

[0048] The conversion unit 34 updates the Local PSN in the conversion table 31 to a value incremented according to the number of RDMA transmission packets P input after the connection is established. The conversion unit 34 sets the updated Local PSN to the Local PSN of the row in the history table 32 corresponding to the number of RDMA transmission packets P input after the connection is established. As shown in FIG. 5(a), when the first RDMA transmission packet P is input after the connection is established, the Local PSN in the first row is set to 0X4444, which is the PSN of the BTH of the first RDMA transmission packet P. When the second RDMA transmission packet P is input after the connection is established, the Local PSN in the second row is set to 0X4445, which is an increment of 0X4444. When the third RDMA transmission packet P is input, the Local PSN in the third row is set to 0X4446, which is an increment of 0X4445.

[0049] The conversion unit 34 updates the Remote PSN in the conversion table 31 to a value incremented according to the number of RDMA transmission packets P input after the connection is established. The conversion unit 34 sets the updated Remote PSN as the Remote PSN in the row of the history table 32 corresponding to the number of RDMA transmission packets P input after the connection is established. As shown in FIG. 5(a), the Remote PSN in the first row of the history table 32 is set to 0X2222, which is the Starting PSN acquired from the remote terminal R by REP when the connection with the first remote terminal R1 is established. When the first RDMA transmission packet P is input after the connection is established, the Remote PSN in the history table 32 is already set to a value. When the second RDMA transmission packet P is input after the connection is established, the Remote PSN in the second row is set to 0X2223, which is obtained by incrementing 0X2222. When the third RDMA transmission packet P is input, the Remote PSN in the third row is set to 0X2224, which is an increment of 0X2223.

[0050] The conversion unit 34 may refer to the BTH Destination QP of the RDMA transmission packet input from the duplication unit 20 to determine whether to process the RDMA transmission packet. If the BTH Destination QP of the second RDMA transmission packet P matches the BTH Destination QP of the first RDMA transmission packet P, the conversion unit 44 transmits the converted second RDMA transmission packet to the first remote terminal. If they do not match, the conversion unit 34 discards the second RDMA transmission packet P. If the BTH Destination QP of the newly received RDMA transmission packet P is set to the same value as that of the previously received RDMA transmission packet P, the newly received RDMA transmission packet P is determined to be a valid packet transmitted from the same source as the previously received RDMA transmission packet P. If a different value is set, the newly received RDMA transmission packet P is determined to be an invalid packet transmitted from a source different from that of the previously received RDMA transmission packet P, and is discarded.

[0051] The second remote side control unit 40 generates an RDMA transmit packet P2 by converting the header of the input RDMA transmit packet P, and transmits the RDMA transmit packet P2 to the second remote terminal R2. As shown in FIG. 3, the second remote side control unit 40 includes a conversion table 41, a history table 42, an establishment unit 43, and a conversion unit 44. Each piece of data is stored in a storage device such as a memory 902 or a storage 903. Each function is implemented in a CPU 901.

[0052] As shown in Fig. 4(b), the conversion table 41 has the same data configuration as the conversion table 31 of the first remote-side control unit 30. As shown in Fig. 5(b), the history table 42 has the same data configuration as the history table 32 of the first remote-side control unit 30. The establishment unit 43 and the conversion unit 44 have the same functions as the establishment unit 33 and the conversion unit 34 of the first remote-side control unit 30, respectively.

[0053] The establishment unit 43 establishes a connection with the second remote terminal R2. When establishing the connection, the establishment unit 43 acquires a QPN and a Starting PSN from the second remote terminal R2. The establishment unit 43 sets the acquired QPN as the dQPN in the conversion table 41. The establishment unit 43 sets the Starting PSN as the Remote PSN in the conversion table 41 and as the Remote PSN in the first row of the history table 42. The establishment unit 43 sets the Source IP address and Source MAC address of the first remote terminal R1 as the IP address and MAC address in the conversion table 41.

[0054] The conversion unit 44 converts the BTH Destination QP and PSN of the RDMA transmit packet P input from the duplicating unit 20. The conversion unit 44 converts the Source IP address and Source MAC address into the IP address and MAC address of the second remote side control unit 40. The conversion unit 44 converts the Destination IP address and Destination MAC address into the IP address and MAC address registered in the conversion table 41, specifically, the IP address and MAC address of the second remote terminal R2. The conversion unit 44 transmits the converted RDMA transmit packet P2 to the second remote terminal R2.

[0055] The conversion of the BTH Destination QP and PSN of the RDMA transmission packet P will be explained.

[0056] The conversion unit 44 converts the value of the Destination QP of the BTH of the RDMA transmission packet P input from the duplicating unit 20 into the value of the dQPN of the conversion table 41, specifically, the QPN acquired from the second remote terminal R2. At this time, the conversion unit 44 sets the value of the Destination QP of the BTH of the RDMA transmission packet P input from the duplicating unit 20 to the value of the dQPN of the conversion table 41.

[0057] When a first RDMA transmission packet P is input for the first time after the connection is established, the conversion unit 44 sets the PSN of the BTH of the first RDMA transmission packet P to the Local PSN of the conversion table 41 and the Local PSN in the first row of the history table 42. The conversion unit 44 converts the value of the PSN of the BTH of the first RDMA transmission packet P to the value of the Remote PSN of the conversion table 41, specifically, the value of the Starting PSN acquired from the second remote terminal R2. At this time, the conversion unit 44 sets the Destination QP of the BTH of the RDMA transmission packet P to the Local dQPN of the conversion table 41.

[0058] When a second RDMA transmission packet P is input after a connection is established and a first RDMA transmission packet P is input, the conversion unit 44 increments each of the Local PSN and Remote PSN in the conversion table 41 and sets the incremented values as the Local PSN and Remote PSN in the second row of the history table 42. The conversion unit 44 converts the PSN value of the BTH of the second RDMA transmission packet P to the Remote PSN value in the conversion table 41, specifically, to a PSN value obtained by incrementing the Starting PSN acquired from the second remote terminal R2.

[0059] The conversion unit 44 updates the Local PSN and Remote PSN in the conversion table 41 to values incremented according to the number of RDMA transmission packets P input after the connection was established. The conversion unit 44 sets the updated Local PSN and Remote PSN as the Local PSN and Remote PSN in the row of the history table 42 corresponding to the number of RDMA transmission packets P input after the connection was established.

[0060] (Connection established) The connection establishment process in the processing system 5 will be described with reference to FIGS.

[0061] First, a connection is established between the local terminal L and the local side control unit 10. In step S11, the local terminal L sends a REQ to the local side control unit 10. The REQ includes the Local QPN and Starting PSN of the local terminal L. In step S12, the local side control unit 10 sends a REP. The REP includes the Local QPN and Starting PSN of the local side control unit 10. In step S13, the local terminal L sends an RTU. In step S14, a connection is established between the local terminal L and the local side control unit 10.

[0062] Next, a connection is established between the first remote side control unit 30 and the first remote terminal R1. In step S21, the first remote side control unit 30 sends a REQ to the first remote terminal R1. The REQ includes the Local QPN and Starting PSN of the first remote side control unit 30. In step S22, the first remote terminal R1 sends a REP. The REP includes the Local QPN and Starting PSN of the first remote terminal R1.

[0063] In step S23, the first remote-side control unit 30 updates the conversion table 31 and the history table 32 using the Local QPN and Starting PSN included in the REP. The first remote-side control unit 30 registers the Local QPN received in step S22 as the dQPN in the conversion table 31. The first remote-side control unit 30 registers the Starting PSN received in step S22 as the Remote PSN in the conversion table 31 and as the Remote PSN in the first row of the history table 32. The first remote-side control unit 30 further sets the Source IP address and Source MAC address included in the REP as the IP address and MAC address in the conversion table 31.

[0064] In step S24, the first remote control unit 30 transmits the RTU. In step S25, a connection is established between the first remote control unit 30 and the first remote terminal R1.

[0065] A connection is also established between the second remote side control unit 40 and the second remote terminal R2. In step S31, the second remote side control unit 40 sends a REQ to the second remote terminal R2. The REQ includes the Local QPN and Starting PSN of the second remote side control unit 40. In step S32, the second remote terminal R2 sends a REP. The REP includes the Local QPN and Starting PSN of the second remote terminal R2.

[0066] In step S33, the second remote-side control unit 40 uses the Local QPN and Starting PSN included in the REP to update the conversion table 41 and the history table 42. The second remote-side control unit 40 registers the Local QPN received in step S32 as the dQPN in the conversion table 41. The second remote-side control unit 40 registers the Starting PSN received in step S32 as the Remote PSN in the conversion table 41 and as the Remote PSN in the first row of the history table 42.

[0067] In step S34, the first remote control unit 30 transmits the RTU. In step S35, a connection is established between the second remote control unit 40 and the second remote terminal R2.

[0068] (Data Transfer) The data transfer process in the processing system 5 will be described with reference to FIG.

[0069] In step S51, the local terminal L transmits an RDMA transmission packet P, which is received by the local-side control unit 10. In step S52, the local-side control unit 10 transmits the RDMA transmission packet P to the duplicating unit 20.

[0070] The duplicating unit 20 transmits the received RDMA transmission packet P to the first remote-side control unit 30 in step S53, and to the second remote-side control unit 40 in step S57.

[0071] When the first remote side control unit 30 receives the RDMA transmit packet P, it updates the conversion table 31 and the history table 32 in step S54. The first remote side control unit 30 sets the Destination QP of the BTH of the RDMA transmit packet P input from the duplicating unit 20 to the Local dQPN of the conversion table 31. If the received RDMA transmit packet P is the first RDMA transmit packet received after the connection is established, the first remote side control unit 30 sets the PSN of the BTH of the received RDMA transmit packet P to the Local PSN of the conversion table 31 and the Local PSN in the first row of the history table 32. If the received RDMA transmit packet P is the second or subsequent RDMA transmit packet P received after the connection is established, the first remote side control unit 30 updates the Local PSN and Remote PSN of the conversion table 31 to values incremented according to the number of RDMA transmit packets P input after the connection is established. The first remote-side control unit 30 sets the updated Local PSN and Remote PSN as the Local PSN and Remote PSN in the row of the history table 32 corresponding to the number of RDMA transmission packets P input after the connection was established.

[0072] In step S55, the first remote side control unit 30 refers to the updated conversion table 31 and converts the header of the input RDMA transmit packet P to generate an RDMA transmit packet P1. The first remote side control unit 30 sets the Destination QP of the BTH of the RDMA transmit packet P input from the duplicating unit 20 to the Local dQPN of the conversion table 31. The first remote side control unit 30 converts the value of the PSN of the BTH of the RDMA transmit packet P to the value of the Remote PSN of the conversion table 31. The first remote side control unit 30 converts the Source IP address and the Source MAC address to the IP address and the MAC address of the first remote side control unit 30. The first remote side control unit 30 converts the Destination IP address and the Destination MAC address to the IP address and the MAC address registered in the conversion table 31, specifically, the IP address and the MAC address of the first remote terminal R1.

[0073] In step S56, the first remote-side control unit 30 transmits the RDMA transmission packet P1, the header of which has been converted in step S55, to the first remote terminal R1.

[0074] In steps S58 to S60, the same processing as in steps S54 to S56 is performed. When the second remote side control unit 40 receives the RDMA transmit packet P, it updates the conversion table 41 and the history table 42 in step S58. In step S59, the second remote side control unit 40 refers to the updated conversion table 41 and converts the header of the input RDMA transmit packet P to generate an RDMA transmit packet P2. In step S60, the second remote side control unit 40 transmits the RDMA transmit packet P2, whose header has been converted in step S59, to the second remote terminal R2.

[0075] Referring to Figure 9, an example of the header of the RDMA transmit packet P transmitted from the local terminal L in step S51 of Figure 8 and an example of the header of the RDMA transmit packet P1 whose header has been converted by the first remote side control unit 30 in step S55 will be described.

[0076] 9(a) is an example of the header of an RDMA transmit packet P transmitted from a local terminal L. In the RDMA transmit packet P, the MAC address, IP address, and UDP port number of the local terminal L are set in the Src (Source) MAC address, IP address, and UDP port number. The MAC address, IP address, and UDP port number of the Dst (Destination) MAC address, IP address, and UDP port number of the local-side control unit 10 are set in the Dst (Destination) MAC address, IP address, and UDP port number. The QPN of the local-side control unit 10 is set in the dQPN. The PSN of the local terminal L is set in the PSN.

[0077] 9(b) is an example of the header of an RDMA transmit packet P1 whose header has been converted by the first remote-side control unit 30. In the RDMA transmit packet P1, the MAC address, IP address, and UDP port number of the first remote-side control unit 30 are set in the Src (Source) MAC address, IP address, and UDP port number. The MAC address, IP address, and UDP port number of the Dst (Destination) are set in the Dst (Destination) MAC address, IP address, and UDP port number. The QPN of the first remote terminal R1 is set in the dQPN. The PSN of the first remote terminal R1 is set in the PSN.

[0078] 9(a) and 9(b), a randomly determined number is set as the source UDP port number. When an RDMA connection is established using the RoCEv2 mechanism, a fixed number is set as the destination UDP port number. Therefore, in the RDMA transmit packet P2, the number assigned to the first remote-side control unit 30 is set as the source UDP port number, and the same number "4791" as that of the RDMA transmit packet P1 is set as the destination UDP port number.

[0079] The process of the establishing unit 33 of the first remote-side control unit 30 will be described with reference to FIG.

[0080] In step S101, the establishing unit 33 transmits a REQ to the first remote terminal R1. In step S102, the establishing unit 33 receives a REP from the first remote terminal R1.

[0081] In step S103, the establishment unit 33 sets the value acquired from the header of the REP in the conversion table 31 and the history table 32. Specifically, as shown in FIG. 11 , the establishment unit 33 sets the Source IP address in the IP header of the REP to the IP address in the conversion table 31. The establishment unit 33 sets the Source MAC address in the Eth header to the MAC address in the conversion table 31. The establishment unit 33 sets the Local QPN in the RDMACM header to the dQPN in the conversion table 31. The establishment unit 33 sets the Starting PSN in the RDMACM header to the Remote PSN in the conversion table, and further sets it to the Remote PSN in the first row of the history table 32.

[0082] When the setting of the conversion table 31 and the history table is completed, in step S104 the establishing unit 33 transmits an RTU to the first remote terminal R1.

[0083] The processing of the conversion unit 34 of the first remote-side control unit 30 will be described with reference to FIG.

[0084] In step S151, the conversion unit 34 receives the RDMA transmission packet P from the duplicating unit 20. In step S152, the conversion unit 34 determines whether the RDMA transmission packet P is the first RDMA transmission packet received after the connection is established.

[0085] If it is determined in step S152 that this is the first reception, the process proceeds to step S153. In step S153, the conversion unit 34 sets the conversion table 31 and the history table 32. Specifically, as shown in FIG. 13, the conversion unit 34 sets the Destination QP of the BTH of the RDMA transmit packet P to the Local dQPN in the conversion table 31. The conversion unit 34 sets the PSN of the BTH to the PSN in the conversion table, and also sets it to the Local PSN in the first row of the history table 32. After the settings, the process proceeds to step S158.

[0086] If it is determined in step S152 that the packet is not the first received packet, the process proceeds to step S154. In step S154, the conversion unit 34 compares the Destination QP of the BTH of the received packet with the Local dQPN in the conversion table 31, and determines whether they match in step S155. If they do not match, in step S156 the conversion unit 34 determines that the source of the received packet is not the local terminal L, drops the packet, and ends the process.

[0087] If it is determined in step S152 that the packet is not the first received packet, and if it is determined in step S155 that the Destination QP of the BTH of the received packet matches the Local dQPN in the conversion table 31, then in step S157 the conversion unit 34 updates the conversion table 31 and the history table 32. Specifically, as shown in FIG. 14 , the conversion unit 34 updates the current value of the Local PSN in the conversion table 31 by incrementing it by one. The conversion unit 34 updates the current value of the Remote PSN in the conversion table 31 by incrementing it by one. The conversion unit 34 sets the incremented Local PSN and Remote PSN in the conversion table 31 to the Local dQPN and Remote PSN, respectively, in the row of the history table 32 corresponding to the number of packets received after the connection was established.

[0088] In step S158, the conversion unit 34 converts the header of the received RDMA transmit packet P to generate an RDMA transmit packet P1. The conversion unit 34 sets the Destination QP of the BTH of the RDMA transmit packet P input from the duplicating unit 20 to the Local dQPN in the conversion table 31. The conversion unit 34 converts the PSN value of the BTH of the RDMA transmit packet P to the Remote PSN value in the conversion table 31. The conversion unit 34 converts the Source IP address and Source MAC address to the IP address and MAC address of the first remote-side control unit 30. The conversion unit 34 converts the Destination IP address and Destination MAC address to the IP address and MAC address registered in the conversion table 31, specifically the IP address and MAC address of the first remote terminal R1.

[0089] In step S159, the conversion unit 34 transmits the converted RDMA transmission packet P1 to the first remote terminal R1.

[0090] In the processing system 5 according to the embodiment of the present invention, the processing device 1 generates and transmits an RDMA transmit packet P1 addressed to the first remote terminal R1 and an RDMA transmit packet P2 addressed to the second remote terminal R2 from an RDMA transmit packet P transmitted from the local terminal L. Regardless of the number of remote terminals R, the local terminal L only needs to generate and transmit the RDMA transmit packet P addressed to the processing device 1, thereby reducing the burden on the local terminal L.

[0091] (Variation) In the processing system 5 according to the embodiment of the present invention, the local terminal L, the processing device 1, the first remote terminal R1, and the second remote terminal R2 are implemented on physically different computers, thereby reducing the burden on the local terminal that transfers data to multiple remote terminals. More specifically, in the processing system 5, the duplication unit 20 is implemented on a computer that is physically or virtually different from the local terminal L, the first remote terminal R1, and the second remote terminal R2, thereby reducing the burden on the local terminal L that transfers data to multiple remote terminals R.

[0092] The functions of the local-side control unit 10, the duplication unit 20, the first remote-side control unit 30, and the second remote-side control unit 40 of the processing device 1 may be implemented on different computers. Furthermore, each function may be implemented on a computer having other functions. For example, the local-side control unit 10 of the processing device 1 may be implemented on a network interface card (NIC) of the local terminal L, the first remote-side control unit 30 may be implemented on a NIC of the first remote terminal R1, or the second remote-side control unit 40 may be implemented on a NIC of the second remote terminal R2.

[0093] In the embodiment of the present invention, the duplicating unit 20 is described as being implemented as a function of a computer, but this is not limiting. The duplicating unit 20 may also be implemented as a function of a communication control device. In this case, packets may be duplicated by electrical or optical processing. In the case of duplication by electrical processing, a device such as an IP router's multicast function, a packet broker, a network tap, or an L2 switch's port mirroring function electrically converts signals into data and duplicates the electrically converted data. In the case of duplication by optical processing, a device such as an optical splitter or an optical tap splits signals while maintaining the optical physical phenomenon.

[0094] As described above, the processing system 5 can be implemented in a variety of forms.

[0095] The processing device 1 of the present embodiment described above is, for example, a general-purpose computer system including a CPU (Central Processing Unit, processor) 901, a memory 902, a storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), a communication device 904, an input device 905, and an output device 906. In this computer system, the CPU 901 executes a program loaded on the memory 902, thereby realizing each function of the processing device 1.

[0096] The processing device 1 may be implemented in one computer or in multiple computers, or may be a virtual machine implemented in a computer.

[0097] The program of the processing device 1 can be stored on a computer-readable recording medium such as an HDD, SSD, USB (Universal Serial Bus) memory, CD (Compact Disc), or DVD (Digital Versatile Disc), or can be distributed via a network.

[0098] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the present invention. [Explanation of symbols]

[0099] 1 Processing equipment 5 Processing System 10 Local side control unit 20 Reproduction Department 30, 40 Remote control unit 31,41 Conversion Table 32,42 History Table 33,43 Establishment Department 34,44 conversion unit 901 CPU 902 memory 903 Storage 904 Communication equipment 905 Input Device 906 Output Device L local terminal R Remote Terminal

Claims

1. a local terminal, a first remote terminal, a second remote terminal, and a processing device; The local terminal Sending an RDMA transmission packet to the processing device, the RDMA transmission packet containing processing data to be transferred to a memory of each accelerator of the first remote terminal and the second remote terminal; The processing device includes: a local-side control unit that establishes a connection with the local terminal and receives the RDMA transmission packet from the local terminal; a first remote-side control unit that establishes a connection with the first remote terminal; a second remote side control unit that establishes a connection with the second remote terminal; a duplicating unit that inputs the RDMA transmission packet to the first remote-side control unit and the second remote-side control unit; The first remote control unit includes: Upon establishing a connection, obtain a QPN from the first remote terminal; Converting a Destination QP of a Base Transport Header (BTH) of the RDMA transmission packet to a QPN obtained from the first remote terminal, and transmitting the converted RDMA transmission packet to the first remote terminal; The second remote control unit includes: Upon establishing a connection, obtain a QPN from the second remote terminal; Converting a Destination QP of the BTH of the RDMA transmission packet into a QPN obtained from the second remote terminal, and transmitting the converted RDMA transmission packet to the second remote terminal; The first remote terminal receiving the converted RDMA transmit packet and transferring the processed data to a memory of an accelerator; The second remote terminal receiving the converted RDMA transmit packet and transferring the processed data to a memory of an accelerator; Processing system.

2. a local-side control unit that establishes a connection with a local terminal and receives, from the local terminal, an RDMA transmission packet in which processing data to be transferred to memories of accelerators of the first remote terminal and the second remote terminal is set; a first remote-side control unit that establishes a connection with the first remote terminal; a second remote side control unit that establishes a connection with the second remote terminal; a duplicating unit that inputs the RDMA transmission packet to the first remote-side control unit and the second remote-side control unit; The first remote control unit includes: Upon establishing a connection, obtain a QPN from the first remote terminal; Converting a Destination QP of the BTH of the RDMA transmission packet into a QPN obtained from the first remote terminal, and transmitting the converted RDMA transmission packet to the first remote terminal; The second remote control unit includes: Upon establishing a connection, obtain a QPN from the second remote terminal; Converting the Destination QP of the BTH of the RDMA transmission packet into the QPN obtained from the second remote terminal, and transmitting the converted RDMA transmission packet to the second remote terminal. Processing equipment.

3. The first remote control unit includes: Upon establishing a connection, obtain a Starting PSN from the first remote terminal; After the connection is established, when a first RDMA transmission packet is input, the PSN of the BTH of the first RDMA transmission packet is converted into a Starting PSN obtained from the first remote terminal, and the converted first RDMA transmission packet is transmitted to the first remote terminal. The processing device according to claim 2 .

4. The first remote control unit includes: After the connection is established and the first RDMA transmission packet is input, when a second RDMA transmission packet is input, the PSN of the BTH of the second RDMA transmission packet is converted into a PSN obtained by incrementing the Starting PSN acquired from the first remote terminal, and the converted second RDMA transmission packet is transmitted to the first remote terminal. The processing device according to claim 3 .

5. The first remote control unit includes: If a BTH Destination QP of the second RDMA transmission packet matches a BTH Destination QP of the first RDMA transmission packet, transmitting the converted second RDMA transmission packet to the first remote terminal; If they do not match, discard the second RDMA transmission packet. The processing device according to claim 4 .

6. The local terminal transmits, to the processing device, an RDMA transmission packet containing processing data to be transferred to the memory of each accelerator of the first remote terminal and the second remote terminal; the processing device establishes a connection with the local terminal and receives the RDMA transmission packet from the local terminal; a first remote side control unit of the processing device establishes a connection with the first remote terminal; a second remote side control unit of the processing device establishes a connection with the second remote terminal; the processing device inputs the RDMA transmission packet to the first remote side control unit and the second remote side control unit; the first remote side control unit of the processing device, when establishing a connection, acquires a QPN from the first remote terminal, converts a Destination QP of a BTH (Base Transport Header) of the RDMA transmission packet to the QPN acquired from the first remote terminal, and transmits the converted RDMA transmission packet to the first remote terminal; the second remote side control unit of the processing device, when establishing a connection, acquires a QPN from the second remote terminal, converts a BTH Destination QP of the RDMA transmission packet to the QPN acquired from the second remote terminal, and transmits the converted RDMA transmission packet to the second remote terminal; the first remote terminal receives the converted RDMA transmission packet and transfers the processed data to a memory of an accelerator; The second remote terminal receives the converted RDMA transmission packet and transfers the processed data to a memory of an accelerator. Processing method.

7. A program for causing a computer to function as the processing device according to any one of claims 2 to 5.

Images