Communication system, transmission device, and reception device

Abstract

This communication system transfers data between a local application 50A running on a local node 100A and a remote application 50B running on a remote node 100B. Upon receiving a request from the local application 50A to transmit data, the local node 100A acquires a transmission time stamp, imparts the transmission time stamp to the data, and sends out the data. The remote node 100B receives the data, acquires a reception time stamp when notifying the remote application 50B of the completion of data reception, holds a transmission / reception history including the transmission time stamp and the reception time stamp, and transmits the transmission / reception history to the local node 100A. The local node 100A and the remote node 100B calculate latency from the difference between the transmission time stamp and the reception time stamp in the held transmission / reception history, and return statistical information of the latency.

Description

Communication system, transmitting device, and receiving device

[0001] The present disclosure relates to a communication system, a transmitting device, and a receiving device.

[0002] Remote Direct Memory Access (RDMA) enables data transfer without going through the operating system (OS), and can achieve high-throughput, low-latency communications (Non-Patent Document 1). Because RDMA was developed for high-performance computing, it is increasingly being used in cluster configurations that connect nodes over short distances and within data centers.

[0003] In use cases where processing must be completed at a specific time, such as controlling manufacturing robots and conveyor belts in factories, and large-volume data backups for disaster recovery where the end time is uncertain, it is important not only to achieve low-latency communication, but also to understand the latency and statistical information obtained using the latency (jitter, minimum value, percentile value, etc.).

[0004] In this disclosure, latency refers to the difference in time between when a sending application prepares data in memory and is ready to send it, and when a receiving application receives all the data and is ready to process it. In other words, latency is the time it takes for data to be transmitted from a sending application to a receiving application, and indicates the time it takes from when the sending application completes preparations for data transmission, to when the data is sent via a network interface card (NIC), and until the receiving application completes reception of all the data via the receiving NIC.

[0005] Generally, there are two methods for measuring latency: a method using Round Trip Time (RTT) and a method using timestamps.

[0006] In the RTT measurement method, data sent from the sender is returned from the receiver to the sender. The sender measures the RTT from the difference between the start time of transmission and the end time of reception. A specific example is a method using ping, which uses ICMP.

[0007] In the timestamp measurement method, when the sender sends data, it assigns a transmission timestamp to the data and sends it. After the receiver receives the data, it saves the reception timestamp and measures the latency from the difference between the transmission timestamp and the reception timestamp. In this case, the reference time must be synchronized between the sender and receiver.

[0008] David.A.Deming, “InfiniBand Architecture Overview”, https: / / www.snia.org / sites / default / files / files2 / files2 / SDC2013 / presentations / Hardware / DavidDeming_Infiniband_Architectural_Overview.pdf, Storage Developer Conference, 2023.NVIDIA DOCS, “Time-Stamping”, https: / / docs.nvidia.com / networking / display / ofedv502180 / time-stamping#src-37849150_TimeStamping-RoCETime-Stamping, Last updated on Oct 23, 2023.

[0009] The RTT measurement method requires that the route from the sender to the receiver be the same for both the outbound and return routes, but since the outbound and return routes are never exactly the same, it is an approximate measurement. Therefore, half the RTT value cannot be used as an accurate latency.

[0010] When measuring latency between a sending application and a receiving application using a timestamp measurement method, each application must have a way to record the timestamp. There are NICs that can acquire a timestamp just before sending or just after receiving data, but this timestamp does not include the time from when the application sends the data to when the data is sent by the NIC, or the time from when the NIC receives the data to when it is passed to the application, so it is not possible to measure the latency between applications accurately. Another issue is that it is device-dependent.

[0011] The timestamp measurement method allows the receiving side to understand the latency, but in order for the sending side to understand the latency, it needs to receive a received timestamp from the receiving side, and each application must have a way to feed back the timestamp.

[0012] Furthermore, even if the reference time is synchronized between the sending side and the receiving side, it is not possible to verify whether the sending timestamp and the receiving timestamp are recorded based on the same reference clock.

[0013] In RDMA, there are transport types that do not perform recovery such as retransmission. If data with a timestamp is lost, it is not possible to obtain latency based on that data.

[0014] The present disclosure has been made in view of the above, and aims to more accurately measure the latency of data transfer between applications.

[0015] A communication system according to one aspect of the present disclosure is a communication system for transferring data between a transmitting device and a receiving device, wherein the transmitting device acquires transmission attribute information when starting a data transmission process, attaches the transmission attribute information to the data and sends it out, the receiving device acquires reception attribute information when completing reception of the data, retains a transmission / reception history including the transmission attribute information and the reception attribute information, and transmits the transmission / reception history to the transmitting device, and the transmitting device and the receiving device return statistical information acquired using the transmission attribute information and the reception attribute information that they retain.

[0016] According to the present disclosure, the latency of data transfer between applications can be measured more accurately.

[0017] FIG. 1 is a diagram illustrating an example of the configuration of a communication system. FIG. 2 is a sequence diagram illustrating an example of the processing flow of the communication system. FIG. 3 is a sequence diagram illustrating an example of the processing flow of the communication system. FIG. 4 is a diagram illustrating processing for saving data in an area where metadata is written. FIG. 5 is a diagram illustrating processing for saving data in an area where metadata is written. FIG. 6 is a sequence diagram illustrating an example of the processing flow of the communication system. FIG. 7 is a sequence diagram illustrating an example of the processing flow of the communication system. FIG. 8 is a diagram illustrating a memory size required when assigning metadata to each piece of divided data. FIG. 9 is a diagram illustrating an example of processing for dividing data. FIG. 10 is a diagram illustrating an example of processing for dividing data. FIG. 11 is a diagram illustrating an example of processing for integrating divided data. FIG. 12 is a diagram illustrating an example of processing for dividing data. FIG. 13 is a diagram illustrating an example of the hardware configuration of a timestamp system and a node.

[0018] [About RDMA] RDMA is a technology that allows a local node to write data directly to the memory of a remote node, and a local node to read data directly from the memory of a remote node. In TCP / IP communication, the OS processes various types of data before passing the data as packets to the NIC for transmission and reception. On the other hand, RDMA offloads the data processing previously performed by the OS to a smart NIC.

[0019] RDMA can be used over lines such as InfiniBand and Ethernet. The present disclosure is applicable to any lower-level communication protocol that implements RDMA. In the embodiments, RDMA over Converged Ethernet version 2 (RoCEv2) is used as the basis for implementing RDMA over Ethernet, but InfiniBand and iWARP can also be used.

[0020] RDMA transport types include Reliable Connection (RC), Unreliable Connection (UC), and Unreliable Datagram (UD). RDMA selects one of the transport types, RC, UC, or UD, to transfer data. RC and UC establish a connection, while UD does not. RC performs error recovery (retransmission), while UC and UD do not. In RoCEv2, data is ultimately sent and received between the local node and the remote node as UDP packets. In other words, the transport type can be considered a virtual connection established on top of UDP.

[0021] RDMA data transmission methods (hereinafter referred to as operation types) include SEND (which is almost identical to SEND with immediate, so there is no distinction and it will be referred to as SEND hereinafter), WRITE, WRITE with immediate, READ, and ATOMIC. The operation types that can be used for each transport type are different. Specifically, all operation types can be used with RC, SEND, WRITE, and WRITE with immediate can be used with UC, and only SEND can be used with UD.

[0022] Also, the direction of data transmission and whether or not a completion notification is sent differ depending on the operation type. For example, SEND, WRITE, and WRITE with immediate send data from the local node to the remote node, while READ sends data from the remote node to the local node. WRITE and READ do not send a completion notification to the remote node.

[0023] Next, we will explain the basic operation of RDMA data transfer. Before sending or receiving data, control information must be exchanged between the application on the local node (hereinafter referred to as the local app) and the application on the remote node (hereinafter referred to as the remote app). For example, the RDMA Communication Manager (CM) can be used to exchange control information.

[0024] In RDMA, a Queue Pair (QP), which is a pair of a Send Queue (SQ) and a Receive Queue (RQ), is configured between the local node and the remote node, and data is transferred using the QP. The unit of communication in RDMA is a communication request called a Work Request (WR). WRs are divided into send requests and receive requests. For send requests, the WR is submitted to the SQ. For receive requests, the WR is submitted to the RQ. The WR submitted to the SQ / RQ is called a Work Queue Element (WQE).

[0025] The NIC retrieves the WQE in order according to the FIFO, and performs data transmission or reception processing using the control information written in the WQE (start address of the memory data to be transferred, memory address length, etc.).

[0026] When data transmission or reception is completed according to the WQE, a Completion Queue Entry (CQE) is placed in the Completion Queue (CQ). You can check whether transmission or reception is complete by checking the CQE stored in the CQ. The extracted CQE is called a Work Completion (WC). For example, in the case of a send request, the WC contains a status code indicating whether the transmission was successful or not, and in the case of a receive request, it contains a status code indicating whether the reception was successful or not, as well as the size of the received data.

[0027] [Device Configuration] Referring to FIG. 1, an example of the configuration of a communication system including a local node 100A and a remote node 100B that transfer data using RDMA is described. The local node 100A and the remote node 100B shown in the figure include hardware such as a NIC 11, a CPU 12, and a memory 13. They operate using software such as an OS 21, a library 22 (the library is described as software that directly requests data transmission from the NIC; however, if this function is provided in this communication system, it can operate even without the library), and a time synchronization function 23. A timestamp system including a message processing unit 31, a memory management unit 32, a statistical information management unit 33, a session control unit 34, and a time reference unit 35 operates. A local application 50A is executed on the local node 100A, and a remote application 50B is executed on the remote node 100B. By transferring data between the local application 50A and the remote application 50B via the timestamp system, the latency of data transmission can be obtained. Each component of the timestamp system is described below.

[0028] The message processing unit 31 receives a request from the local application 50A to send or read a message (a block of data to be transferred via RDMA), and executes message sending or reading processing. When the message processing unit 31 receives a data sending request from an application, it acquires a sending timestamp and assigns metadata (also called attribute information) including the sending timestamp to the data. When data reception is complete, that is, when the message processing unit 31 notifies the application that data reception is complete, it acquires a receiving timestamp. If necessary, the message processing unit 31 divides and combines messages.

[0029] The memory management unit 32 secures and manages the memory area requested by the local application 50A. The memory management unit 32 secures memory of a size equal to the requested size plus the size of the necessary metadata. The memory management unit 32 manages the secured memory area using a memory ID.

[0030] The statistical information management unit 33 stores a data transmission / reception history, obtains latency information and a data loss rate from the transmission / reception history, and manages the information as statistical information. The statistical information management unit 33 returns the statistical information in response to an inquiry from the local application 50A or the remote application 50B.

[0031] The session control unit 34 establishes a control information exchange session for exchanging control information other than data between the local application 50A and the remote application 50B.

[0032] In response to a request from the message processing unit 31, the time reference unit 35 acquires and returns a timestamp and the clock ID of the reference clock that is the basis for the timestamp.

[0033] [Example 1] The processing flow of the communication system of this embodiment will be described below using several examples that differ in RDMA operation type and whether or not message splitting is used. In the examples, time synchronization and session establishment are set as follows. The protocol used for time synchronization between the local node 100A and the remote node 100B is Precision Time Protocol (PTP), and the clock source is the HWC 111 provided in the NIC 11. Time synchronization is performed with the local node 100A as the master and the remote node 100B as the slave. To establish a session for exchanging control information, rdma-cm is used with the local node 100A as the client and the remote node 100B as the server. However, rdma-cm does not necessarily have to be used for the control information exchange described below, and other alternative functions may be used.

[0034] An example of the processing flow of the first embodiment will be described with reference to the sequence diagrams of Figures 2 and 3. In the first embodiment, the operation type is SEND and no message division is performed. Any of RC, UC, and UD may be used as the transport type.

[0035] In step S11, the local node 100A and the remote node 100B synchronize their time between the nodes. In this embodiment, the local node 100A provides a reference clock, and time synchronization is performed between the local node 100A and the remote node 100B. The local node 100A distributes the reference clock in accordance with PTP defined in IEEE1588. However, PTP is merely an example, and other protocols such as NTP may also be used. The time synchronization function 23 of the remote node 100B reflects the reference clock in the HWC 111.

[0036] The time synchronization function 23 manages the clock ID of the reference clock reflected in the HWC 111 in an internal database. In this embodiment, the clock ID is an ID that indicates the HWC 111 of the local node 100A (master). After the reference clock is reflected in the HWC 111, each node synchronizes the time of the system clock managed by the OS 21 based on the HWC 111 for which time synchronization has been completed. The above example is an example of time synchronization based on the HWC 111 of the local node 100A. However, it is also possible to use the system clock managed by the OS 21 as a reference, reflect the system clock in the HWC 111, and then use that HWC 111 to reflect the system clock in the local node 100A. Furthermore, in this embodiment, time synchronization is performed before other procedures, but it is sufficient if it is performed before the transmission timestamp is required.

[0037] In step S12, the session control unit 34 of each node establishes a control information exchange session and exchanges control information as preparation for data transmission.

[0038] In response to a connection request from an application, the session control unit 34 starts establishing a session. The session establishment may be performed by either the local application 50A or the remote application 50B. In this embodiment, rdma-cm is used to establish a session, and the remote application 50B (server) waits and requests a connection from the local application 50A (client). The local application 50A knows the connection destination information (e.g., IP address and port number) of the remote application 50B, and the remote application 50B knows the connection destination information of the local application 50A. Generally, UDP port 4791 is used for RoCEv2 traffic.

[0039] The session control unit 34 of the local node 100A sends a session establishment request to the session control unit 34 of the remote node 100B. After that, a session is established according to the standard procedure defined by rdma-cm. After the session is established, the local application 50A and the remote application 50B exchange the following information in order.

[0040] First, Active MTU (AMTU) information is exchanged. AMTU indicates the maximum size of an RDMA message that can fit into one packet, as configured by the NIC 11 of each node. AMTU can be obtained by querying the device information of the NIC 11. Currently, AMTU is defined as 256, 512, 1024, 2048, or 4096 bytes.

[0041] After exchanging AMTU information, the AMTUs of the NICs 11 of each node are compared, and the smaller AMTU value is adopted as the Path MTU (PMTU). PMTU is the maximum size that a packet can be transmitted over a path without being fragmented.

[0042] Next, metadata structures are exchanged. Metadata is information that is added to the data to be transmitted. Metadata is composed of sub-metadata, such as a transmission timestamp, a clock ID, a sequence number, a transmission message size, and a transmission request ID. Details of the metadata will be described later. When exchanging metadata structures, the numbers and order of the sub-metadata to be used are exchanged. The size of each sub-metadata is assumed to be agreed upon in advance. By exchanging the metadata structures, the offset from the start address of the metadata and the metadata size for obtaining each sub-metadata can be determined. Note that a metadata size of 0 means that no metadata exists.

[0043] Next, whether or not message segmentation is performed is exchanged. Generally, when transferring data using RDMA, if the data size exceeds the PMTU, it is segmented into PMTU pieces and transferred. If the size of the data to be transmitted with metadata added is larger than the PMTU, the message processing unit 31 can segment the data based on the PMTU, and it is possible to select whether or not to segment the data. If the size of the data to be transmitted with metadata added is equal to or smaller than the PMTU, the message processing unit 31 does not segment the data. If message segmentation is performed, the message processing unit 31 on the transmitting side segments the data to be transmitted, and the message processing unit 31 on the receiving side integrates the segmented and transmitted data. Details of message segmentation will be described later. In the first embodiment, message segmentation is not performed, so information indicating no message segmentation is exchanged.

[0044] Next, the maximum transmission data size is exchanged, which is the maximum data size of data that the application requests to be transmitted.

[0045] Finally, a termination signal is exchanged to complete the control information exchange process.

[0046] The RDMA messages used for exchanging control information are, for example, equal to or smaller than the minimum size of PMTU (256 bytes).

[0047] In step S13, the local application 50A requests a memory area by passing the requested memory area size (hereinafter referred to as the user-requested memory size) and the maximum transmission data size to the memory management unit 32. Note that the user-requested memory size is assumed to be larger than the maximum transmission data size. The memory management unit 32 reserves a memory area equal to the combined size of the user-requested memory size and the metadata size (hereinafter referred to as the total memory size), and returns a memory ID that uniquely identifies the reserved memory area and a starting address to the local application 50A. The memory management unit 32 manages the starting address and the total memory size for each memory ID.

[0048] When message division is performed, the number of metadata to be added varies depending on the maximum transmission data size, so the memory management unit 32 reserves a memory area equal to the user-requested memory size and the number of required metadata. Memory allocation when metadata division is performed will be described later.

[0049] In step S14, the remote application 50B reserves a memory area having a size equal to the combined size of the maximum transmission data size and the metadata size so that all data transmitted from the local application 50A can be written.

[0050] When the time synchronization, the exchange of control information, and the allocation of memory areas are completed, the local application 50A and the remote application 50B establish a message transmission / reception session in step S15. Two sessions exist between the local application 50A and the remote application 50B: a control information exchange session and a message transmission / reception session.

[0051] In step S16, the local application 50A stores the data to be sent in the memory area secured in step S13, and sends the start address of the memory area from which transmission is to begin, the transmission data size, and the memory ID to the message processing unit 31, requesting message transmission via RDMA.

[0052] In step S17, upon receiving the message transmission request, the message processing unit 31 of the local node 100A acquires a transmission timestamp and a clock ID via the time reference unit 35. The time reference unit 35 acquires the transmission timestamp by referencing the HWC 111 or the system clock, acquires the clock ID of the reference clock from the time synchronization function 23, and returns the transmission timestamp and the clock ID to the message processing unit 31. Whether the HWC 111 or the system clock is to be referenced may be specified by the message processing unit 31 or may be determined in advance.

[0053] In step S18, the message processing unit 31 of the local node 100A attaches metadata to the data to be transmitted (hereinafter referred to as the transmitted message). In the first embodiment, the message is not divided, and the metadata is attached to the end of the transmitted message. The data in which the metadata is attached to the transmitted message is referred to as the final transmitted message.

[0054] An example of the metadata will now be described. As described above, the metadata in this embodiment is made up of sub-metadata of a transmission timestamp, a clock ID, a sequence number, a transmission message size, and a transmission request ID.

[0055] The transmission timestamp is time information at the time when the local application 50A transmits data. Immediately after receiving the message transmission request from the local application 50A in step S16, the message processing unit 31 acquires the transmission timestamp in step S17.

[0056] The clock ID is an ID that uniquely identifies the reference clock that is the base of the transmission timestamp. The clock ID can be acquired together with the transmission timestamp in step S17. The clock ID is a means for identifying the reference clock that is the synchronization source.

[0057] The sequence number is a value that indicates the order of messages when data is divided and transmitted. For example, the sequence number starts from 1 and the final number is -1. In the first embodiment, messages are not divided, so sequence numbers are not included in the metadata.

[0058] The transmission message size is the size of the transmission message contained in the packet.

[0059] The transmission request ID is an ID for uniquely identifying a message transmission request from the local application 50 A. For example, the transmission request ID starts from 1 and is incremented for each message transmission request from the local application 50 A.

[0060] When attaching metadata, the message processing unit 31 temporarily saves the data in the memory area where the metadata is written, and returns the saved data to its original position when transmission is complete. The process of saving data will be described later.

[0061] When message division is performed, the message processing unit 31 divides the data to be transmitted before adding the metadata.

[0062] In step S19 of FIG. 3, the message processing unit 31 of the local node 100A specifies a start address, a memory ID, a final transmission message size, and an operation type, and uses the library 22 to request the NIC 11 to transmit a final transmission message. The library 22 is a library that provides the NIC with a function for making a transmission request via RDMA. The start address of the final transmission message is the same as that of the transmission message, but the final transmission message size is a value obtained by adding the metadata size to the transmission message size. In the first embodiment, SEND is selected as the RDMA operation type. The library 22 issues a transmission request using RoCEv2 to the NIC 11. Thereafter, data is transmitted according to the standard RDMA procedure.

[0063] In step S20, the NIC 11 of the local node 100A uses Direct Memory Access (DMA) to read data of the final transmission message size from the memory at the start address, transfers the data to the NIC 11, and transmits the data to the remote node 100B as a RoCEv2 packet.

[0064] When packet transmission starts, in step S21, the NIC 11 of the remote node 100B receives the packet and writes the received data to the memory area reserved by the remote application 50B in step S14. It is assumed that the remote node 100B has already started waiting for packet reception. When data reception is complete, the NIC 11 sends a reception completion notification (RDMA CQE) to the message processing unit 31.

[0065] In step S22, the message processing unit 31 of the remote node 100B checks the memory area where the last transmitted message has been written. If the status code included in the reception completion notification indicates success, the message processing unit 31 acquires the acquired message size included in the reception completion notification.

[0066] In step S23, the message processing unit 31 of the remote node 100B acquires sub-metadata from the metadata based on the start address, acquired message size, and metadata size specified by the remote application 50B. The metadata starts from an address obtained by adding the acquired message size to the start address and subtracting the metadata size. The message processing unit 31 acquires each sub-metadata from the metadata address according to the metadata structure exchanged in advance. At this time, the message processing unit 31 confirms that the sum of the transmission message size and the metadata size acquired from the sub-metadata is equal to the acquired message size. Note that if message splitting is performed, the message processing unit 31 integrates the split and transmitted data.

[0067] At this timing, the message processing unit 31 notifies the remote application 50B that the data has been received, and the remote application 50B becomes able to process the received data.

[0068] In step S24, when the message processing unit 31 of the remote node 100B has completed receiving the final transmission message, it acquires a reception timestamp and a clock ID via the time reference unit 35. The time reference unit 35 acquires the reception timestamp by referencing the HWC 111 or the system clock, and also acquires the clock ID of the reference clock for the timestamp from the time synchronization function 23, and returns the reception timestamp and the clock ID to the message processing unit 31.

[0069] When the transmission of the final transmission message is completed, in step S25, the message processing unit 31 of the local node 100A receives a transmission completion notification (RDMA CQE) from the NIC 11. The message processing unit 31 confirms that the status code included in the transmission completion notification indicates success. The message processing unit 31 returns the data saved in step S18 to the original memory area.

[0070] In step S26, the message processing unit 31 of the remote node 100B registers the transmission and reception history acquired in steps S23 and S24 in the statistical information management unit 33. For example, the transmission and reception history includes a transmission timestamp, a transmission clock ID, the IP address of the local application 50A, a reception timestamp, a reception clock ID, the IP address of the remote application 50B, the transmission message size, the RDMA operation type, and the RDMA transport type. It is important that the transmission clock ID and the reception clock ID match. The message processing unit 31 may register the transmission and reception history in the statistical information management unit 33 only if the transmission clock ID and the reception clock ID match. Note that the statistical information management unit 33 may calculate latency from the difference between the reception timestamp and the transmission timestamp and add it to the transmission and reception history.

[0071] In step S27, the session control unit 34 of the remote node 100B transmits the transmission and reception history registered in step S26 to the local node 100A. The session control unit 34 may transmit the transmission and reception history to the local node 100A only if the transmission clock ID and the reception clock ID match.

[0072] In step S28, the local node 100A registers the received transmission and reception history in the statistical information management unit 33.

[0073] Through the above processing, a data transmission / reception history from the local application 50A to the remote application 50B is registered. The statistical information management unit 33 of each node obtains statistical information such as latency information and data loss rate from the registered transmission / reception history and returns the statistical information in response to an inquiry from an application. An application other than the local application 50A or the remote application 50B may also inquire about the statistical information.

[0074] The statistical information management unit 33 calculates the average latency, percentile value, jitter value, minimum value (best value), and maximum value (worst value) for each operation type, transport type, transmission message size, and clock ID, and manages them as latency information. The latency of each transmission and reception history is calculated from the difference between the reception timestamp and the transmission timestamp. The statistical information management unit 33 calculates the data loss rate from missing transmission request IDs and missing sequence numbers.

[0075] When an application queries for statistics, it may specify the message size, operation type, transport type, and / or clock ID.

[0076] [Data Saving Process] Here, with reference to FIG. 4, a process of saving data when adding metadata and restoring it to its original state will be described.

[0077] When the local application 50A requests a user-requested memory area, a total memory area equal to the combined size of the user-requested memory size and the metadata size is secured (FIG. 4(i)). The local application 50A can freely use the user-requested memory area, but cannot use the memory area after the total memory that is equal to the metadata size (hereinafter referred to as the save memory area).

[0078] The local application 50A places a message to be sent in the user request memory area (FIG. 4(ii)), and requests message transmission from the message processing unit 31. The start address of the message to be sent does not have to be the same as the start address of the user request memory area.

[0079] The message processing unit 31 saves data of the metadata size in the memory area behind the transmission message in the save memory area (FIG. 4(iii)).

[0080] The message processing unit 31 writes the metadata in the memory area after the outgoing message (FIG. 4(iv)), and requests the NIC 11 to transmit the final outgoing message. The NIC 11 transmits the outgoing message and the metadata.

[0081] When the transmission of the final transmission message is completed, the message processing unit 31 restores the saved data saved in the save memory area to the original memory area (FIG. 4(v)). As a result, the state of the user request memory area is restored to the state of FIG. 4(ii).

[0082] The location of the metadata is not limited to after the outgoing message. For example, the metadata may be placed before or in the middle of the outgoing message. In this case, the data of the metadata size in the memory area after the outgoing message is also saved to the save memory area.

[0083] An example of placing metadata before an outgoing message will now be described with reference to FIG.

[0084] The process is the same as that shown in FIGS. 4(i) and 4(ii) until the local application 50A secures a memory area and places the message to be sent in the user request memory area.

[0085] The message processing unit 31 saves data of the metadata size in the memory area after the transmission message in the backup memory area, and then moves the first data of the metadata size of the transmission message to the memory area after the transmission message (Figure 5 (iii)).

[0086] The message processing unit 31 writes the metadata at the location of the first data (FIG. 5(iv)), and requests the NIC 11 to transmit the final transmission message. The NIC 11 transmits the metadata, the remaining data, and the first data.

[0087] When the transmission of the final transmission message is completed, the message processing unit 31 restores the saved data saved in the save memory area to the original memory area (FIG. 5(v)).

[0088] In addition, since data is stored in the memory area of ​​remote node 100B in the following order: metadata, remaining data, and first data, the message processing unit 31 of remote node 100B needs to move the first data before the remaining data after acquiring the metadata.

[0089] By performing the same process as in FIG. 5, it is also possible to place metadata at the middle position of the transmitted message.

[0090] The method of placing metadata at the end of the data stream in Figure 4 is more efficient than the method in Figure 5 because it requires fewer memory copies. The method in Figure 4 also does not require rearranging the received data. It is also possible to transmit data with metadata attached using a non-contiguous memory area using a Scatter Gather List supported by RDMA. In this case, the exchange process of metadata and saved data described in Figures 4 and 5 is unnecessary, but the memory area becomes non-contiguous, which is likely to result in inefficiency.

[0091] [Embodiment 2] Next, embodiment 2 will be described. Embodiment 2 differs from embodiment 1 in that the operation type is WRITE with immediate. RC and UC can be used as transport types. Other settings are the same as embodiment 1. Differences from embodiment 1 will be mainly described with reference to the sequence diagrams of Figures 2 and 3. Descriptions of processes that are the same as embodiment 1 will be omitted.

[0092] As in the first embodiment, time synchronization, exchange of control information, and reservation of memory areas are performed in steps S11 to S14.

[0093] In step S15, after establishing a message transmission / reception session, the local application 50A and the remote application 50B exchange information on the writable start address of the write destination memory area of ​​the remote application 50B, the memory ID of the write destination memory area, and the memory size of the write destination memory area, in addition to the control information exchanged in step S15 in Example 1. Note that the write start address does not have to be the start address of the write destination memory area and depends on the information presented by the remote application 50B.

[0094] In step S16, the local application 50A transmits to the message processing unit 31 the start address of the memory area from which transmission will begin, the transmission data size, the memory ID, as well as the write start address and the memory ID of the write destination for the data to be transmitted, and requests message transmission via RDMA.

[0095] In steps S17 and S18, the message processing unit 31 acquires the transmission timestamp and clock ID, and adds metadata to the transmitted message.

[0096] In step S19, the message processing unit 31 specifies the start address, memory ID, final transmission message size, write start address, write destination memory ID, and operation type, and requests the NIC 11 to transmit the final transmission message using the library 22. In the second embodiment, WRITE with immediate is specified as the operation type. The write start address is also stored in Immediate data. The library 22 issues a transmission request to the NIC 11 using RoCEv2.

[0097] In step S20, the NIC 11 of the local node 100A transmits the packet to the remote node 100B.

[0098] In step S21, the NIC 11 of the remote node 100B receives the packet and writes the received data to the write start address specified by the local application 50A.

[0099] In step S22, the message processing unit 31 of the remote node 100B uses the write start address stored in the Immediate data to check the memory area where the data has been written.

[0100] In step S23, when the message processing unit 31 of the remote node 100B acquires the metadata, the write start address stored in the Immediate data is used as the start address.

[0101] The process when the transmission of the final transmission message is completed is the same as step S25 in the first embodiment.

[0102] The message processing unit 31 of the remote node 100B acquires the reception timestamp and clock ID in step S24, and registers the transmission and reception history in the statistical information management unit 33 in step S26.

[0103] In steps S27 and S28, the transmission and reception history is transmitted and registered in the statistical information management unit 33 of the local node 100A.

[0104] [Embodiment 3] Next, embodiment 3 will be described. Embodiment 3 differs from embodiment 1 in that the operation type is WRITE. RC and UC can be used as transport types. Other settings are the same as embodiment 1. Differences from embodiment 1 will be mainly described with reference to the sequence diagrams of Figures 2 and 3. Processing that is the same as embodiment 1 will not be described. Note that WRITE does not include a reception completion notification, so the processing corresponding to this will also be described.

[0105] As in the first embodiment, time synchronization, exchange of control information, and reservation of memory areas are performed in steps S11 to S14.

[0106] In steps S15 and S16, as in the second embodiment, the write destination information of the remote application 50B is exchanged, and a message transmission by RDMA including the write destination information is requested.

[0107] In steps S17 and S18, the message processing unit 31 acquires the transmission timestamp and clock ID, and adds metadata to the transmitted message.

[0108] In step S19, the message processing unit 31 specifies the start address, memory ID, final transmission message size, write start address, write destination memory ID, and operation type, and requests the NIC 11 to transmit the final transmission message using the library 22. In the third embodiment, WRITE is specified as the operation type.

[0109] In step S20, the NIC 11 of the local node 100A transmits the packet to the remote node 100B.

[0110] In step S21, the NIC 11 of the remote node 100B receives the packet and writes the received data to the write start address specified by the local application 50A.

[0111] Since the remote application 50B does not receive a reception completion notification from the NIC 11, when the local application 50A receives a transmission completion notification, in step S25, the local application 50A uses the control information exchange session to notify the remote application 50B of the write start address, transmission request ID, and maximum transmission data size as a completion notification.

[0112] When the remote application 50B receives the completion notification from the local application 50A, in steps S22 and S23, it uses the notified write start address, transmission request ID, and maximum transmission data size to confirm the memory area where the data has been written and acquires the metadata. The message processing unit 31 confirms that the transmission request ID included in the metadata is the same as the notified transmission request ID. The notified maximum transmission data size is used as the acquired message size.

[0113] The message processing unit 31 of the remote node 100B acquires the reception timestamp and clock ID in step S24, and registers the transmission and reception history in the statistical information management unit 33 in step S26.

[0114] In steps S27 and S28, the transmission and reception history is transmitted and registered in the statistical information management unit 33 of the local node 100A.

[0115] [Fourth Embodiment] Next, a fourth embodiment will be described. The fourth embodiment differs from the first embodiment in that the operation type is READ. RC can be used as the transport type. Other settings are the same as those in the first embodiment.

[0116] 6 and 7, the following description will focus on differences from the first embodiment. The same processes as those in the first embodiment will be omitted. Note that, since READ transmits data from the remote application 50B to the local application 50A, the data transmission direction is opposite to that of SEND, WRITE with immediate, and WRITE, and further, since there is no completion notification to the remote application 50B, the corresponding processes will also be described.

[0117] As in the first embodiment, time synchronization, control information exchange, and memory area reservation are performed in steps S11 to S14. Note that when the local application 50A reserves the memory area in step S13, a memory area with a size equal to the sum of the maximum transmission data size and the metadata size is reserved so that all data transmitted from the remote application 50B can be written. When the remote application 50B reserves the memory area in step S14, a memory area with a size equal to the sum of the user request memory size and the metadata size is reserved.

[0118] In step S15, after establishing a message transmission / reception session, the local application 50A and the remote application 50B exchange information regarding the readable start address of the read destination memory area of ​​the remote application 50B, the memory ID of the read destination memory area, and the memory size of the read destination memory area. Note that the read start address does not have to be the start address of the read destination memory area and depends on the information presented by the remote application 50B.

[0119] In step S31, the local application 50A requests message reading by RDMA by transmitting the read start address, read memory size, and read source memory ID of the data to be read to the message processing unit 31. The local application 50A also transmits the write start address for writing the read data and the acquired data size to the message processing unit 31.

[0120] In step S32, the message processing unit 31 transmits a message read request to the remote application 50B via the session control unit 34. The message read request includes a read start address, a read memory size, and a memory ID of the read source.

[0121] In step S33, the message processing unit 31 of the remote node 100B acquires the transmission timestamp and the clock ID via the time reference unit 35. The process of acquiring the transmission timestamp and the clock ID is the same as the process of step S17 in the first embodiment.

[0122] In step S34, the message processing unit 31 of the remote node 100B assigns metadata to the data to be sent (the data to be read) based on the read start address, the read memory size, and the memory ID of the read source. The process of assigning metadata is the same as the process in step S18 of the first embodiment.

[0123] In step S35, the remote application 50B notifies the local application 50A via the session control unit 34 that the message read preparation is complete.

[0124] 7, the message processing unit 31 of the local node 100A specifies the read start address, read memory size, memory ID of the read source, and operation type, and requests the NIC 11 to read a message using the library 22. The read memory size is a value to which the metadata size is added. In the fourth embodiment, READ is specified as the operation type.

[0125] In step S37, the NIC 11 of the local node 100A transmits a READ message to the remote node 100B.

[0126] In step S38, when the NIC 11 of the remote node 100B receives the READ message, it uses DMA to read data of the read memory size from the memory at the read start address, transfers the data to the NIC 11, and transmits it to the local node 100A as a RoCEv2 packet.

[0127] When packet transmission starts, in step S39, the NIC 11 of the local node 100A receives the packet and writes the received data to the memory area reserved by the local application 50A in step S13. When data reception is complete, the NIC 11 transmits a reception completion notification (RDMA CQE) to the message processing unit 31.

[0128] In step S40, the message processing unit 31 of the local node 100A checks the memory area where the last transmitted message was written. If the status code included in the reception completion notification indicates success, the message processing unit 31 acquires the acquired message size included in the reception completion notification. The message processing unit 31 then transmits a read completion notification to the remote application 50B via the session control unit 34.

[0129] In step S41, the message processing unit 31 of the local node 100A acquires sub-metadata from the metadata based on the write destination start address, the acquired data size, and the metadata size specified by the local application 50A. This process is similar to the process of step S23 in the first embodiment, except that it is performed by the local node 100A.

[0130] In step S42, when the message processing unit 31 of the local node 100A completes reception of the transmission message, it acquires the reception timestamp and the clock ID via the time reference unit 35. This process is similar to the process of step S24 in the first embodiment, except that it is performed by the local node 100A.

[0131] Meanwhile, in step S43, the message processor 31 of the remote node 100B receives the read completion notification via the session controller 34. Upon receiving the read completion notification, the message processor 31 returns the data saved in step S34 to the original memory area.

[0132] In step S44, the message processing unit 31 of the local node 100A registers the transmission and reception history acquired in steps S41 and S42 in the statistical information management unit 33. This process is similar to the process of step S26 in the first embodiment, except that it is performed by the local node 100A.

[0133] In step S45, the local node 100A transmits the transmission and reception history registered in step S44 to the remote node 100B.

[0134] In step S46, the remote node 100B registers the received transmission / reception history in the statistical information management unit 33.

[0135] [Fifth Embodiment] Next, a fifth embodiment will be described. The fifth embodiment differs from the first embodiment in that the operation type is ATOMIC. RC can be used as the transport type. Other settings are the same as those in the first embodiment.

[0136] In an ATOMIC operation, the local node 100A performs either a Fetch-and-Add or a Compare-and-Swap on the memory 13 of the remote node 100B. In the case of a Fetch-and-Add, the local node 100A transmits the start address of the remote node 100B and the data increment to the remote node 100B. The remote node 100B transmits the data before the change to the local node 100A. In the case of a Compare-and-Swap, the local node 100A transmits the start address of the remote node 100B, the value to be compared, and the changed value if the value matches the value to be compared to the remote node 100B to the remote node 100B. As with the Fetch-and-Add, the remote node 100B transmits the data before the change to the local node 100A.

[0137] The fifth embodiment includes both patterns of data transmission from the local node 100A to the remote node 100B and data transmission from the remote node 100B to the local node 100A. Therefore, since the local node 100A only needs to record the transmission timestamp and the reception timestamp, there is no need to embed the timestamp in the metadata. Therefore, time synchronization is also unnecessary.

[0138] For example, in the fifth embodiment, the message processing unit 31 of the local node 100A acquires a transmission timestamp when it receives a request for atomic processing from the local application 50A, and acquires a reception timestamp when it has completed receiving the data before the change from the remote node 100B. As in steps S44, S45, and S46 of the fourth embodiment, each node registers the transmission and reception history in the statistical information management unit 33.

[0139] [Embodiment 6] Next, embodiment 6 will be described. Embodiment 6 differs from embodiment 1 in that message division is performed. Since the transport type RC has a retransmission function, the target transport types in embodiment 6 are UC and UD. Other settings are the same as embodiment 1. Differences from embodiment 1 will be mainly described with reference to the sequence diagrams in Figures 2 and 3. Description of the same processes as embodiment 1 will be omitted.

[0140] As in the first embodiment, time synchronization and control information exchange are performed in steps S11 and S12. In the control information exchange, information with message division is exchanged.

[0141] In reserving a memory area for local application 50A in step S13, memory is reserved that is equal to the size of the metadata size obtained by dividing the maximum transmission data size by the transmission message size that can be transferred in one packet. If the maximum transmission data size is X, the metadata size is Y, and the maximum size that can be transferred without dividing packets is PMTU, the transmission message size that can be transferred in one packet is PMTU-Y. If the user-requested memory size is Z, the total memory size to be reserved can be calculated using the following formula. Note that user-requested memory size Z is equal to or greater than the maximum transmission data size X.

[0142] Total memory size = Z + (X / (PMTU-Y) + 1) * Y

[0143] On the other hand, in the remote node 100B, data with metadata added for the number of divisions is written to memory, so when allocating memory space for the remote application 50B in step S14, if the maximum transmission data size is X, the total memory size to be allocated can be calculated using the following formula.

[0144] Total memory size = X + (X / (PMTU-Y) + 1) * Y

[0145] 8, the maximum transmission data size X can be divided by the size of PMTU-Y into 5 parts. Therefore, the total memory size is X+5*Y, which is the maximum transmission data size X plus 5 parts of the metadata size Y.

[0146] In step S15, the local application 50A and the remote application 50B establish a message transmission / reception session and exchange control information in the same manner as in the first embodiment.

[0147] In step S16, the local application 50A requests the message processing unit 31 to transmit a message by RDMA.

[0148] In step S17, the message processing unit 31 of the local node 100A acquires the transmission timestamp and the clock ID.

[0149] In step S18, the message processing unit 31 of the local node 100A divides the data to be transmitted and adds metadata to each of the divided transmission messages.

[0150] An example of the process of dividing data will now be described with reference to FIGS.

[0151] If the data to be sent is larger than the PMTU, the data to be sent is divided by the size of the PMTU. In the example of Figure 9, the data to be sent is divided into three pieces of data A, B, and C in units of PMTU. The sizes of data A and B are the same as the PMTU, and the size of data C is smaller than the PMTU.

[0152] For data in PMTU units, data of the metadata size is moved backward to secure an area for adding metadata (hereinafter referred to as the metadata area). In the example of Fig. 9, data A and B are data in PMTU units. Data A-2 and B-2 of the metadata size are moved from data A and B respectively to the rear of data C. Data A-1 and metadata, and data B-1 and metadata are each transmitted as one packet.

[0153] If data D, which is a combination of data C, data A-2, and data B-2, can be divided in units of PMTU, data D is divided and the process of allocating a metadata area is repeated.

[0154] The division of data D will be described with reference to Figure 10. Because data D is larger than the PMTU, data D is divided into data E and data F in PTMU units. The size of data E is the same as the PMTU, and the size of data F is smaller than the PMTU. Data E-2, which is the size of the metadata, is moved from data E to the rear of data F. Data E-1 and the metadata are sent as one packet.

[0155] If data G, which is a combination of data F and data E-1, can be divided in units of PMTU, data G is divided and the process of allocating a metadata area is repeated.

[0156] In the example of Figure 10, data G is smaller than the PMTU, so a process is performed to add a metadata area to data G. Since the size of data G with the added metadata area is larger than the PMTU, data G-2 is moved backward, and metadata areas are added to both data G-1 and data G-2. Data G-1 and the metadata, and data G-2 and the metadata, are each transmitted as a single packet.

[0157] By the above process, the data to be transmitted can be divided and a metadata area can be allocated to each of the divided data.

[0158] After dividing the data to be transmitted, the message processing unit 31 writes metadata into the metadata area of ​​each divided transmission message. As described above, the message processing unit 31 writes metadata consisting of sub-metadata of a transmission timestamp, a clock ID, a sequence number, a transmission message size, and a transmission request ID. The transmission timestamp and clock ID are the same for each divided transmission message. By assigning the same transmission timestamp and the same clock ID to each divided transmission message, it is possible to increase the tolerance against packet loss in latency measurement. The sequence number is assigned a value that increases by one from the first transmission message onwards, and -1 is assigned to the last transmission message.

[0159] In step S19, the message processing unit 31 of the local node 100A specifies the start address, memory ID, final transmission message size, and operation type, and requests the NIC 11 to transmit the final transmission message using the library 22. When UD is used as the transport type, a transmission request is made for the number of divisions.

[0160] In step S20, the NIC 11 of the local node 100A transmits the packet to the remote node 100B. Since SEND can deliver imm_data separately, the local node 100A transmits the transmission request ID and sequence number as imm_data.

[0161] When packet transmission starts, in step S21, the NIC 11 of the remote node 100B receives the packet and writes the received data to the memory area reserved by the remote application 50B in step S14. When the NIC 11 completes receiving the data, it transmits a reception completion notification to the message processing unit 31.

[0162] In step S22, if the status code included in the reception completion notification indicates success, the message processing unit 31 of the remote node 100B acquires the acquired message size included in the reception completion notification. When UD is used as the transport type, the remote node 100B waits in advance for packet reception with the memory area shifted for each PMTU. The message processing unit 31 receives the reception completion notification for the number of divisions. Each time the message processing unit 31 receives a reception completion notification, it checks the status code and confirms that the transmission request ID included in imm_data matches the received data, that all sequence numbers included in imm_data have been received correctly, and that data indicating the last sequence number included in imm_data has been received.

[0163] In step S23, the message processing unit 31 of the remote node 100B acquires sub-metadata from the metadata and integrates the divided data. Since metadata is assigned to each piece of data divided by PMTU, the message processing unit 31 checks the transmission request ID and sequence number for all metadata.

[0164] An example of the process of integrating data will now be described with reference to Fig. 11. Data integration can be achieved by performing the processes in Figs. 9 and 10 in reverse order.

[0165] First, the metadata of the last data G-2 is obtained. Since data G-2 was moved from the previous (starting) metadata area, the previous metadata is obtained and data G-2 is moved to the previous metadata area. This creates data that combines data G-1 and data G-2.

[0166] Next, the previous metadata is acquired, and data E-2, the size of which is the metadata, is moved from the end of the data combining data G-1 and data G-2 to the area of ​​the previous metadata. This creates data combining data E-1, data E-2, and data F.

[0167] Next, the previous metadata is acquired, and data B-2 and data A-1, which are the size of the metadata, are moved from the end of the combined data of data E-1, data E-2, and data F to the area of ​​the previous metadata.

[0168] Through the above processing, the divided data is integrated into the original data in which data A-1, data A-2, data B-1, data B-2, and data C are concatenated.

[0169] The process when the transmission of the final transmission message is completed is the same as step S25 in embodiment 1. When UD is used as the transport type, the message processing unit 31 of the local node 100A receives a transmission completion notification for each transmission request, and therefore, it is necessary to wait until all transmission requests are completed.

[0170] After integrating the data, the message processing unit 31 of the remote node 100B acquires the reception timestamp and clock ID in step S24, and registers the transmission and reception history in the statistical information management unit 33 in step S26.

[0171] In steps S27 and S28, the transmission and reception history is transmitted and registered in the statistical information management unit 33 of the local node 100A.

[0172] [Another Data Division Method] Here, an example of another process for dividing data will be described. The data division method may be a method other than the data division method shown in Figs.

[0173] For example, as shown in Fig. 12, an area for assigning metadata is provided while shifting the data to be transmitted by the metadata size. Specifically, the data to be transmitted is divided into data A of the size PMTU-Y and the remainder of data A, and the remainder of data A is moved back by the metadata size to provide a metadata area for data A. The combined size of data A and the metadata becomes PMTU. Similarly, the remainder of data A is divided into data B of the size PMTU-Y and the remainder of data B, and the remainder of data B is moved back by the metadata size to provide a metadata area for data B. By repeating this process, the data to be transmitted is divided into data A to E, and a metadata area is provided after each data.

[0174] 9 and 10 are more efficient than the division method shown in FIG. 12 because they require fewer memory copies. Data transmission with metadata attached using discontinuous memory areas using a Scatter Gather List supported by RDMA is also possible. In this case, the exchange process of metadata and saved data described in FIG. 9 and FIG. 10 is unnecessary, but the memory areas are discontinuous, which is likely to result in inefficiency.

[0175] [Seventh embodiment] Next, a seventh embodiment will be described. The seventh embodiment differs from the sixth embodiment in that the operation type is WRITE with immediate. UC can be used as the transport type. Other settings are the same as those of the sixth embodiment, including the point of message division. Differences from the sixth embodiment will be mainly described with reference to the sequence diagrams of Figures 2 and 3. The same processes as those of the sixth embodiment will not be described.

[0176] As in the sixth embodiment, time synchronization, exchange of control information, and reservation of memory area are performed in steps S11 to S14.

[0177] In step S15, after establishing a message transmission / reception session, the local application 50A and the remote application 50B exchange information on the writable start address of the write destination memory area of ​​the remote application 50B, the memory ID of the write destination memory area, and the memory size of the write destination memory area, in addition to the control information exchanged in step S15 in Example 1. Note that the write start address does not have to be the start address of the write destination memory area and depends on the information presented by the remote application 50B.

[0178] In step S16, the local application 50A transmits to the message processing unit 31 the start address of the memory area from which transmission will begin, the transmission data size, the memory ID, as well as the write start address and the memory ID of the write destination for the data to be transmitted, and requests message transmission via RDMA.

[0179] In step S17, the message processing unit 31 of the local node 100A acquires the transmission timestamp and the clock ID.

[0180] In step S18, the message processing unit 31 of the local node 100A divides the data to be transmitted, and adds metadata to each of the divided transmission messages, as in the sixth embodiment.

[0181] In step S19, the message processing unit 31 specifies the start address, memory ID, write memory size, write start address, write destination memory ID, and operation type for each message divided from the final transmission message, and requests the NIC 11 to transmit the message using the library 22. In the seventh embodiment, WRITE with immediate is specified as the operation type. The library 22 issues a transmission request to the NIC 11 using RoCEv2.

[0182] The message processing unit 31 specifies the write start address while shifting it by the write memory size for each message transmission request. Since the write memory size is PMTU for messages other than the last one to be transmitted, the PMTU is added to the write start address for each message transmission request.

[0183] The write start address of each message is also stored in the Immediate data.

[0184] In step S20, the NIC 11 of the local node 100A transmits the packet to the remote node 100B.

[0185] When packet transmission starts, in step S21, the NIC 11 of the remote node 100B receives the packet and writes the received data to the memory area specified by the write start address. When the NIC 11 completes receiving the data, it sends a reception completion notification to the message processing unit 31. The number of reception completion notifications sent is the same as the number of divisions of the final transmission message.

[0186] In step S22, if the status code included in the reception completion notification indicates success, the message processing unit 31 of the remote node 100B acquires the acquired message size included in the reception completion notification. The memory area where the data is written can be identified by the write start address stored in the Immediate date.

[0187] In step S23, the message processing unit 31 of the remote node 100B acquires the sub-metadata from the metadata and integrates the divided data.

[0188] The process when the transmission of the final transmission message is completed is the same as step S25 in the sixth embodiment.

[0189] After integrating the data, the message processing unit 31 of the remote node 100B acquires the reception timestamp and clock ID in step S24, and registers the transmission and reception history in the statistical information management unit 33 in step S26.

[0190] In steps S27 and S28, the transmission and reception history is transmitted and registered in the statistical information management unit 33 of the local node 100A.

[0191] [Embodiment 8] Next, embodiment 8 will be described. Embodiment 8 differs from embodiment 6 in that the operation type is WRITE. UC can be used as the transport type. Other settings are the same as embodiment 6, including message division. Differences from embodiment 6 will be mainly described with reference to the sequence diagrams of Figures 2 and 3. Processing that is the same as embodiment 6 will not be described. Note that WRITE does not include a reception completion notification, so the processing corresponding to this will also be described.

[0192] As in the sixth embodiment, time synchronization, exchange of control information, and reservation of memory area are performed in steps S11 to S14.

[0193] In step S15, after establishing a message transmission / reception session, the local application 50A and the remote application 50B exchange information on the writable start address of the write destination memory area of ​​the remote application 50B, the memory ID of the write destination memory area, and the memory size of the write destination memory area, in addition to the control information exchanged in step S15 in Example 1. Note that the write start address does not have to be the start address of the write destination memory area and depends on the information presented by the remote application 50B.

[0194] In step S16, the local application 50A transmits to the message processing unit 31 the start address of the memory area from which transmission will begin, the transmission data size, the memory ID, as well as the write start address and the memory ID of the write destination for the data to be transmitted, and requests message transmission via RDMA.

[0195] In step S17, the message processing unit 31 of the local node 100A acquires the transmission timestamp and the clock ID.

[0196] In step S18, the message processing unit 31 of the local node 100A divides the data to be transmitted, and adds metadata to each of the divided transmission messages, as in the sixth embodiment.

[0197] In step S19, the message processing unit 31, as in the seventh embodiment, specifies the start address, memory ID, write memory size, write start address, write destination memory ID, and operation type for each message divided from the final transmission message, and requests the NIC 11 to transmit the message using the library 22. In the eighth embodiment, WRITE is specified as the operation type. The library 22 issues a transmission request to the NIC 11 using RoCEv2.

[0198] In addition, in Example 8, since the write start address cannot be notified using immediate data as in Example 7, the local application 50A notifies the remote application 50B of the write start address and transmission request ID using a control information exchange session.

[0199] In step S20, the NIC 11 of the local node 100A transmits the packet to the remote node 100B.

[0200] When packet transmission starts, in step S21, the NIC 11 of the remote node 100B receives the packet and writes the received data into the memory area specified by the write start address.

[0201] Since the remote application 50B does not receive a reception completion notification from the NIC 11, when the local application 50A receives a transmission completion notification, in step S25, the local application 50A uses the control information exchange session to notify the remote application 50B of the transmission completion notification and the maximum transmission data size.

[0202] When the remote application 50B receives the completion notification from the local application 50A, in step S22, it uses the notified write start address, transmission request ID, transmission completion notification, and maximum transmission data size to check the memory area where the data has been written.

[0203] In step S23, the message processing unit 31 of the remote node 100B acquires the sub-metadata from the metadata and integrates the divided data.

[0204] After integrating the data, the message processing unit 31 of the remote node 100B acquires the reception timestamp and clock ID in step S24, and registers the transmission and reception history in the statistical information management unit 33 in step S26.

[0205] In steps S27 and S28, the transmission and reception history is transmitted and registered in the statistical information management unit 33 of the local node 100A.

[0206] Next, a ninth embodiment will be described. The ninth embodiment differs from the previous embodiments in that each node is synchronized with a Global Navigation Satellite System (GNSS). Other settings are the same as those of the previous embodiments.

[0207] In the ninth embodiment, a time synchronization protocol such as PTP is not used, and therefore a clock ID obtained by time synchronization cannot be obtained. In the ninth embodiment, the time reference unit 35 generates a clock ID based on the data type obtainable from the first column of RMC, which is a sentence that summarizes positioning information in a minimal manner. For example, in the case of an arbitrary satellite combination, the combination becomes GN, and therefore the clock ID is set to GNRMC (other IDs include GP=GPS / SBAS, GL=GLONASS, GA=Galileo, GB=BeiDou, GQ=QZSS, etc.).

[0208] The other processing is the same as in the previous embodiments.

[0209] As described above, the communication system of this embodiment is a communication system that transfers data between the local application 50A running on the local node 100A and the remote application 50B running on the remote node 100B. When the local node 100A receives a data transmission request from the local application 50A, the local node 100A acquires a transmission timestamp and sends the data with the transmission timestamp attached. When the remote node 100B receives the data and notifies the remote application 50B that data reception has been completed, the remote node 100B acquires a reception timestamp, stores a transmission / reception history including the transmission timestamp and the reception timestamp, and transmits the transmission / reception history to the local node 100A. The local node 100A and the remote node 100B calculate latency from the difference between the transmission timestamp and the reception timestamp in the stored transmission / reception history and return latency statistical information. This makes it possible to determine the latency when data is transferred between the local application 50A and the remote application 50B. Furthermore, because the timestamp systems included in the local node 100A and the remote node 100B acquire timestamps and manage statistical information and data loss rates based on the timestamps, the applications do not need to be equipped with a means for acquiring latency. The timestamp system of each node shares the timestamp between the nodes, so that the local application 50A can also grasp the latency.

[0210] In the communication system of this embodiment, the time synchronization function 23 manages a clock ID indicating the reference clock reflected in the HWC 111, and the message processing unit 31 acquires the clock ID along with the timestamp, and only when the transmission clock ID and the reception clock ID are the same, registers the transmission and reception history in the statistical information management unit 33. This ensures that the transmission timestamp and the reception timestamp are acquired based on the same reference clock.

[0211] In the communication system of this embodiment, the message processing unit 31 divides and transmits data, thereby providing the user with an API that can transfer data larger than the data transfer amount supported by RDMA (for example, 4096 bytes in the case of UD) in a single request.

[0212] In the communication system of this embodiment, the message processing unit 31 assigns metadata including a transmission timestamp to each piece of divided data, thereby making it possible to increase the tolerance to packet loss in latency measurement.

[0213] Each of the nodes and time stamp systems described above can be implemented using a general-purpose computer system including a central processing unit (CPU) 901, memory 902, storage 903, a communication device 904, an input device 905, and an output device 906, as shown in Fig. 13. In this computer system, each node and time stamp system is realized by the CPU 901 executing a predetermined program loaded onto the memory 902. This program can be recorded on a computer-readable non-transitory recording medium such as a magnetic disk, optical disk, or semiconductor memory, or can be distributed via a network.

[0214] 100A Local node 100B Remote node 50A Local application 50B Remote application 11 NIC 111 HWC 12 CPU 13 Memory 21 OS 22 Library 23 Time synchronization function 31 Message processing unit 32 Memory management unit 33 Statistical information management unit 34 Session control unit 35 Time reference unit

Claims

1. A communication system for transferring data between a transmitting device and a receiving device, wherein the transmitting device acquires transmission attribute information when starting a data transmission process, and sends out the data with the transmission attribute information attached; the receiving device acquires reception attribute information when completing reception of the data, and retains a transmission / reception history including the transmission attribute information and the reception attribute information, and transmits the transmission / reception history to the transmitting device; and the transmitting device and the receiving device return statistical information acquired using the transmission attribute information and the reception attribute information retained.

2. A communication system according to claim 1, wherein the transmission attribute information includes a transmission timestamp, and the reception attribute information includes a reception timestamp, and the transmitting device and the receiving device each hold an identifier indicating a reference clock to which an internal clock for acquiring the timestamp is synchronized, acquire the identifier together with the timestamp, and store the identifier as a transmission / reception history in addition to the transmission timestamp and the reception timestamp.

3. A communication system according to claim 1, wherein the transmitting device divides the data and transmits it, and assigns the transmission attribute information to each piece of divided data before sending it out.

4. A communication system according to claim 1, wherein the transmitting device and the receiving device transfer data using remote direct memory access.

5. A transmitting device that transmits data to a receiving device, the transmitting device acquiring transmission attribute information when starting a data transmission process, attaching the transmission attribute information to the data and sending it out, receiving and retaining a transmission / reception history from the receiving device that includes the transmission attribute information and reception attribute information acquired when the receiving device has completed receiving the data, and returning statistical information acquired using the retained transmission attribute information and reception attribute information.

6. A transmitting device according to claim 5, wherein the transmission attribute information includes a transmission timestamp obtained when starting data transmission processing.

7. A receiving device that receives data from a transmitting device, the receiving device receiving the data to which transmission attribute information obtained when the transmitting device starts the data transmission process is attached, obtaining reception attribute information when the receiving device completes the data reception, retaining a transmission / reception history including the transmission attribute information and the reception attribute information, transmitting the transmission / reception history to the transmitting device, and returning statistical information obtained using the retained transmission attribute information and the reception attribute information.

8. A receiving device according to claim 7, wherein the reception attribute information includes a reception timestamp obtained when reception of data is completed.

Images