Segmentation determination system

Abstract

Provided is a segmentation determination system 300 that determines a segmentation method for data transferred between a transmission node 100 and a reception node 200. The transmission node 100 is provided with: a delay measurement unit 32 that, for each data segmentation method, measures the delay when data is segmented using each segmentation method and transferred; a statistical information management unit 34 that retains statistical delay information obtained from the delay measurement results for each segmentation method; a segmentation determination unit 31 that determines a segmentation method for data to be transferred, on the basis of the statistical delay information for each segmentation method; and a message segmentation unit 33 that segments and transfers the data using the segmentation method determined by the segmentation determination unit 31.

Description

Division Judgment System

[0001] The present disclosure relates to a division determination system.

[0002] Remote Direct Memory Access (RDMA) enables data transfer without going through the operating system (OS), achieving high throughput and low latency. RDMA was developed for high-performance computing, so it is increasingly being used in cluster configurations that connect nodes over short distances and in data centers.

[0003] When transferring memory data (hereinafter referred to as a message) from a sender to a receiver, low latency is required in a data center (Non-Patent Documents 1 and 2).

[0004] Marcos K. Aguilera, et al., "Microsecond Consensus for Microsecond Applications," 14th USENIX Symposium on Operating Systems Design and Implementation, 2020, https: / / www.usenix.org / system / files / osdi20-aguilera.pdfLuiz Barroso, et al., "Attack of the killer microseconds," Communications of the ACM, April 2017, Volume 60, Issue 4, pp. 48-54"RDMA Core Userspace Libraries and Daemons," https: / / github.com / linux-rdma / rdma-core

[0005] RDMA is becoming popular as a large-capacity transfer method, but SmartNICs (hereinafter referred to as NICs) and general-purpose libraries (Non-Patent Document 3) for using RDMA are not sufficiently tuned for each device, and data cannot be transferred with optimized latency. This is because the RDMA accelerators that process RDMA on each SmartNIC differ from vendor to vendor.

[0006] Because each vendor's NIC behaves differently, users had to individually tune parameters to suit the system and vendor in which the NIC was installed.

[0007] The present disclosure has been made in view of the above, and aims to improve latency in RDMA.

[0008] A division judgment system according to one aspect of the present disclosure is a division judgment system that judges a division method for data to be transferred between a first communication device and a second communication device, wherein the first communication device comprises: a delay measurement unit that measures, for each data division method, the delay when the data is divided and transferred using that division method; a statistical information management unit that holds delay statistical information obtained from the delay measurement results for each division method; a division judgment unit that determines a division method for the data to be transferred based on the delay statistical information for each division method; and a division unit that divides and transfers the data using the division method determined by the division judgment unit.

[0009] According to the present disclosure, it is possible to improve the latency in RDMA.

[0010] FIG. 1 is a diagram illustrating an example of a communication system. FIG. 2 is a diagram illustrating an example of the delay reduction effect when a message is divided. FIG. 3 is a diagram illustrating an example of the configuration of a transmitting node. FIG. 4 is a diagram illustrating an example of the configuration of a receiving node. FIG. 5 is a sequence diagram illustrating an example of a process performed before transmitting and receiving delay measurement traffic. FIG. 6 is a sequence diagram illustrating an example of a process for transmitting and receiving delay measurement traffic. FIG. 7 is a sequence diagram illustrating an example of a process for registering delay statistical information. FIG. 8 is a flowchart illustrating an example of a process for extracting a division pattern that satisfies desired conditions. FIG. 9 is a sequence diagram illustrating an example of a process for transmitting and receiving delay measurement traffic. FIG. 10 is a diagram illustrating an example of the hardware configuration of a terminal and a relay device.

[0011] [About RDMA] RDMA is a technology that allows a sending node to write data directly to the memory of a receiving node, and a sending node to read data directly from the memory of a receiving node. In TCP / IP communication, the OS processes various types of data before passing the data as packets to the Network Interface Card (NIC) for transmission and reception. On the other hand, RDMA offloads the data processing previously performed by the OS to a smart NIC.

[0012] RDMA can be used over lines such as InfiniBand and Ethernet. The present disclosure is applicable to any lower-level communication protocol that implements RDMA without any restrictions. In the embodiments, the explanation is based on RDMA over Converged Ethernet version 2 (RoCEv2), which is widely used in data centers, but InfiniBand and iWARP can also be used.

[0013] RDMA transport types include Reliable Connection (RC), Unreliable Connection (UC), and Unreliable Datagram (UD). RDMA selects one of the transport types, RC, UC, or UD, to transfer data. RC and UC establish a connection, while UD does not. RC performs error recovery (retransmission), while UC and UD do not. In RoCEv2, data is ultimately sent and received between the sending and receiving nodes as UDP packets. In other words, the transport types can be considered virtual connections established on top of UDP.

[0014] RDMA data transmission methods (hereinafter referred to as operation types) include SEND (which is almost identical to SEND with immediate, so there is no distinction and it will be referred to as SEND hereinafter), WRITE, WRITE with immediate, READ, and ATOMIC. The operation types that can be used for each transport type are different. Specifically, all operation types can be used with RC, SEND, WRITE, and WRITE with immediate can be used with UC, and only SEND can be used with UD.

[0015] Additionally, the direction of data transmission and whether or not a completion notification is given differ depending on the operation type. For example, SEND, WRITE, and WRITE with immediate send data from the sending node to the receiving node. READ sends data from the receiving node to the sending node. WRITE and READ do not send a completion notification on the receiving node side.

[0016] Next, we will explain the basic operation of RDMA data transfer. Before sending or receiving data, control information must be exchanged between the application on the sending node (hereafter referred to as the sending application) and the application on the receiving node (hereafter referred to as the receiving application). For example, the RDMA Communication Manager (CM) can be used to exchange control information.

[0017] In RDMA, an application issues a request to the NIC in communication units called Work Requests (WRs). WRs for send requests are placed in the Send Queue (SQ), and WRs for receive requests are placed in the Receive Queue (RQ). A WR placed in an SQ / RQ is called a Work Queue Element (WQE). A pair of an SQ and an RQ is called a Queue Pair (QP). Typically, in RC and UC, a QP is established in a one-to-one relationship between a sending node and a receiving node. In this disclosure, this QP is called a session.

[0018] The NIC retrieves the WQE in order according to the FIFO, and performs data transmission or reception processing using the control information written in the WQE (start address of the memory data to be transferred, memory address length, etc.).

[0019] When data transmission or reception is completed according to the WQE, a Completion Queue Entry (CQE) is placed in the Completion Queue (CQ). You can check whether transmission or reception is complete by checking the CQE stored in the CQ. The extracted CQE is called a Work Completion (WC). For example, in the case of a send request, the WC contains a status code indicating whether the transmission was successful or not, and in the case of a receive request, it contains a status code indicating whether the reception was successful or not, as well as the size of the received data.

[0020] [Configuration of Communication System] An example of a communication system 1 according to this embodiment will be described with reference to Fig. 1. The communication system 1 shown in the figure includes a sending node 100 and a receiving node 200. The sending node 100 and the receiving node 200 are connected via a network. A sending application 150 runs on the sending node 100, and a receiving application 250 runs on the receiving node 200. Data sent by the sending application 150 is split into appropriate sizes by a division determination system 300, and transferred to the receiving node 200 by a NIC 11 with an RDMA function, without going through the OS 21.

[0021] In communication system 1, in the first phase, division determination system 300 measures the delay for each data division method, obtains statistical information about delay such as average delay, minimum delay, maximum delay, standard deviation of delay, and 99th percentile value of delay (hereinafter referred to as delay statistical information), and extracts a division method that improves the delay statistical information that the user values ​​based on the obtained delay statistical information. In the second phase after the first phase, transmission application 150 uses division determination system 300 to divide data using the division method extracted in the first phase and transfers the data.

[0022] In the present disclosure, the delay refers to the difference in time from when the transmission application 150 prepares data in memory and completes preparation for transmission until the reception application 250 receives all the data and is ready to process the data.

[0023] Figure 2 shows a comparison of the average latency and 99th percentile value when messages are divided into a specified fragment size and transferred without fragmentation. The horizontal axis represents message size, and the vertical axis represents fragment size. The shaded boxes represent fragmentation patterns that reduced the average latency and percentile value compared to when messages were not fragmented. For example, when a 2MB message was divided into four 512KB messages and transferred, the average latency was 0.166μs and the 99th percentile value was 0.648μs, improving the average latency and 99th percentile value compared to when messages were not fragmented. As shown in Figure 2, dividing a message into a certain fragment size and transferring it can sometimes improve the desired latency statistics compared to when messages are not fragmented.

[0024] [Device Configuration] An example of the configuration of a sending node 100 will be described with reference to Figure 3. The sending node 100 shown in the figure includes hardware such as a NIC 11, a CPU 12, and a memory 13, and operates on software such as an OS 21, an RDMA general-purpose library 22, a time synchronization function 23, and a system clock 24. A sending application 150 runs on the sending node 100. Each part of the sending node 100 will be described below.

[0025] In the first phase, the division determination unit 31 requests the delay measurement unit 32 to measure delay statistical information for all determined division patterns (also called division methods) by specifying the number of trials, and extracts a division pattern that will improve the delay statistical information desired by the user based on the obtained delay statistical information. A division pattern indicates how a message of a specified size is divided. For example, a division pattern may be to divide a 1024-byte message into four 256-byte portions. Division patterns also include dividing a 1024-byte message into 1024-byte portions, i.e., no division. The sending node 100 manages division patterns using division pattern numbers. A division pattern number is a number assigned to a combination of message size and division method. Table 1 below shows examples of division patterns. The maximum message size is 1 GB.

[0026]

[0027] In Table 1, each row indicates the message size, and each column indicates the number of divisions. The combination of row number and column number (row number, column number) is used as the division pattern number. For example, a division pattern number of (10, 4) indicates that a 1024-byte message is divided into 16 parts (each of 64 bytes). The division pattern is not limited to the above, and any arbitrary pattern can be used.

[0028] The delay measurement unit 32 starts delay measurement upon receiving a request from the division determination unit 31. In delay measurement, the delay measurement unit 32 transmits a message transmission request according to the division pattern to the message division unit 33. The message transmission request includes the start address of the message to be transmitted, the message size, the RDMA operation type, and information related to division (e.g., the division pattern number and the division size). The delay measurement unit 32 repeats transmitting the message transmission request the number of times specified by the division determination unit 31.

[0029] Upon receiving a message transmission request, message divider 33 divides the message in a specified method and executes message transmission processing. In the delay measurement mode of the first phase, message divider 33 acquires a transmission timestamp, divides the message in a division pattern specified by delay measurement unit 32, and assigns metadata including the transmission timestamp and division pattern number to the message. In the automatic division mode of the second phase, upon receiving a message transmission request from transmission application 150, message divider 33 divides the message in the optimal division pattern obtained in the first phase.

[0030] The statistical information management unit 34 obtains delay statistical information from the receiving node 200 and manages the delay statistical information for each division pattern. The statistical information management unit 34 may manage the delay statistical information for each operation type or each transport type.

[0031] The session control unit 35 establishes a control information exchange session between the sending node 100 and the receiving node 200 for exchanging control information.

[0032] In response to a request from the message division unit 33, the time reference unit 36 ​​acquires and returns a time stamp.

[0033] An example of the configuration of the receiving node 200 will be described with reference to Figure 4. The receiving node 200 shown in the figure operates using the same hardware and software as the sending node 100, and a division determination system 300 runs on it. A receiving application 250 is executed on the receiving node 200. The division determination system 300 of the receiving node 200 includes a message integration unit 37, a receiving-side delay measurement unit 38, a session control unit 35, and a time reference unit 36. The session control unit 35 and the time reference unit 36 ​​are the same as those of the sending node 100, so a description thereof will be omitted here.

[0034] The message integration unit 37 executes message reception processing and assembles the divided and transmitted messages. In the delay measurement mode of the first phase, when message reception is completed, the message integration unit 37 acquires the division pattern number and the transmission timestamp from the metadata, acquires the reception timestamp from the time reference unit 36, and transmits the division pattern number, the transmission timestamp, and the reception timestamp to the delay measurement unit 38.

[0035] The delay measurement unit 38 calculates the delay from the transmission timestamp and the reception timestamp. The delay measurement unit 38 calculates delay statistical information for the division pattern number from the delay obtained by measuring the number of trials, and transmits the delay statistical information to the sending node 100.

[0036] [Message Division Mode] Next, a description will be given of three modes provided by the message division unit 33. The message division unit 33 provides three modes: an automatic division mode, a delay measurement mode, and a designated division mode.

[0037] The automatic segmentation mode is a mode used by the transmission application 150 and can be used in the second phase. In the automatic segmentation mode, the message segmentation unit 33 segments the message using a segmentation pattern that improves the desired delay statistical information found in the first phase. When using the automatic segmentation mode, the transmission application 150 passes a memory ID, a start address, a transmission message size, and an RDMA operation type to the message segmentation unit 33. If the RDMA operation type is WRITE, WRITE with immediate, or READ, the transmission application 150 passes the memory ID and start address of the memory area reserved by the receiving node 200 in addition to the above information. When UD is used as the transport mode, the transmission application 150 passes a QP number and a security key.

[0038] The delay measurement mode is a mode used by the delay measurement unit 32 when measuring delay in the first phase. In the delay measurement mode, the message division unit 33 divides a message using a specified division pattern, assigns a transmission timestamp, and transmits the divided messages. The delay measurement unit 32 passes a memory ID, a start address, a transmission message size, an RDMA operation type, a division pattern list, and a division pattern number to the message division unit 33. If the RDMA operation type is WRITE, WRITE with immediate, or READ, the delay measurement unit 32 passes the memory ID and start address of the memory area reserved by the receiving node 200 in addition to the above information. If UD is used as the transport mode, the delay measurement unit 32 passes a QP number and a security key.

[0039] The specified division mode is a mode in which the transmission application 150 specifies the division method, and can be used even when the first phase has not been executed. In the specified division mode, the message division unit 33 divides the message using a specified division pattern. When using the specified division mode, the transmission application 150 passes a memory ID, a start address, a transmission message size, an RDMA operation type, and a division pattern list to the message division unit 33. When the RDMA operation type is WRITE, WRITE with immediate, or READ, the transmission application 150 passes the memory ID and start address of the memory area reserved by the receiving node 200 in addition to the above information. When UD is used as the transport mode, the transmission application 150 passes a QP number and a security key.

[0040] [Example 1] The processing flow of the communication system of this embodiment will be described below using several examples that differ in RDMA operation type and whether or not message splitting is used. In the examples, time synchronization and session establishment are set as follows. The protocol used for time synchronization between the sending node 100 and the receiving node 200 is Precision Time Protocol (PTP), and the clock source is the HWC 111 provided in the NIC 11. Time synchronization is performed with the sending node 100 as the master and the receiving node 200 as the slave. To establish a session for exchanging control information, an RDMA Communication Manager (rdma-cm) is used with the sending node 100 as the client and the receiving node 200 as the server. However, rdma-cm does not necessarily have to be used for exchanging control information, and other alternative functions may be used.

[0041] An example of the flow of processing in the first phase of the first embodiment will be described with reference to the sequence diagrams of Figures 5 to 7. In the first embodiment, the operation type is SEND and the transport type is RC. The division determination unit 31 starts the delay measurement process. The division determination unit 31 transmits a delay measurement request to the delay measurement unit 32 together with the number of attempts (e.g., 1000), connection destination information (e.g., the IP address and port number of the receiving node 200), and the division pattern number.

[0042] First, an example of a process that is performed before transmitting and receiving delay measurement traffic will be described with reference to the sequence diagram of FIG.

[0043] In step S11, the sending node 100 and the receiving node 200 synchronize time between the nodes. In this embodiment, the sending node 100 provides a reference time (reference clock), and time synchronization is achieved between the sending node 100 and the receiving node 200. The sending node 100 distributes the reference clock in accordance with PTP defined in IEEE1588. However, PTP is merely an example, and other protocols such as NTP may also be used. The time synchronization function 23 of the receiving node 200 reflects the reference clock in the HWC 111.

[0044] The time synchronization function 23 reflects the time obtained from the reference clock in the HWC 111. The time synchronization function 23 manages identification information of the reference clock reflected in the HWC 111 in an internal database. After reflecting the reference clock in the HWC 111, each node synchronizes the time of the system clock 24 managed by the OS 21 based on the HWC 111 for which time synchronization has been completed. The time synchronization function 23 may receive reference time from an external Global Navigation Satellite System (GNSS) and reflect it in the HWC 111. In this case, after reflecting the reference clock from the GNSS to the HWC 111, the sending node 100 operates as a master and distributes the reference clock to slaves.

[0045] In this embodiment, time synchronization is performed before other procedures, but it is sufficient if it is performed before the transmission timestamp is required.

[0046] In step S12, the session control unit 35 of each node establishes a control information exchange session and exchanges control information in preparation for message transmission. In response to a delay measurement request from the division determination unit 31, the session control unit 35 starts session establishment. Session establishment may be initiated by either the sending side or the receiving side. In this embodiment, rdma-cm is used to establish a session, with the receiving side's session control unit 35 (server) waiting and the sending side's session control unit 35 (client) requesting a connection. Generally, UDP port 4791 is used for RoCEv2 traffic, but a different port number may also be used. It is assumed that the port number is known by the session control unit 35. The session control unit 35 of the sending node 100 sends a session establishment request to the session control unit 35 of the receiving node 200. The session is then established using rdma-cm according to a standard procedure.

[0047] After the session is established, the sending node 100 and the receiving node 200 exchange the following information in order via the control information exchange session.

[0048] First, Active MTU (AMTU) information is exchanged. AMTU indicates the maximum size of an RDMA message that can fit into one packet, as configured by the NIC 11 of each node. AMTU can be obtained by querying the device information of the NIC 11. Currently, AMTU is defined as 256, 512, 1024, 2048, or 4096 bytes.

[0049] After exchanging AMTU information, the AMTUs of the NICs 11 of each node are compared, and the smaller AMTU value is adopted as the Path MTU (PMTU). PMTU is the maximum size that a packet can be transmitted over a path without being fragmented.

[0050] Next, metadata structures are exchanged. The metadata includes the transmission timestamp and segmentation information written in the delay measurement message. The metadata includes, for example, the transmission timestamp, the segmentation pattern number, and a segmentation-related flag and sequence number. When exchanging the metadata structure, information such as the location and size of the transmission timestamp, the location and size of the segmentation pattern number, and the segmentation-related flag and size is exchanged. The location of the transmission timestamp is the memory location of the transmission timestamp to be embedded in the delay measurement message and is indicated by an offset value from the beginning of the message. The location of the segmentation pattern number is the memory location of the segmentation pattern number to be embedded in the delay measurement message and is indicated by an offset value from the beginning of the message. The segmentation-related flags include a flag indicating the start of segmentation (segmentation start flag), a flag indicating the end of segmentation (segmentation end flag), and a segmentation sequence number (segmentation sequence number). In addition to the metadata structure, the maximum number of segments is also exchanged.

[0051] Next, the maximum message size of the delay measurement traffic is exchanged.

[0052] Next, the number of trials is exchanged. The number of trials is provided by the division determination unit 31.

[0053] Finally, a termination signal is exchanged to complete the control information exchange process.

[0054] The RDMA messages used for exchanging control information are, for example, equal to or smaller than the minimum size of PMTU (256 bytes).

[0055] In step S13, the delay measurement unit 32 on the sending side reserves memory for the maximum message size to be measured using the RDMA general-purpose library 22. For example, if the maximum message size is 1 GB, 1 GB of memory is reserved. The delay measurement unit 32 manages a memory ID that uniquely identifies the reserved memory area and a start address.

[0056] In step S14, the receiving-side delay measurement unit 38 reserves a memory area equal to or larger than the memory size reserved by the delay measurement unit 32 so that all data transmitted from the transmitting-side delay measurement unit 32 can be written without omission. The memory size to be reserved by the delay measurement unit 38 need only be equal to or larger than the maximum message size exchanged in the information exchange in step S12. The delay measurement unit 38 manages a memory ID that uniquely identifies the reserved memory area and a start address.

[0057] Once time synchronization, control information exchange, and memory area allocation are complete, in step S15, the delay measurement unit 32 and the delay measurement unit 38 establish a delay measurement session for transmitting and receiving delay measurement messages. Two sessions exist between the sending node 100 and the receiving node 200: a control information exchange session and a delay measurement session. In the following explanation, the delay measurement session will be used unless otherwise specified.

[0058] Next, an example of a process for transmitting a message for delay measurement will be described with reference to the sequence diagram of FIG.

[0059] Once the delay measurement session is established, in step S16, the delay measurement unit 32 transmits to the message division unit 33 a memory ID indicating the memory area reserved in step S13, a start address, the size of the message to be transmitted, an RDMA operation type, a division pattern list, and a division pattern number, requesting message transmission via RDMA. In the first embodiment, the RDMA operation type is SEND. The division pattern list is a list indicating the number of bytes in the message after division. For example, when a 16-byte message is divided into four (row number 4, column number 2 in Table 1), the division pattern list is 4, 4, 4, 4. The delay measurement unit 32 specifies a delay measurement mode for the message division unit 33.

[0060] The delay measurement unit 38 on the receiving side also prepares for message reception by RDMA by transmitting the memory ID indicating the memory area secured in step S14, the start address, the memory size, the RDMA operation type, the division pattern list, and the division pattern number to the message integration unit 37. The delay measurement unit 38 specifies the delay measurement mode to the message integration unit 37.

[0061] When the message division unit 33 receives the message transmission request, in step S17, it acquires a transmission timestamp via the time reference unit 36. The time reference unit 36 ​​acquires the transmission timestamp by referring to the HWC 111 or the system clock 24. Whether the clock to be referenced, the HWC 111 or the system clock 24, may be specified by the message division unit 33 or may be determined in advance.

[0062] In step S18, the message divider 33 writes a transmission timestamp and a flag or sequence number related to division at a position obtained by adding a specified offset to the start address. Regarding the flags related to division, the message divider 33 writes a division start flag and a maximum number of divisions to the divided message to be sent first, a division end flag to the divided message to be sent last, and a division sequence number to the other divided messages. If no division is performed, both a division start flag and a division end flag are written. A divided message is each part of the original message after division. The transmission timestamp may be written to all divided messages, or may be written to any of the divided messages (for example, the first or last divided message to be sent).

[0063] After writing the metadata, in step S19, the message division unit 33 requests message transmission by passing the memory ID, start address, size of the divided messages, and RDMA operation type (SEND operation in the first embodiment) from the RDMA general-purpose library 22 to the NIC 11 via the NIC driver. The RDMA general-purpose library 22 provides a common Verbs API such as Open Fabric Enterprise Distribution (OFED) for executing a transmission request by RDMA. RoCEv2 can be used to request message transmission. Note that because the message is divided and transmitted, the start address is shifted backward each time a divided message is transmitted.

[0064] In step S20, the NIC 11 reads out the message fragments by the size from the memory at the start address and transmits them to the receiving node 200 as RoCEv2 packets.

[0065] The receiving node 200 waits for reception based on information about the memory ID, start address, and memory size of the memory area to which the received fragmented messages will be written. The NIC 11 of the receiving node 200 receives the RoCEv2 packet and writes the received fragmented messages to the memory area allocated by the delay measurement unit 38.

[0066] The message divider 33 repeats the process from step S18 to step S20 until the entire message has been transmitted, that is, the process from step S18 to step S20 is repeated the number of times equal to the number of divisions of the message.

[0067] When the NIC 11 has completed receiving the divided messages, it sends a reception completion notification to the message integration unit 37 in step S21 (submits a CQE to the CQ).

[0068] In step S22, if the status code included in the reception completion notification indicates success, the message integration unit 37 acquires metadata such as a transmission timestamp, a division pattern number, and a flag related to division from the received divided messages.

[0069] The message integration unit 37 repeats the processes from step S20 to step S22 while checking the flags related to the division and the division sequence numbers until all the divided messages have been received.

[0070] When the message integration unit 37 receives all the divided messages, in step S23, it acquires the reception timestamp via the time reference unit 36 ​​and passes the delay measurement result including the division pattern number, transmission timestamp, and reception timestamp to the delay measurement unit 38.

[0071] The process of transmitting and receiving the message for delay measurement from step S16 to step S23 shown in FIG. 6 is repeated the number of times of trial.

[0072] Next, an example of a process for registering delay statistical information will be described with reference to the sequence diagram of FIG.

[0073] Once delay measurement results for the number of trials have been obtained, the delay measurement unit 38 calculates statistical information for the delay measurement results for the number of trials in step S24. Specifically, the delay measurement unit 38 calculates the delay time by taking the difference between the reception timestamp and the transmission timestamp, and calculates delay statistical information such as the average, minimum, maximum, percentile, and standard deviation of the delay time.

[0074] In step S25, the delay measurement unit 38 transmits the division pattern number and the delay statistical information to the transmitting node 100 via the control information exchange session.

[0075] In step S26, the statistical information management unit 34 registers the received division pattern number and delay statistical information.

[0076] The division determination unit 31 executes the processes from step S16 to step S26 in FIGS. 6 and 7 for all division pattern numbers.

[0077] [Determining Division Pattern] Next, an example of a process for determining a division pattern that improves a parameter that a user values ​​based on the obtained delay statistical information will be described.

[0078] Once delay statistical information for all division pattern numbers has been obtained, the division judgment unit 31 extracts the division pattern that improves the parameters that the user values ​​for each transmission message size (each row of Table 1), and determines the division pattern that the message division unit 33 will use in automatic division mode.

[0079] For example, if the user desires a division pattern with the smallest average delay time, the division determination unit 31 extracts the division pattern number with the smallest average delay time for each transmission message size, i.e., for each row in Table 1, and sets the division pattern for each transmission message size in the message division unit 33. This allows the message division unit 33 to divide the message using the division method with the smallest average delay time in the automatic division mode.

[0080] As another example, an example of processing for extracting a division pattern that has a smaller standard deviation and the smallest percentile value compared to a case where no division is performed will be described with reference to the flowchart in Fig. 8. The division determination unit 31 performs the processing in Fig. 8 for each transmission message size.

[0081] In step S101, the division determination unit 31 acquires percentile value A and standard deviation B for the case where no division is performed.

[0082] In step S102, the division determination unit 31 acquires percentile values ​​and standard deviations of unconfirmed division patterns.

[0083] In step S103, the division determination unit 31 determines whether the standard deviation acquired in step S102 is smaller than the standard deviation B acquired in step S101.

[0084] If the standard deviation is greater than B, the division determination unit 31 advances the process to step S106 to process the next division pattern.

[0085] If it is smaller than the standard deviation B, the division determination unit 31 determines whether the percentile value acquired in step S102 is smaller than the held percentile value A. The initial value of the percentile value A is the percentile value acquired in step S101.

[0086] If it is greater than percentile value A, the division determination unit 31 advances the process to step S106 to process the next division pattern.

[0087] If it is smaller than percentile value A, in step S105, the division determination unit 31 updates percentile value A to the percentile value acquired in step S102.

[0088] In step S106, the division determination unit 31 determines whether or not all division patterns have been checked.

[0089] If there is a division pattern that has not been confirmed, the division determination unit 31 executes the processes of steps S102 to S105 for the next division pattern.

[0090] If all division patterns have been confirmed, in step S107, the division determination unit 31 selects the division pattern of percentile value A and registers the presence or absence of division, the RDMA operation type, and the division pattern list for each transmission message size to the message division unit 33.

[0091] The above algorithm for selecting a division pattern is an example, and various algorithms can be set according to parameters desired by the user.

[0092] The above processing completes the first phase. In the second phase, the automatic division mode of the message divider 33 can be used. However, the sending application 150 does not necessarily have to use the automatic division mode, and may use the designated division mode.

[0093] [Embodiment 2] An example of the flow of processing in the first phase of embodiment 2 will be described with reference to the sequence diagrams of Figures 5 to 7. In embodiment 2, the operation type is WRITE / WRITE with immediate, and the transport type is RC. Below, differences from embodiment 1 will be mainly described.

[0094] Steps S11 to S14 in FIG. 5, including time synchronization, exchange of control information, and reservation of memory area, are the same as those in the first embodiment, and therefore redundant explanations will be omitted here.

[0095] In the WRITE / WRITE with immediate operation, the sending node 100 needs to know the memory ID and memory address of the write destination of the connected party. In the second embodiment, when the delay measurement session is established in step S15, the sending node 100 and the receiving node 200 use the control information exchange session to exchange the memory ID reserved by the receiving node 200 and the start address of the write destination.

[0096] In step S16 of FIG. 6, the delay measurement unit 32 requests the message division unit 33 to transmit a message via RDMA by transmitting the memory ID, start address, message size to be transmitted, RDMA operation type, division pattern list, and division pattern number, as well as the memory ID and start address of the write destination notified by the receiving node 200.

[0097] As in the first embodiment, when the message division unit 33 receives a message transmission request, it acquires a transmission timestamp in step S17, and writes the transmission timestamp, flags related to division, etc. into the divided messages in step S18.

[0098] In step S19, the message division unit 33 requests message transmission by passing the memory ID of the receiving node 200 and the start address of the write destination to the NIC 11 via the NIC driver from the RDMA general-purpose library 22, in addition to the memory ID, start address, size of the divided messages, and RDMA operation type (WRITE with immediate operation in the second embodiment). Because the message is divided and transmitted, the start address of the write destination shifts backward each time a divided message is transmitted.

[0099] In the WRITE operation, the receiving node 200 is not notified of the completion of transfer, so in the delay measurement mode, the WRITE with immediate operation is used to send the divided messages. The delay statistical information obtained using the WRITE with immediate operation is used as the delay statistical information for the WRITE operation and the WRITE with immediate operation. For the WRITE operation, the delay statistical information is approximate. It is also possible to use the WRITE operation for delay measurement, and have the sending node 100 notify the receiving node 200 of the completion of transfer using a control information exchange session, but since this basically slows down the notification, the optimal method is to use the delay statistical information obtained using the WRITE with immediate operation as the delay statistical information for the WRITE operation.

[0100] In step S20, the NIC 11 transmits the divided messages to the receiving node 200.

[0101] The NIC 11 of the receiving node 200 receives the RoCEv2 packet and writes the fragmented messages to a memory. In the second embodiment, the destination to write the fragmented messages is determined by the memory ID and start address written in the RoCEv2 packet.

[0102] The message divider 33 repeats the process from step S18 to step S20 until the entire message has been transmitted, that is, the process from step S18 to step S20 is repeated the number of times equal to the number of divisions of the message.

[0103] When the NIC 11 has completed receiving the divided messages, it sends a reception completion notification to the message integration unit 37 in step S21.

[0104] In step S22, if the status code included in the reception completion notification indicates success, the message integration unit 37 acquires metadata such as a transmission timestamp, a division pattern number, and a division-related flag from the received divided messages. At this time, the message integration unit 37 acquires the metadata while shifting the position by the size of the divided messages from the start address secured in step S14.

[0105] The message integration unit 37 repeats the processes from step S20 to step S22 while checking the flags related to the division and the division sequence numbers until all the divided messages have been received.

[0106] When the message integration unit 37 receives all the divided messages, in step S23, it acquires the reception timestamp via the time reference unit 36 ​​and passes the delay measurement result including the division pattern number, transmission timestamp, and reception timestamp to the delay measurement unit 38.

[0107] The process of transmitting and receiving the message for delay measurement from step S16 to step S23 shown in FIG. 6 is repeated the number of times of trial.

[0108] The process of registering the delay statistical information and the determination of the division pattern in FIG. 7 are the same as those in the first embodiment, so a duplicated description will be omitted here.

[0109] An example of the flow of processing in the first phase of the third embodiment will be described with reference to the sequence diagrams of Fig. 5 and Fig. 9. In the third embodiment, the operation type is READ and the transport type is RC. Unlike the first and second embodiments, the third embodiment transmits delay measurement traffic from the receiving node 200 to the transmitting node 100.

[0110] In the third embodiment, the sending node 100 acquires both the sending timestamp and the receiving timestamp, so the time synchronization in step S11 in FIG. 5 is not necessary.

[0111] In the third embodiment, since it is not necessary to embed a transmission timestamp, the metadata does not include the transmission timestamp. In the exchange of control information in step S12, a metadata structure that does not include a transmission timestamp is exchanged.

[0112] A memory area is reserved in steps S13 and S14, and a delay measurement session is established in step S15.

[0113] In the READ operation, the sending node 100 needs to know the memory ID and memory address of the read destination of the connected party. In the third embodiment, when the delay measurement session is established in step S15, the sending node 100 and the receiving node 200 use the control information exchange session to exchange the memory ID reserved by the receiving node 200 and the start address of the read destination.

[0114] When a delay measurement session is established, in step S31 of Figure 9, the delay measurement unit 32 requests the message division unit 33 to transmit a message via RDMA by transmitting the memory ID, start address, message size to be transmitted, RDMA operation type, division pattern list, and division pattern number, as well as the memory ID and start address of the read destination notified by the receiving node 200.

[0115] When the message division unit 33 receives a message transmission request, it acquires a transmission timestamp in step S32. Note that in the third embodiment, the acquired transmission timestamp is not embedded in the message.

[0116] In step S33, the message division unit 33 requests message transmission by passing the memory ID of the receiving node 200 and the start address of the read destination to the NIC 11 via the NIC driver from the RDMA general-purpose library 22, in addition to the memory ID, start address, size of the divided messages, and RDMA operation type (READ operation in the third embodiment). Because the message is divided and transmitted, the start address of the read destination shifts backward each time a divided message is transferred.

[0117] In step S34 , the NIC 11 of the sending node 100 sends a READ message to the receiving node 200 .

[0118] In step S35, upon receiving the READ message, the NIC 11 of the receiving node 200 uses DMA to read the divided messages from the memory at the start address of the read destination, and transmits the divided messages to the sending node 100 as RoCEv2 packets.

[0119] When the NIC 11 has completed receiving the divided messages, it transmits a reception completion notification to the message divider 33 in step S36.

[0120] The message division unit 33 repeats the processes from step S33 to step S36 until all the divided messages are received.

[0121] When the message division unit 33 receives all the divided messages, it acquires the reception timestamp via the time reference unit 36 ​​in step S37.

[0122] The process of transmitting and receiving the message for delay measurement from step S31 to step S37 in FIG. 9 is repeated the number of times of trial.

[0123] When delay measurement results for the number of trials are obtained, in step S38, the delay measurement unit 32 calculates statistical information on the delay measurement results for the number of trials.

[0124] In step S39, the statistical information management unit 34 registers the received division pattern number and delay statistical information.

[0125] The division determination unit 31 executes the processes from step S31 to step S39 in FIG. 9 for all division pattern numbers.

[0126] The division pattern is determined in the same manner as in the first embodiment, so a duplicated explanation will be omitted here.

[0127] [Examples of Operation Types UC and UD] When the operation type is UC, the first phase processing can be executed in the same manner as in the first and second embodiments.

[0128] Even when the operation type is UD, the first phase processing can be executed in the same way as in the first embodiment. However, when the operation type is UD, there is no concept of a session, and a handler is constructed. A handler indicates the relationship with the connection partner, and is equivalent to the concept of a session except that there is no status management on the NIC. Therefore, the session in the first embodiment can be read as a handler, and the first phase processing can be executed in the same way.

[0129] The division determination system 300 of this embodiment determines a division method for data to be transferred between the sending node 100 and the receiving node 200. The sending node 100 includes a delay measurement unit 32 that measures, for each data division method, the delay when data is divided and transferred using that division method, a statistical information management unit 34 that holds delay statistical information obtained from the delay measurement results for each division method, a division determination unit 31 that determines a division method for data to be transferred based on the delay statistical information for each division method, and a message division unit 33 that divides and transfers data using the division method determined by the division determination unit 31. This makes it possible to divide and transfer data using the optimal division method.

[0130] The sending node 100 and receiving node 200 described above can be, for example, a general-purpose computer system including a central processing unit (CPU) 901, memory 902, storage 903, a communication device 904, an input device 905, and an output device 906, as shown in Fig. 10. In this computer system, the sending node 100 and receiving node 200 are realized by the CPU 901 executing a predetermined program loaded onto the memory 902. This program can be recorded on a computer-readable non-transitory recording medium such as a magnetic disk, optical disk, or semiconductor memory, or can be distributed via a network.

[0131] REFERENCE SIGNS LIST 1 Communication system 100 Sending node 150 Sending application 200 Receiving node 250 Receiving application 300 Division determination system 11 NIC 111 HWC 12 CPU 13 Memory 21 OS 22 RDMA general-purpose library 23 Time synchronization function 24 System clock 31 Division determination unit 32 Delay measurement unit 33 Message division unit 34 Statistical information management unit 35 Session control unit 36 ​​Time reference unit 37 Message integration unit 38 Delay measurement unit

Claims

1. A division determination system that determines a division method for data to be transferred between a first communication device and a second communication device, wherein the first communication device comprises: a delay measurement unit that measures, for each data division method, the delay when the data is divided and transferred using that division method; a statistical information management unit that holds delay statistical information obtained from the delay measurement results for each division method; a division determination unit that determines a division method for the data to be transferred based on the delay statistical information for each division method; and a division unit that divides and transfers the data using the division method determined by the division determination unit.

2. A division determination system according to claim 1, wherein the division method is formed by a combination of the size of the data and the number of divisions of the data, and the division determination unit determines the division method according to the size of the data to be transferred.

3. A division determination system according to claim 1, wherein the division determination unit selects a division method that improves desired delay statistical information from the delay statistical information.

4. A division judgment system according to claim 1, wherein the delay measurement unit embeds a transmission timestamp at the start of data transfer into the data, and the second communication device measures the delay from the difference between the transmission timestamp and a reception timestamp when all of the data has been received.

5. A division judgment system according to claim 1, wherein the delay measurement unit requests the second communication device to divide and transfer the data using a specified division method, and measures the delay from the difference between a sending timestamp when the data transfer is requested and a receiving timestamp when all of the data is received.

6. A division judgment system as described in claim 1, wherein the first communication device and the second communication device transfer data using remote direct memory access, the statistical information management unit holds the delay statistical information for each remote direct memory access operation type, and the division judgment unit determines a division method for the data to be transferred based on the operation type and the delay statistical information for each division method.

Images