Parallel and distributed computing system

Abstract

According to the present invention, in the environment that a plurality of computers are connected to each other through a plurality of data links, and perform communication and synchronization with each other by MBCF, the breaking of memory consistency due to disturbance of the order of packet arrival is eliminated. In the case where a plurality of computers 2 are connected to each other through a plurality of data links, and perform communication and synchronization with each other by MBCF, a transmission-side process has a transmission number-counting unit 201 that counts, for each reception-side process, the number of times transmission of an operation request packet has been made, and attaches the count of the operation request packet before or after counting up to the operation request packet. The reception-side process has a reception number-counting unit 202 that counts, for each transmission-side process, the number of times reception of the operation request packet has been made. When the count of the reception number-counting unit 202 is continuous to the count attached to the operation request packet, a transmission destination computer 2(2Y)2 processes the operation request packet, and updates the count of the reception number-counting unit 202 using the count attached to the operation request packet.

Description

[Technical field]

[0001] The present invention relates to a parallel distributed computing system in which multiple computers each equipped with a processor having a translation lookaside buffer (TLB), physical memory, and a network interface controller (NIC) capable of directly accessing the physical memory are interconnected via multiple data links. [Background technology]

[0002] The inventors of this application are developing a Memory Based Communication Facility (MBCF), which is one of the communication synchronization mechanisms based on the memory-based communication synchronization method, as shown in Non-Patent Document 1. This MBCF does not require any special communication synchronization hardware, but uses a general network interface card (NIC) to achieve high-speed, highly functional communication synchronization through remote memory operation using only software. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Matsumoto, T.: A Study on Memory-Based Communications and Synchronization in Distributed-Memory Systems. Dissertation Thesis, Graduate School of Science, Univ. of Tokyo (February 2001). Summary of the Invention [Problem to be solved by the invention]

[0004] Specifically, the MBCF is configured using a computer equipped with a processor having a translation lookaside buffer (TLB), a physical memory, and a network interface controller (NIC) that can directly access the physical memory. For example, a process of a source computer (hereinafter, a sending process) transmits an operation request packet including an identifier of an operation target process (hereinafter, a receiving process) that specifies a process of a destination computer, an operation target address that specifies a memory area of ​​the receiving process, a data size to be written, and a data string. Then, the destination computer receives the operation request packet transmitted by the sending process, and stores the data string in a memory area specified by the receiving process and the operation target address.

[0005] The inventor of this application is considering constructing a parallel distributed computing system in which multiple computers are interconnected via multiple data links (link aggregation connection). In this case, it is assumed that each node (computer) has multiple data links, but because multiple data links exist, there is a possibility that the arrival order of remote memory operation request packets will be disrupted and memory consistency will be broken by distributing the load of packets to the multiple links.

[0006] The present invention has been made to solve the above problems, and its main objective is to maintain memory consistency in a situation where the arrival order of packets is disturbed in a parallel distributed computing system in which multiple computers each equipped with a processor having a translation lookaside buffer (TLB), physical memory, and a network interface controller (NIC) capable of directly accessing the physical memory are interconnected via multiple data links. [Means for solving the problem]

[0007] That is, the parallel distributed computing system according to the present invention is a parallel distributed computing system in which a plurality of computers each having a processor with a translation lookaside buffer (TLB), a physical memory, and a network interface controller (NIC) capable of directly accessing the physical memory are interconnected via a plurality of data links, in which a process of a source computer (hereinafter, a sending process) transmits an operation request packet including an identifier of an operation target process (hereinafter, a receiving process) that specifies a process of a destination computer, an operation target address that specifies a memory area of ​​the receiving process, a data size to be written, and a data string, and the destination computer receives the operation request packet and determines whether the receiving process and the operation target address are identical to each other. and storing the data string in a memory area defined by a plurality of addresses, the sending process having a sending number counting unit which counts the number of times the operation request packet has been sent for each receiving process and which attaches to the operation request packet a count number before or after the operation request packet was counted up, the receiving process having a receiving number counting unit which counts the number of times the operation request packet has been received for each sending process, and the destination computer processes the operation request packet when the count number in the receiving number counting unit and the count number attached to the operation request packet are consecutive, and updates the count number in the receiving number counting unit with the count number attached to the operation request packet.

[0008] In such a parallel distributed computing system, in which multiple computers are interconnected via multiple data links and communicate with each other in synchronization using MBCF, the sending process has a sending number counting unit which attaches a count number before or after counting up the operation request packet to the operation request packet, and the receiving process has a receiving number counting unit which counts the number of operation request packets received for each sending process, and when the count number in the receiving number counting unit and the count number attached to the operation request packet are consecutive, the destination computer processes the operation request packet and updates the count number in the receiving number counting unit with the count number attached to the operation request packet, thereby making it possible to eliminate disruptions to the arrival order of operation request packets as necessary.

[0009] As a specific processing mode when there is an operation request packet that has not arrived, when the arrival sequence at the receiving process is disturbed and there is an operation request packet that has not arrived, the destination computer does not update the count number in the reception number counting section only up to the amount up to the operation request packet that has not arrived, but processes the operation request packets that arrive after the operation request packet that has not arrived, and records the count number attached to the operation request packet in the already arrived count recording section separately from the reception number counting section.

[0010] As a specific processing mode after the arrival of an operation request packet that has not yet arrived, when the arrival of the operation request packet that has not yet arrived causes the recorded contents of the arrival record to the receiving process to be continuous from the reception count count unit, the destination computer uses the already arrived count recording unit to update the reception count count unit within that continuous range and deletes the updated record from the already arrived count recording unit.

[0011] In order to guarantee the processing order of operation request packets, the sending process conveys its intention to prohibit the operation request packet it is about to send from being operated on until the processing of the preceding operation request packet is completed, and adds to the operation request packet a count of the number of preceding operation request packets sent that it wishes to ensure have been completed as multiple link synchronization information.

[0012] As a specific processing mode of operation request packets for ensuring the processing order of operation request packets, when the destination computer receives an operation request packet to which the multiple link synchronization information is attached, it refers to the reception count count unit of the receiving process, and if the count number described in the multiple link synchronization information is less than or equal to the count number of the reception count unit, it processes the operation request packet and updates the reception count count unit and the already arrived count recording unit.

[0013] As a specific processing mode of operation request packets for guaranteeing the processing order of operation request packets, when the destination computer receives an operation request packet to which the multiple link synchronization information is attached, it refers to the reception number count unit of the receiving process, and if the count number described in the multiple link synchronization information is greater than the count number of the reception number count unit, it delays processing of the operation request packet until the count number of the reception number count unit increases to be greater than or equal to the count number described in the multiple link synchronization information, and also delays the updating of the reception number count unit and the already arrived count recording unit. Effect of the Invention

[0014] According to the present invention configured in this manner, in a parallel distributed computing system in which multiple computers are interconnected via multiple data links and the multiple computers communicate with each other and synchronize using MBCF, it is possible to maintain memory consistency at low cost in a situation in which the arrival order of packets is disrupted. [Brief description of the drawings]

[0015] [Figure 1]1 is a diagram illustrating an overall configuration of a parallel distributed computing system according to an embodiment of the present invention. [Diagram 2] 2 is a schematic diagram showing the physical configuration of a source computer and a destination computer of the embodiment. FIG. [Diagram 3] FIG. 2 is a schematic diagram showing an environment in which MBCF_WRITE is used in the embodiment. [Figure 4] 11 is a schematic diagram showing an operation procedure of transmitting an operation request packet, including a packet generation process on the operation request side of the embodiment. FIG. [Diagram 5] 11 is a schematic diagram showing a state in which an operation request packet arrives at a receiving node in the embodiment. FIG. [Figure 6] 13 is a schematic diagram showing MBCF-related processing in a packet reception interrupt routine in a receiving node of the embodiment. FIG. [Figure 7] 11 is a schematic diagram showing a state in which operation request packets arrive out of sequence; FIG. [Figure 8] FIG. 2 is a functional block diagram of a computer according to the embodiment. [Figure 9] 1 is a schematic diagram showing a transmission number counting section and a reception number counting section of node Na as a representative of nodes Na, Nb, and Nc. [Figure 10] 13 shows a first example of a remote memory operation in which synchronization information is inserted according to the embodiment. [Figure 11] 13 is a second example of a use of a remote memory operation in which synchronization information is inserted according to the embodiment. [Explanation of symbols]

[0016] 100 Parallel and distributed computing system 2...calculator 2X...Sender computer 2Y...Destination computer 21 Processor 22 Physical Memory 23 Network Interface Controller (NIC) 3. Data Link 201 Transmission number count unit 202 Reception number count unit 203: Arrival count record section DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] A parallel distributed computing system 100 according to an embodiment of the present invention will be described below with reference to the drawings.

[0018] 1, a parallel distributed computing system 100 of this embodiment is configured by interconnecting a plurality of computers 2 via a plurality of data links 3. By interconnecting a plurality of computers 2 via a plurality of data links 3, a link aggregation coupled parallel distributed computing system is configured.

[0019] As shown in FIG. 2, each computer 2 is equipped with a processor 21 having a translation look-aside buffer (TLB), a physical memory 22, and a network interface controller (NIC; Network Interface Card) 23 that can directly access the physical memory 22.

[0020] The parallel distributed computing system 100 does not require any special communication synchronization hardware, but instead uses a general network interface card (NIC) 23 to construct a memory based communication facility (MBCF) that realizes high speed and high performance communication synchronization through remote memory operation using only software. Specifically, the parallel distributed computing system 100 constructs a memory based communication facility (MBCF) using an operating system (OS) stored in the kernel space of each computer 2.

[0021] The parallel distributed computing system 100 has a variety of operation command variations, such as a write command (MBCF_WRITE) for writing to a remote memory and a read command (MBCF_READ) for reading from a remote memory, as shown below.

[0022] For example, a process of the sending computer 2 (2X) (hereinafter, the sending process) sends an operation request packet including an identifier of an operation target process (hereinafter, the receiving process) that specifies a process of the destination computer 2 (2Y), an operation target address that specifies the memory area of ​​the receiving process, the data size to be written, and a data string, and the destination computer 2 receives the operation request packet and stores the data string in the memory area specified by the receiving process and the operation target address (MBCF_WRITE).

[0023] In addition, the sending process sends an operation request packet including an identifier of the operation target process (hereinafter referred to as the receiving process) that specifies the process of the destination computer 2 (2Y), an operation target address that specifies the memory area of ​​the receiving process, the size of the data to be read, and the data storage area address of the sending process, and the destination computer 2 (2Y) receives the operation request packet, reads out the data string from the memory area specified by the receiving process and the operation target address, and returns it to the data storage area of ​​the sending process (MBCF_READ).

[0024] Here, the procedure of MBCF_WRITE will be described in detail with reference to Figures 3 to 6. Note that the basic procedure is the same for other commands as well.

[0025] Figure 3 shows the environment in which MBCF_WRITE is used, where Pnode1 is the source node (source computer) of the operation request packet, and Pnode2 is the receiving node (destination computer). The processor of the source node creates a packet image in the NIC DMA area of ​​its memory, including a header with delivery information and a payload. The NIC DMA area memory can be directly accessed by the NIC for sending or receiving. When the generation of the packet image for sending is complete, the processor issues an instruction (kick operation) to the NIC to start the sending operation by DMA reading. The receiving node (Pnode2) has a ring buffer for packets that arrive in the NIC DMA area of ​​its memory. The NIC of the receiving node determines whether the packet is addressed to its node (usually by the MAC address), and for packets addressed to itself, creates a copy of the packet in the ring buffer. The NIC then generates a hardware interrupt to notify the processor of the receiving node of the arrival of the packet.

[0026] Next, referring to FIG. 4, the operation procedure for transmitting an operation request packet, including the packet generation process on the operation request side, will be described.

[0027] The sending process (requesting task) prepares parameters consisting of the identifier [Ltask1] of the receiving process (requested task), the memory address to be operated on by the receiving process [Laddr1], the access key [AccessKey] for memory space operation of the receiving process, the MBCF command type [MBCF_WRITE], the data size [n] for remote writing, and the pointer to the beginning of the area where the data to be written is stored [Laddr0]. Then, it calls the MBCF request sending system call with these parameters. Upon receiving the system call, the OS refers to the task table of the sending process and converts the logical task ID indicating the receiving process into a physical task ID [(Pnode2, Ptask5)]. Since the physical task ID includes the physical node ID Pnode2, this information can be used to set the route information (destination information) to the receiving node. If the network used is Ethernet, the MAC address is used as the destination information. This destination information enables the NIC to deliver the operation request packet to the receiving node. The OS then has the NIC send the operation request packet.

[0028] Next, a procedure for receiving an operation request packet at the operation request destination will be described with reference to FIG. 5 and FIG. Figure 5 shows the situation when an operation request packet arrives at the receiving node. The operation request packet is carried by the network to the receiving node [Pnode2]. When the packet arrives, the NIC of the receiving node copies the data image of the operation request packet to the ring buffer by DMA, and then generates an interrupt signal to inform the processor of the receiving node that the packet has arrived.

[0029] Figure 6 shows the MBCF-related processing in the packet reception interrupt routine in the receiving node. When a reception interrupt occurs from the NIC, the processor of the receiving node switches control to the packet reception routine, and first executes the low-level packet reception procedure requested by the NIC. In the packet reception routine, the receiving process is first identified from the physical task ID (Ptask5 in the figure). Specifically, a pointer to the process structure of that process is obtained. Next, it is checked whether the AccessKey in the operation request packet matches that of the receiving process, and memory operations in the memory space of the receiving process are permitted only if they match. If they match, the context of the memory space is switched to that of the receiving process, and the n bytes of data carried in the operation request packet are written from the logical address to be operated on (Laddr1) using a store command with user authority rather than a privilege level. Then, the context of the memory space is returned to that at the time of the interrupt occurrence. This is the end of the processing in the interrupt routine.

[0030] Since the parallel distributed computing system 100 of this embodiment has a plurality of data links 3, there is a possibility that the arrival order of operation request packets may be disturbed, as shown in Fig. 7. For this reason, the parallel distributed computing system 100 of this embodiment has a function of guaranteeing the arrival order of operation request packets to the receiving process within the range intended by the user.

[0031] Specifically, the guarantee only within the range intended by the user means that there are variables A, B, C, D, and F in the memory of the receiving process, and the sending process performs remote writing to A, B, C, and D, respectively, using MBCF_WRITE, and finally writes a value (for example, 1) indicating that the writing has been completed to F using MBCF_WRITE. In this case, it is not necessary to guarantee the order between the MBCF_WRITEs of A, B, C, and D, but it is essential to guarantee the order between these four MBCF_WRITEs and the MBCF_WRITE to F. Conversely, guaranteeing the order between all MBCF_WRITEs is an excessive guarantee, and there is a possibility that extra overhead will occur. Therefore, in this embodiment, synchronization information that indicates that the writing of F is performed after all MBCF operation request packets up to the writing of D have been processed is added only to the MBCF_WRITE request packet of F.

[0032] Specifically, as shown in Fig. 8, the function of guaranteeing the arrival order of operation request packets within the range intended by the user is such that each sending process of the source computer 2 (2X) has a sending number counting unit 201 which counts the number of sending operation request packets for each receiving process and attaches to the operation request packet the count number before or after the count-up of the operation request packet. Also, each receiving process of the destination computer 2 (2Y) has a receiving number counting unit 202 which counts the number of receiving operation request packets for each sending process.

[0033] Then, if the count number in the reception number counting unit 202 and the count number attached to the operation request packet are consecutive, the destination computer 2 (2Y) processes the operation request packet and updates the count number in the reception number counting unit 202 with the count number attached to the operation request packet.

[0034] Here, in the case where the destination computer 2 (2Y) has an operation request packet that has not yet arrived due to a disturbance in the arrival order at the receiving process, the destination computer 2 (2Y) only updates the count up to the operation request packet that has not yet arrived in the reception count unit 202. As for the arrival of an operation request packet after the operation request packet that has not yet arrived, if the packet does not have multiple link synchronization information described below attached thereto, the destination computer 2 (2Y) processes the operation request packet and records the count attached to the operation request packet in the already arrived count recording unit 203 separately from the reception count count unit 202.

[0035] Then, when the arrival of an operation request packet that has not yet arrived causes the recorded contents of the arrival record to the receiving process to be continuous from the reception count count unit 202, the destination computer 2 (2Y) uses the already arrived count recording unit 203 to update the reception count count unit 202 within that continuous range and deletes the updated record from the already arrived count recording unit 203.

[0036] At this time, if the sending process wants to prohibit the operation request packet it is about to send from being operated on until the preceding operation request packet has been processed, in other words, if the user (sending process) wants to guarantee the order of arrival, it adds a count of the number of operation request packets sent that it wants to be sure have been processed to the operation request packet as multiple link synchronization information.

[0037] When the destination computer 2 (2Y) receives an operation request packet with multiple link synchronization information attached, it refers to the reception number count unit 202 of the receiving process, and if the count number described in the multiple link synchronization information is less than or equal to the count number of the reception number count unit 202, it processes the operation request packet and updates the reception number count unit 202 and the already arrived count recording unit 203.

[0038] In addition, when the destination computer 2 (2Y) receives an operation request packet with multiple link synchronization information attached, it refers to the reception number count unit 202 of the receiving process, and if the count number described in the multiple link synchronization information is greater than the count number in the reception number count unit 202, it delays processing of the operation request packet until the count number in the reception number count unit 202 increases and becomes greater than or equal to the count number described in the multiple link synchronization information, and also delays the update work of the reception number count unit 202 and the already arrived count recording unit 203.

[0039] Next, an example of a method for updating the count number of reception number counter 202 using a bit vector in already-arrived count recording unit 203 will be described. When it becomes possible to count up the counter using the bit vector with no gaps in the packet numbers, update the counter and the bit vector. The specific update method is as follows:

[0040] If the bit vector length is 8 bits, and packet 101 is missing, while packets 102 and 103 have arrived, the counter value will be 100, and the bit vector will be 00000110 in binary. In other words, the (102-100)th and (103-100)th bits from the least significant will be set to 1.

[0041] If the 105th packet arrives here, the counter value will remain at 100, and the bit vector will be 00010110 in binary. In other words, the (105-100)th bit will be set to 1.

[0042] In this state, when the 101st packet arrives, the (101-100)th bit is set to 1, and the bit vector becomes 00010111 in binary. Since the rightmost bit is set to 1, this means that consecutive packets have arrived in the counter value, so the bit vector is shifted right until the rightmost bit becomes zero, and the shift count is added to the counter.

[0043] In other words, the shift number is 3, so the counter value is 103 and the bit vector in binary is 00000010. When the 104th packet arrives, the (104-103)th bit is set to 1, the bit vector in binary is 00000011, and just like before, 2 followed by 1 is added to the counter, making the counter value 105 and the bit vector in binary 00000000. When the bit vector is all zero, it means that there are no missing packets and that packets have arrived up to the counter value.

[0044] The following method may also be used. Now, if the counter value is 105, the bit vector is 00000000 in binary, and packet number 106 has arrived (not out of order), the (106-105)th bit of the bit vector is initially set to 1. At this time, the counter value is 105, the bit vector is 00000001 in binary, and because the rightmost bit of the bit vector is set to 1, it is shifted one bit to the right, so the counter value is +1, becoming 106, and the bit vector is 00000000 in binary.

[0045] This method makes it possible to process data in the same way as when data is missing, without the need for a procedure to check that the data is in the correct order.

[0046] In the above, an 8-bit bit vector is used, but assuming link aggregation of up to 8 data links, a bit vector of about 32 bits will pose no practical problems. In the unlikely event that a confusion in the order occurs that exceeds the bit width of the bit vector, packets with numbers that exceed the bit width of the bit vector will be refused acceptance and will be subject to retransmission.

[0047] Next, as shown in FIG. 9, three computers 2 are designated as nodes Na, Nb, and Nc, At node Na, processes Ta1, Ta2, Ta3, Node Nb has processes Tb1, Tb2, Tb3, Node Nc has processes Tc1, Tc2, Tc3, Consider the case where there are three processes (also called tasks) for each of the above, and they communicate and synchronize with each other using MBCF. The sending process can use MBCF to send memory operation requests to other processes, including itself, on the same computer as part of remote memory operations.

[0048] In this case, the process Ta1 has nine transmission number counters Xa1_Ta1, Xa2_Ta1, Xa3_Ta1, Xb1_Ta1, Xb2_Ta1, Xb3_Ta1, Xc1_Ta1, Xc2_Ta1, and Xc3_Ta1 corresponding to the processes Ta1, Ta2, Ta3, Tb1, Tb2, Tb3, Tc1, Tc2, and Tc3. Here, it is assumed that sequence control is performed for the process itself and for the processes of the same node. Note that in these cases, sequence control can be omitted because disturbance of the packet arrival order is not assumed, but the fact that the user or programmer must be aware of whether the receiving process of the communication partner exists in the same computer increases the difficulty of program creation. For this reason, the system is configured to use an MBCF operation request packet including the same specification of transmission number count and multiple link synchronization information regardless of which computer the receiving process exists on.

[0049] For example, when the process Ta1 issues an operation request packet to the process Tb2, the transmission number counter Xb2_Ta1 is counted up. Similarly, each of the other processes has nine transmission number counters.

[0050] In addition, process Ta1 has nine reception number counting units 202 (reception number counters): Ra1_Ta1, Ra2_Ta1, Ra3_Ta1, Rb1_Ta1, Rb2_Ta1, Rb3_Ta1, Rc1_Ta1, Rc2_Ta1, and Rc3_Ta1, corresponding to processes Ta1, Ta2, Ta3, Tb1, Tb2, Tb3, Tc1, Tc2, and Tc3.

[0051] For example, when node Na receives an operation request packet addressed to process Ta1 from process Tc2, if the count value in the operation request packet is (Rc2_Ta1+1), the reception number counter Rc2_Ta1 is counted up after the memory operation. If the count value in the operation request packet is greater than (Rc2_Ta1+1), it is recorded in the already arrived count recording unit 203 (configured using a bit vector) corresponding to the reception number counter Rc2_Ta1 after the memory operation, and the count up of the reception number counter Rc2_Ta1 is delayed until the end of the memory operation of the overtaken operation request packet.

[0052] If multiple link synchronization information (synchronization option) is added to the operation request packet from process Tc2 to process Ta1, if the value of the reception number counter Rc2_Ta1 is equal to or greater than the count value described in the synchronization option, the same processing is performed as for an operation request packet without a synchronization option.

[0053] On the other hand, if the value of the reception count counter Rc2_Ta1 is smaller than the count value described in the synchronization option, execution is delayed, including memory operations and registration in the already received count recording unit (bit vector), until the value of the reception count counter Rc2_Ta1 becomes equal to or greater than the count value described in the synchronization option.

[0054] <2. Effects of this embodiment> According to the parallel distributed computing system 100 of this embodiment configured as described above, a plurality of computers 2 are interconnected via a plurality of data links 3, and the plurality of computers 2 perform communication synchronization with each other using MBCF. In this system, the sending process has a transmission number counting unit 201 which attaches to an operation request packet a count number before or after the count-up of the operation request packet, and the receiving process has a reception number counting unit 202 which counts the number of operation request packets received for each sending process. When the count number in the reception number counting unit 202 and the count number attached to the operation request packet are consecutive, the destination computer 2 (2Y) processes the operation request packet and updates the count number in the reception number counting unit 202 with the count number attached to the operation request packet. Therefore, in an environment in which the arrival order of operation request packets is disrupted, the memory operation order can be guaranteed within the range intended by the user, that is, memory consistency can be maintained.

[0055] Conventionally, in a distributed shared memory parallel computer that allows shared memory access to distributed memories, in order to guarantee the order of memory operations, an acknowledgement (hereinafter, Ack) is returned to the requester for all remote memory operations, and when the order must be guaranteed, a remote memory operation for which the order should be guaranteed is requested after confirming that Acks have been returned for all previous remote memory operations. This mechanism for guaranteeing the order is called a memory barrier. In contrast, in the present invention, Ack is not required for the remote memory operation to guarantee the order, and it is possible to request a remote memory operation for which the order should be guaranteed following the remote memory operation requested most recently. This means that it is not necessary to wait for the return of Ack for the preceding remote memory operation, which enables a significant improvement in performance. In addition, the sending process can add multiple link synchronization information using the value of the transmission number count unit 201 held by itself, and the destination computer 2 (2Y) can judge whether synchronization is established or not using the value of the reception number count unit 202 belonging to the receiving process in the same computer. In other words, the information required for synchronization can be obtained at extremely low cost by both the sending process and the destination computer 2 (2Y).

[0056] By using this invention, there is a possibility that overhead costs due to synchronization can be reduced in addition to eliminating the need to wait for an Ack. This possibility will be explained using a concrete example. A receiving process has memory areas A1, B1, and C1, and a flag variable F1 indicating that values ​​have been stored in these areas. Similarly, the same receiving process has memory areas A2, B2, C2, A3, B3, and C3, and a flag variable F2 indicating that values ​​have been stored in A2, B2, and C2, and a flag variable F3 indicating that values ​​have been stored in A3, B3, and C3. It is assumed that the storage of values ​​in all memory areas and flags is realized by MBCF_WRITE. There is an order relationship that must be maintained between remote writes A1, B1, C1 and F1, A2, B2, C2 and F2, and A3, B3, C3 and F3, but there is no order relationship between An, Bn, and Cn, or between A and B and Cn. F m, FThere is no order relationship between n (where m ≠ n). Simply put, as shown in FIG. 10, when an MBCF_WRITE request packet is issued, it must be guaranteed that writing to F1, F2, and F3 occurs after the immediately preceding writing to C1, C2, and C3, respectively. In the presence of multiple links, it is not guaranteed that the requests will arrive in the order they were issued, so there is a high possibility that the destination computer 2 (2Y) must delay the operation request packet processing for synchronization. In response to this, as shown in FIG. 11, by inserting an unrelated MBCF_WRITE between the write to the memory area where the order should be guaranteed and the write to the flag variable, the count value of the link synchronization information to be added can be the value of the transmission count section 201 of the transmission count section of the previous packet, rather than the value of the MBCF request count issued immediately before. This means that it does not matter if the request packet arrives at the destination computer 2 (2Y) earlier than the two immediately preceding packets, and the case of Fig. 11 is clearly less likely to cause writing to the flag variable to be delayed at the destination computer 2 (2Y) due to synchronization wait than the case of Fig. 10. In other words, by using this invention to optimize the issuance order of operation requests that require synchronization and operation requests that do not, as well as operation requests with no order relationship, it is possible to further reduce synchronization overhead costs.

[0057] Furthermore, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications are possible without departing from the spirit and scope of the present invention. [Industrial Applicability]

[0058] The present invention makes it possible to maintain memory consistency at low cost in a parallel distributed computing system in which multiple computers are interconnected via multiple data links and communicate with each other in synchronization using MBCF, even in a situation in which the arrival order of packets is disrupted.

Claims

1. A parallel and distributed computing system in which a plurality of computers each having a processor with a translation lookaside buffer (TLB), a physical memory, and a network interface controller (NIC) capable of directly accessing the physical memory are interconnected via a plurality of data links, A process of a source computer (hereinafter, a sending process) transmits an operation request packet including an identifier of an operation target process (hereinafter, a receiving process) that specifies a process of a destination computer, an operation target address that specifies a memory area of ​​the receiving process, a data size to be written, and a data string; the destination computer receives the operation request packet and stores the data string in a memory area defined by the receiving process and the operation target address; the sending process has a transmission number counting unit that counts the number of transmissions of the operation request packet for each receiving process and attaches to the operation request packet the count number before or after the count-up of the operation request packet, the receiving process has a reception number counting unit that counts the number of receptions of the operation request packet for each of the sending processes, The destination computer is when the count number of the reception number counting unit and the count number attached to the operation request packet are consecutive, processing the operation request packet and updating the count number of the reception number counting unit with the count number attached to the operation request packet; When the order of arrival at the receiving process is out of order and there is an operation request packet that has not yet arrived, the reception number counting unit updates the count number only up to the operation request packet that has not yet arrived, and for the operation request packets that arrive after the operation request packet that has not yet arrived, the operation request packets are processed, and the count number attached to the operation request packet is recorded in an already-arrived count recording unit separately from the reception number counting unit; when the recorded contents of the arrival record to the receiving process are continuous from the reception count unit by using the already-arrived count recording unit due to the arrival of a previously unarrived operation request packet, the reception count count unit is updated within the continuous range, and the updated record is deleted from the already-arrived count recording unit; In this parallel distributed computing system, when the sending process wants to prohibit the operation request packet it is about to send from being operated on until the processing of the preceding operation request packet is completed, the sending process adds a count of the number of operation request packets whose operations it wants to be surely completed to the operation request packet as multiple link synchronization information.

2. 2. The parallel distributed computing system according to claim 1, wherein when the destination computer receives an operation request packet to which the multiple link synchronization information is attached, the destination computer refers to the reception count unit of the receiving process, and if the count number described in the multiple link synchronization information is less than or equal to the count number of the reception count unit, processes the operation request packet and updates the reception count unit and the already arrived count recording unit.

3. 3. The parallel distributed computing system according to claim 2, wherein when said destination computer receives an operation request packet to which said multiple link synchronization information is added, said destination computer refers to said reception count unit of said receiving side process, and if the count number described in said multiple link synchronization information is greater than the count number in said reception count unit, delays processing of the operation request packet until the count number in said reception count unit increases to be greater than or equal to the count number described in said multiple link synchronization information, and also delays updating of said reception count unit and said already arrived count record unit.

Images