Reusable barriers for synchronization between multiple processes

Abstract

The present disclosure relates to reusable barriers for synchronization between multiple processes. In one aspect, a system and method for facilitating synchronization between processes is provided. During operation, the system can execute multiple processes in parallel on one or more computing nodes. In response to a first process calling a barrier function, the system can suspend execution of the first process, and in response to determining that the first process has gained access to a variable shared by at least a subset of the multiple processes, the system can update the shared variable. The system can release the shared variable to a second process in the subset to update the shared variable when the second process calls the barrier function, and determine whether all processes in the subset have updated the shared variable. In response to the shared variable having been updated by all processes in the subset, the system can resume execution of all processes in the subset.

Description

Technical Field

[0001] This disclosure relates to synchronization between a group of concurrently executing processes. More specifically, this disclosure relates to a reusable barrier mechanism for implementing synchronization between processes running different executable code. Background Technology

[0002] Efficient synchronization and communication between single-threaded processes (often called processing units (PEs)) play a crucial role in high-performance computing (HPC) environments. These mechanisms are essential for maximizing computational efficiency and protecting data integrity. In the HPC field, complex computations and massive data processing are the norm, making seamless coordination of PEs critical. By implementing effective synchronization strategies, HPC systems can precisely orchestrate the execution of multiple processes, thereby minimizing idle time and optimizing resource utilization.

[0003] Many HPC applications utilize Message Passing Interface (MPI) programming to achieve optimal performance and scalability. The MPI programming model allows for a flexible approach where processes can execute different code or programs on different datasets (known as the Multi-Program Multi-Data (MPMD) programming paradigm). This versatility enables developers to tailor their applications to complex computational tasks requiring different algorithms or methods. However, adopting MPMD can present significant challenges when working with certain parallel programming libraries such as OpenSHMEM and Cray OpenSHMEMX. Attached Figure Description

[0004] Figure 1 The diagram illustrates the operating principle of a reusable barrier system according to one aspect of this disclosure.

[0005] Figure 2 An example of multiple synchronization groups according to one aspect of this disclosure is illustrated.

[0006] Figure 3 A flowchart illustrating an example of the synchronization operation of a process implementing a reusable barrier system according to one aspect of this disclosure is presented.

[0007] Figure 4 A flowchart illustrating an example operation process of a computing node that executes multiple processes in parallel according to one aspect of this disclosure is presented.

[0008] Figure 5 An example computer system facilitating synchronization between concurrent processes is illustrated according to one aspect of this disclosure.

[0009] Figure 6 The illustration depicts a computer-readable medium that facilitates synchronization between processes running different programs, according to one aspect of this application.

[0010] In these accompanying drawings, the same reference numerals refer to the same elements. Detailed Implementation

[0011] Existing parallel programming libraries based on shared memory (e.g., OpenSHMEM and Cray OpenSHMEMX) provide built-in synchronization mechanisms for processes running the same program (i.e., Single Program Multiple Data (SPMD)), but do not support synchronization between processes running different programs (i.e., MPMD). This is because, in MPMD mode, the various functionalities within these libraries cannot function fully. Without strong support for MPMD mode, synchronization is hindered, exacerbating the challenge of coordinating processes running different programs. To overcome these challenges, according to some aspects of this disclosure, a novel reusable barrier system can be used to facilitate synchronization between processes running different programs on the same or different computing nodes. This reusable barrier scheme is independent of the evolving functionality of the library and can provide a reliable synchronization mechanism now and in the future, thereby enhancing system performance and reliability.

[0012] Depending on several aspects, reusable barrier systems can rely on semaphores and shared memory to synchronize processes executing on the same compute node, and on remote locking mechanisms and remote memory (e.g., fabric-attached memory, FAM) to synchronize processes executing on different compute nodes. Using a single-node scenario as an example, during operation, different processes can call the barrier function at different times and wait for all processes to reach the barrier point before proceeding. The caller of the barrier in each process can be configured to increment a global counter when it holds the semaphore. Because only one process can hold the semaphore at any given time, the counter can be incremented in a controlled manner. A pointer to the counter can be mapped to a shared memory segment accessible to all processes. By monitoring the counter value, the barrier logic can determine whether all processes in the synchronization group have reached the barrier point. The barrier system can be reused for future synchronization rounds by introducing a "round" value, where each process can update / increment its local "round" value to proceed to the next synchronization round after all processes have reached the barrier point. Multiple barrier systems can be set up for different synchronization groups, with each group using its own semaphore and counter.

[0013] Figure 1 The diagram illustrates the operating principle of a reusable barrier system according to one aspect of this disclosure. More specifically, Figure 1The diagram illustrates the progress of multiple processes (e.g., processes 102-108) to be synchronized over time (indicated by arrow 100). These processes may run different programs (e.g., different executables) and may communicate with each other (e.g., exchange data). To ensure data consistency and correctness, a barrier system can be implemented. In one example, each program may include code that calls a barrier interface, thereby pausing the process until all processes reach the barrier point.

[0014] Concurrently running processes may encounter barriers at different times (e.g., by calling a barrier interface). In this example, process 102 encounters a barrier before all other processes (i.e., processes 104, 106, and 108) and waits for other processes to reach the barrier point. While waiting, process 102 can also increment a global counter and monitor the counter value to determine if all other processes have also incremented their counters (i.e., reached the barrier point). Similarly, when a different process (e.g., process 104 or 106) reaches the barrier point, that process can also increment its counter and wait for other lagging processes. After the last process (e.g., process 108) reaches the barrier point and increments its counter, the barrier can be lifted to allow these processes to resume execution at synchronization point 110. Depending on some aspects, the total increment of the counter value can be compared to the number of processes to be synchronized to determine if all processes have incremented the global counter (i.e., reached the barrier point). For example, if there are ten processes to be synchronized, and the global counter has been incremented ten times, the barrier can be lifted because all ten processes have reached the barrier point.

[0015] Barrier systems are reusable, meaning the same data structures (e.g., a global counter and its mapping to shared memory) can be used in subsequent rounds of synchronization. In some examples, the barrier system can use additional, process-local variables (e.g., round values ​​from the beginning) to track the number of synchronization rounds. Figure 1 In the example shown, after determining that all processes have reached the barrier point, each process can increment the round value to indicate that it is moving to the next synchronization round.

[0016] exist Figure 1In the example shown, in the second round of synchronization, process 104 encounters the barrier first (e.g., the program calls the barrier interface), while process 106 encounters the barrier last. As in the previous round, each process can call the barrier interface and wait for other slower processes to reach the barrier. Calling the barrier interface may result in an increment of the global counter. By determining the increment of the global counter in this round, each process can determine whether all processes have reached the barrier. In some respects, this comparison can be made based on the global counter value and the local round value. For example, the current counter value can be compared to the product of the total number of processes and the round value. If the current counter value equals the product, then all processes have reached the barrier in this synchronization round. In one example, there are 5 processes, the current counter value is 23, and the current round value is 5 (meaning this is the 5th synchronization round). Since the current counter value is less than the product of the number of processes and the number of synchronization rounds (i.e., 23 < 5 * 5), there are processes that have not yet reached the barrier in the current synchronization round.

[0017] In some aspects, multiple barrier systems can be implemented to facilitate synchronization between different groups of processes. In this implementation, each synchronization group may include a subset of concurrently running processes and may rely on a unique data structure corresponding to that group (e.g., a global counter value and a local round value) to synchronize processes within that group.

[0018] Figure 2 An example of multiple synchronization groups according to one aspect of this disclosure is illustrated. In this example, concurrent processes can be grouped into multiple synchronization groups, including synchronization groups 202 and 204. These groups can overlap, meaning that a particular process can participate in multiple groups. In this example, synchronization group 202 includes processes P0, P1, and P2, and synchronization group 204 includes processes P0 and P2. In this example, processes P0 and P2 are present in both groups.

[0019] Synchronization group 202 can implement a reusable barrier system comprising a global counter (e.g., counter_1) accessible to all processes in the group and a process-local round variable (e.g., round value_1). During each round of synchronization, a process invoking the barrier interface can update the global counter value. For example, process P0 arrives at the barrier point before processes P1 and P2 and invokes the barrier interface. While waiting for P1 and P2, process P0 can increment the global counter (e.g., the value of counter_1). Similarly, process P2 can increment counter_1 while waiting for process P1 to invoke the barrier. In response to determining that all processes within synchronization group 202 have arrived at the barrier point (i.e., have incremented counter_1), each process in the group can increment its own round variable (e.g., round value_1). The barrier can then be removed at synchronization point 206, and all processes can resume execution. The updated local round value (e.g., round value_1) can be used in subsequent synchronization rounds to determine whether all processes in synchronization group 202 have arrived at the barrier point.

[0020] The reusable barrier system implemented in synchronization group 204 may include different counters (e.g., counter_2) and different round variables (e.g., round value_2). In this example, process P2 encounters the barrier first. While waiting for process P0, process P2 increments counter_2. When process P2 reaches the barrier point, it invokes the barrier interface to increment counter_2. After counter_2 has been incremented by both processes, each process (e.g., P0 or P2) can then increment its local round variable (e.g., round value_2). The barrier can be removed at synchronization point 208, and all processes in the group can resume execution. The updated local round value (e.g., round value_2) can be used in subsequent synchronization rounds to determine whether all processes in synchronization group 204 have reached the barrier point.

[0021] In some respects, to prevent race conditions associated with a global counter, a reusable barrier system can (when processes are running on a single node) implement semaphores to control access to the global counter. Semaphores have been used as a signaling mechanism to coordinate the execution order of multiple processes executing the same program. In some aspects of this disclosure, a reusable barrier system for processes running different programs can use binary semaphores to ensure that only one process can increment the global counter, such that the instantaneous counter value reflects the number of processes arriving at the barrier point. More specifically, when a process invokes the barrier, it must be checked whether the process holds a semaphore (e.g., whether the semaphore value is 1). Note that the initial value of the semaphore can be set to 1, so that the first process to invoke the barrier will hold the semaphore and (e.g., by setting the semaphore value to 0) prevent other processes from holding the semaphore. A process can increment the global counter if and only if it holds the semaphore. After incrementing the global counter, the process can release the semaphore (e.g., by resetting its value to 1) to allow different processes to hold the semaphore to increment the global counter.

[0022] When these processes run on multiple nodes, a reusable barrier system can implement remote locks to control access to a global counter. Note that a remote lock is a synchronization mechanism that allows a process to acquire a lock on a resource managed by a dedicated server core or node, rather than directly competing for the lock with other processes. Depending on several aspects, a reusable barrier system can implement remote locks using any distributed locking primitive. In one example, a remote lock can be implemented using a centralized lock manager or a distributed consensus algorithm. A process attempts to acquire the lock before accessing the shared resource. If successful, the process performs its critical section operation and then releases the remote lock to allow other processes to acquire it.

[0023] Figure 3 A flowchart illustrating an example synchronization operation of a process implementing a reusable barrier system according to one aspect of this disclosure is presented. The process may be one of multiple processes executing in parallel on a computing node. The reusable barrier system may include data structures (e.g., a global counter) created in shared memory and functions (e.g., barrier functions) embedded in executable code.

[0024] During operation, to initialize the reusable barrier system, a process can create or open (e.g., for a single-node scenario, by calling shm_open() provided by the standard C library) a shared memory object (operation 302). If a shared memory object exists, the process opens the object; otherwise, the process creates the object. The shared memory object can be accessed by all participating processes running different programs. These processes can execute on different processors of one or more compute nodes. In one example, a shared memory object can be opened or created with read / write permissions. Creating shared memory may also include storing a file descriptor and setting the memory size to accommodate integer data.

[0025] Below is sample code for creating / opening shared memory objects:

[0026]

[0027] The process can then map the shared memory object to a global counter variable (operation 304). In one example, the shared memory object can be mapped (e.g., by making an mmap() system call) to a global counter variable as an integer pointer with read / write permissions. Example code for memory mapping is shown below:

[0028]

[0029]

[0030] Note that the MAP_SHARED flag in the mmap() function above indicates that updates to the mapping are visible to other processes mapping the same region. Upon success, mmap() returns a pointer to the mapped region (e.g., a pointer to a global counter). This operation essentially stores a pointer to the global counter in shared memory.

[0031] The process can then create or open a semaphore or remote lock (Operation 306). A semaphore can be created using read / write permissions and initialized to 1. Example code for creating a semaphore is shown below:

[0032] netma_sem

[0033] =sem_open(" / netma_global_barrier_sem",O_CREAT,0666,1).

[0034] Note that semaphores are used here in a single-node scenario. When the process to be synchronized executes on multiple nodes, a remote locking mechanism (e.g., any distributed locking primitive) can be used to control access to a global counter. If no semaphore / remote lock exists in the system, the process creates one. Otherwise, the process can open an existing semaphore / remote lock.

[0035] The system can trigger or continue the execution of a process (operation 308) and determine whether a barrier needs to be invoked (operation 310). When the process reaches a barrier point (e.g., a code segment containing a barrier function), the system invokes the barrier. If not, the system can determine whether the program has completed (operation 312). The program terminates when it has successfully executed all its instructions or when a termination condition is triggered (e.g., when the program encounters a predetermined event or state). If yes, the process terminates. If no, the system continues execution of the process. If a barrier needs to be invoked, the process can invoke the barrier interface to pause execution (operation 314). The process can attempt (for a single-node scenario) to acquire a semaphore or (for a multi-node scenario) to acquire a remote lock (operation 316). In one example, the process can use several variables to invoke the barrier interface, including the number of processes, the semaphore or remote lock, a counter, and a round variable. Example code for invoking a barrier is shown below:

[0036]

[0037] When the barrier function is called, the process suspends execution and determines whether it holds a semaphore (for a single-node scheme) or a remote lock (for a multi-node scheme) (operation 318). In an example of a single-node scheme, determining whether a process holds a semaphore may include determining whether the current value of the semaphore is 1. If not, the process attempts to acquire the semaphore or the remote lock (operation 318). If the process holds the semaphore (e.g., semaphore = 1) or the remote lock, the process increments a counter (e.g., increments it by one or another predetermined value) and releases the semaphore or the remote lock (operation 320).

[0038] The process can further determine whether all processes in the synchronization group have incremented the global counter in the current synchronization round (operation 322). In one example, the process can compare the current counter value with the product of the current round value and the number of processes. If the current counter value equals the product of the current round value and the number of processes in the synchronization group, then all processes in the group have incremented the global counter, indicating that these processes have reached the barrier point. Therefore, the process can increment its local round value (operation 324). Otherwise, the process waits until all other processes have incremented the global counter (operation 322). In one example, all processes increment their local round values ​​so that those round values ​​are synchronized for use in the next synchronization round. In one example, all processes can increment their local round values ​​by a predetermined amount (e.g., by one). Subsequently, the process can resume execution (operation 326) and continue execution to operation 308. The barrier system (i.e., data structures such as global counters and semaphores) can be reused in subsequent synchronization rounds. Example code for the barrier function is shown below:

[0039]

[0040]

[0041] exist Figure 3 The example shown implements a single barrier system. In alternative examples, parallel-executing processes could be grouped into multiple (and potentially overlapping) synchronization groups implementing multiple barriers, similar to... Figure 2 The example shown. In this case, multiple unique data structures can be established during the initialization of the barrier system. For example, operations 302 to 306 can be performed for different barriers. If a process participates in multiple synchronization groups, that particular process can perform operations 302 to 306 multiple times, for each synchronization group (e.g., Figure 2 Synchronization group 202 or 204 shown once.

[0042] For a specific synchronization group, the barrier interface can be invoked using group-specific variables, including the number of processes in the synchronization group, semaphores / remote locks, counters, and round variables. When multiple barrier systems are implemented, the execution order of these processes can be determined by all the barriers. Figure 2 In the example shown, processes P0 and P2 can first synchronize at synchronization point 208, regardless of the progress of process P1. Subsequently, processes P0 and P2 resume execution until they each invoke the barrier of synchronization group 202. Then, all three processes (i.e., P0, P1, and P2) can be synchronized at synchronization point 206.

[0043] Figure 4A flowchart illustrating an example operational process of a computing node executing multiple processes in parallel according to one aspect of this disclosure is presented. The computing node can implement a reusable barrier system to synchronize processes. The reusable barrier system may include data structures created in shared memory and functions embedded in each process participating in the synchronization group. Compared to conventional barriers that only work on processes running the same program (or executable file), the disclosed reusable barrier system can synchronize processes running different programs on one or more computing nodes. Furthermore, the same data structures can be used for multiple rounds of synchronization.

[0044] The operation begins with initializing the reusable barrier system (operation 402). Depending on several aspects, initializing the reusable barrier system includes: each process to be synchronized creating or opening a shared memory object with read and write permissions; storing file descriptors (e.g., reference IDs) for the shared memory object; and setting the size of the file descriptors to accommodate integer data. Initialization may include each process mapping the shared memory to a global counter variable as an integer pointer with read and write permissions. The mapping may be marked as shared, such that updates to the mapping are visible to all processes mapping the same memory region. For a single-node scheme, initialization may further include opening or creating a semaphore with read and write permissions in each process. The initial value of the semaphore may be set to one. For a multi-node scheme, initialization may include setting metadata for remote locks.

[0045] After initializing the reusable barrier system, compute nodes can execute multiple processes in parallel (Operation 404). These processes can run different programs (e.g., with different computer-executable code) and can exchange data during execution. To ensure data consistency and correctness, these processes may need to be synchronized using the reusable barrier system.

[0046] Different processes can have different progress and will encounter barriers (i.e., call the barrier function) at different times. In response to the first process calling the barrier function, the compute node can pause the execution of the first process (operation 406). Pausing execution allows the first process to wait for other processes in the synchronization group to catch up.

[0047] In response to determining that a first process has gained access to a variable shared by a subset of multiple processes, the compute node can update the shared variable (operation 408). According to some aspects, the first process may participate in a synchronization group comprising a subset of processes. Processes in the synchronization group may invoke a barrier function to synchronize at a predetermined critical point. According to some aspects, determining that the first process has gained access to the shared variable includes determining that the first process holds a semaphore (e.g., the current semaphore value is one). The shared variable may include a global counter, and updating the shared variable may include incrementing the global counter by a predetermined value (e.g., one). A pointer to the global counter may be mapped to a shared memory segment accessible to all processes in the subset.

[0048] Subsequently, the first process can release the shared variable to the second process so that the shared variable can be updated when the second process calls the barrier function (operation 410). Releasing the shared variable may include releasing a semaphore (e.g., resetting the semaphore value to one).

[0049] The compute node can determine whether all processes in the subset have updated the shared variables (operation 412). According to one aspect, the compute node can compare the increment of the global counter within the current synchronization round with the number of processes in the synchronization group. If all processes have invoked the barrier function and incremented the global counter, the increment value of the global counter equals the number of processes. According to a further aspect, an additional local variable (i.e., the round value) can be used to determine the number of synchronization rounds since the start of process execution. In this case, each process executing on the compute node can compare its current counter value with the product of its current round value and the total number of processes in the subset.

[0050] If all processes in the subset have updated the shared variables, the compute node can resume execution of all processes in the subset (operation 414). Otherwise, the compute node waits for all processes to catch up (i.e., calls the barrier function and updates the shared variables).

[0051] exist Figures 1 to 3 In the example shown, the processes to be synchronized can execute on a single compute node (e.g., on one or more processors of the compute node) that implements shared memory accessible to all processors. In an alternative example, the processes to be executed can execute on different compute nodes without shared memory. In this case, the processes executing on different compute nodes can access remote memory, such as networked memory (FAM). Interfaces provided by libraries (e.g., OpenFAM) can be used to facilitate data access and synchronization across multiple processes in remote memory (e.g., FAM).

[0052] For example, instead of using semaphores, a reusable barrier system can use remote locks to control access to critical sections (e.g., sections that increment a global counter). A remote lock can be any distributed locking primitive that can be used to coordinate access to remote shared variables. After acquiring the remote lock, a process can enter the critical section where it can increment the remote shared counter. In the lock release path, remote unlocking can be used instead of `sem_post()`. The global counter can include any remote variable (e.g., a FAM variable) that is visible to processes executing on different nodes. Other data structures used in the barrier system (i.e., round values ​​and the number of processes in a synchronization group) can remain constant because round values ​​are process-local variables, and the number of processes is constant for each group.

[0053] Figure 5 An example computer system facilitating synchronization between concurrent processes is illustrated according to one aspect of this disclosure. Computer system 500 may include one or more processing resources (e.g., processing resource 502), one or more storage devices (e.g., storage device 504), private memory 506, and shared memory 508.

[0054] In the examples described herein, processing resources may include, for example, one or more processors included in a single computing device or distributed across multiple computing devices. In some examples, concurrent processes may execute on a single computing device or multiple computing devices. As used herein, a “processor” may be at least one of a central processing unit (CPU), a semiconductor-based microprocessor, a graphics processing unit (GPU), a field-programmable gate array (FPGA) configured to fetch and execute instructions, other electronic circuitry suitable for fetching and executing instructions stored on a computer-readable storage medium, or combinations thereof. In the examples described herein, processing resources may fetch, decode, and execute instructions stored on a storage medium to perform the function described with respect to the instructions stored on the computer-readable medium. In other examples, the function described with respect to any instructions described herein may be implemented in the form of electronic circuitry, executable instructions encoded on a computer-readable medium, or a combination thereof. The computer-readable storage medium may be located in or away from the computing device executing the instructions but accessible (e.g., via a computer network) for execution. In the examples illustrated herein, a node may be implemented by one or more computer-readable storage media.

[0055] Computer system 500 may be coupled to peripheral I / O user devices 510 (e.g., display device 512, keyboard 514, and pointing device 516). Storage device 504 includes a non-transitory computer-readable storage medium and stores operating system 518, process synchronization instructions 520, and data 540. Computer system 500 may include... Figure 5 The entities shown are fewer or more entities.

[0056] The process synchronization instructions 520 may include instructions that, when executed by the computer system 500, cause the computer system 500 to perform the methods and / or procedures described in this disclosure. The process synchronization instructions 520 may be executed, at least on the processing resource 502.

[0057] Process synchronization instructions 520 may include instructions 522 for initializing a reusable barrier system, as described above. Figure 3 Operations 302-306 shown are Figure 4 The operation 402 shown describes a reusable barrier system that may include data structures created in shared memory (for a single-node scheme) or remote memory (for a multi-node scheme) and functions embedded in each process participating in the synchronization operation. For a single-node scheme, instruction 522 may include instructions for setting a shared memory object, instructions for mapping shared memory to a global counter variable as an integer pointer with read / write permissions, and instructions for setting a semaphore with read / write permissions. For a multi-node scheme, instruction 522 may include instructions for setting a remote lock.

[0058] The process synchronization instruction 520 may include instruction 524 for executing multiple processes in parallel, as described above. Figure 4 The operation 404 shown describes the process. These processes can run different programs (e.g., with different computer-executable code) on the same computing device or multiple computing devices.

[0059] The process synchronization instruction 520 may include instruction 526 for suspending the execution of the first process in response to the first process calling a barrier function, as described above. Figure 3 Operations 310 and 312 shown and Figure 4 As described in operation 406, different processes may encounter barriers at different times.

[0060] Process synchronization instructions 520 may include instructions 528 for updating a shared variable in response to determining that a first process has gained access to a variable shared by a subset of multiple processes, as described above. Figure 3 Operations 314 and 316 shown are as follows Figure 4The operation 408 shown describes the following. For a single-node scheme, instruction 528 may include instructions for determining whether a first process holds a semaphore and instructions for incrementing a counter variable accessible to all processes in a subset if the first process holds the semaphore. For a multi-node scheme, instruction 528 may include instructions for determining whether a first process holds a remote lock.

[0061] Process synchronization instruction 520 may include instruction 530 for releasing a shared variable to a second process so that the shared variable can be updated when the second process calls a barrier function, as described above. Figure 4 The operation 410 shown is described. More specifically, instruction 530 may include instructions to release a semaphore or remote lock to a second process.

[0062] Process synchronization instruction 520 may include instruction 532 for determining whether all processes in a subset have updated shared variables, as mentioned above. Figure 3 Operations 318 and shown Figure 4 The operation 412 shown describes this. More specifically, instruction 530 may include instructions for comparing the increment of the global counter with the number of processes in the subset. When multiple synchronization rounds have been executed, instruction 530 may be used to compare the current counter value with the product of the current round value and the total number of processes in the subset.

[0063] Process synchronization instruction 520 may include instruction 534 for resuming execution of all processes in the subset in response to determining that all processes in the subset have updated shared variables, as described above regarding... Figure 3 Operations 322 and 322 shown Figure 4 The operation 414 shown is described.

[0064] Process synchronization instruction 520 can include ratios Figure 4 The instructions shown are further instructions. For example, process synchronization instruction 520 may include instructions for incrementing the round value (e.g., increasing it by one) in response to all processes in a determined subset having updated a shared variable.

[0065] Figure 6The illustration depicts a computer-readable medium facilitating synchronization between processes running different programs according to one aspect of this application. CRM 600 may be a non-transitory computer-readable medium or device storing instructions that, when executed by a computer or processing resource, cause the computer or processing resource to perform a method. As used herein, a "computer-readable storage medium" can be any electronic, magnetic, optical, or other physical storage device for containing or storing information such as executable instructions, data, etc. For example, any computer-readable storage medium described herein may be any one or a combination of RAM, EEPROM, volatile memory, non-volatile memory, flash memory, storage drives (e.g., HDD, SSD), any type of storage disk (e.g., compact disk, DVD, etc.). Furthermore, any computer-readable storage medium described herein may be non-transitory.

[0066] CRM 600 can store instructions 610 for initializing a reusable barrier system, as mentioned above. Figure 3 Operations 302-306 shown are Figure 4 The operation 402 shown describes the instruction 620 for executing multiple processes in parallel, as described above regarding... Figure 4 The operation 404 shown describes the instruction 630 for suspending the execution of the first process in response to the first process calling the barrier function, as described above regarding... Figure 3 Operations 310 and 312 shown and Figure 4 Operation 406 described herein; instruction 640 for updating a shared variable in response to determining that a first process has gained access to a variable shared by a subset of multiple processes, as described above regarding Figure 3 Operations 314 and 316 shown are as follows Figure 4 Operation 408 is described above; instruction 650, used to release a shared variable to the second process so that the shared variable can be updated when the second process calls the barrier function, as described above regarding... Figure 4 The operation 410 shown describes the instruction 660 used to determine whether all processes in the subset have updated shared variables, as described above regarding... Figure 3 Operations 318 and shown Figure 4 The operation 412 shown describes this; and the instruction 660 for resuming execution of all processes in the subset in response to determining that all processes in the subset have updated shared variables, as described above regarding Figure 3 Operations 322 and 322 shown Figure 4 The operation 414 shown is described.

[0067] CRM 600 can include more Figure 6The instructions shown are further instructions. For example, CRM 600 may include instructions for incrementing the round value (e.g., adding one) in response to the determination that all processes in a subset have updated a shared variable.

[0068] Reusable barrier systems offer numerous advantages, including but not limited to flexibility (facilitating efficient synchronization between processes running different programs on the same or different compute nodes), simplicity (introducing a simplified implementation that streamlines process synchronization without relying on SHMEM call overhead), resource optimization (minimizing resource consumption and enhancing overall system performance by reusing synchronization mechanisms across multiple synchronization points), and library independence. While existing libraries can continue to evolve and enhance their support for MPMD patterns, reusable barrier systems provide a resilient alternative for ensuring effective process synchronization. Their adaptability and reliability make them a valuable asset for pushing HPC capabilities beyond the limitations of current libraries.

[0069] Generally, aspects of this disclosure provide a solution to the technical problem of synchronizing processes running different programs on a single compute node or multiple compute nodes. More specifically, reusable barrier systems can be implemented without relying on functions in existing programming libraries (OpenSHMEM and Cray-OpenSHMEMx libraries) (or waiting for their completion) to allow developers of next-generation network interface controllers (NICs) to perform pre-silicon verification using network simulators in MPMD mode. When processes are executed on the same node, the reusable barrier system can rely on semaphores and global counters mapped to shared memory to synchronize these processes. When processes are executed on different nodes, the reusable barrier system can rely on remote (distributed) locking mechanisms (instead of semaphores) and remote variables (e.g., FAM variables) to synchronize these processes.

[0070] One aspect of this application provides a system and method for facilitating synchronization between processes. During operation, the system can execute multiple processes in parallel on a computing node. In response to a first process calling a barrier function, the system can suspend the execution of the first process, and in response to determining that the first process has acquired access to a variable shared by at least a subset of the multiple processes, the system can update the shared variable. The system can release the shared variable to a second process in the subset to update the shared variable when the second process calls the barrier function. The system can determine whether all processes in the subset have updated the shared variable. In response to the shared variable having been updated by all processes in the subset, the system can resume the execution of all processes in the subset.

[0071] In this regard, determining the first process's access rights to shared variables may include determining that the first process holds a semaphore.

[0072] In this regard, shared variables may include global counter values, and updating shared variables may include incrementing the global counter value.

[0073] In another variation, determining whether all processes in a subset have updated shared variables may include determining the current global counter value.

[0074] In another variation, after determining that all processes in the subset have updated the shared variable, the system can increment the round value by each process in the subset, and determining whether all processes in the subset have updated the shared variable may include comparing the current global counter value with the product of the current round value and the total number of processes in the subset.

[0075] In another variation, the global counter value is mapped to a pointer to a shared memory segment associated with the corresponding process.

[0076] In this regard, the processes in the subset form a first synchronization group. The system can create a second synchronization group consisting of a second subset of multiple processes, and the corresponding processes in the second subset are used to update a second variable shared by all processes in the second subset.

[0077] In this regard, at least two processes in the subset run different computer-executable code.

[0078] One aspect of this application provides a computer system including a processing resource and a non-transitory machine-readable storage medium, the non-transitory machine-readable storage medium including instructions executable by the processing resource to perform the following operations: executing a plurality of processes in parallel; suspending the execution of a first process in response to a first process calling a barrier function; updating a shared variable in response to determining that the first process has acquired access to a variable shared by at least a subset of the plurality of processes; releasing the shared variable to a second process in the subset to update the shared variable when the second process calls the barrier function; determining whether all processes in the subset have updated the shared variable; and resuming the execution of all processes in the subset in response to the shared variable having been updated by all processes in the subset.

[0079] The methods and processes described in the Detailed Description section can be embodied in code and / or data, which can be stored in a computer-readable storage medium as described above. When a computer system reads and executes the code and / or data stored on the computer-readable storage medium, the computer system executes the methods and processes embodied in data structures and code and stored within the computer-readable storage medium.

[0080] The methods and processes described above may be included in hardware modules or devices. Hardware modules or devices may include, but are not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), dedicated or shared processors that execute specific software modules or code at specific times, and other programmable logic devices now known or developed later. When a hardware module or device is activated, it executes the methods and processes included therein.

[0081] The foregoing description is presented to enable any person skilled in the art to make and use the aspects and examples, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects and applications without departing from the spirit and scope of this disclosure. Therefore, the aspects described herein are not limited to those shown, but are intended to be consistent with the maximum scope of the principles and features disclosed herein.

[0082] Furthermore, the foregoing descriptions of the various aspects have been presented solely for illustrative and descriptive purposes. These descriptions are not intended to be exhaustive or to limit the aspects described herein to the disclosed forms. Accordingly, many modifications and variations will be apparent to those skilled in the art. Additionally, the foregoing disclosure is not intended to limit the aspects described herein. The scope of the aspects described herein is defined by the appended claims.

Claims

1. A computer-implemented method, comprising: Execute multiple processes in parallel on one or more computing nodes; In response to the first process calling the barrier function, the execution of the first process is suspended; In response to determining that the first process has gained access to a variable shared by at least a subset of the plurality of processes, the shared variable is updated; Release the shared variable to the second process in the subset; The shared variable is updated when the barrier function is called in the second process; Determine whether all processes in the subset have updated the shared variable; In response to the shared variable being updated by all processes in the subset, execution of all processes in the subset is resumed.

2. The computer-implemented method as described in claim 1, wherein, Determining that the first process has access to the shared variable includes determining that the first process holds a semaphore or a remote lock.

3. The computer-implemented method as described in claim 1, wherein, The shared variable includes a global counter value, and updating the shared variable includes incrementing the global counter value.

4. The computer-implemented method as described in claim 3, wherein, Determining whether all processes in the subset have updated the shared variable includes determining the current global counter value.

5. The computer-implemented method of claim 4, further comprising: After determining that all processes in the subset have updated the shared variable, each process in the subset increments the round value; and Determining whether all processes in the subset have updated the shared variable includes comparing the current global counter value with the current round value and the total number of processes in the subset.

6. The computer-implemented method as described in claim 3, wherein, The global counter value is mapped to a pointer to a shared memory segment associated with the corresponding process.

7. The computer-implemented method as described in claim 1, wherein, The processes in the subset form a first synchronization group, wherein the method further includes creating a second synchronization group comprising a second subset of the plurality of processes, and wherein the corresponding processes in the second subset update a second variable shared by all processes in the second subset.

8. The computer-implemented method as described in claim 1, wherein, At least two processes in the subset run different computer-executable code.

9. A computer system, comprising: One or more processing resources; as well as A non-transitory machine-readable storage medium, the non-transitory machine-readable storage medium including instructions that can be executed by the processing resource to perform the following operations: Execute multiple processes in parallel; In response to the first process calling the barrier function, the execution of the first process is suspended; In response to determining that the first process has gained access to a variable shared by at least a subset of the plurality of processes, the shared variable is updated; Release the shared variable to the second process in the subset; The shared variable is updated when the barrier function is called in the second process; Determine whether all processes in the subset have updated the shared variable; and In response to the shared variable being updated by all processes in the subset, execution of all processes in the subset is resumed.

10. The computer system of claim 9, wherein, Determining that the first process has access to the shared variable includes determining that the first process holds a semaphore or a remote lock.

11. The computer system of claim 9, wherein, The shared variable includes a global counter value, and updating the shared variable includes incrementing the global counter value.

12. The computer system of claim 11, wherein, Determining whether all processes in the subset have updated the shared variable includes determining the current global counter value.

13. The computer system as described in claim 12, in, The non-transitory machine-readable storage medium further includes instructions that can be executed by the processing resource to increment the round value by each process in the subset after determining that all processes in the subset have updated the shared variable; and Determining whether all processes in the subset have updated the shared variable includes comparing the current global counter value with the current round value and the total number of processes in the subset.

14. The computer system of claim 11, wherein, The global counter value is mapped to a pointer to a shared memory segment associated with the corresponding process.

15. The computer system of claim 9, wherein, The processes in the subset form a first synchronization group, wherein the non-transitory machine-readable storage medium further includes instructions executable by the processing resource to create a second synchronization group comprising a second subset of the plurality of processes, and wherein the corresponding processes in the second subset update a second variable shared by all processes in the second subset.

16. The computer system of claim 9, wherein, At least two processes in the subset run different computer-executable code.

17. A non-transitory computer-readable storage medium storing instructions for performing the following operations: Execute multiple processes in parallel on one or more computing nodes; In response to the first process calling the barrier function, the execution of the first process is suspended; In response to determining that the first process has gained access to a variable shared by at least a subset of the plurality of processes, the shared variable is updated; Release the shared variable to the second process in the subset; The shared variable is updated when the barrier function is called in the second process; Determine whether all processes in the subset have updated the shared variable; and In response to the shared variable being updated by all processes in the subset, execution of all processes in the subset is resumed.

18. The non-transitory computer-readable storage medium of claim 17, wherein, Determining that the first process has access to the shared variable includes determining that the first process holds a semaphore or a remote lock.

19. The non-transitory computer-readable storage medium of claim 17, wherein, The shared variable includes a global counter value, and wherein updating the shared variable includes incrementing the global counter value, and wherein determining whether all processes in the subset have updated the shared variable includes determining the current global counter value.

20. The non-transitory computer-readable storage medium of claim 19, wherein, The instruction is further used for: After determining that all processes in the subset have updated the shared variable, each process in the subset increments the round value; and Determining whether all processes in the subset have updated the shared variable includes comparing the current global counter value with the current round value and the total number of processes in the subset.

Images