Collective operation using network attached memory

Abstract

The invention relates to collective operation using network attached memory. In some examples, a processor receives a first request to allocate a memory region for a collective operation by a processing entity in a plurality of computer nodes. In response to the first request, the processor creates a virtual address of the memory region and allocates the memory region in a network-attached memory coupled to the plurality of computer nodes over the network. The processor associates the virtual address with the address of the memory region in the mapping information. The processor identifies a memory region in the network attached memory by obtaining an address of the memory region from the mapping information using the virtual address in the second request. In response to the second request, the processor performs a collective operation.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0001] Some embodiments of the disclosure are described with respect to the following figures.

[0002] Figure 1 is a block diagram of an arrangement of a cluster including computer nodes coupled to a network attached storage, according to some examples.

[0003] Figure 2 is a block diagram of a memory region with memory segments assigned to ranks according to some examples.

[0004] FIG. 3A to FIG. 3D Examples of collective operations that may be performed using systems according to some examples are illustrated.

[0005] Figure 4 Depicted is a flow of memory allocation according to some examples.

[0006] Figure 5 Depicted is the flow of an all-gather collective operation according to some examples.

[0007] FIG. 6A to FIG. 6B A reduction collective operation according to some examples is illustrated.

[0008] Figure 7 Depicted is the flow of an all-reduce collective operation according to some examples.

[0009] Figure 8 A flow of collective operations according to some examples is depicted.

[0010] Fig. 9 is a block diagram of a system according to some examples.

[0011] Fig.10 is a block diagram of a storage medium storing machine-readable instructions according to some examples.

[0012] Throughout the drawings, the same reference numerals refer to similar, but not necessarily identical, elements. The drawings are not necessarily drawn to scale, and the dimensions of some parts may be exaggerated to more clearly illustrate the examples shown. In addition, the drawings provide examples and / or implementations consistent with the description; however, the description is not limited to the examples and / or implementations provided in the drawings. DETAILED DESCRIPTION

[0013] "Collective operation" is an operation of a communication (and possibly computing) mode between processing entities that are included in multiple computer nodes. In some systems, the problem associated with collective operation is that a relatively large amount of messages may be transmitted between the processing entities participating in the collective operation. As an example, the message for collective operation transmitted between processing entities may include data distributed in the processing entity as a part of collective operation. If there are P (P ≥ 2) processing entities, for some types of collective operations (such as collective operations involving data broadcasting, data dispersion, data aggregation, or other operations (hereinafter further discussed as examples of collective operations) performed by multiple processing entities), there may be P* (P-1) messages between processing entities. In large systems with many processing entities (for example, hundreds or thousands of processing entities), the communication of a large amount of messages can bring a large load to the resources of computer nodes, which include processing resources, communication resources and / or memory resources.

[0014] In some systems, another problem associated with collective operations is that collective operations may involve calculations that burden the processing resources of the computer nodes. The calculations may include aggregation operations (e.g., summing data values, calculating the average or median of data values, or any other operation that aggregates data values), sorting operations, merging operations, or other operations involving data movement and / or data-based calculations. If the processing resources of the computer nodes are overloaded, the performance of the collective operations may be affected.

[0015] According to some embodiments of the present disclosure, in order to solve one or more of the above problems, a technology or mechanism for performing collective operations using a memory area allocated in a network attached memory is provided, and the network attached memory is coupled to a computer node cluster through a network. The collective operation initiated using the collective operation function enabled (invoke) by the processing entity can access the memory area in the network attached memory, so that the computer node can use the memory area instead of using the message between the computer nodes to exchange data. Therefore, assuming that there are P processing entities across computer nodes, the amount of communication involved in the collective operation based on the use of the allocated memory area can be reduced to 2*P compared to the P*(P-1) communications based on the message passing between the P processing entities. In addition, the network attached memory may include a memory server with an available processor, which can be used to perform the calculation of the collective operation. According to some examples of the present disclosure, the calculation of the collective operation can be offloaded to the available processor of the memory server to reduce the possibility of overloading the processing resources of the computer node.

[0016] A parallel computing architecture includes a cluster of computer nodes capable of performing operations in parallel. A program executed in one or more computer nodes in the cluster can initiate operations that include tasks distributed across the computer cluster.

[0017] In an example, collective operations in a parallel computing architecture can be implemented based on the use of an interface that transmits messages between processing entities in a computer node. An example of such an interface is a message passing interface (MPI), which defines the syntax and semantics of library routines to support parallel computing across computer nodes, including MPI collective operations. In other examples, collective operations can be implemented using other types of interfaces (such as interfaces according to parallel programming models) that provide communication between processing entities in a computer node. An example of a parallel programming model is a global address space programming interface (GASPI) model.

[0018] Collective operations can be performed in response to requests from requesting entities (such as users, programs, or machines). When performing collective operations, processing entities in a computer node can communicate with each other through a communication interface when the collective operation is performed in the computer node. A processing entity can be an instance of one or more programs being executed in a computer node. Such a processing entity can also be referred to as a "process" or "rank" of (multiple) programs.

[0019] The communication interface provides a mechanism by which processing entities can send information to each other. The information may include data to which the collective operation is applied, as well as control information, such as status information indicating the status of the collective operation, commands to trigger requested tasks, synchronization events to support synchronization of processing entities during the collective operation, or another type of information used to perform control or management tasks.

[0020] An example of a communication interface is provided by MPI. MPI implements a programming interface (called an application programming interface or API) that includes functions (also called routines or methods) that can be called (invoked) by processing entities executing on computer nodes. The functions include MPI collective functions that perform different collective operations when invoked.

[0021] As mentioned above, a problem associated with collective operations (such as those performed using MPI collective functions) is that a relatively large number of messages may be passed between ranks (processing entities) involved in the collective operation. Another problem associated with collective operations is that collective operations may involve computations that burden the processing resources of a computer node, which may overload the computer node.

[0022] According to some embodiments of the present disclosure, in order to solve one or more of the above problems, a technology or mechanism for using a network attached memory is provided. An example of a network attached memory is a "networked memory (FAM)". A memory allocation function can be enabled by a processing entity in a computer node to allocate a memory area in the network attached memory, wherein the memory area is available to computer nodes across a computer node cluster. Once the memory area is allocated using the memory allocation function, the processing entity can enable a collective operation function to initiate a collective operation across a computer node cluster. The collective operation initiated using the collective operation function is able to access the memory area so that the computer node can use the memory area instead of using messages between computer nodes to exchange data. Therefore, assuming that there are P ranks across computer nodes, the amount of communication involved in the collective operation based on the use of the allocated memory area can be reduced to 2*(P-1) compared to P*(P-1) communications based on message passing between P ranks.

[0023] Furthermore, according to some examples of the present disclosure, computations for collective operations may be offloaded to available processors of a storage server of a network attached storage to reduce the likelihood of overloading processing resources of a computer node.

[0024] Techniques or mechanisms according to some embodiments of the present disclosure may improve computer functionality in a computing system including computer nodes and a network attached storage storage server by: enabling various functions to implement collective operations that utilize the network attached storage to reduce the amount of information transferred between processing entities; and offloading computations from computer nodes to the network attached storage storage server to reduce the likelihood of computer node overload.

[0025] Figure 1 1 is a block diagram of an example system 100 including a plurality of computer nodes 102-1 to 102-N (N≥2). Computer nodes 102-1 to 102-N are part of a cluster of computer nodes. Computer nodes 102-1 to 102-N are interconnected to FAM 106 (or another type of network attached storage) via network 104. FAM 106 includes a plurality of memory servers 108-1 to 108-M (M≥2). It should be noted that M may be the same as or different from N. More generally, N≥1 and M≥1, which means that system 100 may include one or more computer nodes and / or one or more memory servers.

[0026] Examples of network 104 may include an interconnect (e.g., a high throughput, low latency interconnect) between processors and memory, which may be based on open source standards, protocols defined by standards bodies, or proprietary implementations. A "computer node" may refer to a computer, a portion of a computer, or a collection of multiple computers.

[0027] Each memory server includes a corresponding processor and persistent memory. For example, memory server 108-1 includes processor 110-1 and persistent memory 112-1, and memory server 108-M includes processor 110-M and persistent memory 112-M. More generally, a memory server may include one or more processors. Furthermore, in a different example, the persistent memory may be external to the memory server; in this example, the memory server is connected to the persistent memory and is able to access the persistent memory.

[0028] The persistent memory may be implemented with one or more persistent memory devices, such as flash memory devices, disk-based storage devices, or other types of memory devices capable of retaining data when power is removed. The processor of the memory server may refer to a central processing unit (CPU) that executes the operating system (OS) and other machine-readable instructions of the memory server, including firmware such as a basic input / output system (BIOS) and application programs. Alternatively or additionally, the processor of the memory server may refer to another type of processor, such as a graphics processing unit (GPU) that handles specific calculations in the memory server.

[0029] In other examples, the memory server includes volatile memory, i.e., memory that loses data if power is removed from the memory. Volatile memory can be implemented with one or more volatile memory devices, such as dynamic random access memory (DRAM) devices, static random access memory (SRAM) devices, or other types of memory devices that retain data when power is on but lose data when power is removed. Alternatively, the memory server includes both volatile memory and persistent memory.

[0030] Each storage server further includes a network interface controller (NIC) to allow the storage server to communicate over the network 104. The NIC may include a transceiver that transmits and receives signals. Further, the NIC may include one or more protocol layers that perform communications over the network 104 according to corresponding one or more network protocols. The storage server 108-1 includes the NIC 114-1, and the storage server 108-M includes the NIC 114-M.

[0031] Each computer node also includes a processor and a local memory. For example, computer node 102-1 includes processor 115-1 and local memory 116-1, and computer node 102-N includes processor 115-N and local memory 116-N. More generally, a computer node may include one or more processors. Local memory may be implemented using one or more memory devices, which may be any one or some combination of the following: a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, a flash memory device, a disk-based storage device, or any other type of memory device.

[0032] Each computer node also includes a NIC to communicate over network 104. Computer node 102-1 includes NIC 117-1, and computer node 102-N includes NIC 117-N.

[0033] The processor of a computer node is capable of executing one or more processing entities (PEs). A PE is a rank associated with a program being executed in a computer node. The program can be an application or another type of program, such as an OS, firmware, or another type of machine-readable instruction. Figure 1 As shown, processor 115-1 executes PE 118-1, and processor 115-N executes PE118-P-1 to 118-P. A total of P PEs are executed across computer nodes 102-1 to 102-N. The P PEs constitute P ranks, which can collaborate to perform collective operations. The P ranks are part of a communication group that performs collective operations together. Although a specific number of PEs are shown in computer nodes 102-1 to 102-N, in other examples, a different number of PEs can be executed in each computer node. In a further example, the same number of PEs can be executed in each computer node.

[0034] Each PE can access data of FAM 106. Access can include read access for reading data from FAM 106 or write access for writing data to FAM 106. Additionally, the PE can perform computations on data read from FAM 106 or data to be written to FAM 106.

[0035] like Figure 1As further shown in FIG. 1 , computing node 102-1 includes a collective operation programming interface 120-1. In some examples, collective operation programming interface 120-1 may be an API, such as an MPI API or another type of API. Collective operation programming interface 120-1 includes a memory allocation function 122-1 that may be called (invoked) by a PE (e.g., 118-1) in computing node 102-1 to allocate a memory region in FAM 106 to perform a collective operation.

[0036] The collective operation programming interface 120 - 1 also includes a collective operation function 124 - 1 that can be called by a PE (eg, 118 - 1 ) in the computer node 102 - 1 to initiate a corresponding collective operation that can use a memory region allocated in the FAM 106 .

[0037] Computer node 102-N similarly includes a collective operation programming interface 120-N, which includes a memory allocation function 122-N and a collective operation function 124-N. The collective operation programming interface 120-1 can be accessed by a PE (e.g., 118-P-1 or 118-P) in computer node 102-N. In an example in which MPI is used, memory allocation functions 122-1 to 122-N are MPI_Alloc_mem() functions, and collective operation functions 124-1 to 124-N are MPI collective functions.

[0038] In the example, in response to PE 118-1 calling memory allocation function 122-1, memory allocation function 122-1 allocates a memory region in FAM 106 for use by the ranks involved in the collective operation. Assume that the ranks may include PEs 118-1 to 118-P. In other examples, the ranks involved in the collective operation may include a subset (less than all) of PEs 118-1 to 118-P.

[0039] In the following discussion, references to "persistent storage" refer to the set of any one or more of the persistent storage 112-1 through 112-M of the storage servers 108-1 through 108-M.

[0040] After the memory region is allocated by the memory allocation function 122-1, the PEs 118-1 to 118-P may write data to the allocated memory region and read data from the allocated memory region. Information related to the allocated memory region may be stored as allocated memory region information in each of the computer nodes 102-1 to 102-N. For example, the allocated memory region information 126-1 is stored in the local memory 116-1 of the computer node 102-1, and the allocated memory region information 126-N is stored in the local memory 116-N of the computer node 102-N.

[0041] After a memory region has been allocated via a memory allocation function enabled by a given PE, any of the PEs 118 - 1 through 118 -P may enable a collective operation function to initiate a collective operation utilizing the allocated memory region.

[0042] exist Figure 1 In the example of , computer node 102-1 further includes FAM programming interface 128-1, and computer node 102-N includes FAM programming interface 128-N. The FAM programming interface may include an API, a library, or any other program-accessible subsystem in a computer node that is enabled to perform operations on FAM 106, including read operations, write operations, computations offloaded to FAM 106, or another operation involving data.

[0043] In an example, the FAM programming interface may include an API that includes functions that can be used by a computer node (or more specifically, by a PE in a computer node) to manage the FAM and access data of the FAM. An example of such an API is the OpenFAM API.

[0044] Each computer node further includes a FAM client, which is a program that executes in the computer node to manage access to FAM 106 (eg, in response to calls to the FAM programming interface). Computer node 102-1 includes FAM client 130-1, and computer node 102-N includes FAM client 130-N.

[0045] refer to Figure 2, the memory allocation function 122-1, when enabled, allocates a memory region 200 in the persistent memory of the FAM 106. A corresponding segment of the memory region is allocated to each PE (rank 1 to P). In some examples, the memory portion of the memory region allocated to a given rank includes a memory portion with a specified memory portion size at a corresponding fixed offset. "Offset" refers to a memory location of the persistent memory (e.g., represented by a memory address). For P ranks, the memory region 200 is divided into P memory segments 202-1 to 202-P allocated to the corresponding P ranks (rank 1 to P). In some examples, it is assumed that each of the memory segments 202-1 to 202-P has the same memory segment size. The memory segment 202-j (j=1 to P) allocated to rank j starts at offset j. Therefore, in Figure 2 In the example of , memory segment 202-1 assigned to rank 1 starts at offset 1, memory segment 202-2 assigned to rank 2 starts at offset 2, memory segment 202-3 assigned to rank 3 starts at offset 3, and memory segment 202-P assigned to rank P starts at offset P. Although the examples discussed herein involve ranks starting at 1, in other examples, ranks may start at other numbers, such as 0 or a different starting number.

[0046] In some examples, the offset is determined based on the product of the rank number (j) and the memory segment size. The rank number is an identifier of the rank. For example, PEs 118-1 to 118-P may have rank numbers 1 to P.

[0047] One rank (e.g., rank 1) calls the memory allocation function 122-1 to allocate the memory region 200. Each of the other ranks (e.g., ranks 2 to P) similarly calls the memory allocation function of the collective operation programming interface to obtain information about the allocated memory region 200 so that the other ranks can access their allocated memory segments of the allocated memory region 200.

[0048] Examples of collective operations

[0049] The following provides Figure 1 Various examples of collective operations performed by ranks in computer nodes 102-1 through 102-N.

[0050] Figure 3AAn example broadcast collective operation is shown. Array 300 includes rows representing ranks 1-6 and columns representing data units. A "data unit" may refer to a portion of data to be communicated between ranks. A data unit may include a message, an information element, or any other portion of data.

[0051] Figure 3A The broadcast collective operation broadcasts data unit A1 from rank 1 (at 302 ) to all other ranks 2 - 6 , which causes all ranks 1 - 6 to include a copy of data unit A1 , which is represented by array 304 .

[0052] Figure 3B An example scatter collective operation and an example gather collective operation are shown.The scatter collective operation scatters (at 312) data units A1-A6 from rank 1 (as represented by array 310) to ranks 1-6 (as represented by array 314).

[0053] The gather collective operation gathers (at 316) data units A1-A6 (as represented by array 314) from ranks 1-6 to rank 1 (as represented by array 310). In some cases, a scatter-gather collective operation may be initiated that performs a scatter and gather of data units between ranks 1-6.

[0054] Figure 3CAn example all-gather collective operation is shown that aggregates (at 322) data units A1-A6 (as represented by array 320) from ranks 1-6 to all other ranks so that each rank has data units A1-A6 (as represented by array 324). As represented by array 320, data unit A1 is initially stored at rank 1, data unit A2 is initially stored at rank 2, data unit A3 is initially stored at rank 3, data unit A4 is initially stored at rank 4, data unit A5 is initially stored at rank 5, and data unit A6 is initially stored at rank 6. In the all-gather collective operation, rank 1 gathers data units A1-A6 from ranks 1-6, rank 2 gathers data units A1-A6 from ranks 1-6, rank 3 gathers data units A1-A6 from ranks 1-6, rank 4 gathers data units A1-A6 from ranks 1-6, rank 5 gathers data units A1-A6 from ranks 1-6, and rank 6 gathers data units A1-A6 from ranks 1-6. More generally, given P ranks 1 to P, where each rank j (j=1 to P) initially stores data unit Aj, the all-gather collective operation distributes data units A1-AP from the corresponding ranks 1 to P to ranks 1 to P, so that data units A1-A6 are stored on each rank j.

[0055] Figure 3DAn example all-to-all collective operation is shown that distributes (at 332) data units A(1,1) to A(6,6) originally stored at ranks 1-6 (as represented by array 330) among ranks 1-6 so that each rank has data units originally stored at other ranks (as represented by array 334). As represented by array 330, data units A(1,1) to A(1,6) are originally stored at rank 1, data units A(2,1) to A(2,6) are originally stored at rank 2, data units A(3,1) to A(3,6) are originally stored at rank 3, data units A(4,1) to A(4,6) are originally stored at rank 4, data units A(5,1) to A(5,6) are originally stored at rank 5, and data units A(6,1) to A(6,6) are originally stored at rank 6. In the all-to-all collective operation, rank 1 aggregates the data units A(1,1), A(2,1), A(3,1), A(4,1), A(5,1), and A(6,1) from ranks 1-6, rank 2 aggregates the data units A(1,2), A(2,2), A(3,2), A(4,2), A(5,2), and A(6,2) from ranks 1-6, rank 3 aggregates the data units A(1,3), A(2,3), A(3,3), A(4,3), A(5,3), and A(6,3) from ranks 1-6, rank 4 aggregates the data units A(1,4), A(2,4), A(3,4), A(4,4), A(5,4), and A(6,4) from ranks 1-6, and rank 5 aggregates the data units A(1,6), A(2,7), A(3,8), A(4,9), A(5,10), and A(6,11) from ranks 1-6. 1-6 data units A(1,5), A(2,5), A(3,5), A(4,5), A(5,5), A(6,5), and rank 6 aggregates data units A(1,6), A(2,6), A(3,6), A(4,6), A(5,6), A(6,6) from ranks 1-6. More generally, given P ranks 1 to P, where each rank j (j=1 to P) initially stores data unit A(j,k) (j=1 to P and k=1 to P), the all-to-all collective operation distributes each data unit A(j,k) originally stored by rank j to rank k.

[0056] Other types of collective operations can be performed. For example, the following combines Fig. 6A and Figure 6B Further discussion of reduction collective operations.

[0057] Example AllGather Collective Operation

[0058] The following describes an example of a full aggregation collective operation according to some embodiments of the present disclosure. Figure 3C As shown, if each rank were to individually send data units from that rank to all other ranks as part of an all-gathering collective operation, 6*(6-1) or 30 data units would be exchanged in 30 separate messages. If the number of ranks P is large, then the P*(P-1) exchanges of separate messages may overburden the processing and communication resources of the cluster of computer nodes.

[0059] According to some embodiments of the present disclosure, instead of exchanging P*(P-1) individual messages between P ranks, a memory allocation function (e.g., Figure 1 Any one of 122-1 to 122-N in the example ...N) allocates a memory area in a network attached storage (eg, FAM 106).

[0060] In some examples, if MPI is used, the memory allocation function may be as follows:

[0061] int MPI_Alloc_mem(MPI_Aint Size,

[0062] MPI_Info lnfo,

[0063] void * baseptr),

[0064] Wherein, the input parameters include Size and Info, and the output parameters include *baseptr. Although a specific example of a memory allocation function assuming the use of MPI is depicted, in other examples, other memory allocation functions can be used for other types of collective operation programming interfaces. Size represents the size of the memory area to be allocated (including the total size of the memory area for the memory segments of P ranks). Info includes information related to the memory area to be allocated. In some examples, Info may include an information element that specifies the type of memory area to be allocated. The type may be a local memory type or a FAM type. If Info specifies a local memory type, then in the local memory (for example, Figure 1 On the other hand, if the Info specifies a FAM type, a memory area is allocated in the FAM 106 (and in the local memory). The Info may also include other information related to the memory area to be allocated.

[0065] The output parameter *baseptr is returned by MPI_Alloc_mem() and contains the virtual address (VA) to be used by the rank (PE) when accessing the allocated memory area. More specifically, *baseptr is a pointer to the allocated virtual memory, which starts at the virtual address and whose size is specified by the input parameter Size. The MPI_Alloc_mem() function allocates a memory segment to each rank in the ranks involved in the collective operation.

[0066] Figure 4 is a flow chart illustrating a process for memory allocation for full-gather collective operations according to some examples. In other examples, Figure 4 The tasks can be performed in a different order, some tasks can be omitted, and other tasks can be added.

[0067] Each of the ranks 1-P (e.g., Figure 1 Any one of the PEs 118-1 to 118-N in the 402nd embodiment calls (at 402) a memory allocation function in the collective operation programming interface 420 to allocate a memory region for a full aggregate collective operation. The collective operation programming interface 420 is Figure 1 In an example where the collective operation programming interface 420 is an MPI API, the memory allocation function called is an MPI_Alloc_mem() function. In response to a call from rankj (j=1 to P), the MPI_Alloc_mem() function allocates the requested memory area, which includes a memory segment for the corresponding rank in the FAM 106 (e.g., Figure 2 202-1 to 202-P in FIG. ...2 to 202-P in FIG. 202-3 to 202-4. Each memory segment is offset from a corresponding offset (eg, Figure 2 The MPI_Alloc_mem() function allocates memory segments in the local memory for each rank. The memory segments allocated in the local memory may be cache memory for the corresponding memory segments allocated in the FAM 106.

[0068] In some examples, allocating a memory region in the FAM 106 is performed by calling (at 404) an allocation function in the FAM programming interface 428 for allocating a memory region in the FAM 106 via an MPI_Alloc_mem() function. Figure 4 In the example of , the allocation function is called fam_allocate() function. The FAM programming interface 428 is Figure 1An example of any one of the FAM programming interfaces 128-1 to 128-N in.

[0069] In an example, the fam_allocate() function may be an OpenFAM memory allocation function. In some examples, fam_allocate() may have the following syntax:

[0070] fam_allocate(Name, nbytes, ..., Region),

[0071] Where Name represents the name of the memory region allocated for the collective operation, nbytes represents the size of the memory region (e.g., in bytes), and Region represents a descriptor of the memory region (referred to as a "FAM descriptor"). Although a specific example assuming the use of an allocation function of OpenFAM is depicted, in other examples, other allocation functions may be used to handle other types of FAM programming interfaces between entities.

[0072] The fam_allocate() function allocates (at 406) a memory region in the FAM 106. The memory region includes P memory segments for P ranks. The fam_allocate() function causes a memory server (e.g., Figure 1 108-1 to 108-N) in the FAM 106 to allocate a memory area.

[0073] The fam_allocate() function returns (at 408) the FAM descriptor of the allocated memory area. The FAM descriptor includes the starting address of the allocated memory area. The FAM descriptor may also include other information related to the memory area. In some examples discussed further above, the offset j of the memory segment in the memory area for the corresponding rank is calculated based on the product of the rank number (i) and the memory segment size. The calculated offset j is relative to the starting address of the memory area included in the FAM descriptor. For example, the offset 1 of rank 1 is the offset of the address of the memory segment for rank 1 when added to the starting address of the memory area, and the offset P of rank P is the offset of the address of the memory segment for rank P when added to the starting address of the memory area.

[0074] The FAM descriptor of the allocated memory area is received by the MPI_Alloc_mem() function. In response to receiving the FAM descriptor, the MPI_Alloc_mem() function adds (at 412) entry 410-1 to the mapping data structure 410 of rank j. The mapping data structure 410 can be stored in the local memory of the computer node. In some examples, the mapping data structure 410 can be a mapping table.

[0075] Entry 410-1 associates the virtual address (*baseptr) with the FAM descriptor returned by the fam_allocate() function. Additional entries may be added to mapping data structure 410 for other allocations of memory areas requested by rank j, such as for other collective operations. It should be noted that in some examples, there is one mapping data structure per rank, so that for P ranks, there are P mapping data structures that associate the virtual addresses used by the P ranks with the FAM descriptors of the allocated memory areas.

[0076] The mapping data structure used by each rank is Figure 1 An example of the allocated memory region information 126-1 to 126-N is depicted in FIG.

[0077] The MPI_Alloc_mem() function called by rankj returns (at 418) the virtual address (VA) to rankj. Similarly, the MPI_Alloc_mem() function called by each other rank (other than j) returns the virtual address corresponding to the allocated memory region for use by that other rank.

[0078] Figure 5 is based on Figure 4 A flow chart of a process for performing a full-gather collective operation after memory allocation has been performed. In other examples, Figure 5 The tasks can be performed in a different order, some tasks can be omitted, and other tasks can be added.

[0079] Because the full-aggregate collective operation is an operation involving multiple ranks (e.g., ranks 1 to P), the multiple ranks jointly call (at 502) the full-aggregate function in the collective operation programming interface 520 to initiate the full-aggregate collective operation. The collective operation programming interface 520 is Figure 1 In the example where the collective operation programming interface 520 is an MPI API, the all-gather function called is the MPI_Allgather() function, which has the following syntax:

[0080] int MPI_Allgather(

[0081] const void*sendbuf,int

[0082] sendcount, MPI_Datatype

[0083] sendtype,

[0084] void*recvbuf,

[0085] int recvcount, MPI_Datatype

[0086] recvtype,MPI_Comm comm);

[0087] in:

[0088] *sendbuf is the starting address of the send buffer,

[0089] sendcount is the number of elements in the send buffer,

[0090] sendtype is the data type of the send buffer element in the send buffer,

[0091] *recvbuf is the starting address of the receive buffer,

[0092] recvcount is the number of elements received from any rank,

[0093] recvtype is the data type of the receive buffer element in the receive buffer, and

[0094] comm represents the communicator group including the ranks involved in the collective operation.

[0095] Although a specific example is depicted assuming the use of MPI's all-aggregate functions, in other examples, other all-aggregate functions may be used with other types of collective operation programming interfaces.

[0096] The send buffer contains data to be sent by the rank to the destination, which in the example of the present disclosure is a memory area allocated for the collective operation in FAM 506 (assuming that the Info parameter is set to indicate the FAM type in the MPI_Alloc_mem() function that is called to allocate the memory area). FAM 506 is Figure 1 An example of a FAM 106. The address of the send buffer (*sendbuf) is the address of the send buffer in the local memory of the computer node in which the rank is executed.

[0097] The receive buffer contains data to be received by the rank from other ranks. In the example where a memory area has been allocated in FAM 506, the receive buffer includes the memory area, and *recvbuf is the address of the memory area. The address *recvbuf is set to the virtual address returned by the MPI_Alloc_mem() function for the collective operation.

[0098] The MPI_Allgather() function uses the virtual address *recvbuf to look up (at 504) the mapping data structure 510 to obtain the corresponding FAM descriptor. The mapping data structure 510 is similar to Figure 4 4. If there is an entry (e.g., 510-1) in the mapping data structure 510 containing the virtual address *recvbuf, the corresponding FAM descriptor in the entry is returned in the search. A successful search for the corresponding FAM descriptor in the mapping data structure 510 indicates that the allocated memory area of ​​the FAM 506 is to be used for the full-gather collective operation.

[0099] The MPI_Allgather() function issues a single call (at 505) to the put function in the FAM programming interface 528 to write (at 508) the data in each rank's send buffer (pointed to by *sendbuf) to the allocated memory area in the FAM 506. Figure 4 In the example, the put function is called fam_put() function. The FAM programming interface 528 is Figure 1 1 to 128-N in the FAM programming interface 528. It should be noted that data of different ranks are written to corresponding different memory segments (at different offsets) of the memory area. If the FAM programming interface 528 is an OpenFAM API, the fam_put() function is an OpenFAM function. Although a specific example assuming the use of the put function of OpenFAM is depicted, in other examples, other allocation functions can be used to handle other types of FAM programming interfaces between entities.

[0100] When multiple ranks are writing data from their respective send buffers to corresponding memory segments of a memory region, a synchronization mechanism (e.g., in the form of a barrier) may be implemented to cause a rank to wait for the writes of all of the multiple ranks to be completed.

[0101] Once all ranks have written their data to the memory area, the MPI_Allgather() function issues a single call to the get function in the FAM programming interface 128 (at 511) to read the data at the virtual address (*recvbuf) of the allocated memory area in the FAM 106 (at 512). Figure 4 In the example of , the get function is called fam_get() function. If the FAM programming interface 128 is an OpenFAM API, the fam_get() function is an OpenFAM function. The data in the memory area is transferred (at 514) to the rank to complete the all-gather collective operation.

[0102] according to Figure 5 The all-gather collective operation includes P data writes from P ranks to the memory area (in response to fam_put() calls), and P reads by P ranks of the memory area (in response to fam_get() calls), resulting in 2*P communications between the ranks and FAM 106.

[0103] It should be noted that legacy programs that do not support collective operations using FAM 106 can continue to perform collective operations using local memory. In addition, even if FAM 106 does not exist, programs can use local memory to perform collective operations.

[0104] Other types of collective operations, such as scatter operations, gather operations, all-to-all operations, or other operations involving multiple processing entities, may be performed in a similar manner.

[0105] Example reduction operation

[0106] Another type of collective operation includes a reduction collective operation. Fig. 6A Depicted are reduction operations involving ranks 1, 2, and 3 that initially store data units A1, A2, and A3, respectively. Fig. 6A The reduction operation collects data units A1, A2, and A3 from ranks 1, 2, and 3, and applies a reduction operation F (e.g., a sum, maximum, minimum, median, average, or another operation that aggregates the data) to the data units. The output value produced by the reduction operation F based on A1, A2, and A3 is S, and the output value is stored at rank 1.

[0107] Figure 6BAn all-reduce collective operation is depicted. Ranks 1-4 participate in the all-reduce collective operation. Each rank initially stores a vector with two data elements. For example, rank 1 initially stores the vector [5, 1], rank 2 initially stores the vector [2, 3], rank 3 initially stores the vector [7, 8], and rank 4 initially stores the vector [4, 9]. Figure 6B The reduction function F in applies an indexed vector and operation, where the data values ​​at the first index of the vector are added together (i.e., 5+2+7+4) to produce 18, and the data values ​​at the second index are added together (i.e., 1+3+8+9) to produce 21. Therefore, the reduction function F produces an output vector (18, 21), which is distributed to each of ranks 1, 2, 3, and 4, as shown in Figure 6B shown.

[0108] Figure 7 is according to Figure 4 A flow chart of a process for an all-reduce collective operation after memory allocation has been performed. In other examples, Figure 7 The tasks can be performed in a different order, some tasks can be omitted, and other tasks can be added.

[0109] Because the all-reduce collective operation is an operation involving multiple ranks (e.g., ranks 1 to P), the multiple ranks jointly call (at 702) the all-reduce function in the collective operation programming interface 720 to initiate the all-reduce collective operation. The collective operation programming interface 720 is Figure 1 In the example where the collective operation programming interface 720 is an MPI API, the all-reduce function called is the MPI_Allreduce() function, which has the following syntax:

[0110] int MPI_Allreduce(

[0111] const void*sendbuf, void*recvbuf,

[0112] int count, MPI_Datatype

[0113] datatype, MPI_Op op,

[0114] MPI_Comm comm);

[0115] in:

[0116] *sendbuf is the starting address of the send buffer,

[0117] *recvbuf is the starting address of the receive buffer,

[0118] count represents the number of elements in the send buffer.

[0119] datatype indicates the data type in the send buffer.

[0120] op represents the reduction operation to be performed, and

[0121] comm means communicator (group).

[0122] Although a specific example is depicted assuming the use of an all-reduce function using MPI, in other examples, other all-reduce functions may be used with other types of collective operation programming interfaces.

[0123] In the example where a memory area has been allocated in FAM 706, the receive buffer includes the memory area and *recvbuf is the address of the memory area. FAM 706 is Figure 1 106. The address *recvbuf is set to the virtual address returned by the MPI_Alloc_mem() function for the collective operation.

[0124] The MPI_Allreduce() function uses the virtual address *recvbuf to look up (at 704) the mapping data structure 710 to obtain the corresponding FAM descriptor. The mapping data structure 710 is similar to Figure 4 410 of the mapping data structure 710. If there is an entry (e.g., 710-1) containing the virtual address *recvbuf in the mapping data structure 710, the corresponding FAM descriptor in the entry is returned in the search. A successful search to obtain the corresponding FAM descriptor in the mapping data structure 710 indicates that the allocated memory area of ​​the FAM 706 is to be used for the all-reduce collective operation.

[0125] The MPI_Allreduce() function issues a single call (at 705) to the put function in the FAM programming interface 128 (referred to as fam_put() in the example) to write (at 708) the data in the send buffer of each rank (pointed to by *sendbuf) to the allocated memory area in the FAM 706. It should be noted that data of different ranks are written to corresponding different memory segments (at different offsets) of the memory area.

[0126] When multiple ranks are writing data from their corresponding send buffers to corresponding memory segments of a memory region, a synchronization mechanism may be implemented to cause the rank to wait for the writes of all multiple ranks to be completed.

[0127] Once all ranks have written their data to the memory area, the MPI_Allreduce() function issues a call to an offload function (at 711) to initiate a memory-side operation at the memory servers (e.g., 108-1 to 108-M) of the FAM 706 to perform the reduction operation specified by the parameter op. In an example, the offload function may be a function that is part of the FAM programming interface 728. The FAM programming interface 728 is Figure 1 An example of any one of the FAM programming interfaces 128-1 to 128-N in.

[0128] In some examples, the offload function may cause a FAM client (e.g., one or more of 130-1 to 130-N) to queue computations of the reduce operation into a queue and, after queuing the computations, send (at 712) the computations to a memory server for execution.

[0129] A memory server in the FAM 106 performs (at 714) the offloaded computation of the reduce operation, and the memory server stores (at 716) the result of the reduce operation in a memory area of ​​the FAM 106, such as in a temporary buffer allocated in the memory area.

[0130] In some examples, the wait object waitObj can be used to make rank 1 to P wait for the memory server to complete the calculation of the reduction operation. The wait object causes the PE (rank) to suspend execution to wait for other processes (e.g., threads in the memory server that perform the offloaded calculation) to complete the operation. The wait object may include a status indicator for indicating whether the operation has been completed. In other examples, other types of status indicators may be used to allow the memory server to indicate that the offloaded operation has been completed. When the FAM client detects that the result of the offloaded calculation is available, the FAM client returns the wait object waitObj to one or more waiting PEs. In some examples, the FAM client may detect that the result is available based on an indication (e.g., a signal, a message, an information element, or any other type of indicator) provided by the memory server about the completion of the offloaded calculation at the memory server. (Multiple) waiting PEs may periodically or intermittently check waitObj to determine the status of waitObj. If waitObj indicates that the memory server has completed the offloaded calculation, the result of the offloaded operation may be provided to (multiple) PEs.

[0131] Once the results of the reduce operation are available, the MPI_Allreduce() function issues a call (at 718) to a get function (referred to as fam_get() in the example) in the FAM programming interface 128 to read the results (at 721) at the virtual address (*recvbuf) of the allocated memory area in the FAM 106. The fam_get() function may be called in response to requests from ranks 1 to P. The results in the memory area are transferred (at 722) to the ranks to complete the all-reduce collective operation.

[0132] according to Figure 7 The all-reduce collective operation includes 2*P communications between the ranks and the FAM 706. In addition, the computation of the reduce operation is offloaded to the memory server of the FAM 706, which can utilize the available processing resources of the memory server and reduce the possibility of overburdening the computer nodes 102-1 to 102-N.

[0133] Other types of collective operations involving computations may be performed in a similar manner, where the computations are offloaded to the memory servers of FAM 706 .

[0134] Further examples

[0135] Figure 8 Depicted is a flow of performing collective operations using an allocated memory region of a network attached storage according to some examples. Figure 8 The process includes receiving (at 802) by a processor of a first computer node a first request to allocate a memory region for a collective operation performed by processing entities in a plurality of computer nodes. For example, the first request may include a call to a memory allocation function in a collective operation programming interface. In a more specific example, the memory allocation function may be an MPI memory allocation function MPI_Alloc_mem().

[0136] Figure 8 The process includes creating (at 804) a virtual address of a memory region by a processor in response to a first request, and allocating (at 806) a memory region in a network attached memory that is coupled to the plurality of computer nodes via a network. Creating a virtual address may refer to selecting a virtual address from a plurality of virtual addresses of a virtual address range associated with the network attached memory. Allocating a memory region may refer to designating a region of the network attached memory for use by a requesting entity (e.g., a PE).

[0137] Figure 8 The process includes associating, by the processor at 808, the virtual address with the address of the memory region in mapping information stored in the first computer node. An example of mapping information is, for example, Figure 4 The mapping data structure 410. Associating the virtual address with the address of the memory region in the mapping information may refer to defining an association between the virtual address and the address of the memory region and adding information of the association thus defined to an entry of the mapping information.

[0138] Figure 8 The process includes receiving (at 810) by the processor a second request to perform a collective operation, the second request including a virtual address. The second request may include a call to a collective operation function (e.g., MPI_Allgather, MPI_Allreduce, or another type of operation involving multiple processing entities) in a collective operation programming interface.

[0139] Figure 8 The process includes identifying (at 812) a memory region in the network attached storage by the processor using the virtual address included in the second request to obtain the address of the memory region from the mapping information. Identifying the memory region may refer to obtaining information that allows a requesting entity (such as a processing entity) to reference and use the memory region. For example, identifying includes looking up the mapping data structure 410 using the virtual address to find an entry that includes a FAM descriptor associated with the virtual address.

[0140] Figure 8 The process includes, in response to the second request, performing (at 814) by the processor a collective operation including writing a first data unit of a first computer node to a first segment of a memory area and accessing a second data unit written by a second computer node from a second segment of the memory area. Performing the collective operation may refer to a task of performing the collective operation.

[0141] In some examples, the processor confirms use of the network attached memory for the collective operation in response to successfully performing a lookup of the mapping information using the virtual address.

[0142] In some examples, a plurality of memory servers perform computations of collective operations on a plurality of data units stored in a network attached memory, the plurality of memory servers managing allocation of memory regions and access to data in the network attached memory. In some examples, the computations performed by the plurality of memory servers are performed in response to an offload request from a computer node.

[0143] In some examples, allocating a memory region in the network attached storage includes allocating a first segment of the memory region to a first processing entity in a first computer node and allocating a second segment of the memory region to a second processing entity in a second computer node.

[0144] In some examples, a rank number is assigned to each of a first processing entity and a second processing entity, allocating a first segment of a memory area to the first processing entity includes allocating the first segment starting from a first offset based on the rank number of the first processing entity, and allocating a second segment of the memory area to the second processing entity includes allocating a second segment starting from a second offset based on the rank number of the second processing entity.

[0145] In some examples, allocating a memory region in the network attached memory in response to the first request is performed in response to the first request including first configuration information having a first value indicating that the network attached memory is used for collective operations. For example, the configuration information may include an input parameter Info of an MPI_Alloc_mem() function.

[0146] In some examples, a processor of the first computer node receives a third request to allocate an additional memory region for another collective operation performed by the plurality of computer nodes. The third request includes second configuration information having a second value that indicates to use a local memory of the first computer node for the another collective operation. In response to the third request, the processor allocates the additional memory region in the local memory of the computer node.

[0147] Fig. 9 is a block diagram of a system 900 according to some examples. The system 900 includes a network attached storage 902, such as Figure 1 The system 900 further includes a plurality of computer nodes 904-1, 904-2 coupled to the network attached storage 902 via a network 906. The computer node 904-1 includes one or more hardware processors 907 configured to perform various tasks. The hardware processor may include a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or another hardware processing circuit.

[0148] although Fig. 9 The tasks are shown to be performed by computer node 904-1, but it should be noted that another computer node 904-2 may perform some or all of the tasks of computer node 904-1.

[0149] The tasks of the one or more hardware processors 907 include a memory region allocation request receiving task 908 that receives a first request to allocate a memory region for a collective operation performed by processing entities in the plurality of computer nodes 904. The first request includes a first value indicating use of the network attached memory 902 for the collective operation.

[0150] The tasks of the one or more hardware processors 907 include a virtual address creation and memory region allocation task 910 for creating a virtual address of a memory region and allocating the memory region in the network attached memory in response to the first request including a first value. Creating a virtual address may refer to selecting a virtual address from a plurality of virtual addresses of a virtual address range associated with the network attached memory 902. Allocating a memory region may refer to assigning a region of the network attached memory 902 for use by a requesting entity (e.g., a PE).

[0151] The tasks of the one or more hardware processors 907 include a mapping information update task 912 for adding an entry to the mapping information that associates a virtual address with an address of a memory region in the network attached memory. The mapping information may be, for example, Figure 4 The mapping data structure 410 of FIG. 4. Adding an entry to the mapping information may refer to adding information defining an association between a virtual address and an address of a memory region to the mapping information.

[0152] The tasks of the one or more hardware processors 907 include a collective operation request receiving task 914, which is used to receive a second request for executing a collective operation. The second request includes a virtual address.

[0153] The tasks of the one or more hardware processors 907 include a memory region identification task 916 for identifying a memory region in the network attached storage by obtaining an address of the memory region from mapping information using a virtual address included in the second request. Identifying the memory region may refer to obtaining information that allows a requesting entity (such as a processing entity) to reference and use the memory region.

[0154] The tasks of the one or more hardware processors 907 include a collective operation execution task 918, which is used to execute a collective operation in response to the second request, the collective operation including writing a first data unit of the first computer node to a first segment of the memory area and accessing a second data unit written by the second computer node from a second segment of the memory area. Executing a collective operation may refer to a task of executing a collective operation.

[0155] Fig.10 is a block diagram of a non-transitory machine-readable or computer-readable storage medium 1000 storing machine-readable instructions that, when executed, cause the system to perform various tasks. The machine-readable instructions may be in, for example, Figure 1 The process is executed in the computer nodes 102-1 to 102-N.

[0156] The machine readable instructions include memory region request receiving instructions 1002 for receiving, at a first computer node, a first request to allocate a memory region for a collective operation performed by a plurality of computer nodes.

[0157] The machine-readable instructions include virtual address creation and memory region allocation instructions 1004 for creating a virtual address of a memory region and allocating a memory region in a network attached storage coupled to the plurality of computer nodes via a network in response to a first request.

[0158] The machine-readable instructions include mapping information instructions 1006 for associating a virtual address with an address of a memory region in mapping information stored in the first computer node. For example, the associating includes, for example, Figure 4 An entry is added to the mapping data structure 410.

[0159] The machine-readable instructions include collective operation request receiving instructions 1008 for receiving a second request to perform a collective operation, the second request including a virtual address.

[0160] The machine readable instructions include memory region identifying instructions 1010 for identifying a memory region in the network attached storage by obtaining an address of the memory region from mapping information using a virtual address included in the second request.

[0161] The machine-readable instructions include collective operation execution instructions 1012, which are used to perform a collective operation in response to a second request, the collective operation including writing a first data unit of a first computer node to a first segment of a memory area and accessing a second data unit written by a second computer node from a second segment of the memory area.

[0162] Storage media (e.g. Fig.101000 in ) may include any one or some combination of the following: a semiconductor memory device, such as DRAM or SRAM, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory; a magnetic disk, such as a fixed disk, a floppy disk, and a removable disk; another magnetic medium, including a tape; an optical medium, such as a compact disk (CD) or a digital video disk (DVD); or another type of storage device. Note that the instructions discussed above may be provided on one computer-readable or machine-readable storage medium, or alternatively, may be provided on multiple computer-readable or machine-readable storage media distributed in a large system with possibly multiple nodes. Such one or more computer-readable or machine-readable storage media are considered to be part of an article (or product). An article or product may refer to any manufactured single component or multiple components. One or more storage media may be located in a machine running machine-readable instructions, or at a remote site, from which machine-readable instructions may be downloaded over a network for execution.

[0163] In the present disclosure, unless the context clearly indicates otherwise, the use of the terms "a", "an" or "the" is intended to include the plural forms as well. Similarly, when used in the present disclosure, the terms "includes / including / comprises / comprising", "have / having" specify the presence of the elements, but do not exclude the presence or addition of other elements.

[0164] In the foregoing description, many details are set forth to enable an understanding of the subject matter disclosed herein. However, embodiments may be practiced without some of these details. Other embodiments may include modifications and variations of the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A method comprising: receiving, by a processor of a first computer node, a first request to allocate a memory region for a collective operation performed by processing entities in a plurality of computer nodes; In response to the first request, creating, by the processor, a virtual address of the memory region and allocating the memory region in a network attached storage coupled to the plurality of computer nodes via a network; associating, by the processor, the virtual address with an address of the memory region in mapping information stored in the first computer node; receiving, by the processor, a second request to perform the collective operation, the second request including the virtual address; identifying, by the processor, the memory region in the network attached storage by obtaining an address of the memory region from the mapping information using the virtual address in the second request; as well as In response to the second request, the collective operation is performed by the processor, the collective operation comprising writing a first data unit of the first computer node to a first segment of the memory area and accessing a second data unit written by a second computer node from a second segment of the memory area.

2. The method according to claim 1, further comprising: In response to the second request, the collective operation is performed by a processor of the second computer node, the collective operation comprising writing the second data unit to a second segment of the memory area and accessing the first data unit from a first segment of the memory area.

3. The method according to claim 1, further comprising: Computation of another collective operation is performed on a plurality of data units stored in the network attached storage by a plurality of storage servers that manage allocation of the memory region and access to data in the network attached storage.

4. The method according to claim 3, further comprising: Results of the calculations are retrieved from the network attached storage by the plurality of computer nodes.

5. The method of claim 3, wherein performing the computations by the plurality of memory servers is performed in response to an offload request from the first computer node.

6. The method of claim 1 , wherein allocating the memory area in the network attached storage comprises allocating a first segment of the memory area to a first processing entity in the first computer node, and allocating a second segment of the memory area to a second processing entity in the second computer node.

7. A method according to claim 6, wherein a rank number is assigned to each of the first processing entity and the second processing entity, allocating a first segment of the memory area to the first processing entity includes allocating the first segment starting from a first offset based on the rank number of the first processing entity, and allocating a second segment of the memory area to the second processing entity includes allocating the second segment starting from a second offset based on the rank number of the second processing entity.

8. The method of claim 1, wherein allocating the memory region in the network attached memory in response to the first request is performed in response to the first request including first configuration information having a first value, the first value indicating use of the network attached memory for the collective operation.

9. The method according to claim 8, further comprising: receiving, by a processor of the first computer node, a third request to allocate additional memory area for another collective operation performed by the plurality of computer nodes, the third request comprising second configuration information having a second value, the second value indicating to use a local memory of the first computer node for the another collective operation; as well as The further memory region is allocated, by the processor, in a local memory of the first computer node in response to the third request.

10. The method of claim 1, wherein performing the collective operation by the processor further comprises accessing a third data unit written by a third computer node from a third section of the memory area.

11. The method of claim 1, wherein the collective operation comprises a message passing interface (MPI) collective operation.

12. The method of claim 1, wherein the mapping information associates the virtual address with a descriptor, the descriptor comprising an address of a memory segment in the memory region in the network attached storage, the memory segment being allocated to the processing entity in the plurality of computer nodes.

13. The method according to claim 1, comprising: Responsive to successfully performing a lookup of the mapping information using the virtual address, use of the network attached storage for the collective operation is confirmed.

14. The method of claim 1, wherein the collective operation is performed by the processing entities executed in the plurality of computer nodes, the first request is issued by a first processing entity among the processing entities, and the method comprises: Obtaining the mapping information by a second processing entity among the processing entities; as well as The mapping information is used by the second processing entity in the collective operation.

15. A system comprising: Network Attached Storage; as well as A plurality of computer nodes are coupled to the network attached storage via a network, the plurality of computer nodes comprising a first computer node and a second computer node, and the first computer node comprises a processor configured to perform the following operations: receiving a first request to allocate a memory region for a collective operation performed by processing entities in the plurality of computer nodes, the first request comprising a first value indicating use of the network attached memory for the collective operation; in response to the first request including the first value, creating a virtual address of the memory region and allocating the memory region in the network attached storage; adding to mapping information an entry associating the virtual address with an address of the memory region in the network attached storage; receiving a second request to perform the collective operation, the second request including the virtual address; identifying the memory region in the network attached storage by obtaining an address of the memory region from the mapping information using the virtual address in the second request; as well as In response to the second request, the collective operation is performed, the collective operation comprising writing a first data unit of the first computer node to a first segment of the memory area and accessing a second data unit written by the second computer node from a second segment of the memory area.

16. The system of claim 15, wherein the network attached storage comprises a plurality of storage servers configured to perform computations of another collective operation offloaded from the plurality of computer nodes to the plurality of storage servers. 17 . The system of claim 16 , wherein the plurality of memory servers are configured to store results of the calculations in the memory area, and the plurality of computer nodes are configured to retrieve the results from the memory area.

18. The system of claim 15, wherein allocating the memory region in the network attached storage comprises allocating a first segment of the memory region to a first processing entity in the first computer node and allocating a second segment of the memory region to a second processing entity in the second computer node, and Wherein a rank number is assigned to each of the first processing entity and the second processing entity, allocating a first segment of the memory area to the first processing entity includes allocating the first segment starting from a first offset based on the rank number of the first processing entity, and allocating a second segment of the memory area to the second processing entity includes allocating the second segment starting from a second offset based on the rank number of the second processing entity.

19. A non-transitory machine-readable storage medium comprising instructions that, when executed, cause a system to: receiving, at a first computer node, a first request to allocate a memory region for a collective operation performed by processing entities in a plurality of computer nodes; In response to the first request, creating a virtual address of the memory region and allocating the memory region in a network attached storage coupled to the plurality of computer nodes via a network; associating the virtual address with an address of the memory region in mapping information stored in the first computer node; receiving a second request to perform the collective operation, the second request including the virtual address; identifying the memory region in the network attached storage by obtaining an address of the memory region from the mapping information using the virtual address in the second request; as well as In response to the second request, the collective operation is performed, the collective operation comprising writing a first data unit of the first computer node to a first segment of the memory area and accessing a second data unit written by a second computer node from a second segment of the memory area.

20. The non-transitory machine-readable storage medium of claim 19, wherein the collective operation is performed by P processing entities in the plurality of computer nodes, and the collective operation involves 2*(P-1) communications of data units to complete the collective operation, and wherein, P is greater than or equal to 2.

Images