Distributed computer system, non-transitory machine-readable medium, and method

Abstract

The present application relates to electronic data file access management in a distributed computer system. The electronic data file management system may include a storage node comprising a memory device coupled to a computing node via a network. The computing node is configured to request group access to a file on the storage node, the request being issued by at least two processes running on the computing node. At least one of the computing nodes is configured to determine the type of the memory device comprising the file, so that the plurality of computing nodes collectively access the file in a first access mode, apply a write lock mechanism and disable a collective buffering mechanism to each of the different computing nodes if the memory device is of the first type, and access the file in a second access mode, apply a write lock mechanism and enable a collective buffering mechanism to each of the different computing nodes if the memory device is of the second type.

Description

Technical Field

[0001] The present application relates generally to distributed computer systems. Background Art

[0002] Computing systems utilizing multi-node parallel processing require communication protocols in order to operate. For example, a computing system can adopt a protocol such as a message passing interface (MPI) to manage the operation of parallel processes executed on a computing node and to transmit information between parallel processes. Many of these computing systems also include storage nodes connected to computing nodes configured for storage using a distributed file storage system. In order to manage the access of computing nodes to files on storage nodes, computing nodes require additional communication protocols that enhance the file management system in the operating system used in the computing nodes. For example, MPI has been updated to include an input / output message passing protocol and is referred to as a message passing interface-input / output (MPI-IO). MPI-IO includes two elements or functions for enhancing the standard file system control in the operating system to more efficiently perform read and write access.

[0003] Most storage nodes utilize traditional mass storage memory devices, such as magnetic hard drives and optical drives. Recent technological advances have provided new mass storage memory devices, such as solid-state drives or persistent memory, that can access data within the memory much faster than traditional mass storage memory devices. In addition, due to the development of high-speed interface protocols, such as Non-Volatile Memory Express (NVMe), external access to the memory on these new mass storage memory devices is even faster. Summary of the Invention

[0004] According to one aspect of the present invention, a distributed computer system comprises: at least one storage node coupled to a network, the storage node being configured in a parallel distributed file system and comprising at least one type of memory device; a plurality of computing nodes coupled to the network, each of the plurality of computing nodes comprising at least one processor, the plurality of computing nodes being configured to request group access to a file in the parallel distributed file system, the request being issued by at least two processes, each of the at least two processes running on a different one of the plurality of computing nodes in the network; wherein at least one of the plurality of computing nodes is configured to: determine the type of the memory device comprising the file; if it is determined that the memory device is of a first type, enable the plurality of computing nodes to collectively access the file in a first access mode, the first access mode comprising applying a write locking mechanism to each of the different ones of the computing nodes and disabling a collective buffering mechanism; and if it is determined that the memory device is of a second type, access the file in a second access mode, the second access mode comprising applying the write locking mechanism to each of the different ones of the computing nodes and enabling a collective buffering mechanism.

[0005] According to another aspect of the present invention, a non-transitory machine-readable medium stores an operation image, the operation image including instruction file access management instructions, which, when executed by at least one processor in a computing node, causes the node to instantiate a data file access system to: in response to a request for group access to a file in a parallel distributed file system issued by at least two processes: determine the type of the storage device including the file; if it is determined that the storage device is of a first type, cause the multiple computing nodes to collectively access the file in a first access mode, the first access mode including applying a write lock mechanism to each of the different computing nodes in the computing nodes and disabling a collective buffer mechanism; and if it is determined that the storage device is of a second type, access the file in a second access mode, the second access mode including applying the write lock mechanism to each of the different computing nodes in the computing nodes and enabling a collective buffer mechanism.

[0006] According to another aspect of the present invention, a method for a distributed computer system includes: requesting group access to a file located on a first type of storage device in a parallel distributed file system, the request being issued by at least two processes, each of the at least two processes running on a different one of a plurality of computing nodes in a network; and accessing the file in a first access mode, the first access mode being associated with the first type of storage device, the first access mode including applying a write lock mechanism to each of the different ones of the computing nodes and disabling a collective buffering mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present invention can be understood from the following detailed description, which can be understood alone or in conjunction with the accompanying Figure 1 The accompanying drawings are included to provide a further understanding of the present invention and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate one or more examples of the present teachings and, together with the description, explain certain principles and operations. In the drawings:

[0008] Figure 1 is a block diagram illustrating an example distributed computer system.

[0009] Figure 2 is a flow chart illustrating an example method for managing access to electronic data files in a distributed computer system.

[0010] Figure 3 is a flow chart illustrating another example method for managing access to electronic data files in a distributed computer system.

[0011] Figure 4 is a flow chart illustrating an example method for accessing a file in a first access mode in a distributed computer system.

[0012] Figure 5 is a flow chart illustrating an example method for accessing a file in a first access mode in a distributed computer system.

[0013] Figure 6 is a flow chart illustrating an example method for determining a type of memory device in a distributed computer system.

[0014] Figure 7 is a flow chart illustrating another example method for determining a type of memory device in a distributed computer system.

[0015] Figure 8 is a flow chart illustrating another example method for managing access to electronic data files in a distributed computer system.

[0016] Figure 9An example non-transitory machine-readable medium storing example data file access management instructions is shown.

[0017] Figure 10 An example computing node including a non-transitory machine-readable medium storing example data file access management instructions is shown. DETAILED DESCRIPTION

[0018] Modern computer systems typically include multiple compute nodes that run multiple processes connected to multiple shared storage nodes via a network. More specifically, in some systems (such as high performance computing (HPC) systems), many compute nodes and processes can be tightly coupled together to work together (e.g., as a cluster or distributed computing system) to process the same job, with each process taking on a different part of the job. In some cases, several processes in a compute node may need to access the same file in a shared storage target within a storage node at the same time. This situation is called group access and is not something that can be effectively handled using a standard file management system as part of the operating system included on the compute node. Instead, management of group access to storage nodes is performed by a data transfer protocol (such as MPI-IO).

[0019] The structure of MPI-IO ensures that access to files in storage nodes requested by multiple compute nodes does not result in overlapping data storage within the storage resource. Some file systems associated with MPI-IO include a group locking mechanism and a collective buffering mechanism. The group locking mechanism notifies the standard file management system to disable its normal locking mechanism and allows the process using the group locking mechanism to be responsible for ensuring consistent data storage. The file management system's normal locking mechanism typically prevents multiple processes from writing to a file at once to avoid inconsistent or overlapping writes, but with the normal locking mechanism disabled (due to the group locking mechanism), multiple processes can write to the file simultaneously. The collective buffering mechanism operates in two phases. The first phase involves assigning a subset of processes requesting access to the file as aggregators. The aggregators first collect all data from the processes requesting access to the file. Then, based on the group locking conditions of each aggregator, the aggregators write the data to the file. In this way, the number of processes writing to remote storage during a group write is reduced because a subset of processes (aggregators) rather than all of them write the data for any particular group write call. For example, if there are 20 processes, each of which writes 10 kilobyte (kB) chunks of data, then a subset of these processes (e.g., 4) will act as aggregators and collect 10kB chunks of data from the other processes. Each aggregator will then write a 50kB chunk of data (including its data and the data collected from the other four processes) to remote storage.

[0020] This two-step process in collective buffering works well on more traditional mass storage memory devices such as magnetic hard drives and optical drives. However, with the advent of more high-speed storage targets such as solid-state drives or persistent memory, and the development of high-speed interface protocols such as NVMe, the two-stage collective buffering mechanism has become a performance bottleneck due to the additional processing overhead it requires within the compute nodes. Until now, in systems that utilize or are based on MPI-IO, when group locking is used, this automatically leads to the use of collective buffering. Until now, no messaging protocol has been able to adapt to and take advantage of high-speed storage targets and high-speed interface protocols.

[0021] To address these challenges associated with group access to high-speed storage targets, the examples disclosed herein introduce a new mechanism or mode for accessing data files within a data transfer protocol (e.g., MPI-IO) that retains the advantages of a group locking mechanism but does not force the use of traditional collective buffering. When two or more processes in a computing node issue a group access request to a file, the new access mode applies a write lock similar to a group lock to these processes. The new access mode also allows each process to access the file based on the process's write lock conditions. In other words, the new access mode involves using a group lock or write lock mechanism while disabling the above-mentioned collective buffering mechanism. In addition, the use of the new access mode can be controlled based on determining the type of memory device being accessed for the storage target. More specifically, when the type of memory device is a fast storage target, as described above, the new access mode can be used, while when the type of memory device is a traditional, slower storage target, the traditional access mode using group locking and collective buffering can be used.

[0022] In some examples, access to a file using the new access mode can be enforced by allowing each process requesting access to write data directly to remote storage. In some examples, a data transfer protocol (e.g., MPI-IO) can treat processes as all aggregators in a group write mechanism, aggregators of their own data, but not of other processes. This approach differs from the way aggregators are used in traditional collective buffering operations because (a) all processes write to storage, not just a subset, and (b) no process collects data from other processes to write (they all write only their own data). In this way, the new access mode disables the intermediate step of collecting data from processes before accessing the file. By disabling collective buffering (the two-stage process of which is described herein) while still maintaining the group write lock, processing overhead within the compute node is reduced. As a result, processes requesting access to files in the compute node are better able to keep up with the access capabilities of higher-speed storage devices.

[0023] Turning now to the drawings, various apparatus, systems, and methods consistent with aspects of the present invention will be described.

[0024] Figure 1 is a block diagram illustrating an example computer system 100. It should be understood that Figure 1 It is not intended to illustrate a specific implementation of the components shown, and implementations of computer system 100 may have a different number and arrangement of the components shown, and may also include other elements not shown.

[0025] like Figure 1 As shown, the computer system 100 is configured as a multi-node parallel processing computer system including multiple storage nodes configured in a parallel distributed file system. The computer system 100 includes multiple computing nodes 110 (e.g., computing nodes 110A-n) interconnected with at least one storage node 150 (e.g., storage nodes 150A-p) via a network 140. Figure 1 In FIG, computing nodes 110A-n are shown as examples, but computer system 100 may include any number of computing nodes 110 equal to or greater than 2. Similarly, in Figure 1 , storage nodes 150A-p are shown as examples, but computer system 100 may include any number of storage nodes 150. Network 140 may include one or more network elements (not shown), such as switches, data communication cables, or other network elements that route communications between compute nodes 110 and storage nodes 150. Network 140 may use any network protocol or network technology.

[0026] Each of the computing nodes 110 includes one or more processors 111 (eg, processors 111A-111m), a network interface card (NIC) 112, and local memory 115. Figure 1 , computing nodes 110A-n are shown as having processors 111A-111m, but each computing node 110 does not necessarily have the same number of processors 111, and in some examples, one or more computing nodes 110 may have a single processor. Local memory 115 includes storage for application software and associated data 116, a process thread system 117, and a data file access system 118.

[0027] Each storage node 150 includes a NIC 152, a memory controller 154, and a storage array 156 or a storage array 158. The storage array 158 includes storage media of a first performance class, and the storage array 156 includes storage media of a second performance class, where the performance (e.g., read or write time) of the second performance class is slower than that of the first performance class. For example, the storage array 158 may include solid-state storage media, such as flash memory devices or non-volatile RAM (NVRAM) (e.g., flash memory, memristor RAM, resistive RAM, phase change memory, 3D XPoint memory, etc.), while the storage array 156 may include disk-based storage media. Figure 1 , a plurality of storage nodes 150 are shown with a mix of storage arrays 156 and 158. Specifically, in Figure 1 , storage nodes 150A and 150B include storage array 156, while storage node 150p includes storage array 158. However, in some examples, system 100 may include only storage nodes 150 with storage array 156, while in other examples, system 100 may include only storage nodes 150 with storage array 158. Furthermore, in some examples, only a single storage node 150 is provided (which may have either storage array 156 or storage array 158). As explained in more detail below, in some examples (such as in Figure 1 ), system 100 may use different access modes when writing to storage array 156 compared to storage array 158. These different access modes are described in more detail below.

[0028] The processor 111 in the computing node 110 may include any electronic processing circuit capable of executing machine-readable instructions, such as a central processing unit (CPU), a system on a chip (SoC), a microprocessor, a microcontroller device, a digital signal processor, etc.

[0029] The NIC 112 in the compute node 110 is configured to provide a wired or wireless connection between the compute node 110 and the network 140. Some examples of physical interfaces include a wired Ethernet connection, a wireless 2.4 gigahertz (GHz) transceiver connection, and a 5 GHz transceiver connection. The NIC 112 also includes software programs and instructions and control commands to allow the compute node to communicate with other elements (including other compute nodes and storage nodes 150) using common communication protocols. Some examples of communication protocols include Internet Protocol (IP), cellular 3G, 4G, and 5G, Institute of Electrical and Electronics Engineers (IEEE) standard 802.11 protocols, Fibre Channel (FC), InfiniBand, and HPE / Cray Slingshot.

[0030] The local memory 115 in the compute node 110 may include any machine-readable medium from which a processor may read program instructions and data, including volatile media such as random access memory (RAM) (e.g., dynamic RAM, static RAM, etc.) and / or persistent (non-volatile) media such as non-volatile RAM (NVRAM) (e.g., flash memory, memristor RAM, resistive RAM, phase change memory, 3D XPoint memory, etc.).

[0031] The application and data portion 116 of the local memory 115 is configured to store and retrieve program instructions and any associated data for distributed software applications running on the computing nodes 110. In some instances, the entire software application can be stored in the application and data portion 116 of each local memory 115. In other cases, a portion can be stored in each local memory 115, such as the portion running in the associated storage node 110A-n. Examples of distributed software applications that can be stored in the application and data portion include cloud platform applications for managing businesses using software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS).

[0032] The process thread system 117 of the local memory 115 is configured to store and retrieve program instructions for implementing and managing parallel or distributed processing of software applications across the processors 111 in each of the compute nodes 110A-n. Many process thread systems 117 can utilize a set of instructions and control commands that are included as part of the MPI standard. The MPI instructions and control commands provide a communication mechanism between processes managed by the process thread system 117. Examples of process thread systems 117 include the Portable Operating System Interface (POSIX) and the Compute Unified Device Architecture (CUDA).

[0033] The data file access system 118 of the local storage 115 is configured to store and retrieve program instructions for implementing and managing access to one or more files in the storage nodes 150A-p via the NIC 112. A file is generally defined as an arrangement of linked memory locations within a storage structure (such as the storage node 150). The memory locations may have specific addresses that allow access thereto. The program instructions stored in the data file access system 118 include instructions and commands for managing and controlling access to files by multiple processes executing on the processor 111 as part of a distributed software application. The data file access system 118 may include write lock instructions that may be executed as part of each process running on the processor 110 and requesting access to a file. The write lock instructions may include instructions for removing specific file locks that control access to files as part of the file system used by the operating system for the compute node. The write lock instructions may also include instructions for replacing specific file locks used by the file system with write lock instructions that implement a group write lock that may be provided to a group of processes collectively requesting access to write data to a file. The write lock instruction may also include instructions to allocate a portion of the memory in storage node 150 used for the file to one or more processes requesting access to the file. The data file access system 118 also includes collective buffering instructions that can be executed in each process requesting access to the data file in conjunction with the write lock instruction. The data file access system 118 also includes instructions to execute the write lock instruction without executing some or all of the collective buffering instructions. Examples of the data file access system 118 include MPI-IO and Lustre.

[0034] The application and data components 116, process thread system 117, data file access system 118, and other components and systems (not shown) configured in local memory 115 constitute a distributed computing software stack. An example of a distributed computing software stack is MPI on Chameleon (MPICH).

[0035] In some examples, computing system 100 comprises an HPC system, such as an HPE Cray EX system, an HPE Apollo system, or other HPC system. In some examples, computing system 100 comprises a converged or hyperconverged computing system, such as an HPE Converged System, an HPE SimpliVity system, or other converged / hyperconverged system. In some examples, computing system 100 comprises a collection of individual servers, such as HPE ProLIant servers or other servers. Examples of compute nodes 110 include nodes of an HPC system, converged or hyperconverged computing modules, individual servers, or other computing devices.

[0036] NIC 172 in storage node 150 is configured to provide a wired or wireless connection between storage node 150 and network 140 , and may be similar to those examples described for NIC 112 in compute node 110 .

[0037] The memory controller 154 in the storage node 150 is configured to provide an interface for processing data received over the network (via the NIC) to direct the data to a specific portion or location within the storage array. Examples of memory controllers include Ultra320 Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), Peripheral Component Express (PCIe), and NVMe.

[0038] As described above, in some examples, one or more of the storage nodes 150, such as Figure 1 The storage nodes 150A and 150B in FIG. 1 include a storage array 156 that includes relatively slower second performance level storage media, such as disk-based storage devices. Examples of disk-based storage devices include magnetic disk drives and optical disk drives. In addition, in some examples, one or more of the storage nodes 150, such as Figure 1The storage node 150p in includes a storage array 158, which includes relatively fast first performance level storage media, such as solid-state device (SSD) memory devices. These SSD memory devices are capable of achieving faster access speeds than disk-based memory devices. It is worth noting that the typical access speed of disk-based memory devices is about 300 megabytes per second (MB / s), while the typical access speed of SSD memory devices is about 4000 MB / s. Examples of storage nodes 150A and 150B include Cray ClusterStor L300, HP Eapollo 4000 system with HHD, HP E Nimble system with HHD, or other storage systems. Examples of storage node 150p are Cray ClusterStor E1000, HPE Apollo 4000 system with SSD, HPE 3-PAR system, or other storage systems. These two types of storage arrays 156 and 158 are not necessarily always provided in the same system 100, but in some cases, there may be some benefits to providing these two types of storage arrays 158 and 156. For example, storage array 156 may be more cost-effective in terms of cost per bit of storage capacity, so it may be beneficial to use such storage array 156 for bulk storage of large amounts of data (e.g., to reduce costs), while storage array 158, despite having a higher cost per bit of storage capacity, may provide faster access speeds, so it may be beneficial to use such storage array 158 for data that is expected to be accessed more frequently and / or requires faster access. By combining the two types of storage arrays 156 and 158 in the same system, cost and performance can be balanced. The number and distribution of such storage arrays 156 and 158 are not limited to any particular arrangement. On the other hand, some systems 100 may use only one or the other of storage arrays 156 or 158.

[0039] As described above, compute nodes 110 operate as a multi-node parallel processing computer system. In this manner, program instructions for a distributed software application can be executed by multiple processors 111 on one or more compute nodes 110. For example, the distributed software application can initiate and manage processes executed on multiple processors 111, which perform operations associated with the operation of the distributed software application via process thread system 117. Furthermore, the distributed software application can establish communication with storage node 150 to access files located on storage arrays 156 and 158 in order to write data to the files. The program instructions for this communication can be included as part of data file access system 118.

[0040] The data file access system 118 in the computing node 110 can be started based on instructions from the application and data portion 116 and / or the process thread system 117. Once started, one or more computing nodes execute program instructions to perform specific functions associated with requesting access to and writing data to files located on one or more storage nodes 150. For ease of understanding, these functions are described separately below, but it should be understood that in practice, these functions may overlap and are not necessarily mutually exclusive. Example operations related to these functions are described below, some of which are as follows: Figure 2-8 The instructions for the data file access system program include instructions that, when executed by the processor, cause the data file access system to perform the following operations, for example, including instructions corresponding to Figure 2-8 Instructions for the operations shown in .

[0041] In addition, although the following operations involving accessing files are mainly described as writing data to files, Figure 2-8 Some or all of the operations shown in can also be applied to reading from a file.

[0042] Figure 2 2 is a flow chart illustrating an example process 200 for managing access to electronic data files in a distributed computer system (e.g., system 100). In block 210, group access is requested to a file at a storage node, such as a storage node containing an SSD storage device (e.g., storage node 150p). While access refers to writing data to or reading data from a file, group access refers to access to a file by a group of processes. Two or more processes executed by processor 111 included in compute node 110 request group access. For example, a first process may be running on processor 111A on compute node 110A, and a second process may be running on processor 11B on compute node 110B. In some embodiments, one of the processes in the group may serve as the management process for the group, and the request for group access may be communicated to the management process from the remaining processes in the group. In other embodiments, a separate management process (i.e., not the management process of the process group) in one of processors 111 may receive the request for group access.

[0043] In block 220, two or more processes are instructed, for example by a management process, to access a file using a first access mode. The first access mode includes performing a group write lock while also disabling collective buffering, as described in more detail below with respect to blocks 230 and 240. In some examples, the first access mode is particularly used when accessing files on a storage node that includes an SSD memory device, such as the storage array 158 at storage node 150p. Specific instructions for the first access mode may be included in the local memory 115 of each compute node 110 as part of the data file access system 118.

[0044] As described below, in some examples, other access modes may be used to access files on the storage nodes. In such examples, as part of box 220, the management process may determine to use the first access mode or an additional access mode based on one or more criteria in the data file access system 118 stored in the local memory 115 of each computing node 110A-n. The criteria may include the type of storage medium being accessed, the size of the data being written or read to the storage node, or other criteria. In other words, in some examples, in response to determining that the type of storage medium being accessed is a defined type (e.g., a first performance level), the size of the data being written or read meets some criteria, or meets some other criteria, the first access mode may be used in box 220, while in other cases, a different access mode (e.g., the second access mode described below) may be used. For example, in Figure 1 In the context of , in some embodiments, whenever the accessed file is stored on one of the storage arrays 158, a first access mode is used, and when the accessed file is stored on one of the storage arrays 156, a different access mode is used (e.g., as described below with respect to Figure 3 In other examples, the first access mode may be used regardless of the type of storage media (including on both storage array 156 and storage array 158).

[0045] In block 230, specific instructions for a group write lock mechanism are executed by or for a group of two or more processes. In some examples, at least one of the processes in the group, or an external management process on behalf of the group, may execute instructions for the group write lock mechanism, causing the process executing the instructions to send a request to the file management system to disable its normal locking mechanism with respect to the process group and the files being accessed by the group. As described above, conventional locking mechanisms of a file management system may prevent multiple processes from accessing files simultaneously. In some examples, all processes in the group may individually send requests for a group write lock to the file management system. In other examples, a single process (e.g., a management process) may send a single request to the file management system on behalf of all processes in the group. Each process participating in the group write lock mechanism will be assigned a specific and unique memory allocation within the file being accessed. In some examples, the allocation of memory allocations may be performed by the management process. In other examples, each process may have a predetermined allocation of memory locations. For example, the processes can have a predetermined ordering, and each process can be configured to utilize a memory allocation defined according to its ordering (e.g., the first process takes a first memory location, the second process takes a second memory location at a predefined offset from the first memory location, the third process takes a third memory location at a predefined offset from the second memory location, and so on). The allocations can be determined in any manner based on any criteria (including random), as long as they do not overlap with each other. Allocating memory blocks to each process in conjunction with group write locking (such as implemented by the write locking mechanism described herein) ensures consistency in overwriting memory locations due to access by multiple processes. In other words, because memory locations are allocated between groups of processes that enforce group write locking in a non-overlapping manner, processes can access files concurrently without overwriting each other's data, which is why it is safe to disable the file system's normal file locking (which is intended to prevent such overwrites).

[0046] In block 240, specific instructions are executed by each of the two or more processes to disable the collective buffering mechanism. The instructions to disable the collective buffering mechanism may include instructions that allow each of the two or more processes to access a file for writing its own data to a corresponding memory block assigned to each process by the write lock mechanism in block 230. As described above, in previous systems, collective buffering was typically automatically used whenever a group write lock was engaged. Therefore, disabling the collective buffering mechanism in block 240, along with engaging the group write lock in block 230, changes the system's typical behavior when using a group write lock. As described above, in certain circumstances, collective buffering can cause performance bottlenecks due to the additional processing overhead required for collective buffering. Therefore, by disabling collective buffering in certain circumstances, performance can be improved.

[0047] In block 250, each of the two or more processes instructs the NIC 112 in the compute node 110 to access the file and write data to a designated memory location on the storage array 158 at the storage node 150p. The designated memory location is a memory location assigned according to the group write lock mechanism described above with respect to block 230. Thus, for example, a first process writes data to a first memory location, a second process writes data to a second memory location, and so on. Notably, due to the operations performed as part of the first access mode as described above, each process can communicate and write data during the same time period without the risk of writing data to overlapping memory locations.

[0048] Figure 3 An example method 300 is described for managing access to electronic data files in a distributed computer system that includes the ability to manage file access using multiple access modes when multiple types of storage devices are available at a storage node 150. In block 310, a group of processes running in a computing node 110 requests group access to a file contained on one of the storage nodes 150. The group of processes may include at least two processes running on different ones of the computing nodes 110. The request may be made using software instructions included in the data file access system 118 described above.

[0049] In block 320, the type of storage device used to store the file is determined, for example, by a management process. The determination may be made based on identification information of the storage device and based on a comparison of the identification information with a selection from a list associating various predetermined storage devices with types of storage devices. Alternatively, the determination may be made based on one or more performance characteristics of the storage devices used in the storage node containing the file, and the devices may be identified as different types based on their performance characteristics (e.g., faster devices may be identified as devices of a first type, while slower devices may be considered devices of a second type). In some examples, the type of storage device is selected from two predetermined types: a first type corresponding to a first performance level and a second type corresponding to a second performance level, wherein the first type is faster than the second performance type. For example, in some embodiments, the second type of storage devices includes disk-based storage devices, such as hard disk drives and optical disk drives, for storage devices in storage array 156, and the first type of storage devices includes SSD storage devices, such as solid-state memory drives and persistent memory drives, for storage devices in storage array 158. In other embodiments, a flash-based SSD may be part of the second type (and the slower disk-based storage devices), while the first type may include NVRAM or other persistent memory that is faster than a flash-based SSD. In some implementations, the type of memory device is defined by a numerical range of performance characteristics rather than by a product category.

[0050] In some examples, some aspects of the above-described determination in block 320 may be performed at different times, and some aspects may not necessarily be repeated for each request to access the file. For example, in some embodiments, operations related to collecting information from or about the memory devices (e.g., performing measurements, discovering identification information, etc.) and comparing that information to a list or range to determine the type of the memory device may be performed once (e.g., during startup, on the first set of accesses, or at some other convenient time), and thereafter, these operations are not repeated each time the file is accessed. Instead, in some examples, once the system has made such an identification, the system may maintain a record of the types of the various memory devices, and the determination in block 320 may then be made by reference to that record.

[0051] In block 330, if the type of the memory device is a first type of memory device, each of the set of processes accesses the file in a first access mode. For example, if the first type of memory device is an SSD memory device, the first access mode includes executing program instructions for accessing files on the SSD memory device. The program instructions may be similar to those described above in Figure 2Those instructions described in , that is, to implement a write locking mechanism and disable collective buffering for each of two or more processes.

[0052] In box 340, if the type of the memory device is a second type of memory device, each of the group of processes accesses the file in a second access mode. For example, if the second type of memory device is a disk-based memory device, the second access mode includes program instructions for accessing files on the disk-based memory device. The program instructions may include instructions for each of the two or more processes to execute a write lock mechanism similar to the instructions described above in box 230. The program instructions may also include instructions for each of the two or more processes to implement and enable a collective buffering mechanism. As previously described, the use of collective buffering includes additional processing operations and overhead in the computing node 110 to reduce the number of processes that must access the storage node 150 when writing data to the file. Therefore, collective buffering is more suitable for use with a type of memory device with a slower access speed (such as a disk-based memory device).

[0053] In some embodiments, the determination in block 320 may be performed in a separate process from the two or more process groups requesting access to the file, such as the separate management process described above or some other process that may be running on one of the compute nodes 110. In some examples, the process making the determination in block 320 also determines the access mode that the group of processes should use based on the determined type of memory device, and the process instructs the group of processes which mode to use. In other examples, the process making the determination in block 320 does not determine which access mode the group of processes should use, but rather simply communicates the determined type of memory device to the processor group. In such cases, the process group may independently determine the access mode to use based on the received information, e.g., the management process for the process group may determine which mode to use, or each process may make the determination independently (each process may include the same logic for making this determination, so that, despite making the determination independently, they may all reach the same conclusion).

[0054] In some other embodiments, the determination in block 320 may be performed by one of the two or more processes in the group of computing nodes requesting access to the file. For example, the management process of the group may make the determination in block 320. In some examples, the process making the determination in block 320 also determines the access mode that the group of processes should use based on the determined type of memory device, and the process instructs the group of processes which mode to use. In other examples, the process making the determination in block 320 does not determine which access mode the group of processes should use, but instead simply communicates the determined type of memory device to the other processes in the group. In this case, the group's processes may independently determine the access mode to use based on the received information (each process may include the same logic for making this determination, so that, despite making it individually, they can all reach the same conclusion). In other embodiments, the determination in block 320 may be performed by each process in the group of two or more processes in the computing node. In other words, each process determines for itself the type of memory device being accessed and which of the first access mode and the second access mode to use based on that determination (in this embodiment, each process is programmed with the same logic for making these determinations, so that despite doing so independently, all processes will reach the same conclusion).

[0055] Thus, method 300 allows for two different access modes to be used based on the type of storage medium being accessed, a first access mode to be used when the storage medium is of a first type and a second access mode to be used when the storage medium is of a second type.

[0056] Now go to Figure 4 , additional method 400 will be described. In some embodiments, for implementing Figure 4 The program instructions for the operations described as part of process 400 in block 330 may be used as part of accessing a file in the first access mode described above in block 330. For example, method 400 may be used to cause a compute node (e.g., all or a subset of compute nodes 110) comprising a group of processes requesting group access to a file to access the file using the first access mode. Method 400 may include executing various instructions (or causing execution of such instructions) by one or more compute nodes 110. As described above with respect to method 300, in some cases, a single process may execute the instructions, such as a management process (which may be part of the group of processes performing the group write or may be separate therefrom), while in other cases, all of the instructions may be executed by multiple processes (e.g., each process in the group performing the group write) in a parallel and / or distributed manner.

[0057] In block 410, instructions are executed to initiate a first access mode. For example, the instructions may include initiating a process in the compute node that includes instructions for the first access mode in response to identifying the type of the memory device as the type of the first memory device.

[0058] In block 420, instructions are executed to implement a write lock mechanism on each process in the process group. For example, instructions may include allocating a separate memory block to each process and preventing each process from writing data to the memory device outside of its allocated memory block. Block 420 may be executed in response to the initiation of the first access mode in block 410. In other words, in some examples, the first access mode includes block 420.

[0059] In block 430, instructions are executed to disable the use of the collective buffering mechanism for each process in the process group. For example, instructions may include allowing each process in the process group to write data to a corresponding memory block allocated by the write lock mechanism in block 420. In some embodiments, instructions may include causing the file system in compute node 110 to view each process in the process group as if it were an aggregator that is part of the collective buffering mechanism, even if the collective negotiation mechanism is disabled and each process is configured to access files directly. Block 430 may be executed in response to the initiation of the first access mode in block 410. In other words, in some examples, the first access mode includes block 430.

[0060] In some examples, when the target storage medium for group writes is a relatively fast storage medium (such as an SSD), the first access mode is used, and thus the collective buffering mechanism that is part of the first access mode is disabled. As described above, for fast storage media, the time required to access the media becomes low enough that the processing overhead associated with collective buffering is now a more significant bottleneck than accessing the storage medium, and thus collective buffering now becomes a net negative in terms of overall system performance. Therefore, disabling collective buffering in these situations may result in faster system performance.

[0061] In block 440 , instructions are executed in a manner similar to that described above in block 250 such that each process in the process group writes data from the process to the file with the write lock mechanism active and collective buffering disabled.

[0062] In some embodiments, for implementing Figure 5 Program instructions for the operations described as part of method 500 may be used as part of accessing a file using the second access mode described above in block 340. In block 510, instructions are executed to initiate the first access mode. For example, the instructions may include identifying the type of the memory device as the type of the first memory device in order to initiate a process in the compute node that includes instructions for the second access mode.

[0063] In block 520, instructions are executed to implement a write lock mechanism on each process in the process group. For example, as described in block 420, instructions may be included for allocating a separate memory block and preventing data from being written outside of the allocated memory block. Block 520 may be executed in response to the initiation of the second access mode in block 510. In other words, in some examples, the second access mode includes block 520.

[0064] In block 530, instructions are executed to enable use of the collective buffering mechanism on each process in the process group by selecting a subset of processes to be used to collect data before accessing the file, corresponding to each process in the subset selected as an aggregator, as described above. Block 530 may be executed in response to initiation of the second access mode in block 510. In other words, in some examples, the second access mode includes block 530.

[0065] In some examples, when the target storage medium is a relatively slow storage medium (such as a disk-based device), the second access mode is used, thereby using a collective buffering mechanism. As described above, collective buffering works relatively well with such devices and can help alleviate bottlenecks that may occur when multiple processes attempt to concurrently access such relatively slow media. Specifically, in this case, collective buffering results in a time saving in accessing (e.g., writing to or reading from) the media, and this time saving may outweigh the increased processing overhead caused by collective buffering.

[0066] In block 540, instructions are executed to cause one or more of the selected processes to request and collect data to be written to the file from each process in the process group. In some embodiments, each of the one or more selected processes may request and collect data from a specific process in the process group. In some embodiments, each of the one or more selected processes may request and collect data from all processes in the process group.

[0067] In block 550, instructions are executed such that one or more of the selected processes are started in Figure 2In some embodiments, the data (including the collected data) from the selected processes can be written to the file in the manner described above. In some embodiments, one or more selected processes can receive information about the allocated memory blocks from each process in order to use these allocated memory blocks to write the data (including the collected data) to the file. For example, if the selected process has a memory block allocated from 00ffh to 0ffeh, and data is collected from two processes in a process group that have memory blocks allocated from 0fffh to 1000h and 1001h to 1fffh, respectively, the selected process can write all dates to the file between memory locations 00ffh and 1fffh. In some embodiments, as part of a collective buffering mechanism, one or more selected processes can be allocated separate memory blocks for the collected data, the separate memory blocks being large enough to store all the data.

[0068] In some embodiments, for implementing Figure 6 The program instructions for the operations described as part of method 600 in block 610 may be used as part of determining the type of memory device described in block 320 above. In block 610, the memory device to be accessed by the process requesting access to the file is identified. For example, identification information of the memory device, such as a product name, SKU number, device type or category identifier, or other identification information, may be read from the memory device or some other component (such as a baseboard management controller or system controller). The identification of the memory device may be communicated from one of the processes requesting access to the other processes in the compute node 111. Alternatively, the identification of the memory device may be provided to the compute node 111 by the storage node 150 where the file is located.

[0069] In block 620, the identification information for the memory device is compared to a list that associates the identification information of the memory device with a memory device type. The list of memory devices and types may be pre-programmed and / or user-configurable. For example, in some embodiments, the list may be created by a manufacturer or developer, specifying various known memory devices and categorizing them as a first type or a second type. In some examples where the list is user-configurable, the user may specify additional memory devices or change the categorization of existing devices in the list. In examples where the list is user-configurable, the list may be provided to the user or programmer as part of a user interface mechanism as part of the setup of a distributed software application in computer system 100. Alternatively, in examples where the list is user-configurable, the list of devices and types may be stored in local memory 115 and provided to the user or programmer as part of the operation of the distributed software application in computer system 100. In other examples, the list is not provided to the user and is not user-configurable. In some examples, the list may include identification information for memory devices of the first and second types, as well as an indication of the corresponding type. In other examples, the list may include identification information for memory devices of only one type (e.g., a first type), and it may be implicitly determined that the memory device is of that type if found in the list and of another type if not found in the list.

[0070] In block 630, a determination is made as to whether the memory device selected in block 620 is a memory device of the first type based on the comparison in block 620. If the memory device selected in block 630 is a memory device of the first type, then in block 640, a control command is provided to any process requesting access to a file on the selected memory device to use the first access mode, such as Figure 3 If the memory device selected in block 630 is not a memory device of the first type, then in block 650, a control command is provided to any process requesting access to a file on the selected memory device to use a second access mode, such as Figure 3 as described in block 340 of .

[0071] In some embodiments, for implementing Figure 7 The program instructions for the operations described as part of method 700 in block 710 may be used as part of determining the type of memory device described in block 320 above. In block 710, a memory device to be accessed by a process requesting access to a file is identified. The identification of the memory device may be communicated from one of the processes requesting access to the other processes in compute node 111. Alternatively, the identification of the memory device may be provided to compute node 111 by storage node 150 where the file is located.

[0072] In block 720, a value of a performance characteristic associated with the identified memory device is determined. In some examples, determining the performance characteristic may include system 100 performing a characteristic measurement. In some implementations, the value of the performance characteristic may be determined as part of a self-test process performed by storage node 150, or as a result of a request from compute node 110 to storage node 150 to run a test of the performance characteristic and provide a value from the test. An example of a performance characteristic is access speed, where the performance characteristic is access time. In other examples, the storage node may self-report the performance characteristic to the compute node.

[0073] In block 730, a determination is made as to whether the value of the performance characteristic indicates that the identified memory device is a first type of memory device. This determination may be made by comparing the value determined for the performance characteristic of the identified memory device with the value or range of values ​​for that performance characteristic for each of the different types of available memory devices. For example, in some embodiments, for random writes, the range of values ​​for a first type of memory device (such as an SSD memory device) is 4000 MB / s or greater, and for random writes, the range of values ​​for a second type of memory device (such as a disk-based memory device) is less than 270 MB / s. In such an example, if it is assumed that the value of the access speed of the identified memory device is measured to be 15000 MB / s, the value of the access speed from the identified memory device would indicate that it is a first type of memory device. Other performance characteristics and / or ranges may be used to define the first type and the second type.

[0074] In block 730, if the value of the performance characteristic indicates that the identified memory device is a first type of memory device, then in block 740, a control command is provided to any process requesting access to a file on the selected memory device to use a first access mode, such as Figure 3 If, in block 730, the value of the performance characteristic indicates that the identified memory device is not a first type of memory device, then, in block 750, a control command is provided to any process requesting access to a file on the selected memory device to use a second access mode, such as Figure 3 as described in block 340 of .

[0075] Figure 8 An example method 800 for managing access to electronic data files in a distributed computer system that includes the ability to manage file access when multiple types of memory devices are available at a storage node 150 is described.

[0076] In block 810, a process for managing access to files on storage nodes 150 identifies a group of processes running on multiple nodes that need to perform a group write to the file. The process for managing access can be one of the processes in the group or a separate process from the process group.

[0077] In block 820, the process for managing access determines whether collective buffering has been disabled for the group of processes. As previously described, collective buffering can be enabled or disabled based on the type of storage device being used by one or more storage nodes 150. If collective buffering has been disabled, then in block 830, the process for managing access causes each process in the group of processes to write its own data to the file. In some embodiments, the process for managing access can cause each process in the group of processes to act as an aggregator, without collecting data from any other process, as described above.

[0078] In block 820, if collective buffering has not been disabled, the process for managing access causes a subset of the processes to act as aggregators in block 840. The process for managing access further causes each process that is not an aggregator to forward its own data to one of the aggregators in block 850. In block 860, the process for managing access additionally causes each aggregator to instruct its corresponding NIC 112 in the compute node 110 to write its own data and data received from other processes to the file.

[0079] Figure 9 Illustrated are example processor-executable instructions stored on a non-transitory machine-readable medium 9000. Specifically, data file access management instructions 910 associated with a data file access system 118 in a compute node 110 are stored on the medium 9000.

[0080] The data file access management instructions 910 may include instructions that, when executed, instantiate the data file access system 118 in the computing node 110. Specifically, the data file access management instructions 910 may include instructions for performing any or all of the operations described above as being performed by the computing node 110, including, for example, Figure 2-8 ) instructions for any example operations shown in .

[0081] For example, the data file access management instruction 910 may include a data file access request instruction 920, a memory device type determination instruction 930, a first memory device type instruction 940, and a second memory device type instruction 950, so as to execute Figure 3 and Figure 8 One or both of the operations. The memory device type determination instruction 930 may include executing Figure 6 and Figure 7The first memory device type file access instruction 940 may include executing one or both of the operations in the instruction. Figure 2 and Figure 4 The second memory device type file access instruction 950 may include executing one or both of the operations in the instruction. Figure 5 Instructions for operations in .

[0082] For example, the memory device type determination instructions 930 may include instructions for identifying a memory device accessed by a process requesting access to a file, comparing the identified memory device to a list of memory devices and types, determining whether the selected memory device is a memory device of a first type, and determining whether the selected memory device is a memory device of a first type based on the memory device type. Figure 6 The memory device type determination instructions 930 may include instructions for identifying a memory device accessed by a process requesting access to a file, determining a value of a performance characteristic of the identified memory device, determining whether the value of the performance characteristic indicates that the memory device is a first type of memory device, and determining whether the memory device is a first type of memory device depending on the memory device type. Figure 7 An instruction for determining whether to use the first access mode or the second access mode as described in.

[0083] For example, the first memory device type file access instructions 940 may include implementing write locks and disabling collective buffering on each process and Figure 4 Instructions for writing data from each process to a file as described in .

[0084] For example, the second memory device type file access instructions 950 may include instructions to implement a write lock on each process and enable collective buffering, such as Figure 5 The second memory device type file access instruction 950 may also include an instruction to select a subset of processes to act as an aggregator, which collects data from the processes and writes the collected data to a file, such as Figure 5 and Figure 8 As stated.

[0085] Figure 10 An example computing device 110 is shown that is configured to read instructions stored on a non-transitory machine-readable medium 9000. The non-transitory machine-readable medium 9000 may be included as part of the local memory 115. Specifically, one or more of the processors 111 in the computing device 110 are configured to read and execute some or all of the data file access management instructions 910 stored on the non-transitory machine-readable medium 9000 in order to perform any or all of the operations described above, including, for example, Figure 2-8 Any of the example operations shown in .

[0086] For example, the processor 111 may be configured to read and execute the data file access request instruction 920, the memory device type determination instruction 930, the first memory device type instruction 940, and the second memory device type instruction 950 to perform Figure 3 and Figure 8 Processor 111 may also be configured to read and execute some or all of the determination instructions 930 to perform Figure 6 and Figure 7 The processor 111 may also be configured to read and execute some or all of the first memory device type file access instructions 940 to perform Figure 2 and Figure 4 The processor 111 may also be configured to read and execute some or all of the second memory device type file access instructions 950 to perform Figure 5 Operations in .

[0087] In the above description, various types of electronic circuits are described. As used herein, "electronic" is intended to be broadly understood to include all types of circuits utilizing electricity, including digital and analog circuits, direct current (DC) and alternating current (AC) circuits, as well as circuits for converting electricity into another form of energy and circuits for using electricity to perform other functions. In other words, as used herein, there is no difference between "electronic" circuits and "electrical" circuits. In some cases, some electronic circuits may include processing circuits. A processor or processing circuit includes a circuit configured with logic for performing various operations. The logic of the processing circuit may include dedicated hardware for performing various operations, software (machine-readable and / or processor-executable instructions) for performing various operations, or any combination thereof. In an example where the logic includes software, the processing circuit may include a processor for executing software instructions and a memory device for storing software. The processor may include one or more processing devices capable of executing machine-readable instructions, such as a processor, a processor core, a central processing unit (CPU), a controller, a microcontroller, a system on a chip (SoC), a digital signal processor (DSP), a graphics processing unit (GPU), etc. Where the processing circuitry includes dedicated hardware, in addition to or in lieu of a processor, the dedicated hardware may include any electronic device configured to perform specific operations, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), discrete logic circuits, hardware accelerators, hardware encoders, etc. The processing circuitry may also include any combination of dedicated hardware and a processor plus software.

[0088] It should be understood that both the general description and the detailed description provide examples that are illustrative in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. Various mechanical, component, structural, electronic, and operational changes may be made without departing from the scope of this specification and the claims. In some instances, well-known circuits, structures, and techniques have not been shown or described in detail in order not to confuse the examples. Like numbers in two or more numbered sections represent identical or similar elements.

[0089] In addition, unless the context indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms as well. In addition, the terms "comprises", "includes", "includes" and the like specify the presence of stated features, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components and / or groups. Components described as coupled may be directly coupled, either electronically or mechanically, or they may be indirectly coupled via one or more intermediate components, unless specifically stated otherwise. Unless the context of the description indicates otherwise, mathematical and geometric terms are not necessarily intended to be used in accordance with their strict definitions, as those skilled in the art will understand that, for example, substantially similar elements that function in a substantially similar manner can readily fall within the scope of a descriptive term even if that term also has a strict definition.

[0090] Providing: As used herein, "providing" an item means taking possession and / or control of the item. This may include, for example, forming (or assembling) part or all of the item from its constituent materials, and / or obtaining possession and / or control of an already formed item.

[0091] And / or: This document occasionally uses the phrase "and / or" with a list of items. This phrase is meant to include any combination of items from a single item to all of the items in the list, and any permutation in between. Thus, for example, "A, B, and / or C" means "one of {A}, {B}, {C}, {A, B}, {A, C}, {C, B}, and {A, C, B}."

[0092] Whenever feasible, elements and related aspects described in detail with reference to one example may be included in other examples where they are not specifically shown or described. For example, if an element is described in detail with reference to one example but not with reference to a second example, the element may still be claimed to be included in the second example.

[0093] Unless otherwise stated herein or implied by the context, when approximate terms such as "substantially," "approximately," "about," "approximately," and the like are used, this should be understood to mean that mathematical precision is not required, but rather that a range of variation is included but not strictly limited to the stated value, property, or relationship. In particular, the range of variation implied by the use of such approximate terms includes at least any insignificant variations, as well as those variations that are typical in the relevant art for the type of item in question due to manufacturing or other tolerances, in addition to any ranges expressly stated herein (if any). In any case, unless otherwise stated, a range of variation may include values ​​that are at least within ±1% of the stated value, property, or relationship.

[0094] In view of the disclosure herein, further modifications and alternative examples will be apparent to those skilled in the art. For example, the apparatus and method may include additional components or steps omitted from the diagrams and descriptions to operate clearly. Therefore, this description is to be interpreted as illustrative only and is used to teach those skilled in the art the general manner of performing this teaching. It should be understood that the various examples shown and described herein are to be regarded as exemplary. Elements and materials, as well as the arrangement of these elements and materials, may replace those shown and described herein, parts and processes may be reversed, and certain features of the present invention may be used independently, all of which will be apparent to those skilled in the art after having obtained the benefit of the description herein. Changes may be made to the elements described herein without departing from the teachings of the present invention and the scope of the following claims.

[0095] It is to be understood that the particular examples described herein are non-limiting and that modifications in structure, dimensions, materials, and methods may be made without departing from the scope of the present teachings.

[0096] Other examples according to the invention will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. The specification and examples are to be considered exemplary only, and the following claims are to be entitled to their fullest breadth, including equivalents, under applicable law.

Claims

1. A distributed computer system comprising: at least one storage node coupled to the network, the storage node being configured in a parallel distributed file system and comprising at least one type of memory device; a plurality of computing nodes coupled to the network, each of the plurality of computing nodes comprising at least one processor, the plurality of computing nodes configured to request group access to a file in the parallel distributed file system, the request being issued by at least two processes, each of the at least two processes running on a different computing node among the plurality of computing nodes in the network; Wherein, at least one computing node among the multiple computing nodes is configured as: determining the type of storage device comprising the file; If it is determined that the memory device is of the first type, causing the plurality of computing nodes to collectively access the file in a first access mode, the first access mode including applying a write locking mechanism to each of a different one of the computing nodes and disabling a collective buffering mechanism; and If it is determined that the memory device is of the second type, the file is accessed in a second access mode that includes applying the write locking mechanism to each of the different ones of the computing nodes and enabling a collective buffering mechanism.

2. The system according to claim 1, wherein: Determining the type of memory device further includes selecting the type of memory device from a list of types of memory devices.

3. The system according to claim 1, wherein: Determining the type of memory device also includes determining performance characteristics of the memory device.

4. The system according to claim 3, wherein: The performance characteristic is access speed.

5. The system according to claim 1, wherein: The type of memory device includes at least one of a solid-state memory device and a hard disk drive.

6. The system according to claim 1, wherein: Enabling the collective buffering mechanism also includes selecting a subset of the at least two processes, the subset including at least one process as an aggregator, the aggregator collecting data from at least one other process of the at least two processes and writing the data to the file, and wherein disabling the collective buffering mechanism also includes each process writing its own data to the file.

7. The system according to claim 1, wherein: Disabling the collective buffering includes causing the file system to view each process as if it were an aggregator, except that the process does not collect data from any other process.

8. The system according to claim 1, wherein: The at least two processes are part of a software application that runs on the plurality of computing nodes to perform tasks.

9. The system according to claim 1, wherein: The first access mode and the second access mode are included in a message passing interface MPICH on chameleon software.

10. The system according to claim 1, wherein: The at least one computing node that determines the type of the memory device includes a computing node among the multiple computing nodes that determines the type of the memory device and indicates which of the first access mode and the second access mode to use for other computing nodes based on the determined type of the memory device.

11. The system according to claim 1, wherein: The at least one computing node that determines the type of memory device includes each computing node of the plurality of computing nodes that determines the type of memory device and determines which of the first access mode and the second access mode to use based on the determined type of memory device.

12. A non-transitory machine-readable medium storing an operational image, the operational image comprising instruction file access management instructions that, when executed by at least one processor in a computing node, cause the node to instantiate a data file access system to: In response to requests issued by at least two processes for group access to a file in a parallel distributed file system: determining a type of storage device including the file; If it is determined that the memory device is of the first type, causing the plurality of computing nodes to collectively access the file in a first access mode, the first access mode comprising applying a write locking mechanism to each of a different one of the computing nodes and disabling a collective buffering mechanism; as well as If it is determined that the memory device is of the second type, the file is accessed in a second access mode that includes applying the write locking mechanism to each of the different ones of the computing nodes and enabling a collective buffering mechanism.

13. The non-transitory machine-readable medium of claim 12, wherein: Determining the type of memory device further includes selecting the type of memory device from a list of types of memory devices.

14. The non-transitory machine-readable medium of claim 12, wherein: Determining the type of memory device also includes determining performance characteristics of the memory device.

15. The non-transitory machine-readable medium of claim 12, wherein: Enabling the collective buffering mechanism also includes selecting a subset of the at least two processes, the subset including at least one process as an aggregator, the aggregator collecting data from at least one other process of the at least two processes and writing the data to the file, and wherein disabling the collective buffering mechanism also includes each process writing its own data to the file.

16. The non-transitory machine-readable medium of claim 12, wherein: Disabling the collective buffering includes causing the file system to view each process as if it were an aggregator, except that the process does not collect data from any other process.

17. A method for a distributed computer system, comprising: requesting group access to a file located on a first type of storage device in a parallel distributed file system, the request being issued by at least two processes, each of the at least two processes running on a different compute node among a plurality of compute nodes in a network; as well as The file is accessed in a first access mode associated with a memory device of the first type, the first access mode including applying a write locking mechanism to each of the different ones of the compute nodes and disabling a collective buffering mechanism.

18. The method according to claim 17, wherein Disabling the collective buffering includes causing the file system in each of the different ones of the compute nodes to view each process as if it were an aggregator, except that the process does not collect data from any other process.

19. The method according to claim 17, wherein The first type of memory device is a solid-state memory device.

20. The method according to claim 17, wherein The at least two processes are part of a software application that runs on the plurality of computing nodes to perform tasks.

Images