Computer, media, and method for client memory allocation across storage servers

Abstract

The present disclosure relates to client memory allocation across storage servers. In some examples, a computer identifies a plurality of storage servers accessible by the computer to perform remote access over a network to data stored by the plurality of storage servers, sends an allocation request to allocate a memory segment to place interleaved data of the computer across the plurality of storage servers, and receives, at the computer in response to the allocation request, metadata related to the memory segment at the plurality of storage servers, the metadata including addresses of the memory segment at the plurality of storage servers. The computer accesses, using the metadata, the interleaved data at the plurality of storage servers, the interleaved data including data blocks distributed across the memory segment.

Description

Technical Field

[0001] This disclosure relates to computers, media, and methods for client memory allocation across storage service servers. Background Technology

[0002] Fabric-attached memory refers to memory that can be accessed via a fabric by any of multiple clients. "Fabric" can also refer to the network that allows communication between compute nodes connected to the network. In some examples, fabric-attached memory is implemented using memory devices such as flash memory or other types of persistent memory. Summary of the Invention

[0003] A computer includes: a processor; and a non-transitory storage medium storing client instructions executable on the processor to perform the following operations: implementing a client using the client instructions; identifying a plurality of storage servers accessible by the computer to perform remote access over a network to data stored on the plurality of storage servers; generating an allocation request from the client to allocate memory segments to place interleaved data of the computer across the plurality of storage servers; sending the allocation request to the plurality of storage servers; receiving, in response to the allocation request, metadata at the computer associated with the memory segments at the plurality of storage servers, the metadata including addresses of the memory segments at the plurality of storage servers; and accessing the interleaved data at the plurality of storage servers using the metadata, the interleaved data comprising data blocks distributed across the memory segments. Attached Figure Description

[0004] Some embodiments of this disclosure are described with reference to the following figures.

[0005] Figure 1 This is a block diagram based on some example arrangements, which include a structure-attached memory and a client system, the structure-attached memory including a storage server coupled to a network, and the client system being able to access the structure-attached memory over the network.

[0006] Figure 2 and Figure 3 It is a block diagram based on some examples of interwoven data across storage servers.

[0007] Figure 4 It is a message flow diagram based on some example client systems and storage server processes.

[0008] Figure 5 It is a block diagram of descriptors including metadata, based on some examples.

[0009] Figure 6 This is a flowchart illustrating the process of writing data of a specified size into a structured attached memory, based on some example client systems.

[0010] Figure 7 It is a block diagram based on some examples, including an input / output (I / O) array for storing I / O vectors for I / O requests.

[0011] Figure 8 It is a block diagram of a computer based on some examples.

[0012] Figure 9 It is a block diagram of a storage medium with machine-readable instructions based on some examples of storage.

[0013] Figure 10 It is a flowchart based on some examples.

[0014] Throughout the accompanying drawings, the same reference numerals denote similar but not necessarily identical elements. The drawings are not necessarily drawn to scale, and the dimensions of some parts may be enlarged to illustrate the examples more clearly. Furthermore, the drawings provide examples and / or embodiments consistent with the description; however, the description is not limited to the examples and / or embodiments provided in the drawings. Detailed Implementation

[0015] A client can perform remote access (remote read or remote write) to a structure-attached memory. A “client” can refer to any entity (machine or program) capable of issuing a request to access data in the structure-attached memory. Examples of networks through which a client can access structure-attached memory include any one or a combination of the following: Compute Fast Link (CXL) interconnects, Slingshot interconnects, Infinite Bandwidth interconnects, etc.

[0016] Structured attached storage can include a distributed arrangement of storage servers, where each storage server can include persistent storage or be coupled to persistent storage. In some examples, clients can access structured attached storage over a network using Remote Direct Memory Access (RDMA).

[0017] RDMA data transfer from client to storage server involves data transfer from the client over a network to the persistent storage of the storage server, where the data transfer does not involve work performed by the storage server's main processor. The "main processor" of the storage server can refer to the processor that executes the storage server's operating system (OS) and other machine-readable instructions, including firmware and applications such as the Basic Input / Output System or BIOS.

[0018] In RDMA data transfer, data is transmitted from the persistent storage of the storage server to the client via the storage server's network interface, or vice versa. A "network interface" can refer to network communication components that send and receive signals, messages, packets, etc. (more generally, "information") over a network. A network interface can include network transceivers for sending and receiving information.

[0019] A storage server may include main memory, which comprises a collection of memory devices (a single memory device or multiple memory devices), such as dynamic random access memory (DRAM) devices, static random access memory (SRAM) devices, etc. The storage server may also include cache memory, which may be part of or associated with the main processor. If the main processor uses cache memory to store data used by the main processor, then the cache memory is associated with the main processor.

[0020] Note that because RDMA data transfers occur through the network interface of the storage server, the data can be stored in the network interface's memory and processed by the network interface's processor. However, the network interface's processor and memory are separate and distinct from the main processor and its cache memory, which is part of or associated with the main processor.

[0021] In some cases, because RDMA data transfers do not involve the main processor of the storage servers, the placement and distribution of data across a collection of storage servers is not managed by the storage servers themselves. Note that while a storage server can control where data is placed in its main memory, it does not control which data goes to which storage server within the collection. In some situations, data placement across storage servers can be skewed, where a given client load may place significantly more data on the first storage server than on the other (multiple) storage servers. Skewed data placement can cause a bottleneck on the first storage server, resulting in data operations taking longer to complete on the first storage server than on the other (multiple) storage servers.

[0022] Furthermore, using RDMA eliminates the need for a centralized controller to manage data placement across storage servers, as clients write data directly to the persistent storage of the storage server. Conversely, using a centralized controller increases the complexity and cost associated with using structured attached storage.

[0023] According to some embodiments of this disclosure, clients capable of remotely accessing a storage service server over a network can control the placement of data across storage segments of the storage service server without relying on the coordination of a central controller to manage data placement. Data for a given data item can be distributed across storage servers. A "data item" can refer to a file, object, collection of files or objects, or any other identifiable unit of data.

[0024] Data distribution across storage servers can include interleaved data. Data interleaving refers to distributing portions of data across storage servers in a specified order or other arrangement. For example, data can be interleaved by sequentially placing data portions across storage servers. As an example, in an arrangement with four storage servers (storage servers 0 to 3), interleaving places data portion 0 at storage server 0, data portion 1 at storage server 1, data portion 2 at storage server 2, data portion 3 at storage server 3, data portion 4 at storage server 0, data portion 5 at storage server 2, data portion 6 at storage server 3, and so on. This interleaving arrangement can be called a polling interleaving arrangement, where sequential data portions are placed in the storage servers according to the order of the storage servers (starting from the first storage server and proceeding to the last storage server in the sequence), and when the last storage server in the sequence is reached, the next data portion is placed in the first storage server in the sequence, and this process continues until the end of these data portions is reached. Placing a data portion on a given storage server means writing the data portion into the main memory of the storage server.

[0025] Figure 1 This is a block diagram of an example arrangement of a structured attached memory 102, which includes a distributed arrangement of storage servers 104 connected to a network 106. Client systems 108-1 to 108-N (N ≥ 1) are also connected to the network 106. Each client system has access to the structured attached memory 102.

[0026] Figure 1The arrangement of components in client system 108-1 is shown. The remaining client systems(s) (including 108-N) may have the same or different component arrangements. Client system 108-1 includes a network interface 110 for communication via network 106 and a processor 112 for executing machine-readable instructions (including firmware, operating system (OS), applications, etc.) of client system 108-1. Client system 108-1 also includes local memory 114 for storing local data within client system 108-1. A “processor” may include a microprocessor, the core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or other hardware processing circuitry.

[0027] Examples of client systems include any one or a combination of the following: desktop computers, laptop computers, tablet computers, server computers, smartphones, gaming devices, Internet of Things (IoT) devices, etc.

[0028] A client system is an example of a "client" that can issue a request to perform remote access to the structure-attached memory 102. A program (including machine-readable instructions) executed in client system 108 can be another example of a "client".

[0029] According to some embodiments of this disclosure, the client system 108-1 includes a remote memory data placement management engine 116. As used herein, "engine" may refer to one or more hardware processing circuits, which may include any one or a combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or another hardware processing circuit. Alternatively, "engine" may refer to a combination of one or more hardware processing circuits and machine-readable instructions (software and / or firmware) executable on the one or more hardware processing circuits.

[0030] The remote storage data placement management engine 116 can manage the allocation of storage regions across storage servers 104 for placing data. Additionally, the remote storage data placement management engine 116 can manage data placement across storage servers 104, such as by placing interleaved data in the allocated storage regions across storage servers 104.

[0031] Each storage server 104 includes a main processor 118 for executing the OS 120 of the storage server 104 and other machine-readable instructions, such as firmware (e.g., BIOS code), applications, etc. Note that each storage server 104 may include multiple processors of different types (including the main processor 118) for performing their respective functions. A "main processor" may include a single processor or multiple processors.

[0032] Each storage server 104 also includes persistent memory 122 (which is the main memory of storage server 104) capable of storing data representing client systems 108-1 to 108-N. "Persistent" memory refers to memory that retains data stored in it even in the event of a power outage. For example, persistent memory may include non-volatile memory, wherein the stored data is retained even in the event of a power outage (i.e., it is not lost). In another example, persistent memory may refer to an arrangement in which, in the event of a power loss, the data in the memory is erased and dumped to a backup storage device (e.g., a disk-based storage device, etc.), and then the data is restored from the backup storage device to the memory after power is restored.

[0033] Storage server 104 also includes a network interface 124 that enables storage server 104 to communicate via network 106.

[0034] In an example where a client system, such as client system 108-1, can perform RDMA access to the structure-attached memory 102, the RDMA data transfer will cause data to be transferred from the client system via network 106 to a selected storage server 104 of the structure-attached memory 102. The RDMA data transfer will be performed via the network interface 124 of each of the selected storage servers 104 and written to the persistent memory 122 of each of the selected storage servers 104, without involving the main processor 118 of each of the selected storage servers 104. During the RDMA data transfer, no processing cycles of the main processor 118 of the storage server 104 are used to place the data into the persistent memory 122.

[0035] In some examples of this disclosure, the remote storage data placement management engine 116 can identify multiple storage servers 104 accessible to the client system 108-1 to perform remote access to data stored by the multiple storage servers 104 via network 106. The remote storage data placement management engine 116 sends allocation requests to allocate memory regions including memory segments (e.g., which may contain data stripes) to place the interleaved data of the client system 108-1 across the multiple storage servers 104.

[0036] The remote storage data placement management engine 116 receives metadata related to a storage region comprising storage segments at multiple storage servers 104 in response to an allocation request. The metadata includes the addresses of the storage segments at the multiple storage servers 104.

[0037] The program in client system 108-1 uses metadata to access interleaved data at multiple storage servers 104. In some examples, the interleaved data includes data blocks distributed across memory segments.

[0038] Figure 2 This is a block diagram illustrating an example of allocating a memory region (“Memory Region A”) to place interleaved data across multiple storage servers 104 (in this example, storage servers 1, 2, 3, and 4). A “Memory Region” refers to the amount of allocated memory allocated to clients for storing data in structure-attached memory 102.

[0039] exist Figure 2 In the example, two data items D1 and D2 are stored in memory region A and distributed across storage servers 1, 2, 3, and 4. Each part of a data item in a storage server is called a stripe. In this example, data item D1 is divided into four stripes, which are placed in the corresponding storage servers 1, 2, 3, and 4. Each stripe includes a certain number of blocks of data item D1. Data item D1 includes blocks B1, B2, B3, B4, B5, B6, B7, B8, B9, etc., where B1 is the starting block of data item D1. Stripe 1 of data item D1 (placed in storage server 1) includes blocks B1, B5, B9, etc.; stripe 2 of data item D1 (placed in storage server 2) includes blocks B2, B6, B10, etc.; stripe 3 of data item D1 (placed in storage server 3) includes blocks B3, B7, B11, etc.; and stripe 4 of data item D1 (placed in storage server 4) includes blocks B4, B8, B12, etc.

[0040] Similarly, data item D2 includes blocks B1, B2, B3, B4, B5, B6, B7, B8, B9, etc. Stripe 1 of data item D2 (placed in storage server 3) includes blocks B1, B5, B9, etc.; stripe 2 of data item D2 (placed in storage server 4) includes blocks B2, B6, B10, etc.; stripe 3 of data item D2 (placed in storage server 1) includes blocks B3, B7, B11, etc.; and stripe 4 of data item D2 (placed in storage server 2) includes blocks B4, B8, B12, etc.

[0041] Note that the starting block B1 of data item D1 is placed in storage server 1, while the starting block B1 of data item D2 is placed in a different storage server 3.

[0042] Placing the starting blocks of different data items on different storage servers allows for more distributed access to data on storage servers 1, 2, 3, and 4, reducing the likelihood of any storage server becoming a bottleneck when blocks of data items D1 and D2 are accessed. For example, a client might request access to both data items D1 and D2. By placing the starting blocks of data items D1 and D2 on different storage servers (1 and 3 respectively), access to the data items can be performed in parallel; for example, a client can access block B1 of data item D1 on storage server 1 while simultaneously accessing block B1 of data item D2 on storage server 3, preventing the client from simultaneously attempting to access blocks B1 of D1 and D2 from the same storage server (such as storage server 1).

[0043] The interleaving of data item D1 across storage servers 1, 2, 3, and 4 is accomplished by writing blocks of data item D1 in the following sequence: B1 to storage server 1, B2 to storage server 2, B3 to storage server 3, B4 to storage server 4, B5 to storage server 1, B6 to storage server 2, B7 to storage server 3, B8 to storage server 4, and so on, i.e., writing blocks across storage servers 1, 2, 3, and 4 in a round-robin fashion.

[0044] The interleaving of data item D2 across storage servers 1, 2, 3, and 4 is accomplished by writing blocks of data item D2 in the following sequence: B1 to storage server 3, B2 to storage server 4, B3 to storage server 1, B4 to storage server 2, B5 to storage server 3, B6 to storage server 4, B7 to storage server 1, B8 to storage server 2, and so on, i.e., writing blocks across storage servers 1, 2, 3, and 4 in a round-robin fashion.

[0045] Figure 3 This is a block diagram illustrating an example of the addresses associated with blocks containing data items D1 and D2 in storage servers 1, 2, 3, and 4. Figure 3 In the example, it is assumed that the interleaving size (or equivalent, block size) for each data item is 4096 bytes. In other examples, the interleaving size may be different.

[0046] Storage server 1 stores blocks B1, B5, etc., of stripe 1 for data item D1, and blocks B3, B7, etc., of stripe 3 for data item D2. Block B1 of stripe 1 for data item D1 contains data at addresses 0-4095, and block B3 of stripe 3 for data item D2 contains data at addresses 8192-12287. Figure 3 In the example, each address is the byte offset of the corresponding byte in the identifier block.

[0047] For example, in storage server 1, byte offset 0 of data item D1 identifies byte 0 in block B1 of stripe 1 of data item D1, byte offset 1 of data item D1 identifies byte 1 in block B1 of stripe 1 of data item D1, ..., and byte offset 4095 of data item D1 identifies byte 4095 in block B1 of stripe 1 of data item D1.

[0048] Similarly, in storage server 1, byte offset 8192 of data item D2 identifies byte 0 in block B3 of stripe 3 of data item D2, byte offset 8193 of data item D2 identifies byte 1 in block B3 of stripe 3 of data item D2, ..., and byte offset 12287 of data item D2 identifies byte 4095 in block B3 of stripe 3 of data item D2.

[0049] Figure 4 This is a message flow diagram of the processes involving client system 108-1 and storage servers 1, 2, 3 and 4 that can be executed in some examples.

[0050] Client system 108-1 receives (at 402) a FAM_Allocate request, which is a request to allocate a memory region at structure-attached memory 102, including multiple storage servers, including storage servers 1, 2, 3, and 4. The FAM_Allocate request can be issued in response to a request from a program within client system 108-1 or from a requester outside client system 108-1. The FAM_Allocate request can specify the size of the memory region to be allocated. In some examples, the FAM_Allocate request can be triggered based on a program or another requester accessing an application programming interface (API) that provides routines for memory allocation and other functions related to structure-attached memory 102.

[0051] In response to the FAM_Allocate request, the client system 108-1 issues the Allocate(NAME,SIZE) command (at 404), which is a command to allocate a memory region with an identifier specified by NAME and a size specified by SIZE.

[0052] Client system 108-1 also discovered (at 406) an available storage server that can allocate storage regions. In some examples, client system 108-1 can communicate with metadata server 130 ( Figure 1The metadata server 130 interacts to discover available storage servers. The metadata server 130 may be a server computer that maintains information about the collection of storage servers attached to the storage server 102. The maintained information may include any one or a combination of the following: information about the available storage capacity of each storage server (indicating how much storage capacity is available for storing data in the persistent storage of the storage server), information about the health status of each storage server (e.g., related to error rates and any failures encountered by the storage server), information about the data rate of each storage server (e.g., related to the speed at which data can be written to or read from the storage server), and so on.

[0053] Client system 108-1 can send a request to metadata server 130 specifying the size of a storage region to be allocated. Metadata server 130 can use its information to identify which(s) of the metadata server set have sufficient capacity to store at least a portion of the storage region. Metadata server 130 can then return a list of the identified storage servers(s) to client system 108-1. Figure 4 In the example, assume that the list returned by metadata server 130 identifies storage servers 1, 2, 3 and 4 (that is, the list includes the identifiers of storage servers 1, 2, 3 and 4).

[0054] In response to the list from metadata server 130, client system 108-1 issues an Allocate_Memory(SIZE / M) request to allocate a memory region (the size of which is specified by SIZE) across the storage servers in the list. In the example, the storage servers in the list are storage servers 1, 2, 3 and 4 (and therefore M = 4).

[0055] Client system 108-1 sends an Allocate_Memory(SIZE / M) request to storage server 1 (at 408-1), an Allocate_Memory(SIZE / M) request to storage server 2 (at 408-2), an Allocate_Memory(SIZE / M) request to storage server 3 (at 408-3), and an Allocate_Memory(SIZE / M) request to storage server 4 (at 408-4). Each Allocate_Memory(SIZE / M) request can be sent to metadata server 130, which forwards the Allocate_Memory(SIZE / M) request to the appropriate storage server. Alternatively, client system 108-1 can send each Allocate_Memory(SIZE / M) request directly to the appropriate storage server.

[0056] In other examples, instead of sending four separate Allocate_Memory(SIZE / M) requests, a single Allocate_Memory(SIZE / M) request can be multicast to storage servers 1, 2, 3, and 4, either directly or via metadata server 130.

[0057] according to Figure 4 For example, in response to the Allocate_Memory(SIZE / M) request sent (at 408-1), storage server 1 registers the first segment of the memory region at storage server 1 (at 410-1); storage server 2 registers the second segment of the memory region at storage server 2 (at 410-2) in response to the Allocate_Memory(SIZE / M) request sent (at 408-2); storage server 3 registers the third segment of the memory region at storage server 3 (at 410-3) in response to the Allocate_Memory(SIZE / M) request sent (at 408-3); and storage server 4 registers the fourth segment of the memory region at storage server 4 (at 410-4) in response to the Allocate_Memory(SIZE / M) request sent (at 408-4).

[0058] In some examples, each storage server can register the appropriate segments of a memory region by calling library routines from the libfabric library, an open-source library that supports various functions, including defining memory for client use. In other examples, each storage server may use a different mechanism to register the appropriate segments of a memory region.

[0059] Additionally, storage server 1 (directly or via metadata server 130) sends an allocation response to client system 108-1 (at 412-1), storage server 2 (directly or via metadata server 130) sends an allocation response to client system 108-1 (at 412-2), storage server 3 (directly or via metadata server 130) sends an allocation response to client system 108-1 (at 412-3), and storage server 4 (directly or via metadata server 130) sends an allocation response to client system 108-1 (at 412-4). Each allocation response includes status information (indicating whether the allocation of the corresponding segment of the memory region was successful or failed), a key, and the corresponding base address.

[0060] Keys can be values ​​(such as numbers, strings, etc.) that client systems can use to perform operations on the allocated segments of the memory region at the corresponding storage server. A key is a tag that is identified as associated with a segment of the memory region at the storage server (such as by the libfabric library located at the storage server).

[0061] The base address refers to the address that serves as the starting address of an allocated segment of a memory region at the corresponding storage server (e.g., a virtual address).

[0062] The information returned in the allocation responses from storage servers 1, 2, 3, and 4 constitutes metadata that can be used at client system 108-1 to access the allocated memory region segment. Client system 108-1 stores the received metadata contained in the allocation response (at 414) in a descriptor, which can be stored at client system 108-1 (e.g., stored at [location not specified]). Figure 1 (In the local memory 114 shown).

[0063] Figure 5 This is a block diagram of descriptor 500, which can store the aforementioned metadata. Descriptor 500 can be a data structure in any of various formats, such as tables, files, etc. For example, the descriptor can be stored in... Figure 1 The client system 108-1 is located in the local storage 114.

[0064] Descriptor 500 includes four entries: 500-1, 500-2, 500-3, and 500-4 (in such cases) Figure 4 (As shown in the example with four storage servers). Entry 500-1 includes the identifier M1 of storage server 1, the key K1 of the allocated segment of the memory region on storage server 1, and the base address V1 of the allocated segment of the memory region on storage server 1.

[0065] Entry 500-2 includes the identifier M2 of storage server 2, the key K2 of the allocated segment of the memory region at storage server 2, and the base address V2 of the allocated segment of the memory region at storage server 2.

[0066] Entry 500-3 includes the identifier M3 of storage server 3, the key K3 of the allocated segment of the memory region on storage server 3, and the base address V3 of the allocated segment of the memory region on storage server 3.

[0067] Entry 500-4 includes the identifier M4 of storage server 4, the key K4 of the allocated segment of the memory region on storage server 4, and the base address V4 of the allocated segment of the memory region on storage server 4.

[0068] Descriptor 500 includes a list of storage servers 502 that are allocated across its memory regions, a list of structure keys for the corresponding memory region segments 504, and a list of base addresses for the corresponding memory region segments 506.

[0069] Each allocated memory region segment at the storage server can be used to store the corresponding stripe of a data item. For example, in Figure 2 In the memory region A, the allocated segment storage data item D1 stripe 1 and data item D2 stripe 3 in storage server 1; the allocated segment storage data item D1 stripe 2 and data item D2 stripe 4 in storage server 2; the allocated segment storage data item D1 stripe 3 and data item D2 stripe 1 in storage server 3; and the allocated segment storage data item D1 stripe 4 and data item D2 stripe 2 in storage server 4.

[0070] Once a memory region is allocated across storage servers for a given data item, when accessing (reading and / or writing) any offset within the given data item, the client system 108-1 can obtain the storage server identifier based on the interleaving size and the number of storage servers across which the allocated memory region is located (e.g., in...). Figure 5 (any of M1 to M4 in the example) and the byte offset within the corresponding stripe for the given data item.

[0071] Therefore, for a given read / write operation with a specified offset and the size of the data to be accessed (e.g., the number of bytes), the list of storage servers to perform the corresponding remote access operation and the data size of each remote access operation at each storage server can be calculated by the client system 108-1.

[0072] Based on the information specified in a given read / write operation and stored in a descriptor (e.g., Figure 5The metadata in the 500) can be remotely processed from the client system 108-1 without additional intervention or coordination from a central controller or any storage server.

[0073] Figure 6 Refer to the example of distributing a 1-gigabyte (1GB) data item across four storage servers with an interleaving size of 1 megabyte (MB) (or equivalent, a block size). In the example, client system 108-1 can receive a write request to perform an operation to write data (with a data size of 2 MB) to the structure-attached memory 102. In the example, the write request specifies that the written data will be written at an offset of 256 (i.e., 256 bytes) within the data item.

[0074] although Figure 6 A specific order of tasks is described, but it should be noted that in other examples, tasks may be performed in a different order, and / or some tasks may be omitted, and / or other tasks may be added.

[0075] Client system 108-1 stores the write data of the write request (at 602) in the input buffer 140 of client system 108-1. Figure 1 In the process, client system 108-1 will retrieve a portion of the data to be written from input buffer 140 and write it to the storage server. In this example, it is assumed that there are M (= 4) storage servers, and the data to be written will be interleaved across these storage servers.

[0076] Client system 108-1 determines (at 603) whether the data item targeted by the write request is interleaved across multiple storage servers. If the data item is not interleaved (meaning the data item is stored on only one storage server), client system 108-1 continues (at 640) to issue a write request to the single storage server.

[0077] The following describes the task performed in response to the client system 108-1 determining (at 603) the interleaving of data items.

[0078] The client system 108-1 initializes (at 604) various parameters related to the positions in the storage server, stripes, blocks, and input buffer 140. The initialization parameters are as follows.

[0079] The storage server index MEMORY_SERVER_INDEX is initialized to refer to the storage server (among M storage servers) where a write operation will begin.

[0080]

[0081] In the calculation above, the operator "%" is the modulo operator, which returns the remainder of (OFFSET / INTERLEAVE_SIZE) divided by M. The parameter OFFSET represents the offset of the write operation (256 in the example above). The parameter INTERLEAVE_SIZE represents the interleaving size (1 MB or 1,048,576 bytes in the example above).

[0082] Note that if the offset is greater than 1 MB, then MEMORY_SERVER_INDEX will be set to 1 or a larger value to refer to another storage server among the M storage servers.

[0083] The block number within the stripe, BLOCK_NUM_WITHIN_STRIPE, is initialized to indicate the block number within the stripe at the storage server where the write operation will begin.

[0084]

[0085] The offset within the block value, OFFSET_WITHIN_BLOCK, is initialized to indicate the offset within the block where writing will begin, identified by BLOCK_NUM_WITHIN_STRIPE:

[0086]

[0087] The buffer pointer BUFFER_POINTER is set to the value of INPUT_BUFFER, which contains a pointer to the position where the write data of the write request begins in the input buffer 140 of the client system 108-1. In the example below, it is assumed that INPUT_BUFFER is set to 1000, for example, referring to the 1000th byte in the input buffer 140.

[0088]

[0089] After the above parameters are initialized, loop 606 can be executed to create I / O requests to be submitted to the structure-attached memory 102 to perform the write operation specified by the write request. One I / O request is created at a time in loop 606. This will be further elaborated below. Figure 7 Further discussion reveals that the created I / O requests are placed into I / O request data structures for the corresponding M storage servers (e.g., one I / O request data structure per storage server, such as a vector). Each I / O request writes a portion of the data to the structure-attached memory 102.

[0090] Each iteration of loop 606 creates a "current" I / O request IO(i), where i represents the current iteration of loop 606. Additional iterations of loop 606 create other I / O requests.

[0091] Loop 606 includes several descriptor parameters obtained from descriptor 500 (at 608), including the storage server identifier (MEMORY_SERVER_ID), the key (KEY) used to access the memory region segment, and the base address (BASE_ADDRESS) of the memory region segment to be accessed for the write operation. The storage server index MEMORY_SERVER_INDEX is used for lookup. Figure 5 The corresponding entries for descriptor 500 (e.g., one of entries 500-1 to 500-4) are as follows:

[0092]

[0093] In the above text, MEMORY_SERVERS[ ] is the storage server list 502, FABRIC_KEYS[ ] is the structure key list 504, and BASE_ADDRESS_LIST[ ] is the base address list 506.

[0094] Next, using the base address (BASE_ADDRESS) obtained from descriptor 500 via the storage server index MEMORY_SERVER_INDEX, client system 108-1 calculates (at 610) the remote structure attached memory address (REMOTE_FAM_POINTER), which is the address at which the current I / O request IO(i) will write the corresponding write data portion.

[0095]

[0096] Next, client system 108-1 determines (at 612) the size (DATA_CHUNK_SIZE) of the write data portion of the current I / O request, as follows:

[0097]

[0098] The size IO(i) of the write data portion of the current I / O request is the minimum difference between the interleaving size and the write request size (REQUEST_SIZE) and the total completed I / O size (COMPLETED_IO_SIZE) of the write data portions of the (multiple) I / O requests (if any) created in loop 606, minus OFFSET_WITHIN_BLOCK.

[0099] The write request size (REQUEST_SIZE) is the size of the data to be written specified in the write request, which is 2 MB (or 2,097,152 bytes) in this example. The total completed I / O size (COMPLETED_IO_SIZE) of the write data portions of the (multiple) I / O requests (if any) created in loop 606 is the sum of the sizes of the write data portions of the (multiple) I / O requests (if any) created in loop 606.

[0100] The value of OFFSET_WITHIN_BLOCK is subtracted from the minimum of the differences between the interleaving size and the write request size (REQUEST_SIZE) and the total completed I / O size (COMPLETED_IO_SIZE) because for the first block written, the write can begin with a non-zero byte represented by OFFSET_WITHIN_BLOCK.

[0101] Client system 108-1 then constructs the current I / O request IO(i) for the current iteration i of loop 606 (at position 614), represented as:

[0102]

[0103] The current I / O request IO(i) includes a key (KEY) for accessing the corresponding memory region segment, a remote structure attached memory address (REMOTE_FAM_POINTER) that serves as the address where the corresponding write data portion of the current I / O request will be written, a buffer pointer (BUFFER_POINTER) indicating the starting position of the write data in the input buffer 140 of the client system 108-1, and the size of the write data portion of the current I / O request (DATA_CHUNK_SIZE).

[0104] Client system 108-1 adds (at 616) the current I / O request IO(i) to the corresponding I / O request data structure (e.g., I / O vector) of the storage server with the storage server identifier MEMORY_SERVER_ID. Figure 7 I / O vectors 702-1, 702-2, 702-3, and 702-4 are shown for four corresponding storage servers with storage server identifiers M1, M2, M3, and M4. I / O vectors 702-1, 702-2, 702-3, and 702-4 collectively form I / O array 700, which contains I / O requests to be submitted to the structure-attached memory 102 to write write data in response to write requests.

[0105] I / O vector 702-1 contains I / O requests for a storage server with storage server identifier M1, I / O vector 702-2 contains I / O requests for a storage server with storage server identifier M2, I / O vector 702-3 contains I / O requests for a storage server with storage server identifier M3, and I / O vector 702-4 contains I / O requests for a storage server with storage server identifier M4.

[0106] Although Figure 7 The example shows a depth of 4 for each I / O vector, but in other examples, each I / O vector can have a smaller or larger depth.

[0107] Assuming loop 606 is in the first iteration, where i = 1 and MEMORY_SERVER_ID = M1, then IO(1) is added to the first entry of I / O vector 702-1.

[0108] As described below, client system 108-1 updates various parameters (at 618) in preparation for the next iteration.

[0109] Client system 108-1 increments the buffer pointer (BUFFER_POINTER) before the next iteration, as follows:

[0110]

[0111] The buffer pointer (BUFFER_POINTER) is incremented to the size (DATA_CHUNK_SIZE) of the write data portion of the I / O request IO(i) added to the I / O array 700.

[0112] The client system 108-1 increments the total completed I / O size (COMPLETED_IO_SIZE) of the write data portion of the I / O request by adding DATA_CHUNK_SIZE, as follows:

[0113]

[0114] In other words, increment COMPLETED_IO_SIZE to add the size of the write data portion of the current I / O request IO(i).

[0115] Client system 108-1 resets OFFSET_WITHIN_BLOCK to zero (OFFSET_WITHIN_BLOCK = 0). This indicates that the next write (to the I / O request constructed in the next iteration) is performed on a block starting at offset 0.

[0116] Client system 108-1 determines (at 620) whether the number of entries in I / O array 700 exceeds the specified maximum number of entries MAX_SCATTER_SIZE. In some examples, the check at 620 is performed to prevent I / O array 700 from becoming too large (i.e., to prevent it from containing too many unissued I / O requests). If the number of entries in I / O array 700 exceeds the specified maximum number of entries MAX_SCATTER_SIZE, client system 108-1 issues an I / O request for I / O array 700 to structure-attached memory 102 via network 106 (at 622). I / O requests can be issued in distributed I / O operations, where I / O requests are sent in parallel to multiple storage servers. Issued I / O requests are then removed from I / O array 700.

[0117] If the number of entries in I / O array 700 is greater than the specified maximum number of entries MAX_SCATTER_SIZE, then loop 606 continues to add entries to I / O array 700.

[0118] Client system 108-1 increments (at 624) MEMORY_SERVER_INDEX to indicate the next storage server. Note that since there are M (e.g., 4) storage servers in this example, if client system 108-1 determines (at 626) that MEMORY_SERVER_INDEX is greater than M after the increment (at 626), then client system 108-1 resets MEMORY_SERVER_INDEX (at 628) to 1, thus iterating through the storage servers again in a round-robin fashion. Additionally, if MEMORY_SERVER_INDEX is greater than M, client system 108-1 increments (at 630) the block number within the stripe, BLOCK_NUM_WITHIN_STRIPE, to indicate the next block to be accessed in the next iteration. In other words, after creating I / O requests for all M storage servers in a round, the block number within the stripe is incremented to indicate the next block.

[0119] Client system 108-1 determines (at 632) whether COMPLETED_IO_SIZE is equal to the write request size REQUEST_SIZE, that is, the size of the write data specified in the write request (e.g., 2 MB in this example). If not, loop 606 continues for the next iteration by incrementing (at 634) i to construct the next I / O request IO(i).

[0120] If COMPLETED_IO_SIZE equals the write request size REQUEST_SIZE, then the client system 108-1 issues any remaining I / O requests in the I / O array 700 during the distributed I / O operation (at 636).

[0121] Collecting the I / O requests built in loop 606 into I / O array 700 allows I / O requests for each storage server to be collected into corresponding I / O vectors 702-j (j = 1 to 4 in this example). By collecting the I / O requests built in loop 606 into I / O array 700, instead of issuing them when the I / O requests are created, the number of I / O operations on network 106 can be reduced by aggregating I / O requests for the same server into a single I / O operation (instead of multiple I / O operations). Furthermore, I / O operations for different storage servers corresponding to the I / O requests in I / O vectors 702-1 to 702-4 can be issued in parallel for better bandwidth. Each I / O operation into which the I / O requests of the corresponding I / O vector are aggregated performs the write of the corresponding write data portion of the I / O request for that I / O vector.

[0122] Figure 8 This is a block diagram of a computer 800 based on some examples. For example, computer 800 may be a client system (e.g., 108-1 to 108-N).

[0123] Computer 800 includes one hardware processor 802 (or multiple hardware processors). The hardware processor performing a task can refer to a single hardware processor performing a task or multiple hardware processors performing a task.

[0124] Computer 800 includes a non-transitory or computer-readable storage medium 804 storing client instructions that can be executed on hardware processor 802 to perform various tasks. Machine-readable instructions that can be executed on hardware processors can refer to instructions that can be executed on a single hardware processor or instructions that can be executed on multiple hardware processors.

[0125] The client instructions include storage server identification instructions 806, used to identify multiple storage servers accessible by computer 800 to perform remote access to data stored by the multiple storage servers over a network. The multiple storage servers may include, for example... Figure 1 Storage server 104. As an example, the identification of multiple storage servers can be achieved by... Figure 4 The discovery task 406 in the process is executed.

[0126] The client instructions include allocation request sending instruction 808, used to send an allocation request for allocating a memory segment to place interleaved data of computer 800 across multiple storage servers. For example, the allocation request may include an Allocate_Memory(SIZE / M) request sent at locations 408-1 to 408-4. The memory segment may include, for example, Figure 2 The memory region A shown is segmented.

[0127] The client instructions include metadata receiving instructions 810, used to receive metadata related to memory segments at multiple storage servers at computer 800 in response to an allocation request. This metadata includes the addresses of the memory segments at the multiple storage servers. For example, the metadata may include... Figure 4 The allocation response sent at positions 412-1 to 412-2 in the code may include information stored in, for example... Figure 5 Metadata in descriptor 500.

[0128] Client instructions include interleaved data access instructions 812, used by computer 800 to access interleaved data located at multiple storage servers using metadata. This interleaved data includes data blocks distributed across memory segments. Data blocks may include, for example... Figure 2 The data items D1 or D2 in the data block B1, B2, etc.

[0129] In some examples, client instructions store metadata in the computer's 800 memory (e.g., Figure 1 The descriptor in 114) (e.g., Figure 5 In the 500 section, the client instruction access descriptor determines the address offset (e.g., REMOTE_FAM_POINTER) of the set of data blocks (single or multiple data blocks) that the computer 800 wants to access.

[0130] In some examples, client instructions access descriptors to determine the identifier (MEMORY_SERVER_ID) of the storage service server where the data block set is placed.

[0131] In some examples, each data block has an interleaving size. The address offset of the data block set is determined based on the interleaving size and metadata in the descriptor.

[0132] In some examples, client instructions distribute data blocks across memory segments in a polling manner, thereby providing interleaved data.

[0133] In some examples, data interleaving across memory segments across multiple storage servers is controlled by client instructions without the need for coordination from a central controller.

[0134] In some examples, the allocated memory segment is used for the first data item and the second data item (e.g., Figure 2 (D1 and D2 in the original text). The client instruction places the first interleaved data of the first data item across memory segments and the second interleaved data of the second data item across memory segments. Each of the first and second interleaved data includes data blocks distributed across memory segments, wherein the starting block of the first data item is located at a first storage server, and the starting block of the second data item is located at a different second storage server (e.g., in...). Figure 2 In the diagram, the starting block of D1 is located at storage server 1, and the starting block of D2 is located at storage server 3.

[0135] In some examples, the client instruction receives a request to write data, determines the corresponding data block to be written to each of the corresponding storage servers in multiple storage servers, and aggregates the I / O requests for the corresponding data blocks into the corresponding I / O request data structure of the corresponding storage server at computer 800 (e.g., Figure 7 (The I / O vector shown).

[0136] In some examples, this aggregation will be used for multiple I / O request data structures at computer 800 (e.g., Figure 7 I / O requests are generated in sections 702-1 to 702-4, where each of the multiple I / O request data structures includes an I / O request for the corresponding storage server. The client instruction issues the I / O requests of the multiple I / O request data structures as distributed I / O operations, which are then sent in parallel to multiple storage servers.

[0137] In some examples, distributed I / O operations include a first I / O request issued to a first storage server among multiple storage servers, and a second I / O request issued in parallel to a second storage server among multiple storage servers.

[0138] In some examples, client instructions are in a loop (e.g., Figure 6 In loop 606, I / O requests are created iteratively, where a first I / O request created in the first iteration of the loop is for a first storage server, and a second I / O request created in the second iteration of the loop is for a second storage server, wherein the second iteration immediately follows the first iteration in the loop. For example, if the first I / O request is created in iteration m of loop 606, then the second I / O request is created in iteration m + 1 of loop 606.

[0139] In some examples, each corresponding I / O request in an I / O request includes the address of the write data portion written for the corresponding I / O request (e.g., REMOTE_FAM_POINTER), the input buffer (e.g., ... Figure 1 The location of the data to be written (e.g., BUFFER_POINTER) and the size of the data to be written (DATA_CHUNK_SIZE) are stored in 140.

[0140] In some examples, the client instruction receives a request to allocate a memory region of a specified size and generates an allocation request in response to the request to allocate the memory region.

[0141] In some examples, in response to a request to allocate a memory region, the client instruction discovers which storage servers in the set of storage servers are available based on criteria used for allocating the memory region. This discovery identifies multiple storage servers.

[0142] In some examples, the standard is a memory capacity standard, where multiple storage servers are storage servers with available memory capacity that meets the memory capacity standard.

[0143] Figure 9 It is a block diagram of a non-transitory machine-readable or computer-readable storage medium 900 that stores machine-readable instructions that, when executed, enable the computer to perform various tasks.

[0144] The machine-readable instructions in storage medium 900 include metadata storage instructions 902 for storing metadata in the computer's memory. This metadata includes an identifier of the storage server, the placement of interleaved data on the storage server, and the base address of a memory region segment containing the interleaved data. The metadata may, for example, be stored in... Figure 5 In descriptor 500. The storage server is a structure-attached storage (e.g., ...) Figure 1 It is part of the structure attached to memory 102.

[0145] The machine-readable instructions in storage medium 900 include I / O request creation instructions 904 for creating I / O requests for corresponding data blocks of interleaved data using metadata stored in memory. The creation of the first I / O request among these I / O requests includes retrieving a first base address from the base address in the metadata (e.g., ...). Figure 4 (Any of V1 to V4 shown), using the first base address and the interleaving size of the interleaved data to determine the structure address of the first data portion of the first I / O request (e.g., REMOTE_FAM_POINTER), and including the structure address in the first I / O request.

[0146] The machine-readable instructions in the storage medium 900 include I / O request add instructions 906 for adding I / O requests to the corresponding I / O request data structures (e.g., I / O vectors 702-1 to 702-4) of the storage server.

[0147] The machine-readable instructions in the storage medium 900 include an I / O request issuance instruction 908, which sends an I / O request from an I / O request data structure to a structure-attached memory, wherein the I / O request of the first I / O request data structure in the I / O request data structure is sent to the first storage server in parallel, and the I / O request of the second I / O request data structure in the I / O request data structure is sent to the second storage server.

[0148] In some examples, I / O requests for a first I / O request data structure are aggregated into a first I / O operation for a first storage server, and I / O requests for a second I / O request data structure are aggregated into a second I / O operation for a second storage server.

[0149] Figure 10 It is a flowchart of a process that can be executed, for example, by a computer.

[0150] The process includes identifying (at 1002) multiple storage servers accessible by a computer to perform remote access over a network to data stored on the multiple storage servers in a structure-attached memory, wherein the computer will use RDMA to access a given storage server among the multiple storage servers, including data transfer via the network interface of the given storage server, wherein the data transfer does not involve work performed by the main processor running the OS on the given storage server. This identification could be, for example, Figure 4 It was part of the discovery mission 406.

[0151] This process includes sending an allocation request (at 1004) to allocate a segment of memory region in response to a request to allocate a memory region, so as to place the computer's interleaved data across multiple storage servers. For example, the allocation request may include an Allocate_Memory(SIZE / M) request sent at 408-1 to 408-4.

[0152] The process includes receiving, in response to an allocation request, metadata associated with memory region segments across multiple storage servers (at location 1006), including the base addresses of the memory region segments across the multiple storage servers. The base addresses may include, for example... Figure 5 V1 to V4 in the series.

[0153] The process includes executing tasks 1008, 1010, and 1012 in response to requests for access to data.

[0154] Task 1008 includes accessing metadata to create multiple I / O requests for multiple data portions of requested data. Creating multiple I / O requests involves using metadata to calculate structure-attached memory addresses (e.g., REMOTE_FAM_POINTER) to be included in the multiple I / O requests. The structure-attached memory addresses specify the location of the requested data in multiple storage servers attached to the network. For example, the multiple I / O requests can be in the following manner: Figure 6 Created within the loop 606.

[0155] Task 1010 involves collecting I / O requests into corresponding I / O request data structures for the corresponding storage servers across multiple storage servers. The I / O request data structures may include, for example, I / O vectors 702-1 to 702-4.

[0156] Task 1012 includes issuing I / O requests from an I / O request data structure to perform access (e.g., write data) to requested data (e.g., write data) interleaved across multiple storage servers, wherein the interleaving of requested data across multiple storage servers is performed without the coordination of a central controller.

[0157] In this disclosure, unless the context clearly indicates otherwise, the terms “a / an” or “the” are intended to include the plural form as well. Similarly, when used in this disclosure, the terms “includes / including / comprises / comprising” or “have / having” indicate the presence of the said element but do not preclude the presence or addition of other elements.

[0158] Storage media (e.g., Figure 8 804 or Figure 9The 900 in the specification can include any one or a combination of the following: semiconductor memory devices, such as dynamic or static random access memory (DRAM or SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory; disks, such as fixed disks, floppy disks, and removable disks; another magnetic medium, including magnetic tape; optical media, such as optical discs (CDs) or digital video discs (DVDs); or another type of storage device. Note that the instructions discussed above may be provided on a single computer-readable or machine-readable storage medium, or alternatively, on multiple computer-readable or machine-readable storage media distributed across a large system having potentially multiple nodes. Such one or more computer-readable or machine-readable storage media are considered part of an article (or article of manufacture). An article or article of manufacture can refer to any single or multiple manufactured components. One or more storage media may be located in a machine that executes the machine-readable instructions or at a remote site from which the machine-readable instructions can be downloaded via a network for execution.

[0159] In the foregoing description, numerous details have been set forth to facilitate understanding of the subject matter disclosed herein. However, embodiments may be practiced without some of these details. Other embodiments may include modifications and variations of the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A computer, comprising: processor; as well as A non-transitory storage medium storing client instructions that can be executed on the processor to perform the following operations: The client is implemented using the aforementioned client instructions; Identify multiple storage servers accessible by the computer to perform remote access to data stored by the multiple storage servers over a network; The client generates an allocation request to allocate memory segments to place the computer's interleaved data across the multiple storage servers; Send the allocation request to the plurality of storage servers; In response to the allocation request, the computer receives metadata related to the memory segment at the plurality of storage servers, the metadata including the address of the memory segment at the plurality of storage servers; as well as The interleaved data, comprising data blocks distributed across the memory segments, is accessed by the computer using the metadata located at the plurality of storage servers. The client instructions can be executed on the processor to perform the following operations: Receive requests to write data; Determine the appropriate data block to be written to each of the plurality of storage servers; as well as Input / output (I / O) requests for the corresponding data blocks are aggregated into the corresponding I / O request data structure of the corresponding storage server at the computer.

2. The computer as claimed in claim 1, wherein, The client instructions can be executed on the processor to perform the following operations: The metadata is stored in a descriptor in the computer's memory; as well as Access the descriptor to determine the address offset of the set of data blocks the computer needs to access.

3. The computer as claimed in claim 2, wherein, The client instructions can be executed on the processor to perform the following operations: Access the descriptor to determine the identifier of the storage service server that stores the set of data blocks.

4. The computer as claimed in claim 2, wherein, Each data block in the set has an interleaving size, and the address offset of the set of data blocks is determined based on the interleaving size and the metadata in the descriptor.

5. The computer as claimed in claim 1, wherein, The client instructions can be executed on the processor to perform the following operations: The data blocks are distributed across the memory segment in a polling manner to provide the interleaved data.

6. The computer as claimed in claim 1, wherein, Data interleaving across the memory segments at the multiple storage servers is controlled by the client instructions without the need for coordination of a central network device that controls memory allocation for multiple requesting devices, including the computer.

7. The computer as claimed in claim 1, wherein, The allocated memory segment is used for the first data item and the second data item, and wherein the client instructions are executable on the processor to perform the following operations: Place the first interleaved data of the first data item across the memory segment; and Place the second interleaved data of the second data item across the memory segment. Each of the first interleaved data and the second interleaved data includes data blocks distributed across the memory segments, wherein the starting block of the first data item is located at a first storage server among the plurality of storage servers, and the starting block of the second data item is located at a different second storage server among the plurality of storage servers.

8. The computer as claimed in claim 1, wherein, The I / O request is generated from multiple I / O request data structures aggregated at the computer, wherein each of the multiple I / O request data structures includes an I / O request for a corresponding storage server among the multiple storage servers, and wherein the client instructions are executable on the processor to perform the following operations: The I / O requests of the multiple I / O request data structures are issued as distributed I / O operations, which are then sent in parallel to the multiple storage servers.

9. The computer as claimed in claim 8, wherein, The distributed I / O operation includes a first I / O request sent to a first storage server among the plurality of storage servers, and a second I / O request sent in parallel to a second storage server among the plurality of storage servers.

10. The computer as claimed in claim 1, wherein, The client instructions can be executed on the processor to perform the following operations: The I / O requests are created iteratively in a loop, wherein the first I / O request created in the first iteration of the loop is for a first storage server among the plurality of storage servers, and the second I / O request created in the second iteration of the loop is for a second storage server among the plurality of storage servers, wherein the second iteration immediately follows the first iteration in the loop.

11. The computer of claim 10, wherein, Each corresponding I / O request in the I / O request includes the address of the write data portion written for the corresponding I / O request, the location in the input buffer where the write data portion is stored, and the size of the write data portion.

12. The computer of claim 1, wherein, The client instructions can be executed on the processor to perform the following operations: Receive requests to allocate a memory region of a specified size; and The allocation request is generated in response to the request to allocate the memory region.

13. The computer of claim 12, wherein, The client instructions can be executed on the processor to perform the following operations: In response to the request to allocate the memory region, the availability of storage servers in the storage server set is determined based on criteria used for allocating the memory region. The discovery identifies the plurality of storage servers.

14. The computer of claim 13, wherein, The standard is a memory capacity standard, wherein the plurality of storage servers are storage servers having available memory capacity that meets the memory capacity standard.

15. The computer as claimed in claim 1, wherein, The plurality of storage servers include storage servers that manage the placement of data in the attached storage structure.

16. A non-transitory machine-readable storage medium, comprising instructions that, when executed, cause a computer to perform the following operations: Implement the client; Metadata is stored in the computer's memory, the metadata including: an identifier of the storage server, interleaved data placed across the storage server, and a base address of a memory region segment containing the interleaved data and allocated by the client at the computer, wherein... The storage server is part of a structure-attached storage system; Using metadata stored in the memory, the client creates input / output (I / O) requests for corresponding data blocks of the interleaved data, wherein creating the first I / O request in the I / O request includes: Retrieve the first base address from the base addresses in the metadata. The structure address of the first data portion of the first I / O request is determined using the first base address and the interleaving size of the interleaved data. The structure address is included in the first I / O request; Add the I / O request to the corresponding I / O request data structure of the storage service server; and The I / O request is sent from the I / O request data structure to the attached memory, wherein, in parallel, the I / O request of the first I / O request data structure in the I / O request data structure is sent to the first storage server, and the I / O request of the second I / O request data structure in the I / O request data structure is sent to the second storage server. When executed, the instructions also cause the computer to perform the following operations: Receive requests to write data; Determine the appropriate data block to be written to each of the plurality of storage servers; and Input / output (I / O) requests for the corresponding data blocks are aggregated into the corresponding I / O request data structure of the corresponding storage server at the computer.

17. The non-transitory machine-readable storage medium of claim 16, wherein, The I / O requests of the first I / O request data structure are aggregated into a first I / O operation to the first storage server, and the I / O requests of the second I / O request data structure are aggregated into a second I / O operation to the second storage server.

18. A method for client memory allocation, comprising: The client is implemented by the computer; The computer identifies a plurality of storage servers accessible by the computer to perform remote access over a network to data stored on the plurality of storage servers in a structure-attached memory, wherein the computer will use Remote Direct Memory Access (RDMA) to access a given storage server among the plurality of storage servers, including data transfer through the network interface of the given storage server, wherein the data transfer does not involve work performed by the main processor running an operating system (OS) on the given storage server; The client generates a request to allocate a memory region; In response to the request to allocate the memory region, an allocation request for allocating a segment of the memory region is sent to place the interleaved data of the computer across the plurality of storage servers; In response to the allocation request, the computer receives metadata related to memory regions located at the plurality of storage servers, the metadata including the base address of the memory regions located at the plurality of storage servers; and A request for data in response to an access request: Accessing the metadata to create multiple input / output (I / O) requests for multiple data portions of the requested data, wherein creating the multiple I / O requests includes using the metadata to calculate a structure-attached memory address to include in the multiple I / O requests, the structure-attached memory address specifying the location of the requested data in the multiple storage servers attached to the network. The I / O requests are collected into the corresponding I / O request data structures of the corresponding storage servers in the plurality of storage servers, and The I / O request is issued from the I / O request data structure to perform access to the requested data interleaved across the multiple storage servers, wherein the interleaving of the requested data across the multiple storage servers is performed without the coordination of a central controller. The method further includes: Receive requests to write data; Determine the appropriate data block to be written to each of the plurality of storage servers; and Input / output (I / O) requests for the corresponding data blocks are aggregated into the corresponding I / O request data structure of the corresponding storage server at the computer.

19. The method of claim 18, wherein, The computer is a first computer, wherein issuing the I / O request includes sending the I / O request of the first I / O request data structure in the I / O request data structure from the first computer to the first storage server in parallel, and sending the I / O request of the second I / O request data structure in the I / O request data structure from the first computer to the second storage server, and The additional memory region segment can be allocated across the plurality of storage servers by a second computer separate from the first computer, which places interleaved data in the additional memory region segment.

Images