System, method and apparatus for time-synchronized storage delivery

Abstract

Systems, methods, and devices for time-synchronized storage delivery are disclosed. The method includes receiving, at a first computing device, a first input / output (IO) command from a first artificial intelligence processing unit (AI PU), the first IO command associated with a first AI model training operation. The method also includes receiving, at the first computing device, a second IO command from a second AI PU, the second IO command associated with a second AI model training operation. The method also includes assigning a first timestamp to the first IO command based on a first bandwidth allocated to the first AI model training operation. The method also includes assigning a second timestamp to the second IO command based on a second bandwidth allocated to the second AI model training operation.

Description

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 051,883, filed July 14, 2020, and U.S. Application No. 17 / 154,444, filed January 21, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0002] Embodiments of the inventive concept relate to systems, methods, and apparatus for time-synchronized storage delivery. Background Technology

[0003] Large datasets are used to train artificial intelligence (AI) models (e.g., variational autoencoders). AI systems may include thousands of graphics processing units (GPUs) configured to process large datasets in parallel to properly train one or more AI models. Summary of the Invention

[0004] Storage systems and methods for performing time-synchronized storage delivery are disclosed. These systems and methods can be used to support the processing of large datasets during the training of artificial intelligence models.

[0005] A method includes receiving a first input / output (IO) command from a first artificial intelligence processing unit (AI PU), the first IO command being associated with a first AI model training operation. The method also includes receiving a second IO command from a second AI PU, the second IO command being associated with a second AI model training operation. The method further includes assigning a first timestamp to the first IO command based on a first bandwidth allocated to the first AI model training operation. The method also includes assigning a second timestamp to the second IO command based on a second bandwidth allocated to the second AI model training operation.

[0006] A method includes receiving a first request for a first data block in a storage device, the first request including a first timestamp and a first global stream identifier (GFID). The method further includes receiving a second request for a second data block in the storage device, the second request including a second timestamp and a second GFID. The method also includes identifying a first artificial intelligence (AI) model processing unit in the storage device based on the first GFID. Furthermore, the method includes identifying a second AI model processing unit in the storage device based on the second GFID. The method further includes sending the first data block from the storage device to the first AI model processing unit, wherein the timing of the sending of the first data block is based on the first timestamp. Finally, the method includes sending the second data block from the storage device to the second AI model processing unit, wherein the timing of the sending of the second data block is based on the second timestamp.

[0007] A computing device includes a network interface and a processor unit. The processor unit is configured to receive a first input / output (IO) command from a first artificial intelligence processing unit (AI PU), the first IO command being associated with a first AI model training operation. The processor unit is also configured to receive a second IO command from a second AI PU, the second IO command being associated with a second AI model training operation. The processor unit is further configured to assign a first timestamp to the first IO command based on a first bandwidth allocated to the first AI model training operation. The processor unit is also configured to assign a second timestamp to the second IO command based on a second bandwidth allocated to the second AI model training operation.

[0008] A storage device includes a network interface and a processor unit. The processor unit is configured to receive a first request for a first data block, the first request including a first timestamp and a first global stream identifier (GFID). The processor unit is further configured to receive a second request for a second data block, the second request including a second timestamp and a second GFID. The processor unit is further configured to identify a first artificial intelligence (AI) model processing unit based on the first GFID. The processor unit is further configured to identify a second AI model processing unit based on the second GFID. The processor unit is further configured to initiate the transmission of the first data block to the first AI model processing unit via the network interface, wherein the timing of the transmission of the first data block is based on the first timestamp. The processor unit is further configured to initiate the transmission of a second data block to the second AI model processing unit via the network interface, wherein the timing of the transmission of the second data block is based on the second timestamp. Attached Figure Description

[0009] Figure 1 This is a diagram of a system used to perform time-synchronized storage and delivery.

[0010] Figure 2 This is a diagram of a storage device used to perform time-synchronized storage delivery.

[0011] Figure 3 It includes Figure 2 Another diagram of a system with multiple storage devices for performing time-synchronized storage delivery.

[0012] Figure 4 This is a diagram of a global flow identifier database entry.

[0013] Figure 5 This is a flowchart illustrating a method for constructing a stored query plan.

[0014] Figure 6This is a flowchart illustrating a method for assigning timestamps to I / O requests.

[0015] Figure 7 This is a flowchart illustrating a method for outputting data in response to an I / O request.

[0016] Figure 8 This is a sequence diagram illustrating the synchronous processing of I / O commands.

[0017] Figure 9 This is a diagram illustrating a system that includes multiple artificial intelligence host devices and multiple object storage systems. Detailed Implementation

[0018] Reference Figure 1 A diagram of a system 100 for time-synchronized storage delivery is shown. System 100 includes an artificial intelligence (AI) host device 102. AI host device 102 may correspond to one or more computing devices. AI host device 102 includes an AI application 108 and a data loader 110. AI application 108 and data loader 110 may correspond to software executable by a processor (not shown) of AI host device 102. AI host device 102 also includes a network interface controller (NIC) 104, dynamic random access memory (DRAM) 106, and multiple AI model processing units (AI PUs) 112a to 112h. NIC 104 may correspond to a wireless NIC or a wired NIC. The multiple AI PUs include a first AI PU 112a, a second AI PU 112b, a third AI PU 112c, a fourth AI PU 112d, a fifth AI PU 112e, a sixth AI PU 112f, a seventh AI PU 112g, and an eighth AI PU 112h. This example embodiment illustrates eight AI PUs 112a to 112h, but any number can be used. Multiple AI PUs 112a to 112h may include one or more graphics processing units (GPUs), one or more applications executable by the processor, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), central processing units (CPUs), additional processing components, or combinations thereof. In some examples, AI PUs 112a to 112h may be configured to handle workloads associated with AI operations. In some examples, the workload may include matrix multiplication and / or convolution-intensive operations. The AI ​​host device 102 may correspond to a physical device or a virtual device (e.g., a physical or virtual device implemented by a cloud computing system).

[0019] In some implementations, the AI ​​host device 102 and Figure 1The examples shown are configured differently. For example, DRAM 106 may be replaced with a different type of memory cell (e.g., static random access memory, SRAM, etc.). As another example, NIC 104 may be replaced with a different type of network adapter. As another example, AI host device 102 may include a different number of GPUs than the number shown. Furthermore, AI host device 102 may include... Figure 1 Additional components not shown. For example, AI host device 102 may include one or more processors configured to execute instructions corresponding to AI application 108, data loader 110, AI PUs 112a to 112h, or combinations thereof.

[0020] System 100 also includes object storage system 120. Object storage system 120 may correspond to one or more computing devices. In some embodiments, object storage system 120 corresponds to a server (e.g., an object storage server). Such a "server" may be a physical device or a virtual device (e.g., a physical device or virtual device implemented by a cloud computing system).

[0021] Object storage system 120 includes NIC 122, processor unit 121, and memory buffers 124a to 124h. In some embodiments, object storage system 120 includes a number of buffers 124a to 124h equal to the number of AI PUs included in AI host device 102. In other embodiments, object storage system 120 includes a different number of buffers. Processor unit 121 may correspond to a field-programmable gate array configured to perform the described actions, a central processing unit configured to execute instructions stored in a memory device (not shown) to perform the described actions, an application-specific integrated circuit (ASIC) configured to perform the described actions, or other processing means. NIC 122 may correspond to a wireless NIC or a wired NIC. Memory buffers 124a to 124h may correspond to memory space within one or more memory devices. Such memory devices may include DRAM, SRAM, other types of memory, or combinations thereof. In the illustrated example, memory buffers 124a to 124h include a first memory buffer 124a, a second memory buffer 124b, a third memory buffer 124c, a fourth memory buffer 124d, a fifth memory buffer 124e, a sixth memory buffer 124f, a seventh memory buffer 124g, and an eighth memory buffer 124h. Each of the memory buffers 124a to 124h may be associated with a different priority. For example, the first memory buffer 124a may be associated with a first priority, and the second memory buffer 124b may be associated with a second priority. In other examples, the object storage system 120 may include a different number of memory buffers.

[0022] Each of AI PUs 112a to 112h has a separate connection to NIC 122. In some embodiments, these connections are maintained via NIC 104 of AI host device 102. In other embodiments, connections are maintained via separate network adapters for AI PUs 112a to 112h. The connections may be direct or via a public network (e.g., the Internet), a private network, or a combination thereof. AI PUs 112a to 112h are configured to send input / output (IO) commands to object storage system 120 via the connections. For example, the first AI PU 112a may send a first IO command requesting a first data object to object storage system 120. The AI ​​PUs are configured to train one or more AI data models based on the acquired data objects. For example, in response to receiving a first data object, the first AI PU 112a may train an AI model (or a portion of an AI model) and return the trained model (or a portion of a trained model) to AI application 108.

[0023] As further described herein, the processor unit 121 of the object storage system 120 is configured to buffer received commands in memory buffers 124a to 124h based on a global flow identifier (GFID) included in the received I / O command. The processor unit 121 of the object storage system 120 is configured to fetch I / O commands and initiate associated data operations based on the priority of the memory buffers 124a to 124h. For example, based on the relative priority of the first memory buffer 124a and the second memory buffer 124b, the processor unit 121 of the object storage system 120 may fetch I / O commands from the first memory buffer 124a more frequently than from the second memory buffer 124b.

[0024] The processor unit 121 of the object storage system 120 manages data objects 126a and 126b. Figure 1 In the diagram, a first data object 126a and a second data object 126b are labeled. Received I / O commands can identify the data objects (e.g., via key values). As further described herein, the processor unit 121 of the object storage system 120 can be configured to initiate time-synchronized transfers of two or more data objects to one or more AI PUs. The data objects 126a, 126b managed by the object storage system 120 can be stored in one or more storage devices. These storage devices can be connected to the object storage system 120 via NIC 122. Therefore, initiating a transfer of a data object may include sending a request to a storage device requesting the transfer of a data object (or a portion of a data object) to the object storage system 120 or a requested AI PU.

[0025] In some implementations, the processor unit 121 of the object storage system 120 is configured to implement a data erasure coding scheme. The processor unit 121 of the object storage system 120 can perform the data erasure coding scheme on data objects to generate one or more erase-coded (EC) blocks and store the EC blocks on one or more storage devices. In response to an I / O command referencing a data object, the processor unit 121 of the object storage system 120 can be configured to determine which storage device(s) stores the EC blocks of the data object and issue I / O requests to those storage devices.

[0026] In some implementations, the storage device includes a non-volatile memory fast (NVMe) device, an NVMe over fabric (NVMe-oF) target, other storage devices, or combinations thereof.

[0027] Figure 2 An example of a storage device 200 (e.g., an NVMe-oF target) capable of storing EC blocks of one or more data objects is depicted. The storage device 200 includes a NIC 201 and a processor unit 205. The NIC 201 may correspond to a wireless NIC or a wired NIC. The processor unit 205 may correspond to a field-programmable gate array configured to perform the described actions, a central processing unit configured to execute instructions stored in a memory device (not shown) to perform the described actions, an application-specific integrated circuit (ASIC) configured to perform the described actions, or other processing devices. The storage device 200 includes a first drive 208a, a second drive 208b, a third drive 208c, and a fourth drive 208d. Each of drives 208a through 208d may include a solid-state drive, a hard disk drive, different types of computer storage drives, or combinations thereof. Drives 208a through 208d are configured to store data (e.g., EC-encoded data blocks). In some embodiments, the storage device 200 includes... Figure 2 The number of drives shown in the figure varies.

[0028] Storage device 200 includes a first-level buffer 202a, a second-level buffer 202b, a third-level buffer 202c, and a fourth-level buffer 202d. Other embodiments may include different numbers of level buffers. Level buffers 202a to 202d may include DRAM, SRAM, other types of memory, or combinations thereof. Level buffers are configured to store I / O requests received from object storage system 120 via NIC 201, and to store data (e.g., EC-encoded data blocks) or combinations thereof obtained from drives 208a to 208d. For example, all level buffers 202a to 202d may be configured to store I / O requests received from object storage system 120, all level buffers 202a to 202d may be configured to store data blocks from drives, or a first subset of level buffers 202a to 202d may be configured to store data blocks, and a second subset of level buffers 202a to 202d may be configured to store I / O requests. In some embodiments, storage device 200 includes one buffer for each drive included in storage device 200. In other embodiments, the storage device 200 includes a different number of buffers (e.g., one buffer per drive plus one or more output buffers).

[0029] As further described herein, in embodiments where one or more of the level buffers are configured to receive I / O requests, processor unit 205 may assign a specific I / O request to a specific one of level buffers 202a to 202d based on a GFID associated with the I / O request. Level buffers 202a to 202d may have associated priorities, and I / O requests stored in level buffers 202a to 202d may be processed from level buffers 202a to 202d in an order determined based on priority. Similarly, as further described herein, in embodiments where one or more of the level buffers are configured to receive data from drivers 208a to 208d, processor unit 205 may assign specific data (e.g., a specific block) to a specific level buffer based on a GFID associated with an I / O request for the acquisition of specific data. As described above, level buffers 202a to 202d may have associated priorities. Data stored in level buffers 202a to 202d can be sent from level buffers 202a to 202d in a priority-determined order (e.g., to object storage system 120 or to one or more of AI PUs 112a to 112h).

[0030] Storage device 200 also includes bus 206. Bus 206 may include a Peripheral Component Interconnect Fast (PCIe) bus or other types of bus. Bus 206 connects drives 208a to 208d to level buffers 202a to 202d. NIC 201 and / or processor unit 205 may also be connected to bus 206. Figure 2 The diagram depicts I / O 204a to 204d flowing between drives 208a to 208d and level buffers 202a to 202d. These I / O 204a to 204d may include data blocks, I / O requests, or combinations thereof. In some embodiments, the storage device 200 includes additional components that can be further connected to bus 206.

[0031] Reference Figure 3 This diagram illustrates the operation of a system used for time-synchronized storage delivery. Figure 3 In the example, Figure 1 Object storage system 120 and AI PUs 112a to 112h (e.g., directly or via one or more networks) are connected to storage devices 200a to 200d. While four storage devices 200a to 200d are shown, in other examples, object storage system 120 and / or AI PUs 112a to 112h may be connected to a different number of storage devices. Furthermore, in some examples, a different number of AI PUs than shown in the illustrated example may be included. Each of the storage devices 200a to 200d is... Figure 2 The storage device 200 shown is an example. It should be noted that although storage devices 200a to 200d are examples of storage device 200, storage devices 200a to 200d may not be the same. For example, as described above, storage devices 200 may have different configurations (e.g., different numbers or uses of level buffers, etc.), and the first storage device 200a may have one of these configurations, while the second storage device 200b may have a second of these configurations.

[0032] Furthermore, object storage system 120, AI PUs 112a to 112h, and storage devices 200a to 200d (e.g., via NIC 122 and NIC 201, respectively) are connected to GFID database 302. GFID database 302 is implemented on a computing device. In some examples, GFID database 302 is implemented on... Figure 1 The AI ​​host device 102 (e.g., on one of AI PUs 112a to 112h or on a separate processor of the AI ​​host device 102). In some examples, the GFID database 302 is implemented on the object storage system 120 or on one of the storage devices 200a to 200d. In some examples, the GFID database 302 is implemented on the AI ​​PUs 112a to 112h, the storage devices 200a to 200d, and a computing device external to the object storage system 120.

[0033] The GFID database 302 is configured to store metadata that identifies resources associated with an AI model training session (or AI training session). For example, the GFID database 302 may store associations between GFIDs, data associated with an AI PU, data associated with one or more storage devices, and data associated with one or more object storage systems. Data associated with an AI PU may include the AI ​​PU's identifier (ID), the connection identifier associated with the AI ​​PU, the bandwidth associated with the AI ​​PU, or a combination thereof. Data associated with one or more object storage systems may include the object storage system identifier for each object storage system, the connection identifier for each object storage system, the bandwidth associated with each object storage system, or a combination thereof. Data associated with one or more storage devices may include the identifier for each storage device, the connection identifier associated with each storage device, the bandwidth associated with each storage device, or a combination thereof.

[0034] Reference Figure 4 This shows a diagram illustrating example entry 402 in the GFID database 302. Figure 4 In the example shown, entry 402 includes a global stream ID 404, data 406 associated with the AI ​​PU, data 414 associated with one or more object storage systems, and data 422 associated with one or more storage devices. The global stream ID 404 may be a unique identifier that identifies an AI training session (or other data stream).

[0035] Data 406 associated with the AI ​​PU describes the AI ​​PU involved in the AI ​​training session (or other data stream). Data 406 associated with the AI ​​PU includes an entity identifier or name 408. The entity identifier or name 408 may include a network address (e.g., an Internet Protocol address), a hostname, another identifier, or a combination thereof. Data 406 also includes a connection identifier 410 associated with the AI ​​PU. The connection identifier 410 may include a socket identifier, other types of connection identifiers, or a combination thereof that identify the connection through which the AI ​​PU identified by the entity identifier or name 408 can send and / or receive data. Data 406 associated with the AI ​​PU also includes a bandwidth indicator 412. The bandwidth indicator 412 indicates the bandwidth available to the AI ​​PU for receiving data during the AI ​​training session (or other data stream).

[0036] Data 414 associated with one or more object storage systems includes an object storage system identifier 416 for each object storage system associated with the AI ​​training session (or other data stream). The object storage system identifier 416 may include a network address (e.g., an Internet Protocol address), a hostname, other identifiers, or combinations thereof. Data 414 associated with one or more object storage systems also includes a connection identifier 418 for each object storage system associated with the AI ​​training session (or other data stream). The connection identifier 418 may include a socket identifier, other types of connection identifiers, or combinations thereof that identify the connection through which the object storage system identified by the object storage system identifier 416 can send and / or receive data. Data 414 associated with one or more object storage systems also includes a bandwidth indicator 420 for each object storage system associated with the AI ​​training session (or other data stream). The bandwidth indicator 420 indicates the bandwidth available for the object storage system to send and / or receive data during the AI ​​training session (or other data stream).

[0037] Data 422 associated with one or more storage devices includes a target identifier 424 for each storage device associated with an AI training session (or other data stream). The target identifier 424 may include a network address (e.g., an Internet Protocol address), a hostname, another identifier, or a combination thereof. Data 422 associated with one or more storage devices also includes a connection identifier 426 for each storage device associated with an AI training session (or other data stream). The connection identifier 426 may include a socket identifier, other types of connection identifiers, or a combination thereof that identify the connection through which the storage device identified by the target identifier 424 can send and / or receive data. Data 422 associated with one or more storage devices also includes a bandwidth indicator 428 for each storage device associated with an AI training session (or other data stream). The bandwidth indicator 428 indicates the bandwidth available for the storage device to send and / or receive data during the AI ​​training session (or other data stream).

[0038] therefore, Figure 4 An example of an entry in the Global Stream Identifier database is depicted, which associates a global stream identifier with metadata describing the various resources used during an AI training session (or other data stream).

[0039] Return to reference Figure 3During an AI model training session (or other data stream), a first AI PU 112a (or any other AI PU) may send I / O commands (e.g., read requests) to object storage system 120. The first AI PU 112a may include a GFID associated with the AI ​​training session (or other data stream) in the I / O command. In some implementations, the first AI PU 112a may limit the rate at which I / O commands are sent to object storage system 120 based on a bandwidth indicator 420 associated with object storage system 120, as identified by GFID database 302.

[0040] In response to receiving an I / O command, the processor unit 121 of the object storage system 120 may assign a timestamp to the I / O command and then place the I / O command in one of the memory buffers 124a to 124h. Assigning a timestamp to the I / O command may include adding the timestamp to a record (e.g., a command control block) associated with the I / O command. The record may correspond to a metadata block associated with the I / O command. In some examples, the processor unit 121 of the object storage system 120 looks up the GFID in the I / O command within the GFID database 302 and determines which memory buffer to place the I / O command in based on the data associated with the GFID. For example, the processor unit 121 of the object storage system 120 may place the I / O command in the first memory buffer 124a based on the bandwidth associated with the first AIPU 112a as identified in the GFID database 302 (e.g., the bandwidth indicated by the bandwidth indicator 412). In some examples, the processor unit 121 of the object storage system 120 may maintain a priority-to-bandwidth range mapping (e.g., a first bandwidth range may be mapped to a first priority, a second bandwidth range may be mapped to a second priority, etc.). The processor unit 121 of the object storage system 120 can identify the bandwidth range into which the bandwidth of the AI ​​PU associated with the GFID identified by the IO command falls, and then add the IO command to the memory buffers 124a to 124h that correspond to the priority mapped to the bandwidth range.

[0041] In some examples, the timestamp assigned to an IO command by the processor unit 121 of the object storage system 120 is based on a GFID (e.g., based on the bandwidth associated with the GFID). For example, the processor unit 121 of the object storage system 120 may look up a GFID in the GFID database 302 to identify the bandwidth associated with the AI ​​PU associated with the GFID, and then assign a timestamp based on the bandwidth. For example, the assigned timestamp may correspond to the time when the IO command is received by the object storage system 120 plus an offset based on the bandwidth selection. For illustration, the object storage system 120 may receive a first IO command including a first GFID and a second IO command including a second GFID via the NIC 122. The first GFID may be associated with a first AI PU 112a in the GFID database 302, and the second GFID may be associated with a second AI PU 112b in the GFID database 302. The entry for the first GFID may also indicate that the bandwidth associated with the first AI PU 112a is X, while the entry for the second GFID may indicate that the bandwidth associated with the second AI PU 112b is Y. Object storage system 120 may maintain an association (e.g., mapping) between bandwidth and offset (e.g., in external or internal memory). Processor unit 121 of object storage system 120 may set a first offset based on X (e.g., +5 milliseconds (ms)) and a second offset based on Y (e.g., +3 ms). In another embodiment, processor unit 121 may allocate a timestamp (n) of a specific I / O command associated with a GFID based on the bandwidth of the GFID and based on a previous timestamp (n-1) of a previous I / O command associated with the GFID. For illustration, processor unit 121 may allocate timestamp n such that the time between timestamp n-1 and timestamp n satisfies the data rate indicated by the bandwidth associated with the GFID. In some embodiments, processor unit 121 allocates timestamps of I / O commands associated with a GFID such that (e.g., regardless of when the I / O command is received by object storage system 120) the average time between the timestamps of the GFID satisfies the bandwidth associated with the GFID. As further explained below, object storage devices may be configured to use timestamps to schedule responses to commands. Therefore, setting the timestamp of IO commands based on the GFID associated with the AI ​​training session can provide priority-based scheduling for the AI ​​training session, in which data is exchanged between several devices. This priority scheme can be used to ensure that system 100 meets the bandwidth (e.g., the rate at which the AI ​​PU receives data) and latency (latency represents the time difference between when the AI ​​PU sends a command and when the storage device 200 sends a response) requirements of the AI ​​training session.

[0042] The processor unit 121 of the object storage system 120 is configured to process an I / O command from one of the memory buffers 124a to 124h by removing the I / O command from memory buffers 124a to 124h and identifying which of the storage devices 200a to 200d stores the data (e.g., a data block) indicated by the I / O command. The processor unit 121 of the object storage system 120 is configured to generate a request for each data block and send the request to the corresponding storage device. The request includes the GFID from the I / O command, the identifier of the requested data block, and a timestamp assigned to the I / O command. If the I / O command identifies an object stored (or partially stored) in the storage device that is not associated with a GFID in the GFID database 302, the processor unit 121 of the object storage system 120 may (e.g., via NIC 122) return an error message to the AI ​​PU. The processor unit 121 of the object storage system 120 may process the I / O commands from the memory buffers 124a to 124h in an order determined based on the priority associated with the memory buffers 124a to 124h. Furthermore, in some implementations, the processor unit 121 of the object storage system 120 may limit the rate at which requests are sent to a particular one of the storage devices 200a to 200d based on bandwidth 428 associated with the storage device, as identified by the GFID database 302.

[0043] Storage devices 200a to 200d (e.g., processor unit 205 of storage devices 200a to 200d) can be configured to send data (e.g., data blocks) in an order determined based on timestamps included in associated requests. Storage devices 200a to 200d can send data directly to the requesting AI PU or to the object storage system 120 via NIC 201. For example, the first storage device 200a can receive a request for a first data block via NIC 201 of the first storage device 200a. Processor unit 205 of the first storage device 200a can insert requests into one of the level buffers 202a to 202d based on timestamps in the requests. For example, the position where a request is added to a level buffer can be determined based on timestamps, such that requests included in the level buffers are ordered by timestamp order.

[0044] Furthermore, the processor unit 205 may select the level buffer to which the request is inserted based on the bandwidth (e.g., bandwidth 428) associated with the GFID in the request. The request may be processed from the level buffer (e.g., in order of priority based on the level buffer) and sent to bus 206 to be received by a driver (e.g., one of drivers 208a to 208d) that stores the requested data (e.g., a data block).

[0045] The driver can return data to bus 206 for output by storage device 200 via NIC 201 to the AI ​​PU associated with the GFID. For example, processor unit 205 can initiate a transfer of a query to GFID database 302 via NIC 201 to identify which AI PU is associated with the request, and then initiate a transfer of data output by the driver to that AI PU via NIC 201. In some embodiments, storage device 200 sends data (e.g., data blocks) to object storage system 120 for forwarding to the AI ​​PU.

[0046] In some implementations, processor unit 205 adds data to one of the level buffers 202a to 202d associated with the buffered output. Data may be added to the level buffer based on the GFID and / or timestamp associated with the request. Furthermore, in some implementations, processor unit 205 may limit the rate at which data is returned to the AI ​​PU based on a bandwidth indicator 412 associated with the AI ​​PU, such as that identified by the GFID database 302.

[0047] Because the storage device processes requested and output data in timestamp order, objects can be retrieved synchronously. Furthermore, timestamps can be assigned based on priority (e.g., bandwidth) corresponding to the GFID that identifies the AI ​​training session. Therefore, AI training sessions can be prioritized. Additionally, data rates between components within an AI training session can be managed based on the bandwidth linked to the GFID of the AI ​​training session.

[0048] Reference Figure 5 The diagram illustrates a flowchart of a method 500 for generating a storage query plan for an AI training session. Method 500 can be derived from... Figure 1 The AI ​​host device 102 executes.

[0049] The method includes reading next-store service session parameters at 502. For example, AI host device 102 may read parameters (e.g., workload parameters) associated with the AI ​​training session (e.g., from an input file or other source). These parameters may include the target bandwidth of the AI ​​training session, dataset, training time, bandwidth per GPU, objects / data to be read, number of GPUs, size of the prefetch buffer, other parameters, or combinations thereof. Parameters may be input by data loader 110.

[0050] Method 500 also includes reading system and network topology information at 504. For example, AI host device 102 can determine Figure 3The system network topology is shown in the diagram. AI host device 102 can use Internet Control Message Protocol (ICMP), Link Layer Discovery Protocol (LLDP), other tools, or combinations thereof to determine the network topology. Determining the network topology information may also include determining the bandwidth available at various devices in the network. For example, AI host device 102 may determine the available bandwidth of storage device 200, the available bandwidth of object storage system 120, the available bandwidth of AI PUs 112a to 112h, or combinations thereof.

[0051] Method 500 also includes reading storage capacity information at 506. For example, AI host device 102 may query storage devices 200a to 200d, object storage system 120, or a combination thereof to determine the storage capacity available for AI training sessions.

[0052] Method 500 also includes analyzing network topology information and storage capacity information in step 508, creating a storage query plan (SQP), and generating global stream database entries that associate GFID with resources associated with the AI ​​training session.

[0053] In response to determining at 510 that the topology and storage capacity do not support storage system parameters, method 500 includes reading the next storage service session parameters at 502.

[0054] In response to determining at 510 that the topology and storage capacity do indeed support storage system parameters, method 500 includes distributing global streaming database entries at 512. For example, AI host device 102 may distribute database entries to GFID database 302.

[0055] Method 500 also includes setting up and initializing storage queries and related streams at 514. For example, AI host device 102 may initialize a connection to object storage system 120. This connection may have bandwidth indicated by entries stored in GFID database 302. AI host device 102 may also initialize a connection to one or more storage devices identified by GFID database entries.

[0056] Method 500 also includes launching the application at 516. For example, AI host device 102 may launch AI application 108 and begin sending IO commands to object storage system 120 as part of an AI training session.

[0057] Reference Figure 6 The diagram illustrates a flowchart of a method 600 for sending a time synchronization request to a storage device. Method 600 may be performed by an object storage system (such as object storage system 120) or by another type of storage system.

[0058] Method 600 includes receiving a first input / output (IO) command from a first AI PU at 606, the first IO command being associated with a first AI model training operation. For example, object storage system 120 may receive the first IO command from first AI PU 112a via NIC 122. The first IO command may include a first GFID associated with a first AI training session performed by first AI PU 112a. For example, the first GFID identifies the first AI model training operation.

[0059] Method 600 also includes receiving a second IO command from a second AI PU at 608, the second IO command being associated with a second AI model training operation. For example, object storage system 120 may receive the second IO command from second AI PU 112b via NIC 122. The second IO command may include a second GFID associated with a second AI training session performed by second AI PU 112b. For example, the second GFID identifies a second AI model training operation.

[0060] Method 600 further includes assigning a first timestamp to a first IO command at 610 based on a first bandwidth allocated to a first AI model training operation. For example, processor unit 121 of object storage system 120 may query GFID database 302 to determine a first bandwidth indicator 412 associated with a GFID included in the first IO command. Processor unit 121 of object storage system 120 may assign the first timestamp to the first IO command based on the first bandwidth indicated by the first bandwidth indicator 412. In some examples, the first timestamp corresponds to the time when the first IO command is received at object storage system 120 plus an offset determined based on the first bandwidth.

[0061] Method 600 further includes assigning a second timestamp to the second I / O command at 612 based on a second bandwidth allocated to the second AI model training operation. For example, object storage system 120 may query GFID database 302 to determine a second bandwidth indicator 412 associated with a GFID included in the second I / O command. Processor unit 121 of object storage system 120 may assign the second timestamp to the second I / O command based on the second bandwidth indicated by the second bandwidth indicator 412. In some examples, the second timestamp corresponds to the time when the second I / O command is received at object storage system 120 plus an offset determined based on the second bandwidth.

[0062] Data requests sent by object storage system 120 based on I / O commands may include timestamps assigned to the I / O commands. Storage devices receiving data requests can output data in a time-stamp-determined order. Therefore, data can be output by the storage devices in a time-synchronized manner.

[0063] Reference Figure 7The flowchart illustrates a method 700 for sending data from a storage device in a time-synchronized manner.

[0064] Method 700 includes receiving a first request for a first data block at 702. The first request includes a first timestamp and a first global stream identifier (GFID). For example, a first storage device 200a may receive the first request from object storage system 120 via NIC 201. The first request may request the first data block, and the first request may include the first timestamp and may include the first GFID.

[0065] Method 700 further includes receiving a second request for a second data block at 704, the second request including a second timestamp and a second GFID. For example, the first storage device 200a may receive the second request from the object storage system 120 via NIC 201. The second request may request a second data block, and may include the second timestamp and may include the second GFID.

[0066] Method 700 also includes identifying the first AI PU based on the first GFID at 706. For example, the processor unit 205 of the first storage device 200a can query the GFID database 302 via NIC 201 to determine that the first AI PU 112a is associated with the first GFID.

[0067] Method 700 also includes identifying a second AI model processing unit based on a second GFID at 708. For example, the processor unit 205 of the first storage device 200a can query the GFID database 302 via NIC 201 to determine that the second AI PU 112b is associated with a second GFID.

[0068] Method 700 further includes sending a first data block to a first AI PU at 710, wherein the timing of the transmission of the first data block is based on a first timestamp. For example, processor unit 205 of first storage device 200a may initiate the transmission of the first data block to first AI PU 112a via NIC 201. Processor unit 205 of first storage device 200a may schedule the processing of a first request from one of level buffers 202a to 202d and / or schedule the transmission of the first data block to first AI PU 112a based on the first timestamp. In some embodiments, processor unit 205 of first storage device 200a waits until a time indicated by the first timestamp to send the first data block to first AI PU 112a.

[0069] Method 700 further includes sending a second data block to a second AI model processing unit at 712, wherein the timing of the transmission of the second data block is based on a second timestamp. For example, the processor unit 205 of the first storage device 200a may initiate the transmission of the second data block to the second AI PU 112b via NIC 201. The processor unit 205 of the first storage device 200a may schedule the processing of a second request from one of the level buffers 202a to 202d and / or schedule the transmission of the second data block to the second AI PU 112b based on the second timestamp. In some embodiments, the processor unit 205 of the first storage device 200a waits until the time indicated by the second timestamp to send the second data block to the second AI PU 112b.

[0070] Therefore, method 700 can be used by a storage device to send data blocks in a time-synchronized manner. Since the timestamps used by the storage device for synchronous transmission can be based on the AI ​​training session priority, the transmission of data can take into account the relative priority of the AI ​​training session.

[0071] Reference Figure 8 The diagram 800 shows an example of the first AI PU 112a sending three IO commands for a specific AI training session. The sequence diagram shows the first storage device 200a through the nth storage device 200n. As described above, Figure 3 The system may include a different number of storage devices than the four storage devices shown.

[0072] At 802, the sequence includes a first AI PU 112a connecting to the object storage system 120. The first AI PU 112a may initiate the connection in response to determining that a storage query plan has been successfully established for the AI ​​training session and that the storage query plan allocates the object storage system 120 to the AI ​​training session. In the example shown, the AI ​​training session has a GFID of “ABC”. In one example, “ABC” as a GFID could be 100. Therefore, the GFID database 302 stores GFID database entries that associate the GFID “ABC” with the object storage system 120.

[0073] At 804, the sequence includes a first AI PU 112a connected to a first storage device 200a. The first AI PU 112a may initiate this connection in response to determining a storage query plan to allocate the first storage device 200a to an AI training session. A GFID database entry associates the GFID “ABC” with the first storage device 200a.

[0074] At 806, the sequence includes a first AI PU 112a connected to a second storage device 200b. The first AI PU 112a may initiate this connection in response to determining a storage query plan to allocate the second storage device 200b to an AI training session. A GFID database entry associates the GFID “ABC” with the second storage device 200b.

[0075] At 808, the sequence includes a first AI PU 112a connected to the nth storage device 200n. The first AI PU 112a may initiate the connection in response to determining a storage query plan to allocate the nth storage device 200n to an AI training session. A GFID database entry associates the GFID "ABC" with the nth storage device 200n.

[0076] Sequence 800 also includes AI PU 112a sending a first IO command to object storage system 120 at 810, a second IO command to object storage system 120 at 812, and a third IO command to object storage system 120 at 814. In the example shown, the first IO command, the second IO command, and the third IO command are GET commands. Figure 8 In this configuration, the first IO command, the second IO command, and the third IO command are represented by S3 GET Cmd TS0, S3 GET Cmd TS1, and S3 GET Cmd TS2, respectively. The first IO command (e.g., by key-value pair) identifies a first data object, the second IO command identifies a second data object, and the third IO command identifies a third data object. The object storage system 120 assigns a first timestamp to the first IO command, a second timestamp to the second IO command, and a third timestamp to the third IO command. Timestamps are generated based on the priority (e.g., bandwidth) associated with GFID "ABC" in the GFID database 302.

[0077] For each command, object storage system 120 then generates a request for a data block associated with the object identified by the IO command. The request includes a timestamp assigned to the IO command and is sent by object storage system 120 to the storage device identified in GFID database 302 as associated with GFID "ABC".

[0078] exist Figure 8In the example shown, object storage system 120 generates a first request, a second request, and a third request (e.g., EC-0TS0, EC-1TS0, and EC-NTS0) for a data block based on a first IO command. Each of these requests includes a first timestamp assigned to the first IO command. Object storage system 120 also generates a fourth, a fifth, and a sixth request (e.g., EC-0TS1, EC-1TS1, and EC-NTS1) for a data block based on a second IO command. Each of these requests includes a second timestamp assigned to the second IO command. Object storage system 120 also generates a seventh, an eighth, and a ninth request (e.g., EC-0TS2, EC-1TS2, and EC-NTS2) for a data block based on a third IO command. Each of these requests includes a third timestamp assigned to the third IO command.

[0079] In sequence diagram 800, object storage system 120 sends a first request to first storage device 200a at 816, a seventh request to first storage device 200a at 818, an eighth request to second storage device 200b at 820, a fourth request to first storage device 200a at 822, a second request to second storage device 200b at 824, a third request to nth storage device 200n at 826, a ninth request to nth storage device 200n at 828, a fifth request to second storage device 200b at 830, and a sixth request to nth storage device 200n at 832. Therefore, requests can be sent out of order by object storage system 120 (and received by storage devices 200a through 200n).

[0080] Storage devices 200a to 200n are configured to output data blocks in an order determined based on timestamps included in the requests. For example, first storage device 200a may receive a first request, a fourth request, and a seventh request, insert the first, fourth, and seventh requests into a first-level buffer 202a in an order based on their respective timestamps (which may differ from the order in which the corresponding requests were received), and then process the requests from the first-level buffer 202a in a sequence-based order. Alternatively or additionally, first storage device 200a may insert the data block associated with the first request, the data block associated with the fourth request, and the data block associated with the seventh request into a second-level buffer 202b in a timestamp-based order, and then send the data blocks from the second-level buffer 202b in a sequence-based order. In some embodiments, data blocks are removed from the first-level buffer 202b and sent to the first AI PU 112a at a time indicated by a timestamp associated with the data block (or approximately at a time indicated by a timestamp associated with the data block (e.g., within 3 ms)).

[0081] Therefore, in the illustrated sequence 800, at 834, a data block in response to the first request (e.g., 'EC-0TS0') is output to the first AI PU 112a; at 836, a data block associated with the second request (e.g., 'EC-1TS0') is output to the first AI PU 112a; at 838, a data block associated with the third request (e.g., 'EC-NTS0') is output to the first AI PU 112a; at 840, a data block associated with the fourth request (e.g., 'EC-0TS1') is output to the first AI PU 112a; at 842, a data block associated with the fifth request (e.g., 'EC-1TS1') is output to the first AI PU 112a; and at 844, a data block associated with the sixth request (e.g., 'EC-NTS1') is output to the first AI PU 112a. At 846, the data block associated with the seventh request (e.g., 'EC-0TS2') is output to the first AI PU 112a; at 848, the data block associated with the eighth request (e.g., 'EC-1TS2') is output to the first AI PU 112a; and at 850, the data block associated with the ninth request (e.g., 'EC-NTS2') is output to the first AI PU 112a. It should be noted that, in addition to determining the order of returned data, the timestamps set by the object storage system 120 also determine the rate at which data is delivered to the AI ​​PUs. Therefore, the data blocks returned to the first AI PU 112a are time-synchronized and delivered at an appropriate rate. In one example, the data block associated with the first request corresponds to the first erase-coded block of the data object requested by the first IO command, and the data block associated with the second request corresponds to the second erase-coded block of the data object.

[0082] In the illustrated example, object storage system 120 sends a first completion notification (e.g., S3 Completion TS0) to first AI PU 112a at 852, a second completion notification (e.g., S3 Completion TS1) to first AI PU 112a at 854, and a third completion notification (e.g., S3 Completion TS2) to first AI PU 112a at 856. The completion notifications can be sent based on a time associated with a timestamp. For example, the first completion notification can be sent by object storage system 120 at a time indicated by a first timestamp. Therefore, first AI PU 112a can be notified that all data blocks associated with the first IO command should have been sent.

[0083] The above example describes an implementation of an AI host device interacting with an object storage system. However, the system may include more than one AI host device and more than one object storage system. More than one AI host device may interact with an object storage system. Furthermore, an AI host device may interact with more than one object storage system. Figure 9 The example system shown includes multiple AI host devices and multiple object storage systems. Figure 9 The system's composition and operation can be similar to those described above. Figures 1 to 8 The system and operation described.

[0084] The foregoing is illustrative of exemplary embodiments and should not be construed as limiting them. Although some exemplary embodiments have been described, those skilled in the art will readily understand that many modifications can be made to the exemplary embodiments without substantially departing from the novelty and advantages of the exemplary embodiments. Therefore, all such modifications are intended to be included within the scope of the exemplary embodiments as defined in the claims. In the claims, the device plus function clause is intended to cover the structures described herein that perform the functions, and not only structural equivalents but also equivalent structures. Therefore, it should be understood that the foregoing is illustrative of exemplary embodiments and should not be construed as limiting to the specific embodiments disclosed, and modifications to the disclosed exemplary embodiments and other exemplary embodiments are intended to be included within the scope of the appended claims.

[0085] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising,” “having,” and “including” are used in this specification, they indicate the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0086] As used herein, the word “may” in describing embodiments of this disclosure means “one or more embodiments of this disclosure”. As used herein, the terms “use,” “in use,” and “being used” are to be understood as synonyms with the terms “exploit,” “being exploited,” and “being exploited,” respectively. Furthermore, the term “exemplary” is intended to indicate an example or illustration.

[0087] When specific embodiments can be implemented differently, a particular order of processing may be performed in a manner different from that described. For example, two consecutively described processing or method steps may be performed substantially simultaneously or in an order different from that described.

[0088] Electronic or electrical devices and / or any other related devices or components according to embodiments of the present disclosure described herein can be implemented using any suitable hardware, firmware (e.g., application-specific integrated circuits), software, or a combination of software, firmware, and hardware. For example, various components of these devices may be formed on an integrated circuit (IC) chip or a separate IC chip. Furthermore, various components of these devices may be implemented on a flexible printed circuit film, a tape-on-a-package (TCP), a printed circuit board (PCB), or formed on a substrate. Additionally, various components of these devices may be processes or threads executing computer program instructions and interacting with other system components on one or more processors in one or more computing devices to perform the various functions described herein. The computer program instructions are stored in memory, which may be implemented in the computing device using standard memory devices, such as random access memory (RAM). The computer program instructions may also be stored on other non-transitory computer-readable media, such as CD-ROMs, flash drives, etc. Furthermore, those skilled in the art will recognize that, without departing from the spirit and scope of the exemplary embodiments of the present disclosure, the functions of various computing devices may be combined or integrated into a single computing device, or the functions of a particular computing device may be distributed across one or more other computing devices.

[0089] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will also be understood that, unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formalized sense.

Claims

1. A method for time-synchronized storage and delivery, comprising: The first computing device receives a first input / output command from the first artificial intelligence model processing unit, the first input / output command being associated with the first artificial intelligence model training operation; The first computing device receives a second input / output command from the second artificial intelligence model processing unit, the second input / output command being associated with the training operation of the second artificial intelligence model; Based on the first bandwidth allocated to the training operation of the first artificial intelligence model, the first timestamp is assigned to the first input / output command; as well as The second timestamp is assigned to the second input / output command based on the second bandwidth allocated to the training operation of the second artificial intelligence model.

2. The method according to claim 1, wherein, The first input / output command includes a first Global Stream Identifier (GFID) that identifies a first AI model training operation, and the second input / output command includes a second GFID that identifies a second AI model training operation, wherein the method further includes: Based on the first GFID, an indication of the first bandwidth associated with the first artificial intelligence model processing unit is retrieved from the database; and Based on the second GFID, an indication of the second bandwidth associated with the second artificial intelligence model processing unit is retrieved from the database.

3. The method of claim 1 or 2, further comprising: Based on the first input / output command, a first request for the first data block is sent to the first storage device, the first request including a first timestamp.

4. The method of claim 3, further comprising: Based on the first input / output command, a second request for the second data block is sent to the second storage device, the second request including the first timestamp.

5. The method according to claim 4, further comprising: Based on the second input / output command, a third request for the third data block is sent to the third storage device, the third request including the second timestamp.

6. The method according to claim 4, wherein, The first input / output command requests a data object, wherein the first data block corresponds to a first erase code block of the data object, and the second data block corresponds to a second erase code block of the data object.

7. The method according to claim 3, wherein, The first storage device corresponds to the fast NVMe-oF target of non-volatile memory via network architecture.

8. A method for time-synchronized storage and delivery, comprising: The storage device receives a first request for a first data block, the first request including a first timestamp and a first global stream identifier (GFID); The storage device receives a second request for a second data block, the second request including a second timestamp and a second GFID; The storage device identifies the first artificial intelligence model processing unit based on the first GFID; The storage device identifies the second artificial intelligence model processing unit based on the second GFID; Sending a first data block from the storage device to the first artificial intelligence model processing unit, wherein the timing of the sending of the first data block is based on a first timestamp; and The second data block is sent from the storage device to the second artificial intelligence model processing unit, wherein the timing of the sending of the second data block is based on the second timestamp.

9. The method according to claim 8, wherein, The storage device includes a fast NVMe-oF target for non-volatile memory via a network architecture, wherein the NVMe-oF target includes multiple solid-state drives.

10. The method according to claim 8, wherein, The first artificial intelligence model processing unit includes a graphics processor.

11. The method according to claim 8, wherein, The first artificial intelligence model processing unit includes applications that execute on the processor.

12. The method according to claim 8, wherein, The first data block corresponds to the erase code block of the data object.

13. The method according to claim 8, wherein, In response to the first timestamp being before the second timestamp, the first data block was sent before the second data block.

14. The method according to any one of claims 8 to 13, wherein, The step of identifying the first artificial intelligence model processing unit includes sending a query to identify the first GFID to the GFID database.

15. A computing device, comprising: Network interface; as well as The processor unit is configured as follows: Receive a first input / output command from the first artificial intelligence model processing unit; the first input / output command is associated with the training operation of the first artificial intelligence model. The second input / output command is received from the second artificial intelligence model processing unit, and the second input / output command is associated with the training operation of the second artificial intelligence model. Based on the first bandwidth allocated to the training operation of the first artificial intelligence model, the first timestamp is assigned to the first input / output command; as well as The second timestamp is assigned to the second input / output command based on the second bandwidth allocated to the training operation of the second artificial intelligence model.

16. The computing device according to claim 15, wherein, The first input / output command includes a first global stream identifier (GFID) that identifies a first AI model training operation, and the second input / output command includes a second GFID that identifies a second AI model training operation, wherein the processor unit is further configured to: Based on the first GFID, an indication of the first bandwidth associated with the first artificial intelligence model processing unit is retrieved from the database; and Based on the second GFID, an indication of the second bandwidth associated with the second artificial intelligence model processing unit is retrieved from the database.

17. The computing device according to claim 15 or 16, wherein, The processor unit is also configured to: initiate the transmission of a first request for a first data block to a first storage device via a network interface based on a first input / output command, the first request including a first timestamp.

18. The computing device according to claim 17, wherein, The processor unit is also configured to: initiate the transmission of a second request for a second data block to a second storage device via a network interface based on a first input / output command, the second request including a first timestamp.

19. The computing device according to claim 18, wherein, The processor unit is also configured to: initiate the transmission of a third request for a third data block to a third storage device via a network interface based on a second input / output command, the third request including a second timestamp.

20. The computing device according to claim 18, wherein, The first input / output command requests a data object, wherein the first data block corresponds to a first erase code block of the data object, and the second data block corresponds to a second erase code block of the data object.

21. The computing device according to claim 17, wherein, The first storage device corresponds to the fast NVMe-oF target of non-volatile memory via network architecture.

22. A storage device, comprising: Network interface; as well as The processor unit is configured as follows: Receive a first request for a first data block, the first request including a first timestamp and a first global stream identifier (GFID); Receive a second request for a second data block, the second request including a second timestamp and a second GFID; The first artificial intelligence model processing unit is based on the first GFID recognition; The second artificial intelligence model processing unit is based on the second GFID identification; Initiate the transmission of a first data block to a first artificial intelligence model processing unit via a network interface, wherein the timing of the transmission of the first data block is based on a first timestamp; and Initiate the transmission of a second data block to a second artificial intelligence model processing unit via a network interface, wherein the timing of the transmission of the second data block is based on a second timestamp.

23. The storage device according to claim 22, wherein, The storage device includes a fast NVMe-oF target with non-volatile memory via a network architecture, and the NVMe-oF target includes multiple solid-state drives.

24. The storage device according to claim 22, wherein, The first data block corresponds to the erase code block of the data object.

25. The storage device according to claim 22, wherein, In response to the first timestamp being before the second timestamp, the first data block was sent before the second data block.

26. The storage device according to any one of claims 22 to 25, wherein, The step of identifying the first artificial intelligence model processing unit includes sending a query to identify the first GFID to the GFID database.

Images