Method and apparatus for time-synchronized storage delivery - Patents.com

Abstract

To provide a storage system and method for carrying out time-synchronized storage delivery.SOLUTION: The method comprises: a step in which a first computing device receives a first input / output (I / O) command related to a first AI model training operation from a first artificial intelligence model processing unit (AI PU); and a step in which the first computing device receives a second I / O command related to a second AI model training operation from a second AI PU. The method further includes: a step in which a first timestamp is assigned to the first I / O command on the basis of a first bandwidth allocated to the first AI model training operation; and a step in which a second timestamp is assigned to the second I / O command on the basis of a second bandwidth allocated to the second AI model training operation.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The invention disclosed herein relates to a storage system and method for performing time-synchronized storage delivery. [Background technology]

[0002] Artificial intelligence (AI) models (e.g., variational autoencoders) are trained using large datasets. AI systems may include thousands of graphics processing units (GPUs) configured to process large datasets in parallel to adequately train one or more AI models. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 9,825,652 [Patent Document 2] U.S. Pat. No. 1,058,5749 [Patent Document 3] US Patent Publication No. 20170308321 [Patent Document 4] US Patent Publication No. 20190188079 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide a storage system and method for performing time-synchronized storage delivery. [Means for solving the problem]

[0005] A storage system and method for performing time-synchronized storage delivery is disclosed that can be used to support processing large data sets during training of artificial intelligence models.

[0006] The method includes a first computing device receiving a first input / output (I / O) command from a first artificial intelligence model processing unit (AI PU), the first I / O command associated with a first AI model training operation. The method further includes the first computing device receiving a second I / O command from a second AI PU, the second I / O command associated with a second AI model training operation. The method further includes assigning a first timestamp to the first I / O command based on a first bandwidth allocated to the first AI model training operation. The method further includes assigning a second timestamp to the second I / O command based on a second bandwidth allocated to the second AI model training operation.

[0007] The method includes a step of receiving a first request for a first data chunk by the storage device, the first request including a first timestamp and a first global flow identifier (GFID). The method further includes a step of receiving a second request for a second data chunk by the storage device, the second request including a second timestamp and a second GFID. The method further includes a step of identifying a first artificial intelligence (AI) model processor by the storage device based on the first GFID. The method further includes a step of identifying a second AI model processor by the storage device based on the second GFID. The method further includes a step of transferring the first data chunk from the storage device to the first AI model processor, the transfer timing of the first data chunk being based on the first timestamp. The method further includes a step of transferring the second data chunk from the storage device to the second AI model processor, the transfer timing of the second data chunk being based on the second timestamp.

[0008] The computing device includes a network interface and a processor unit. The processor is configured to receive a first input / output (I / O) command from a first artificial intelligence model processing unit (AI PU), the first I / O command being associated with a first AI model training operation. The processor unit is further configured to receive a second I / O command from a second AI PU, the second I / O command being associated with a second AI model training operation. The processor unit is further configured to assign a first timestamp to the first I / O command based on a first bandwidth allocated to the first AI model training operation. The processor unit is further configured to assign a second timestamp to the second I / O command based on a second bandwidth allocated to the second AI model training operation.

[0009] The storage device includes a network interface and a processor unit. The processor unit receives a first request for a first data chunk, the first request being configured to include a first timestamp and a first global flow identifier (GFID). The processor unit receives a second request for a second data chunk, the second request being further configured to include a second timestamp and a second GFID. The processor unit is further configured to identify a first artificial intelligence (AI) model processor based on the first GFID. The processor unit is further configured to identify a second AI model processor based on the second GFID. The processor unit initiates a transfer of the first data chunk of the first AI model processor via the network interface, the transfer timing of the first data chunk being further configured to be based on the first timestamp. The processor unit initiates a transfer of the second data chunk of the second AI model processor via the network interface, the transfer timing of the second data chunk being further configured to be based on the second timestamp. Effect of the Invention

[0010] According to an embodiment, the system and method are used to support the processing of large data sets during training of artificial intelligence models. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a system diagram for performing time-synchronized storage delivery. [Diagram 2] FIG. 2 is a diagram of a storage device for performing time-synchronized storage delivery. [Diagram 3] FIG. 3 is a system diagram for performing time-synchronized storage delivery including the multiple storage devices of FIG. 2. [Figure 4] FIG. 2 is a diagram of a global flow identifier database entry. [Diagram 5] 1 is a flow chart illustrating a method for constructing a storage query plan. [Figure 6] 1 is a flow chart illustrating a method for assigning timestamps to I / O requests. [Figure 7] 1 is a flow chart illustrating a method for outputting data in response to an I / O request. [Figure 8] 1 is a flow chart illustrating synchronized processing of I / O commands. [Figure 9] A diagram showing a system including multiple artificial intelligence host devices and multiple object storage systems. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Referring to FIG. 1, a diagram of a system 100 for providing time-synchronized storage is shown. The system 100 includes an artificial intelligence (AI) host device 102. The AI ​​host device 102 corresponds to one or more computing devices. The AI ​​host device 102 includes an AI application 108 and a data loader 110. The AI ​​application 108 and the data loader 110 correspond to software that may be executed by a processor (not shown) of the AI ​​host device 102. The AI ​​host device 102 further includes a network interface controller (NIC) 104, a dynamic random access memory (DRAM) 106, and a number of AI model processing units (AI PUs) (112a-112h). The NIC 104 may correspond to a wired or wireless NIC. The AI ​​PUs include a first AI PU 112a, a second AI PU 112b, a third AI PU 112c, a fourth AI PU 112d, a fifth AI PU 112e, a sixth AI PU 112f, a seventh AI PU 112g, and an eighth AI PU 112h. Although eight AI PUs (112a-112h) are illustrated in the example embodiment, any number may be used. The AI ​​PUs (112a-112h) may include one or more graphics processor units (GPUs), one or more applications that may be executed by a processor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a central processor unit (CPU), other processing components, or a combination thereof. In some examples, the AI ​​PUs (112a-112h) are configured to process workloads related to AI operations.In some examples, the workload may include matrix multiplication and / or convolution intensive operations. The AI ​​host device 102 may correspond to a physical device or a virtual device (e.g., implemented by a cloud computing system).

[0013] In some implementations, the AI ​​host device 102 is configured differently than the example illustrated in FIG. 1. For example, the DRAM 106 can be replaced with a different type of memory device (e.g., static random access memory (SRAM)). As another example, the NIC 104 can be replaced with a different type of network adapter. As another example, the AI ​​host device 102 can include a different number of GPUs than those illustrated. It is noted that the AI ​​host device 102 can include additional components not illustrated in FIG. 1. For example, the AI ​​host device 102 can include one or more processors configured to execute instructions corresponding to the AI ​​application 108, the data loader 110, the AI ​​PUs (112a-112h), or a combination thereof.

[0014] System 100 further comprises object storage system 120. Object storage system 120 corresponds to one or more computing devices. In some embodiments, object storage system 120 may correspond to a server (e.g., an object storage server). Such a server may be a physical device or a virtual device (e.g., implemented by a cloud computing system).

[0015] The object storage system 120 includes a NIC 122, a processor unit 121, and memory buffers (124a-124h). In some embodiments, the object storage system 120 may include as many buffers (124a-124h) as there are AI PUs included in the AI ​​host device 102. In other embodiments, the object storage system 120 may include a different number of buffers. The processor unit 121 may correspond to a field programmable gate array configured to perform the actions described above, a central processing unit configured to perform instructions stored in a memory device (not shown) to perform the actions described above, an application specific integrated circuit configured to perform the actions described above, or other processing device. The NIC 122 corresponds to a wired or wireless NIC. The memory buffers (124a-124h) correspond to memory spaces in one or more memory devices. Such memory devices may include DRAM, SRAM, other types of memory, or combinations thereof. In the illustrated example, the memory buffers (124a-124h) include a first memory buffer 124a, a second memory buffer 124b, a third memory buffer 124c, a fourth memory buffer 124d, a fifth memory buffer 124e, a sixth memory buffer 124f, a seventh memory buffer 124g, and an eighth memory buffer 124h. Each of the memory buffers (124a-124h) may be associated with a different priority. For example, the first memory buffer 124a may be associated with a first priority and the second memory buffer 124b may be associated with a second priority. In other examples, the object storage system 120 may include a different number of buffers.

[0016] Each of the AI ​​PUs (112a-h) has a separate connection to the NIC 122. In some embodiments, these connections may be maintained through the NIC 104 of the AI ​​host device 102. In other embodiments, the connections may be maintained through separate network adapters of the AI ​​PUs (112a-h). The connections may be direct connections or may be connected through a public network (e.g., the Internet), a private network, or a combination thereof. The AI ​​PUs (112a-h) are configured to forward input / output (I / O) commands to the object storage system 120 through the connections. For example, the first AI PU 112a may forward a first I / O command requesting a first data object to the object storage system 120. The AI ​​PUs (112a-h) may train one or more AI data models based on the retrieved data objects. For example, in response to receiving the first data object, the first AI PU 112a may train an AI model (or a portion thereof) and return the trained model (or a portion thereof) to the AI ​​application 108.

[0017] As described herein, the processor unit 121 of the object storage system 120 is configured to buffer received commands in memory buffers (124a-124h) based on a global flow identifier (GFID) included in the received I / O command. The processor unit 121 of the object storage system 120 is configured to retrieve the I / O command and initiate operations on associated data based on the priority of the memory buffers (124a-124h). For example, the processor unit 121 of the object storage system 120 may retrieve I / O commands from the first memory buffer 124a more frequently than the second memory buffer 124b based on the relative priority of the first and second memory buffers (124a, 124b).

[0018] The processor unit 121 of the object storage system 120 manages the data objects (126a, 126b). In FIG. 1, the first data object 126a and the second data object 126b are labeled. The received I / O command can identify the data objects (e.g., by a key value). As described herein, the processor unit 121 of the object storage system 120 may be configured to initiate a time-synchronized transfer of two or more data objects to one or more AI PUs. The data objects (126a, 126b) managed by the object storage system 120 can be stored in one or more storage devices. The storage devices are coupled to the object storage system 120 via the NIC 122. Thus, initiating the transfer of the data object can include forwarding a request to the storage device to transfer the object (or a portion thereof) to the object storage system 120 or the requesting AI PU.

[0019] In some embodiments, processor unit 121 of object storage system 120 may be configured to implement a data erasure coding scheme. Processor unit 121 of object storage system 120 may perform a data erasure coding scheme on a data object to generate one or more erasure coded (EC) chunks and store the EC chunks on one or more storage devices. In response to an I / O command that references a data object, processor unit 121 of object storage system 120 may be configured to determine which storage device or storage devices store the EC chunks for the data object and issue an I / O request to the storage device.

[0020] In some embodiments, the storage device may include a non-volatile memory express (NVMe) device, an NVMe over fabric (NVMe-oF) target, another storage device, or a combination thereof.

[0021] FIG. 2 illustrates an example of a storage device 200 (e.g., an NVMe-oF target) capable of storing EC code chunks of one or more data objects. The storage device 200 includes a NIC 201 and a processor unit 205. The NIC 201 may correspond to a wired or wireless NIC. The processor unit 205 may correspond to a field programmable gate array configured to perform the actions described above, a central processing unit configured to execute instructions stored in a memory unit (not shown) to perform the actions described above, an application specific integrated circuit configured to perform the actions described above, or other processing unit. The storage device 200 includes a first drive 208a, a second drive 208b, a third drive 208c, and a fourth drive 208d. Each of the drives (208a-208d) may include a solid state drive, a hard disk drive, another type of computer storage drive, or a combination thereof. The drives (208a-208d) are configured to store data (eg, EC code data chunks). In some embodiments, the storage device 200 may include a different number of drives than shown in FIG.

[0022] Storage device 200 includes first staging buffer 202a, second staging buffer 202b, third staging buffer 202c, and fourth staging buffer 202d. Other embodiments may include a different number of staging buffers. The staging buffers (202a-202d) may include DRAM, SRAM, other types of memory, or a combination thereof. Staging buffers (202a-202d) are configured to store I / O requests received from object storage system 120 via NIC 201, store data (e.g., EC code data chunks) retrieved from drives (208a-208d), or a combination thereof. For example, all of the staging buffers (202a-202d) may be configured to store I / O requests received from object storage system 120 and all of the staging buffers (202a-202d) may be configured to store data chunks from a drive, or a first subset of the staging buffers (202a-202d) may be configured to store data chunks and a second subset of the staging buffers (202a-202d) may be configured to store I / O requests. In some embodiments, storage device 200 may include one buffer for each drive included in storage device 200. In other embodiments, storage device 200 may include other numbers of buffers (e.g., one buffer per drive as well as one or more output buffers).

[0023] In embodiments configured to receive I / O requests from one or more of the staging buffers as described herein, the processor unit 205 can assign a particular I / O request to a particular one of the staging buffers (202a-202d) based on a GFID associated with the I / O request. The staging buffers (202a-202d) can have an associated priority, and the I / O requests stored in the staging buffers (202a-202d) can be processed from the staging buffers (202a-202d) in an order determined based on the priority. Similarly, in embodiments configured to receive data from one or more of the drives (208a-208d) from the staging buffers as described herein, the processor unit 205 can assign a particular piece of data (e.g., a particular chunk) to a particular one of the staging buffers (202a-202d) based on a GFID associated with an I / O request related to retrieval of a particular piece of data. As described above, the staging buffers (202a-202d) can have an associated priority. The data stored in the staging buffers (202a to 202d) is transferred from the staging buffers (202a to 202d) (eg, to the object storage system 120 or one or more of the AI ​​PUs (112a to 112h)) in an order determined based on the priority.

[0024] The storage device 200 further includes a bus 206. The bus 206 may include a peripheral component interconnect express (PCIe) bus or other type of bus. The bus 206 couples the drives (208a-208d) to the staging buffers (202a-202d). The NIC 201 and / or the processor unit 205 may also be coupled to the bus 206. FIG. 2 illustrates I / O (204a-204d) flowing between the drives (208a-208d) and the staging buffers (202a-202d). The I / O (204a-204d) may include data chunks, I / O requests, or a combination thereof. In some embodiments, the storage device 200 may include additional components further coupled to the bus 206.

[0025] Referring to FIG. 3, a diagram illustrating the operation of a system for time-synchronized storage delivery is shown. In the example of FIG. 3, the object storage system 120 and the AI ​​PUs (112a-112h) of FIG. 1 are coupled (e.g., directly through one or more networks) to the storage devices (200a-200d). Although four story devices (200a-200d) are shown, in other examples, the storage system 120 and / or the AI ​​PUs (112a-112h) may be coupled to a different number of storage devices. It should be noted that some examples include a different number of AI PUs than the illustrated example. Each of the storage devices (200a-200d) is an example of the storage device 200 shown in FIG. 2. It should be noted that the storage devices (200a-200d) are examples of the storage device 200, but the storage devices (200a-200d) may not be the same. For example, as discussed above, the storage devices 200 may have other configurations (e.g., a different number or use of staging buffers) and the first storage device 200a may have one of these configurations while the second storage device 200b has another of these configurations.

[0026] The object storage system 120, the AI ​​PUs (112a to 112h), and the storage devices (200a to 200d) are connected to a GFID database 302 (e.g., a NIC 122 and a NIC 201, respectively). The GFID database 302 is implemented on a computing device. In some examples, the GFID database 302 may be implemented on the AI ​​host device 102 of FIG. 1 (e.g., any one of the AI ​​PUs (112a to 112h) or another processor of the AI ​​host device 102). In some examples, the GFID database 302 may be implemented on the object storage system 120 or any one of the storage devices (200a to 200d). In some examples, the GFID database 302 may be implemented on a computing device external to the AI ​​PUs (112a to 112h), the storage devices (200a to 200d), and the object storage system 120.

[0027] The GFID database 302 is configured to store metadata identifying resources associated with an AI training session. For example, the GFID database 302 can store an association between a GFID, data associated with an AI PU, data associated with one or more storage devices, and data associated with one or more object storage systems. The data associated with an AI PU may include an identifier for the AI ​​PU, a link identifier associated with the AI ​​PU, a bandwidth associated with the AI ​​PU, or a combination thereof. The data associated with the one or more object storage systems may include an object storage system identifier for each object storage system, the link identifier for each object storage system may include a bandwidth associated with each object storage system, or a combination thereof. The data associated with the one or more storage devices may include an identifier for each storage device, a link identifier associated with each storage device, a bandwidth associated with each storage device, or a combination thereof.

[0028] 4, a diagram is shown depicting an example entry 402 in GFID database 302. In the example of FIG. 4, entry 402 includes a global flow ID 404, data associated with an AI PU 406, data associated with one or more object storage systems 414, and data associated with one or more storage devices 422. Global flow ID 404 may be a unique identifier that identifies an AI training session (or other data flow).

[0029] The data 406 associated with the AI ​​PU describes the AI ​​PU involved in the AI ​​training session (or other data flow). The data 406 associated with the AI ​​PU includes an entity identifier or name 408. The entity identifier or name 408 may include a network address (e.g., an Internet Protocol address), a hostname, other identifiers, or combinations thereof. The data 406 further includes a connection identifier 410 associated with the AI ​​PU. The connection identifier 410 may include a socket identifier, other type of connection identifier, or combinations thereof that identify the connection over which the AI ​​PU identified by the entity identifier or name 408 transmits and / or receives data. The data 406 associated with the AI ​​PU further includes a bandwidth indicator 412. The bandwidth indicator 412 indicates the bandwidth available to the AI ​​PU for receiving data during the AI ​​training session (or other data flow).

[0030] The data 414 associated with the one or more object storage systems includes an object storage system identifier 416 for each object storage system associated with the AI ​​training session (or other data flow). The object storage system identifier 416 may include a network address (e.g., Internet Protocol address), a hostname, other identifiers, or combinations thereof. The data 414 associated with the one or more object storage systems further includes an object storage system-specific connection identifier 418 associated with the AI ​​training session (or other data flow). The connection identifier 418 may include a socket identifier, other type of connection identifier, or combinations thereof that identify a connection over which the object storage system identified by the object storage system identifier 416 can send and / or receive data. The data 414 associated with the one or more object storage systems further includes an object storage system-specific bandwidth indicator 420 associated with the AI ​​training session (or other data flow). The bandwidth indicator 420 indicates the bandwidth available to the object storage system for sending and / or receiving data during the AI ​​training session (or other data flow).

[0031] The data 422 associated with the one or more object storage systems includes a storage-apparatus-specific target identifier 424 associated with the AI ​​training session (or other data flow). The target identifier 424 may include a network address (e.g., Internet Protocol address), a hostname, other identifiers, or combinations thereof. The data 422 associated with the one or more storage systems further includes a storage-apparatus-specific connection identifier 426 associated with the AI ​​training session (or other data flow). The connection identifier 426 may include a socket identifier, other type of connection identifier, or combinations thereof that identify a connection over which the object storage system identified by the target identifier 424 can send and / or receive data. The data 422 associated with the one or more storage systems further includes a storage-apparatus-specific bandwidth identifier 428 associated with the AI ​​training session (or other data flow). The bandwidth identifier 428 indicates the bandwidth available to the storage system for sending and / or receiving data during the AI ​​training session (or other data flow).

[0032] Thus, FIG. 4 shows an example of an entry in a global flow identifier database that associates a global flow identifier with metadata describing various resources used during an AI training session (or other data flow).

[0033] 3, during an AI training session (or other data flow), the first AI PU 112a (or any other AI PU) may issue an I / O command (e.g., a read request) to the object storage system 120. The first AI PU 112a may include a GFID associated with the AI ​​training session (or other data flow) in the I / O command. In some embodiments, the first AI PU 112a may limit the rate at which the I / O command is forwarded to the object storage system 120 based on a bandwidth indicator 420 associated with the object storage system 120 as identified by the GFID database 302.

[0034] In response to receiving the I / O command, the processor unit 121 of the object storage system 120 can assign a timestamp to the I / O command and subsequently place the I / O command in one of the memory buffers (124a-124h). Assigning a timestamp to the I / O command can include appending the timestamp to a record (e.g., a command control block) associated with the I / O command. The record corresponds to a block of metadata associated with the I / O command. In some examples, the processor unit 121 of the object storage system 120 can look up the GFID in the I / O command in the GFID database 302 and determine a memory buffer in which to place the I / O command based on the data associated with the GFID. For example, the processor unit 121 of the object storage system 120 can place the I / O command in the first memory buffer 124a based on a bandwidth associated with the first AI PU 112a (e.g., the bandwidth indicated by the bandwidth indicator 412) as identified in the GFID database 302. In some examples, the processor unit 121 of the object storage system 120 may maintain a mapping of priorities to bandwidth ranges (e.g., a first bandwidth range may be mapped to a first priority, a second bandwidth range may be mapped to a second priority, etc.). The processor unit 121 of the object storage system 120 may identify a bandwidth that falls into the bandwidth of the AI ​​PU associated with the GFID identified by the I / O command, and attach the I / O command to one of the memory buffers (124a-124h) corresponding to the priority that is mapped to the next bandwidth range.

[0035] In some examples, the timestamp assigned by the processor unit 121 of the object storage system 120 to the I / O command is based on the GFID (e.g., the bandwidth associated with the GFID). For example, the processor unit 121 of the object storage system 120 may query the GFID in the GFID database 302 to identify the bandwidth associated with the AI ​​PU associated with the GFID and assign the timestamp based on the next bandwidth. For example, the assigned timestamp corresponds to the time the I / O command is received by the object storage system 120 as well as an offset selected based on the bandwidth. To illustrate, the object storage system 120 may receive a first I / O command including a first GFID and a second I / O command including a second GFID via the NIC 122. The first GFID may be associated with the first AI PU 112a from the GFID database 302 and the second GFID may be associated with the second AI PU 112b from the GFID database 302. The entry for the first GFID may further indicate that the bandwidth associated with the first AI PU 112a is X, while the entry for the second GFID may indicate that the bandwidth associated with the second AI PU 112b is Y. The object storage system 120 may maintain (e.g., in an external or internal memory) an association (e.g., a mapping) between the bandwidth and the offset. The processor unit 121 of the object storage system 120 may set a first offset (e.g., +5 milliseconds (ms)) based on X and a second offset (e.g., +3 ms) based on Y. In other embodiments, the processor unit 121 may assign a timestamp (n) for a particular I / O command associated with the GFID based on the bandwidth for the GFID and a previous timestamp (n-1) for a previous I / O command associated with the GFID.To illustrate, the processor unit 121 may assign a timestamp (n) such that the time between timestamp (n-1) and timestamp (n) satisfies a data transfer rate in terms of a bandwidth associated with the GFID. In some embodiments, the processor unit 121 may assign a timestamp for an I / O command associated with a GFID such that the average time between timestamps for the GFID satisfies a bandwidth associated with the GFID (e.g., regardless of when the I / O command is received by the object storage system 120). As described in more detail below, the storage device may be configured to use timestamps to reserve responses to commands. Thus, setting timestamps for I / O commands based on a GFID associated with an AI training session may provide priority based scheduling for an AI training session in which data is exchanged between multiple devices. This priority scheme may be used to ensure that the system 100 meets the bandwidth (e.g., the rate at which the AP PU receives data) and latency (latency meaning between when the AI ​​PU forwards a command and when a response is forwarded by the storage device 200) requirements of the AI ​​training session.

[0036] The processor unit 121 of the object storage system 120 is configured to process the I / O command from one of the memory buffers (124a-124h) by removing the I / O command from the memory buffers (124a-124h) and identifying one of the storage devices that stores the data (e.g., data chunk) indicated in the I / O command. The processor unit 121 of the object storage system 120 is configured to generate a request for each data chunk and forward the request to the appropriate storage device. The request may include the GFID from the I / O command, an identifier of the requested data chunk, and a timestamp assigned to the I / O command. In a situation where the I / O command identifies an object that is stored (or partially stored) in a storage device that is not associated with a GFID in the GFID database 302, the processor unit 121 of the object storage system 120 may return an error message to the AI ​​PU (e.g., via the NIC 122). The processor unit 121 of the object storage system 120 may process I / O commands from the memory buffers (124a-124h) in an order determined based on the priority associated with the memory buffers (124a-124h). Note that in some embodiments, the processor unit 121 of the object storage system 120 may limit the rate at which requests are forwarded to a particular one of the storage devices (200a-200d) based on the bandwidth 428 associated with the storage device, as identified by the GFID database 302.

[0037] The storage devices (200a-200d) (e.g., the processor unit 205 of the storage devices (200a-200d)) are configured to transfer data (e.g., data chunks) in an order determined based on a timestamp included in the associated request. The storage devices (200a-200d) may transfer the data directly to the requesting AI PU or the object storage system 120 via the NIC 201. For example, the first storage device 200a may receive a request for a first data chunk via the NIC 201 of the first storage device 200a. The processor unit 205 of the first storage device 200a may insert the request into one of the staging buffers (202a-202d) based on the timestamp in the request. For example, the location where the request is added to the staging buffer may be determined based on the timestamp such that the requests included in the staging buffer are sorted in timestamp order.

[0038] Additionally, the staging buffer into which the request is inserted may be selected by processor unit 205 based on the bandwidth (e.g., bandwidth 428) associated with the GFID in the request. The request is processed from the staging buffer (e.g., in order based on the priority of the staging buffer) and forwarded to bus 206 for receipt by a drive (e.g., any of drives (208a-208d)) that stores the requested data (e.g., data chunks).

[0039] The drive can return the data to bus 206 to be output by storage device 200 via NIC 201 to the AI ​​PU associated with the GFID. For example, processor unit 205 can initiate transfer of a query to the GFID database via NIC 201 to ascertain which AI PU is associated with the request, and then initiate transfer of the data to be output by the drive to that AI PU via NIC 201. In some embodiments, storage device 200 can transfer data (e.g., data chunks) to object storage system 120 for delivery to the AI ​​PU.

[0040] In some embodiments, the processor unit 205 can add the data to one of the staging buffers (202a-202d) associated with buffering the output. The data can be added to the staging buffer based on the GFID and / or timestamp associated with the request. Note that in some embodiments, the processor unit 205 can limit the rate at which data is returned to the AI ​​PU based on a bandwidth identifier 412 associated with the AI ​​PU, as identified by the GFID database 302.

[0041] The storage device processes the requests and outputs data in timestamp order so that objects are retrieved in a synchronized manner. Note that the timestamps are assigned based on a priority (e.g., bandwidth) that corresponds to the GFID that identifies the AI ​​training session. Thus, the AI ​​training session is prioritized. Note that the data transfer rate between components within the AI ​​training session may be managed based on the bandwidth that is linked to the GFID of the AI ​​training session.

[0042] 5, a diagram illustrating a method 500 for creating a storage query plan for an AI training session is shown. Method 500 is performed by AI host device 102 of FIG.

[0043] At 502, method 500 includes reading storage service session parameters. For example, AI host device 102 can read parameters (e.g., workload parameters) associated with an AI training session (e.g., from an input file or other source). These parameters can include a target bandwidth for the AI ​​training session, a data set, training time, bandwidth per GPU, objects / data to read, a number of GPUs, a size of a prefetch buffer, other parameters, or a combination thereof. The parameters may be input by data loader 110.

[0044] At 504, method 500 further includes reading system and network topology information. For example, AI host device 102 may determine the network topology of the system shown in FIG. 3. AI host device 102 may determine the network topology using Internet control message protocol (ICMP), link layer discovery protocol (LLDP), other tools, or a combination thereof. Determining the network topology information may further include determining the available bandwidth of various devices on the network. For example, AI host device 102 may determine the available bandwidth of storage device 200, the available bandwidth of object storage system 120, the available bandwidth of AI PUs (112a-112h), or a combination thereof.

[0045] At 506, method 500 further includes reading storage capability information. For example, AI host device 102 can query storage devices (200a-200d), object storage system 120, or a combination thereof to determine the amount of storage capacity available during an AI training session.

[0046] At 508, method 500 further includes analyzing the network information and the storage capacity information and generating a global flow database entry that associates the GFID with a resource associated with the AI ​​training session.

[0047] In response to determining, at 510, that the topology and storage capabilities do not support the parameters of the storage system, at 502, the method 500 includes reading parameters of a next storage service session.

[0048] In response to determining at 510 that the topology and storage capabilities support the parameters of the storage system, the method 500 includes distributing the global flow database entry at 512. For example, the AI ​​host device 102 can distribute the database entry to the GFID database 302.

[0049] At 514, method 500 further includes setting up and initializing flows associated with the storage queries. For example, AI host device 102 may initialize a connection with object storage system 120. The connection may have a bandwidth represented by an entry stored in GFID database 302. AI host device 102 may further initialize connections with one or more storage devices identified in the GFID database entry.

[0050] At 516, the method 500 includes launching an application. For example, the AI ​​host device 102 can launch the AI ​​application 108 as part of an AI training session and begin issuing I / O commands to the object storage system 120.

[0051] 6, a diagram illustrating a method 600 for forwarding a time synchronized request to a storage device is shown. Method 600 may be performed by an object storage system, such as object storage system 120, or other types of storage systems.

[0052] At 606, method 600 includes receiving a first input / output (I / O) command from a first AI PU, the first I / O command associated with a first AI model training operation. For example, object storage system 120 may receive the first I / O command from first AI PU 112a via NIC 122. The first I / O command may include a first GFID associated with a first AI training session performed by first AI PU 112a.

[0053] At 608, method 600 includes receiving a second I / O command from a second AI PU, the second I / O command associated with a training operation of a second AI model. For example, object storage system 120 may receive the second I / O command from second AI PU 112b via NIC 122. The second I / O command may include a second GFID associated with a second AI training session performed by second AI PU 112b.

[0054] At 610, the method 600 includes assigning a first timestamp to the first I / O command based on a first bandwidth allocated to the first AI model training operation. For example, the processor unit 121 of the object storage system 120 may query the GFID database 302 to determine a first bandwidth indicator 412 associated with a GFID included in the first I / O command. The processor unit 121 of the object storage system 120 may assign a first timestamp to the first I / O command based on the identified first bandwidth. In some examples, the first timestamp corresponds to a time of receipt of the first I / O command at the object storage system 120 as well as an offset determined based on the first bandwidth.

[0055] At 612, the method 600 further includes assigning a second timestamp to the second I / O command based on a second bandwidth allocated to the training operation of the second AI model. For example, the object storage system 120 may query the GFID database 302 to determine a second bandwidth indicator 412 associated with the GFID included in the second I / O command. The processor unit 121 of the object storage system 120 may assign a second timestamp to the first I / O command based on the identified second bandwidth. In some examples, the second timestamp corresponds to a time of receipt of the first I / O command at the object storage system 120 as well as an offset determined based on the second bandwidth.

[0056] Data requests forwarded by object storage system 120 based on I / O commands may include a timestamp assigned to the I / O command. A storage device receiving the data request may output the data in an order determined based on the timestamp. Thus, data is output by the storage device in a time-synchronized manner.

[0057] Referring to FIG. 7, a diagram illustrating a method 700 for transferring data from a storage device in a time synchronized manner is shown.

[0058] At 702, the method 700 includes receiving a first request for a first data chunk. The first request includes a first timestamp and a first global flow identifier (GFID). For example, the first storage device 200a may receive the first request from the object storage system 120 via the NIC 201. The first request requests the first data chunk, includes a first timestamp, and may include a first GFID.

[0059] At 704, the method 700 includes receiving a second request for the second data chunk, the second request including a second timestamp and a second GFID. For example, the first storage device 200a may receive the second request from the object storage system 120 via the NIC 201. The second request requests the second data chunk, includes a second timestamp, and may include a second GFID.

[0060] At 706, the method 700 further includes identifying a first AI PU based on the first GFID. The processor unit 205 of the first storage device 200a can query the GFID database 302 via the NIC 201 to determine that the first AI PU 112a is associated with the first GFID.

[0061] At 708, the method 700 further includes identifying a second AI PU based on the second GFID. For example, the processor unit 205 of the first storage device 200a can query the GFID database 302 via the NIC 201 to determine that the second AI PU 112b is associated with the second GFID.

[0062] At 710, the method 700 further includes transferring the first data chunk to the first AI PU, where the transfer time of the first data chunk is based on the first timestamp. For example, the processor unit 205 of the first storage device 200a may initiate the transfer of the first data chunk to the first AI PU 112a via the NIC 201. The processor unit 205 of the first storage device 200a may reserve processing of the first request from one of the staging buffers (202a-202d) and reserve the transfer of the first data chunk to the first AI PU 112a based on the first timestamp. In some embodiments, the processor unit 205 of the first storage device 200a may wait until the time represented by the first timestamp before transferring the first data chunk to the first AI PU 112a.

[0063] At 712, the method 700 further includes transferring the second data chunk to the second AI PU, where the transfer time of the second data chunk is based on the second timestamp. For example, the processor unit 205 of the first storage device 200a may initiate the transfer of the second data chunk to the second AI PU 112b via the NIC 201. The processor unit 205 of the first storage device 200a may reserve processing of the second request from one of the staging buffers (202a-202d) and reserve the transfer of the first data chunk to the first AI PU 112a based on the second timestamp. In some embodiments, the processor unit 205 of the first storage device 200a may wait until the time represented by the second timestamp to transfer the second data chunk to the first AI PU 112a.

[0064] Method 700 can also be used by a storage device to transfer data chunks in a time synchronized manner, and the timestamp used by the storage device to synchronize the transfer can be based on the priority of the AI ​​training session, so that the transfer of data can take into account the relative priority of the AI ​​training session.

[0065] 8, a flowchart 800 is shown illustrating an example in which the first AI PU 112a issues three I / O commands during a particular AI tie session. The flowchart 800 shows first to n-th storage devices (200a to 200n). As described above, the system of FIG. 3 may include a different number of storage devices than the four shown in the figure.

[0066] At 802, the flowchart 800 includes a first AI PU 112a coupled to an object storage system 120. The first AI PU 112a may initiate this coupling in response to a storage query plan being successfully established during an AI training session and determining that the storage query plan assigns the object storage system 120 to the AI ​​training session. In the illustrated example, the AI ​​training session has a GFID of "ABC." Thus, the GFID database 302 stores a GFID database entry that associates the GFID "ABC" with the object storage system 120.

[0067] At 804, the flowchart 800 includes a first AI PU 112a coupled to the first storage device 200a. The first AI PU 112a may initiate this coupling in response to determining that the storage query plan assigns the first storage device 200a to an AI training session. A GFID database entry associates the GFID "ABC" with the first storage device 200a.

[0068] At 806, the flowchart 800 includes a first AI PU 112a coupled to the second storage device 200b. The first AI PU 112a may initiate this coupling in response to determining that the storage query plan assigns the second storage device 200b to the AI ​​training session. A GFID database entry associates the GFID "ABC" with the second storage device 200b.

[0069] At 808, the flowchart 800 includes a first AI PU 112a coupled to the nth storage device 200n. The first AI PU 112a may initiate this coupling in response to determining that the storage query plan assigns the nth storage device 200n to an AI training session. A GFID database entry associates the GFID "ABC" with the nth storage device 200n.

[0070] The flowchart 800 further includes a first AI PU 112a forwarding a first I / O command to the object storage system 120 at 810, a second I / O command to the object storage system 120 at 812, and a third I / O command to the object storage system 120 at 814. In the illustrated example, the first, second, and third I / O commands are GET commands. The first I / O command identifies a first data object (e.g., by a key value), the second I / O command identifies a second data object, and the third I / O command identifies a third data object. The object storage system 120 assigns a first timestamp to the first I / O command, a second timestamp to the second I / O command, and a third timestamp to the third I / O command. The timestamps are generated based on a priority (e.g., bandwidth) associated with GFID “ABC” in the GFID database 302.

[0071] For each command, the object storage system 120 continues to generate a request for the data chunk associated with the object identified by the I / O command. The request includes a timestamp assigned to the I / O command and is forwarded by the object storage system 120 to the storage device identified in the GFID database 302 as being associated with GFID "ABC."

[0072] In the illustrated example of FIG. 8, the object storage system 120 generates a first request for the data chunk, a second request for the data chunk, and a third request for the data chunk based on the first I / O command. Each of these requests includes a first timestamp assigned to the first I / O command. The object storage system 120 generates a fourth request for the data chunk, a fifth request for the data chunk, and a sixth request for the data chunk based on the second I / O command. Each of these requests includes a second timestamp assigned to the second I / O command. The object storage system 120 generates a seventh request for the data chunk, an eighth request for the data chunk, and a ninth request for the data chunk based on the third I / O command. Each of these requests includes a third timestamp assigned to the third I / O command.

[0073] In the flowchart 800, the object storage system 120 forwards the first request to the first storage device 200a in 816, the seventh request to the first storage device 200a in 818, the eighth request to the second storage device 200b in 820, the fourth request to the first storage device 200a in 822, the second request to the second storage device 200b in 824, the third request to the nth storage device 200n in 826, the ninth request to the nth storage device 200n in 828, the fifth request to the second storage device 200b in 830, and the sixth request to the nth storage device 200n in 832. In this way, requests may be forwarded by the failed object storage system 120 (and may be received by the storage devices (200a-200n)).

[0074] The storage devices (200a-200n) are configured to output data chunks in an order determined based on timestamps included in the requests. For example, the first storage device 200a may receive the first request, the fourth request, and the seventh request, insert the first request, the fourth request, and the seventh request in the first stage buffer 202a in an order based on their respective timestamps (which may differ from the order in which the corresponding requests are received), and process the requests from the first staging buffer 202a in an order based on the sequence. Additionally or alternatively, the first storage device 200a may insert data associated with the first request, data associated with the fourth request, and data associated with the seventh request in the second staging buffer 202b in an order based on the timestamps, and transfer the data chunks from the second staging buffer 202b in an order based on the sequence. In some embodiments, the data chunk is removed from the staging buffer 202b and transferred to the first AI PU 112a at the time (or approximately, for example, within 3 ms) indicated by a timestamp associated with the data chunk.

[0075] Thus, in the illustrated flowchart 800, at 834, a data chunk corresponding to a first request is output to the first AI PU 112a, at 836, a data chunk associated with a second request is output to the first AI PU 112a, at 838, a data chunk associated with a third request is output to the first AI PU 112a, at 840, a data chunk associated with a fourth request is output to the first AI PU 112a, at 842, a data chunk associated with a fifth request is output to the first AI PU 112a, at 844, a data chunk associated with a sixth request is output to the first AI PU 112a, at 846, a data chunk associated with a seventh request is output to the first AI PU 112a, at 848, a data chunk associated with an eighth request is output to the first AI PU 112a, and at 850, a data chunk associated with a ninth request is output to the first AI PU 112a. It should be noted that in addition to determining the order in which the data is returned, the timestamps set by the object storage system 120 also determine the rate at which the data is delivered to the AI ​​PUs. Thus, the data chunks returned to the first AI PU 112a can be time synchronized or delivered at an appropriate transfer rate.

[0076] In the illustrated example, the object storage system 120 forwards the first completion notification to the first AI PU 112a at 852, the second completion notification to the first AI PU 112a at 854, and the third completion notification to the first AI PU 112a at 856. The completion notifications are forwarded based on a time associated with a timestamp. For example, the first completion notification is forwarded by the object storage system 120 at a time represented by the first timestamp. Thus, the first AI PU 112a is notified that all data chunks associated with the first I / O command should be transferred.

[0077] The above example illustrates an embodiment in which one AI host device interacts with one object storage system. However, the system may include multiple AI host devices and multiple object storage systems. Multiple AI host devices may interact with an object storage system. Additionally, an AI host device may interact with multiple object storage systems. An example system including many AI host devices and many object storage systems is shown in FIG. 9. The configuration and operation of the system of FIG. 9 is similar to the systems and operations described above with reference to FIGS. 1-8.

[0078] The above description is for the purpose of illustrating example embodiments and should not be construed as limiting the present invention. Although several example embodiments have been described, those skilled in the art will readily appreciate that many variations are possible in the example embodiments without substantially departing from the novel teachings and advantages of the example embodiments. Accordingly, all such modifications are intended to be included within the scope of the example embodiments as defined in the claims. In the claims, means-plus-function clauses are intended to include structures described herein by reciting equivalent structures as well as structural equivalents. Accordingly, the above description is for the purpose of illustrating example embodiments and should not be construed as being limited to the particular embodiments disclosed, but should be understood to include modifications to the example embodiments and other example embodiments set forth within the scope of the appended claims.

[0079] The terms used herein are used to describe specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "a" and "an" are intended to include the plural forms unless the context clearly indicates otherwise. It should be understood that the terms "comprises," "comprising," "have," "having," "includes," and "including" as used herein specify the presence of the mentioned features, numbers, steps, operations, and / or components, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0080] As used herein, when describing embodiments of the present invention, the use of "may" refers to "one or more embodiments of the present invention." As used herein, the terms "use," "using," and "used" are considered synonymous with the terms "utilize," "utilizing," and "utilized," respectively. Additionally, the term "exemplary" is considered an example or illustration.

[0081] When some embodiments are embodied differently, the order of certain processes may be performed differently than that described above, for example, two successively described process or method steps may be performed substantially simultaneously or in a different order than that described.

[0082] The electronic or electrical devices according to the embodiments of the invention described herein and / or any other related devices or elements may be implemented using any suitable hardware, firmware (e.g., application-specific integrated circuits (ASICs)), software, or a combination of software, firmware, and hardware. For example, the various elements of the devices may be formed on a single integrated circuit (IC) chip or on separate IC chips. It is also possible that the various elements of the devices are implemented on a flexible printed circuit film (TCP), a tape carrier package (PCB), a printed circuit board (PCB), or formed on a single substrate. It is also possible that the various elements of the devices are processes or threads implemented in one or more computing devices or in multiple processors that execute computer program instructions and interact with other system elements to perform the various functions described herein. The computer program instructions are stored in a memory implemented in the computing device using standard memory devices such as random access memory (RAM). The computer program instructions may also be stored, for example, on a CD-ROM, a flash drive, or other such non-transitory computer readable media.Those skilled in the art of the present invention should also recognize that the functionality of various computing devices may be integrated or integrated into a single computing device, and that the functionality of a particular computing device may be distributed across one or more other computing devices, without departing from the spirit and scope of the example embodiments of the present invention.

[0083] Unless otherwise defined, all terms (including technical or scientific terms) used herein generally have the same meaning as understood by one of ordinary skill in the art to which this invention belongs. This means that terms defined in commonly used dictionaries are to be understood to have a meaning consistent with their meaning in the context of the relevant art and / or this specification, and should not be interpreted in an idealized or overly formal sense unless expressly defined in this application.

Claims

1. An object storage system comprising: an object storage device; a first processing unit; and a first command received by the object storage device from the first processing unit, the first command being associated with a first operation related to an artificial intelligence (AI) model and for requesting input / output of data stored in one or more storage devices; receiving, by the object storage system, a second command from a second processing unit, the second command being associated with a second operation related to the AI ​​model and for requesting input / output of data stored in one or more storage devices; the object storage system assigning a first timestamp to the first command based on a first characteristic associated with the first operation, the first characteristic comprising a first bandwidth; the object storage system assigning a second timestamp to the second command based on a second characteristic associated with the second operation, the second characteristic comprising a second bandwidth; wherein data transmission from the one or more storage devices to the first processing unit and the second processing unit is performed based on the first timestamp and the second timestamp.

2. The first command includes a first identifier that identifies the first operation, and the second command includes a second identifier that identifies the second operation, and the method further comprises: retrieving an indication of the first characteristic from a database based on the first identifier; and retrieving an indication of the second characteristic from the database based on the second identifier. The method of claim 1.

3. sending a first request for first data to a first storage device based on the first command, the first request including the first timestamp. The method of claim 1.

4. sending a second request for second data to a second storage device based on the first command, the second request including the first timestamp. The method according to claim 3.

5. sending a third request for third data to a third storage device based on the second command, the third request including the second timestamp. The method according to claim 4.

6. The first command identifies a data object, the first data corresponds to a first chunk of the data object, and the second data corresponds to a second chunk of the data object. The method according to claim 4.

7. receiving a first request for first data by a storage device, the first request including a first attribute and a first identifier; receiving a second request for second data by the storage device, the second request including a second attribute and a second identifier; the storage device identifying a first processing unit based on the first identifier; the storage device identifying a second processing unit based on the second identifier; transmitting the first data from the storage device to the first processing unit, the transmission of the first data being based on the first attribute; transmitting the second data from the storage device to the second processing unit, the transmission of the second data being based on the second attribute; wherein the first attribute comprises a first timestamp, the second attribute comprises a second timestamp, and an order of transmission of the first data and the second data is based on the first timestamp and the second timestamp.

8. The storage device includes a non-volatile memory express over fabric (NVMe-oF) target, the NVMe-oF target including a plurality of solid state drives (SSDs). The method of claim 7.

9. The first processing unit includes a graphics processing unit. The method of claim 7.

10. The first processing unit includes an application that runs on a processor. The method of claim 7.

11. The first data corresponds to a chunk of a data object. The method of claim 7.

12. Identifying the first processing unit includes sending a query to a database to identify the first identifier. The method of claim 7.

13. A network interface; a processor unit, the processor unit comprising: receiving a first command from a first processing unit, the first command being associated with a first operation related to a first artificial intelligence (AI) model and for requesting input / output of data stored in one or more storage devices; receiving a second command from a second processing unit, the second command being associated with a second operation related to the AI ​​model and for requesting input / output of data stored in one or more storage devices; assigning a first timestamp to the first command based on a first characteristic associated with the first operation, the first characteristic comprising a first bandwidth; assigning a second timestamp to the second command based on a second characteristic associated with the second operation, the second characteristic including a second bandwidth; wherein data transmission from the one or more storage devices to the first processing unit and the second processing unit is performed based on the first timestamp and the second timestamp.

14. the first command includes a first identifier that identifies the first operation, the second command includes a second identifier that identifies the second operation, and the processor unit retrieving an indication of the first characteristic from a database based on the first identifier; and detecting an indication of the second characteristic from the database based on the second identifier.

14. The computing device of claim 13.

15. The processor unit is further configured to initiate transmission of a first request for the first data to a first storage device via the network interface based on the first command, the first request including the first timestamp.

14. The computing device of claim 13.

16. The processor unit is further configured to initiate transmission, via the network interface, of a second request for the second data to a second storage device based on the first command, the second request including the first timestamp.

16. The computing device of claim 15.

17. The processor unit is further configured to initiate transmission, via the network interface, of a third request for third data to a third storage device based on the second command, the third request including the second timestamp.

17. The computing device of claim 16.

18. The first command identifies a data object, the first data corresponds to a first chunk of the data object, and the second data corresponds to a second chunk of the data object.

17. The computing device of claim 16.

19. A network interface; A storage device comprising: receiving a first request for a first data item, the first request including a first attribute and a first identifier; receiving a second request for second data, the second request including a second attribute and a second identifier; identifying a first processing unit based on the first identifier; identifying a second processing unit based on the second identifier; initiating transmission of the first data to the first processing unit via the network interface, the transmission of the first data being based on a first attribute; initiating transmission of the second data to the second processing unit via the network interface, the transmission of the second data being based on a second attribute; The first attribute includes a first timestamp, the second attribute includes a second timestamp, and an order of transmission of the first data and the second data is based on the first timestamp and the second timestamp.

20. and a non-volatile memory express over fabric (NVMe-oF) target including a plurality of solid state drives (SSDs).

20. The storage device according to claim 19.

21. The first data corresponds to a chunk of a data object.

20. The storage device according to claim 19.

22. Identifying the first processing unit includes sending a query to a database to identify the first identifier.

20. The storage device according to claim 19.

Images