System comprising a storage device

Abstract

Systems including storage devices are disclosed. The storage devices can store data. A load module can read data from the storage devices based at least in part on input / output (I / O) requests. A scheduler can place an I / O request in a queue for delivery to the load module based at least in part on a size of the I / O request being less than a threshold.

Description

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 302,561, filed January 24, 2022, which is incorporated herein by reference for all purposes. Technical Field

[0002] The disclosure generally relates to storage devices, and more specifically, to systems including storage devices that can achieve low read latency. Background Technology

[0003] Some applications may rely on large amounts of data. If this data is not already stored in memory, it can be read from the storage device. However, accessing data from a single storage device can create a bottleneck, slowing down data access and the resulting computations.

[0004] Low-latency access to the data is still required. Summary of the Invention

[0005] Disclosed embodiments may include a system. The system may include a storage device for storing data and a loading module for reading data from the storage device. A scheduler may receive input / output (I / O) requests and transmit I / O requests to the loading module based on the size of the I / O request.

[0006] In one general aspect, a system including a storage device, the system comprising: a storage device for storing data; a loading module for reading the data from the storage device based at least in part on input / output (I / O) requests; and a scheduler for receiving the I / O requests and placing the I / O requests in a queue based at least in part on the size of the I / O requests being less than a threshold, for delivery to the loading module.

[0007] In one general aspect, a method for a system including a storage device is provided, the method comprising: receiving an input / output (I / O) request at a scheduler; determining the size of the I / O request by the scheduler; identifying a queue by the scheduler based at least in part on the I / O request size being less than a threshold; placing the I / O request in the queue by the scheduler for delivery to a loading module; and reading data from the storage device by the loading module based at least in part on the I / O request.

[0008] In one general aspect, an article of manufacture includes a non-transitory storage medium storing instructions that, when executed by a machine, cause: an input / output (I / O) request to be received at a scheduler; the size of the I / O request to be determined by the scheduler; a queue to be identified by the scheduler based at least in part on the size of the I / O request being less than a threshold; the I / O request to be placed in the queue by the scheduler for transmission to a loading module; and data to be read from a storage device by the loading module based at least in part on the I / O request. Attached Figure Description

[0009] The accompanying drawings described below are examples of how the disclosed embodiments may be implemented and are not intended to limit the scope of the disclosed embodiments. The various disclosed embodiments may include elements not shown in certain drawings and / or elements shown in certain drawings may be omitted. The drawings are intended to provide illustration and may not be drawn to scale.

[0010] Figure 1 The diagram illustrates a machine configured to support low-latency access to storage devices during computational processing, according to a disclosed embodiment.

[0011] Figure 2 The following is illustrated according to the disclosed embodiments. Figure 1 Details of the machine.

[0012] Figure 3 The following is illustrated according to the disclosed embodiments. Figure 1 Details of multi-process systems.

[0013] Figure 4 The document illustrates the sending of a document according to a disclosed embodiment. Figure 1 Details of input / output (I / O) requests in a multi-process system.

[0014] Figure 5 The following is illustrated according to the disclosed embodiments. Figure 3 Details of the conversion table.

[0015] Figure 6 The following is illustrated according to the disclosed embodiments. Figure 3 Details of the I / O scheduler.

[0016] Figure 7 The following is illustrated according to the disclosed embodiments. Figure 3 Details of the loaded module.

[0017] Figure 8 The following is illustrated according to the disclosed embodiments. Figure 1 Details of the computing system.

[0018] Figure 9 The use according to the disclosed embodiments is shown. Figure 1 Multi-process system processing Figure 4 The flowchart shows an example program for I / O requests.

[0019] Figure 10 The use according to the disclosed embodiments is shown. Figure 1 Multi-process system processing Figure 4 An alternative flowchart for an example program of I / O requests.

[0020] Figure 11 The following is illustrated according to the disclosed embodiments. Figure 3 The I / O scheduler in Figure 4 The flowchart shows an example program that uses priority queuing when queuing I / O requests.

[0021] Figure 12 The following is illustrated according to the disclosed embodiments. Figure 3 The manager will Figure 4 I / O requests are assigned to Figure 3 The flowchart shows an example program for loading modules.

[0022] Figure 13 The following is illustrated according to the disclosed embodiments. Figure 3 The loading module from Figure 1 A flowchart of an example program for reading data from a storage device.

[0023] Figure 14 The following is illustrated according to the disclosed embodiments. Figure 1 The computing system processes Figure 8 The flowchart shows an example program for calculating the request.

[0024] Figure 15A The following is illustrated according to the disclosed embodiments. Figure 8 computation scheduler deployment Figure 8 Processing elements to process Figure 8 The flowchart shows an example program for calculating the request.

[0025] Figure 15B Continuing with the disclosed embodiments Figure 8 computation scheduler deployment Figure 8 Processing elements to process Figure 8 Example program for computation request Figure 15A The flowchart. Detailed Implementation

[0026] Reference will now be made in detail to the disclosed embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth to enable a thorough understanding of the disclosure. However, it should be understood that those skilled in the art can practice the disclosure without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0027] It will be understood that although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of disclosure, a first module may be referred to as a second module, and similarly, a second module may be referred to as a first module.

[0028] The terminology used in the disclosed description is for the purpose of describing particular embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms are intended to include the plural forms as used in the disclosed description and the appended claims. It will also be understood that the term “and / or” as used herein means and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that, when used in this specification, the terms “comprising” and / or “including” indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Components and features in the drawings are not necessarily drawn to scale.

[0029] Some applications, such as deep learning recommendation models (DLRM), may rely on large amounts of data. DLRM may depend on embedding tables, which can be terabytes in size. Transferring large amounts of data from storage devices to memory for processing can take time. This problem can be exacerbated if multiple applications attempt to process data stored on the storage device, as storage devices may have insufficient bandwidth to handle all the requested data.

[0030] The disclosed embodiments address these problems by incorporating a system with storage devices. The storage device can be, for example, a solid-state drive (SSD). Data can be distributed across multiple storage devices, which reduces the load on a single storage device to provide all requested data. A scheduler can schedule I / O requests to one or more queues based on the size of the data to be retrieved. An input / output (I / O) processing manager can retrieve requests from the queues and can identify loading modules to retrieve data from the storage devices.

[0031] Figure 1 The diagram illustrates a machine configured, according to a disclosed embodiment, to support low-latency access to storage devices during computational processing. Figure 1 The machine 105, also referred to as the host or system, may include a processor 110, a memory 115, and storage devices 120-1 and 120-2 (collectively referred to as storage device 120). The processor 110 can be any type of processor. (For ease of illustration, the processor 110 and other components discussed below are shown external to the machine; however, the disclosed embodiments may include these components internally.) Figure 1 A single processor 110 is shown, but the machine 105 may include any number of processors, each of which may be a single-core or multi-core processor, each of which may implement a Reduced Instruction Set Computer (RISC) architecture or a Complex Instruction Set Computer (CISC) architecture (and other possibilities), and may be mixed in any desired combination.

[0032] Processor 110 may be coupled to memory 115. Memory 115 may be any type of memory (such as flash memory, dynamic random access memory (DRAM), static random access memory (SRAM), persistent random access memory, ferroelectric random access memory (FRAM), or non-volatile random access memory (NVRAM) (such as magnetoresistive random access memory (MRAM)). Memory 115 may be volatile or non-volatile memory as needed. Memory 115 may also be any desired combination of different memory types and may be managed by memory controller 125. Memory 115 may be used to store data that can be referred to as “short-term”: that is, data that is not expected to be stored for a long time. Examples of short-term data may include temporary files, data used locally by the application (which may have been copied from other storage locations), etc.

[0033] Processor 110 and memory 115 may also support an operating system under which various applications can run. These applications can issue requests (also referred to as commands) to read data from or write data to either memory 115. When storage device 120 is used to support applications that read or write data via a file system, storage device 120 can be accessed using device driver 130. Although Figure 1 Two storage devices 120 are shown, but any number (one or more) of storage devices may be present in machine 105. Storage devices 120 may each support any desired one or more protocols (including, for example, the Non-Volatile Memory Fast (NVMe) protocol). Different storage devices 120 may support different protocols and / or interfaces.

[0034] Although Figure 1 The general term "storage device" is used, but the disclosed embodiments may include any storage device format that can benefit from the use of computing storage units, examples of which may include hard disk drives and solid-state drives (SSDs). Any reference to "SSD" below should be understood to include such other embodiments disclosed. Furthermore, different types of storage devices may be mixed. For example, storage device 120-1 may be a hard disk drive, and storage device 120-2 may be an SSD.

[0035] Machine 105 may also include a multi-process system 135 and a computing system 140. The multi-process system 135 may be based on a processor 110 (or...) Figure 1 An application running on a processor (not shown on a remote machine) receives input / output (I / O) requests to manage reading data from storage device 120. I / O requests can request data from storage device 120 that can be used in computational processing. That is, given a computational request, data to be processed by the computational request can first be requested from storage device 120 in the I / O requests handled by the multiprocessing system 135. The multiprocessing system 135 can schedule data readings from storage device 120 based on the size of the I / O request to reduce latency (the time required to complete computational processing, including the time required to read the data and execute appropriate commands on the data). See below for reference. Figure 3 Further discussion on multi-process systems 135.

[0036] Once the data is read by the multi-process system 135, the computing system 140 can perform computational processing to process the data. See below for reference. Figure 8 Further discussion of computing system 140.

[0037] As described above, machine 105 may include multiple storage devices 120. By including more than one storage device 120, data requested in an I / O request can be distributed across multiple storage devices 120. By distributing data across multiple storage devices 120, read requests can be processed by each storage device 120: if these read requests are processed in parallel, the requested data can be read faster than if all data were stored on only one storage device 120. However, the disclosed embodiments may include only one storage device 120 (without the potential benefit of reading data in parallel from multiple storage devices 120).

[0038] Although Figure 1 Machine 105 is shown as including a multi-process system 135 and a computing system 140, but the disclosed embodiments may place these components elsewhere. For example, the multi-process system 135 may be included as part of machine 105, while the computing system 140 may be part of another machine accessible across a network. In practice, the disclosed embodiments may separate the storage device 120, the multi-process system 135, and the computing system 140 onto separate machines 105 connected via some network or communication path.

[0039] Figure 2 The following is illustrated according to the disclosed embodiments. Figure 1 Details of the machine. Figure 2Typically, machine 105 includes one or more processors 110, which may include a memory controller 125 and a clock 205 for coordinating the operation of the machine's components. Processor 110 may also be integrated with memory 115, which, for example, may include random access memory (RAM), read-only memory (ROM), or other state-saving media. Processor 110 may also be integrated with storage device 120 and network connector 210, which may be, for example, an Ethernet connector or a wireless connector. Processor 110 may also be connected to bus 215, and user interface 220 and input / output (I / O) interface ports managed by I / O engine 225, as well as other components, may be connected to the bus.

[0040] In explanation Figure 1 Before considering the structure and operation of the multi-process system 135, it is advisable to use... Figure 1 The various requests processed by machine 105 can be helpful. (It is available in...) Figure 1 The example application running on processor 110 is a deep learning recommendation model (DLRM) as an example of a machine learning algorithm. The DLRM application may have a Service Level Agreement (SLA) established, which manages how long query processing should take. In other words, the DLRM can expect a particular query to take a certain amount of time: if the query takes longer than that amount of time, the DLRM may wait longer than expected before continuing processing.

[0041] In order to execute the query, Figure 1 Machine 105 may need to retrieve the data in question and then perform computational processing on that data. Both retrieving the data and performing computational processing can take some time.

[0042] If the query is relatively small (e.g., involving fewer than 256 data points), data retrieval can be relatively fast, and the entire process of executing the query can meet the SLA. However, if the query is relatively large (e.g., involving more than 256 data points), data retrieval can take a sufficiently long time, causing the SLA to be unmet. Because DLRMs can have variable query sizes, relatively large queries can be expected, and the processing of retrieving stored data can take longer than the time required to perform computational processing. This is because it is not expected... Figure 1 Machine 105 does not meet SLA, therefore, it is expected to reduce the cost of... Figure 1 The time required for the storage device 120 to retrieve data: that is, to achieve low latency in data retrieval.

[0043] Figure 3 The following is illustrated according to the disclosed embodiments. Figure 1 Details of the multi-process system 135. Figure 3In this system, the multi-process system 135 may include an I / O scheduler 305, queues 310-1, 310-2 and 310-3 (which may be collectively referred to as queue 310) and an I / O process / memory pool 315.

[0044] The I / O scheduler 305 can be accessed from... Figure 1 The application running on processor 110 receives I / O requests. Such requests may identify requests to be read from storage device 120 for computation purposes (to be performed by...). Figure 1 The data (the computing system 140 processes computing requests).

[0045] The I / O scheduler determines the size of the I / O request: that is, the amount of data to be read from storage device 120 and used in the computational processing. Using the size of the I / O request, the I / O scheduler selects one of multiple queues 310 to place the I / O request.

[0046] exist Figure 3 The diagram shows three queues 310-1 to 310-3. Each queue 310 can be used to store I / O requests of different sizes. For example, queue 310-1 can be used to store I / O requests for retrieving data (rows of an embedded table) with a size no greater than, for example, 128 vectors; queue 310-2 can be used to store I / O requests for retrieving data with a size greater than, for example, 128 vectors but no greater than, for example, 512 vectors; and queue 310-3 can be used to store I / O requests for retrieving data with a size greater than, for example, 512 vectors. In this way, I / O requests can be grouped roughly based on the amount of data to be retrieved, which can represent the amount of time required to retrieve the data.

[0047] Although Figure 3Three queues 310 are shown, but the disclosed embodiments may include any number (one or more) of queues 310. For example, if priority queuing is used, the disclosed embodiments may include one queue 310. If priority queuing is used, the I / O scheduler 305 may determine the priority of an I / O request based on the size of the data to be read from storage device 120, and may associate a priority label with an I / O request in queue 310, so that the I / O process / storage pool 315 can determine the relative priority of the I / O request. For example, if an I / O request requests to read no more than 128 embedding vectors (rows of an embedding table), the priority label may indicate that the request has priority 1; if an I / O request requests to read more than 128 embedding vectors but no more than 512 embedding vectors, the priority label may indicate that the request has priority 2; and if an I / O request requests to read more than 512 embedding vectors, the priority label may indicate that the request has priority 3. As with the number of queues 310, any number of different priorities may be used: the three priorities described above are merely example numbers of priorities.

[0048] In some of the disclosed embodiments, queue 310 may be a first-in, first-out (FIFO) queue. In other disclosed embodiments, other types of queue 310 may be used.

[0049] Note that even if there is only one queue 310, the choice of queue can still technically be based on the size of the I / O requests (even if all I / O requests can be placed in that queue). Furthermore, if there is only one queue, it is not a FIFO queue. That is, I / O requests can be removed from the queue in a different order than the order in which they were added (for example, a priority 1 I / O request added to the queue later than a priority 2 I / O request can still be removed from the queue first and processed first).

[0050] Once the I / O scheduler 305 has placed an I / O request in queue 310, the I / O process / memory pool 315 can retrieve the I / O request from queue 310. By using multiple queues 310 (or by using different priorities), the I / O process / memory pool 315 can select which I / O request to process next. In this way, I / O requests can be processed by the I / O process / memory pool 315 in conjunction with their requests from queue 310. Figure 1 Applications running on processor 110 are sent to multi-process system 135 for processing in different sequences.

[0051] I / O process / memory pool 315 can retrieve I / O requests from queue 310 using any desired technique. For example, I / O process / memory pool 315 can use round-robin access, retrieving I / O requests from queue 310-1, then from queue 310-2, then from queue 310-3, and then returning to queue 310-1, and so on. (Of course, if there are no I / O requests in queue 310, I / O process / memory pool 315 can skip queue 310 and move to the next queue 310 to retrieve an I / O request.)

[0052] I / O process / storage pool 315 may include manager 320, which is responsible for retrieving I / O requests from queue 310. Manager 320 is also responsible for determining which storage device(s) 120 store the requested data (the data may be stored on a single storage device(s) 120 or on multiple storage devices(s) 120). Once manager 320 has determined which storage device(s) 120 store the requested data, manager 320 may dispatch the I / O request to one or more load modules 325-1 to 325-4 (collectively referred to as one or more load modules 325) to read data from one or more storage devices(s) 120.

[0053] In some disclosed embodiments, each storage device 120 may be considered separate from the other storage devices 120. That is, there may be no predetermined relationship between the storage devices 120 to manage their use. For example, each storage device 120 may be considered not only a physically separate storage device, but also a logically separate storage device (the arrangement of storage devices may be compared, for example, to a redundant array of independent disks (RAID) in which the management of data storage is left to a RAID controller). However, in other disclosed embodiments, the storage devices 120 may be configured as an array (such as RAID).

[0054] To determine which storage device(s) 120 stores the data requested in the I / O request, the manager 320 can access a table (or translation table) 330. Table 330 can function similarly to a flash translation layer in an SSD. However, instead of mapping logical addresses (such as logical block addresses of data used by an application) to physical addresses (on the storage device), table 330 maps logical addresses (or some other identifier of the data) to identifiers of one or more storage devices 120 storing the requested data. Table 330 can be stored in some form of storage device (e.g., volatile storage devices (such as local DRAM) or non-volatile storage devices (such as firmware modules or flash storage devices)). References are as follows. Figure 5 Further discussion on the use of Table 330.

[0055] Once the manager 320 has dispatched an I / O request to one or more loading modules 325, the one or more loading modules 325 can access the requested data from one or more storage devices 120. For example, if the data is stored on storage device 120-1, the manager 320 can dispatch the I / O request to loading modules 325-1 and / or 325-2; if the data is stored on storage device 120-2, the manager 320 can dispatch the I / O request to loading modules 325-3 and / or 325-4. Figure 3 In the diagram, the I / O process / storage pool 315 is shown as including two storage devices 120 and four loading modules 325. The disclosed embodiments may include any number (one or more) of storage devices 120 and any number (one or more) of loading modules 325 (however, at least one loading module 325 should exist for each storage device 120).

[0056] Note that in Figure 3 In the disclosed embodiments, each storage device 120 has two loading modules 325 capable of accessing data to the storage device. In some disclosed embodiments, the storage device 120 may support multi-threaded access: that is, the storage device 120 may support reading data to satisfy multiple requests simultaneously. For example, if the storage device 120 includes multiple channels, each of which can be used independently of the others, one thread may request data stored along one channel, and another thread may request data stored on a second channel, with both threads operating concurrently. In disclosed embodiments that include multiple loading modules 325 capable of simultaneously accessing the storage device 120 but whose data can be accessed by only one loading module 325, the table 330 may also include an identifier for a specific loading module 325 for retrieving the requested data. Furthermore, in some disclosed embodiments, each storage device 120 may have only one loading module 325. The disclosed embodiments may also include combinations of these possibilities: for example, one storage device 120 may support multi-threading and be accessible by multiple loading modules 325, while another storage device 120 may not support multi-threading and therefore can be accessed by only one loading module 325.

[0057] exist Figure 3 In this context, loading module 325 can be a sparse length (SLS) loading module 325. See below for reference. Figure 7 Further discussion on SLS loading module 325.

[0058] Loading module 325 can access storage device 120 using, for example, a user-space non-volatile memory fast (UNVMe) drive. While drives accessing storage device 120 can use a file system, UNVMe drives can access data directly from storage device 120 without using a file system. Loading module 325 can also access data using various application programming interfaces (APIs) provided by storage device 120.

[0059] As shown above (refer to the reference) Figure 1 The storage device 120 discussed can be any desired type of storage device (such as hard disk drives and SSDs). Furthermore, variations of these various storage devices can also be used. For example, the storage device 120 can be an SSD optimized for storing and retrieving data to satisfy DLRM queries: such an SSD may have a different architecture than an SSD designed for general use.

[0060] In some disclosed embodiments, the data requested in an I / O request may be stored on a specific storage device 120. Therefore, multiple I / O requests may be sent to the same storage device 120. When accessing data to a specific storage device 120, the loading module 325 may use a submission queue to manage multiple requests to the same storage device 120. The loading module 325 may also consider the size of the requests and the availability of the submission queue in an attempt to balance the load on the storage device 120.

[0061] In the above discussion, how the data is stored on storage device 120 was not discussed. In some disclosed embodiments, storage device 120 may be preloaded with data and table 330 may be prepared in advance; in other disclosed embodiments, an application may request data to be written to storage device 120, and manager 330 may select which storage device(s) ...)(s))(s)))(s))))))))))),)) the)(s)))))))")))"")))"" 330))"" 330)) 450.)

[0062] It is expected that all storage devices 120 across I / O processes / storage pool 315 will access data approximately equally (this characteristic can be described as load balancing). However, depending on how data is stored across multiple storage devices 120 and what applications are requesting data from storage devices 120, the load on each storage device 120 may not be balanced. For example, although storage devices 120-1 and 120-2 may store the same amount of data in terms of size, it is possible that, for example, 80% of I / O requests request data stored on storage device 120-1 (and only 20% of I / O requests request data stored on storage device 120-2). In such a case, the unbalanced load can result in higher latency than expected when accessing data to storage device 120-1.

[0063] To address this situation, the I / O process / memory pool 315 may include a migration module. Figure 3 (Not shown in the image). The migration module can be responsible for moving data between storage devices 120 to achieve the desired balance. For example, some data may be migrated from storage device 120-1 to storage device 120-2 in an attempt to balance how much data is requested from each storage device.

[0064] There are other reasons why data may move between storage devices 120. While read load balancing may be an important objective, it may also be important to keep the storage devices roughly balanced in capacity (e.g., to support new data being written by another application that can be distributed across multiple storage devices 120). Alternatively, some data may be accessed frequently enough, or considered important enough, to justify storing such data across multiple storage devices 120 to provide redundancy.

[0065] Regardless of why the data is migrated (e.g., for storage capacity balancing, read load balancing, or redundancy), the migration tool can update table 330 to reflect such a change. That is, if data is migrated from storage device 120-1 to storage device 120-2, table 330 can be updated to reflect the data migration.

[0066] Figure 4 The document illustrates the issuance of a document according to a disclosed embodiment. Figure 1 Details of input / output (I / O) requests for the multi-process system 135. Figure 4 The diagram illustrates I / O request 405. I / O request 405 is shown as including identifier 410 and vectors 415-1 to 415-7 (which may be collectively referred to as vector 415). For example, vector 415 may include 64 data points, but disclosed embodiments may include any number of data points per vector. Identifier 410 may be a vector... Figure 1 The identifier of the data requested by the multi-process system 135. For example, identifier 410 could be defined by... Figure 1 The logical address of the data used by the application running on the processor 110; however, the disclosed embodiments may use any desired identifier for the data.

[0067] Vector 415 can identify a specific vector from the data of interest. As discussed above, DLRM queries can use data from an embedded table (embedded vector), which can be very large (up to hundreds of GB or TB). However, queries may depend only on specific data within the table, and reading the entire table can take a very long time relative to the amount of data actually needed. Conversely, I / O request 405 can include vector 415, which identifies a specific vector of interest from the embedded table, and all other vectors can be ignored. Although Figure 4Seven vectors 415 are shown in I / O request 405, but the disclosed embodiments may include any number of vectors.

[0068] Additionally, if I / O request 405 includes vector 415, then Figure 3 The I / O scheduler 305 may be able to determine the size of the data to be read. For example, Figure 3 The I / O scheduler 305 can determine the number of bytes to be read by multiplying the number of vectors 415 in the I / O request 405 by the number of data points in each vector 415, and by the size of each data point in each vector 415. If the number of vectors 415 in the I / O request 405 is, for example, 64, and each vector 415 includes, for example, 128 data points, and each data point requires four bytes, then the size of the data to be read by the I / O request 405 can be determined as 64 (number of vectors 415 in the I / O request 405) × 128 (number of data points in each vector 415) × 4 (number of bytes per data point) = 32,768B.

[0069] Although Figure 4 I / O request 405 is shown as including only identifier 410 and vector 415, but the disclosed embodiments may include other data, or some of the data shown may be removed. For example, instead of including vector 415, I / O request 405 may include an offset of a logical address (in... Figure 4 In this context, the logical address is used as identifier 410, but the disclosed embodiments can distinguish between identifier 410 and the logical address of data. In this case, the logical address can be separate data included in I / O request 405 and the number of bytes to be read (in this case, the number of bytes to be read can be used to determine the size of I / O request 405). Alternatively, I / O request 405 may include... Figure 4 Various labels are not shown. Disclosed embodiments may include any such variations of I / O request 405.

[0070] Figure 5 The following is illustrated according to the disclosed embodiments. Figure 3 Details of Table 330. Figure 5 Table 330 is shown as including three entries. One entry maps the identifier 410-1 of the first data to the identifier 505-1 of the storage device storing the data; another entry maps the identifier 410-2 of the second data to the identifier 505-2 of the storage device storing the data; and a third entry maps the identifier 410-3 of the third data to the identifier 505-3 of the storage device storing the data. Although Figure 5 Table 330 is shown as including three entries, but the disclosed embodiments may include any number (zero or more) of entries.

[0071] Note that identifiers 505-1, 505-2, and 505-3 (collectively referred to as identifier 505) are shown as numerical identifiers of specific storage devices. Table 330 may not store the actual physical address, as this information can be stored by the storage device itself. Identifier 505 may be replaced with other information that uniquely identifies the storage device: for example, information assigned during discovery and / or enumeration. Figure 1 The identifier of the storage device 120, or by using Figure 1 The serial number of the storage device 120 and other possibilities.

[0072] Although Figure 5 This indicates that each identifier 410 can be associated with a single unique identifier 505, but the disclosed embodiments may map identifier 410 to one or more identifiers 505. Table 330 may reflect the fact that data associated with identifier 410-1 is stored on multiple storage devices (e.g., to provide redundancy). If the data in question is stored on... Figure 1 On the multiple storage devices 120, then Figure 3 The manager 320 can have the ability to... Figure 4 I / O request 405 is assigned to Figure 3 The option to select more than one loading module 325. This selection, for example, is for load balancing. Figure 3 In I / O process / memory pool 315 Figure 1 The load on storage device 120 can be useful. For example, if Figure 1 The storage device 120-1 has a relatively large amount of Figure 4 The I / O request is 405 pending processing and the data is available from Figure 1 If both storage devices 120-1 and 120-2 are obtained, then Figure 3 Manager 320 can be selected Figure 3 The loading module 325-3 or 325-4 is used to... Figure 1 Storage device 120-2 executes I / O request 405.

[0073] In some of the disclosed embodiments, Figure 4 The data requested in the I / O request 405 can be stored in Figure 1 On a single storage device 120. In such a disclosed embodiment, identifier 505 can uniquely identify where the data represented by identifier 410 may be located. However, in other disclosed embodiments, data may be located across... Figure 1 Multiple storage devices 120 are distributed. In such a disclosed embodiment, Figure 3 The manager 320 can determine Figure 4 All data requested in the 405 I / O request can be crossed Figure 1Multiple storage devices 120 are distributed and can be used to... Figure 4 The I / O request 405 is divided into multiple different I / O requests, each of which is sent to Figure 3 Different loading modules 325. Table 330 can identify... Figure 1 Which storage device 120 stores what data? Therefore, Table 330 may include multiple entries for different parts of the data.

[0074] As an example, let's consider again Figure 4 The I / O request is 405. Figure 4 The I / O request 405 request comes from Figure 4 The data of five vectors, 415. Figure 3 The manager 320 can determine Figure 4 Vectors 415-1 and 415-4 are stored in Figure 1 On the storage device 120-1, and Figure 4 Vectors 415-2, 415-3, and 415-5 are stored Figure 1 On storage device 120-2. In this case, Figure 3 The manager 320 can send an I / O request to Figure 3 Load module 325-1 or 325-2 to read Figure 4 The vectors 415-1 and 415-4 can be used to send another I / O request to Figure 3 Load module 325-3 or 325-4 to read Figure 4 The vectors are 415-2, 415-3, and 415-5.

[0075] In addition, recall Figure 3 Multiple loading modules 325 are accessible Figure 1 The same storage device 120. In this case, even if the data to be read is only stored in, for example Figure 1 On storage device 120-1, Figure 3 The Manager 320 can also send two I / O requests: one I / O request to Figure 3 Loading module 325-1, another I / O request to Figure 3 Loading module 325-2. In this way, even Figure 3 A single loading module 325 may be able to process from Figure 1 Storage device 120-1 reads all data. Figure 3 Manager 320 can also speed up the process from Figure 1 The storage device 120-1 reads data.

[0076] Figure 6 The following is illustrated according to the disclosed embodiments. Figure 3Details of the I / O scheduler 305. Figure 6 In this configuration, the I / O scheduler 305 may include a size calculator 605, a threshold 610, a comparator 615, and a queue selector 620. (See above reference...) Figures 3 to 4 The I / O scheduler 305, as discussed, can be used Figure 4 The size of the I / O request 405 is used to select Figure 4 I / O request 405 can be placed in Figure 3 Queue 310. Figure 6 Size calculator 605 can determine Figure 4 The size of I / O request 405. Among other data, the size calculator can be used based on... Figure 4 The I / O request 405 indicates the number of bytes to be read from the data, or Figure 4 The number of vectors 415 combined Figure 4 The number of data points in each vector 415 and the number of bytes per data point are used to calculate Figure 4 The size of the I / O request is 405.

[0077] Once the size calculator 605 has been determined Figure 4 The I / O request size is 405, and the comparator 615 can handle it. Figure 4 The size of I / O request 405 is compared with threshold 610. Threshold 610 can be compared with... Figure 4 The size of the I / O request 405 is compared with any expected threshold, and the queue selector 620 can be used to select any expected threshold. Figure 3 Select from queue 310 to place Figure 4 This information is used when making an I / O request 405. If determined by the size calculator 605 according to comparator 615... Figure 4 If the size of I / O request 405 is less than the threshold 610, then queue selector 620 can select... Figure 3 A queue of 310 is used to place Figure 4 I / O request 405; otherwise, queue selector 620 can select Figure 3 Another queue 310 is used to place Figure 4 The I / O request is 405.

[0078] Although Figure 6 A threshold 610 is shown, but the disclosed embodiments may include any number of thresholds 610, and the comparator 615 may... Figure 4 The size of I / O request 405 is compared with each threshold 610 until a value less than 40 is identified. Figure 4 Until the maximum threshold of 610 for the size of the I / O request 405 is reached (or alternatively, until a value greater than 405 is identified). Figure 4The minimum threshold 610 for the size of the I / O request 405 is then used by the queue selector 620 to select the queue. Figure 3 Used for placing Figure 4 The I / O request 405 is in queue 310.

[0079] For example, as referenced above Figure 3 Discussed Figure 3 Queue 310-1 can be used to store I / O requests for retrieving data of no more than, for example, 128 embedded vectors; Figure 3 Queue 310-2 can be used to store I / O requests for retrieving data with, for example, 128 embedding vectors but not, for example, 512 embedding vectors; and Figure 3 Queue 310-3 can be used to store I / O requests for retrieving data with, for example, more than 512 embedding vectors. For this example, two thresholds 610 can be used: one at 128 embedding vectors and another at 512 embedding vectors. Therefore, in some disclosed embodiments, Figure 3 The number of queues 310 can be 1 greater than the number of queues 610 (where the threshold 610 acts as...). Figure 3 (The dividing line between queues 310).

[0080] Figure 7 The following is illustrated according to the disclosed embodiments. Figure 3 Details of loading module 325. See above for reference. Figure 4 Discussed Figure 4 I / O request 405 may indicate that a read is to be made from an embedded table. Figure 4 The specific vector is 415. Because... Figure 4 I / O request 405 Figure 4 The number of vectors 415 can be relatively small compared to the number of vectors in the data, so most of the data can be ignored. Therefore, since most values ​​can satisfy... Figure 4 The destination of the I / O request 405 was ignored, so Figure 1 The data in the storage device 120 can be considered "sparse".

[0081] When load module 325 is a "sparse length sum" load module, load module 325 can read from... Figure 4 The I / O request identified in 405 Figure 4 A specific vector 415 is used, and these vectors are added together to produce a single vector. Then it can be returned. Figure 4 The sum of the vector 415 of the identifier is used as the result of Figure 4 The I / O request is a 405 request for data.

[0082] For operation as an SLS loading module 325, the loading module 325 may include a reader 705 and an adder 710. The reader 705 can access... Figure 1 The storage device 120 reads from Figure 4 The I / O request identified in 405 Figure 4 The specific vector 415. Note that in some disclosed embodiments, the reader 705 may in some way direct to... Figure 1 The storage device 120 accesses data; in other disclosed embodiments, the reader 705 may access data from the storage device 120. Figure 1 The storage device 120 issues an appropriate command, and the storage device 120 can return data to the loading module 325. The adder 710 can then load the data from... Figure 1 The vectors retrieved by the storage device 120 are added together to generate a response to Figure 4 The data returned when the I / O request is 405.

[0083] Figure 8 The following is illustrated according to the disclosed embodiments. Figure 1 Details of the computing system 140. Figure 8 In the middle, the computing system 140 can receive a computing request 805, which can be a processing request from... Figure 1 Multi-process system 135 response Figure 4 The I / O request 405 is a request for data retrieved. The computing system 140 may include a computing scheduler 810, queues 815-1 and 815-2 (collectively referred to as queues 815), and processing elements 820-1, 820-2, and 820-3 (collectively referred to as processing elements 820).

[0084] The compute scheduler 810 can place compute request 805 into one of a plurality of queues 815 based on the workload of compute request 805. For example, compute request 805 may involve resources provided only by one of a plurality of processing elements 820, and processing element 820 can determine which queue 815 should be placed in for compute request 805. As discussed below, compute scheduler 810 may also consider how busy processing element 820 is when allocating compute request 805 to processing element 820.

[0085] In some disclosed embodiments, queue 815 may be a FIFO queue. In other disclosed embodiments, other types of queue 815 may be used.

[0086] Then, processing element 820 can remove computation request 805 from queue 815 and process the request. Processing element 820 can be any desired type of processing element: for example, a central processing unit (CPU), graphics processing unit (GPU), general-purpose GPU (GPGPU), neural processing unit (NPU), tensor processor (TPU), or accelerator (such as a field-programmable gate array (FPGA) or application-specific integrated circuit (ASIC)), and other possibilities. Additionally, for elements comprising multiple cores (e.g., a multi-core CPU), each core can be considered a separate processing element 820.

[0087] In some disclosed embodiments, each processing element 820 may have its own queue 815 from which it receives computation requests for processing. That is, a processing element 820 may process only computation requests specifically assigned to it and may ignore computation requests assigned to other processing elements. In other disclosed embodiments, two or more processing elements may share a queue. For example, as... Figure 8 As shown, processing elements 820-1 and 820-2 can both receive computation requests via queue 815-1, while processing element 820-3 can receive computation requests via queue 815-2. Once a processing element has finished processing a computation request, it can then check the appropriate queue to see if other computation requests are waiting for it to process. If the processing element finds a computation request it can process waiting, it can begin processing that request; otherwise, the processing element can become idle.

[0088] In some disclosed embodiments, the processing element can search for computation requests in multiple queues 815. For example, processing element 820-3 may be able to process any computation request that processing elements 820-1 and 820-2 can process, but may also be able to process some additional computation requests. In this case, processing element 820-3 can retrieve a computation request from queue 815-2 as long as there are computation requests waiting to be processed; if queue 815-2 is empty, processing element 820-3 can retrieve a computation request from queue 815-1.

[0089] The computing system 140 may also include a ready queue 825. When a processing element 820 has finished processing a computing request, it can use the ready queue 825 to notify the computing scheduler 810. In this way, the computing scheduler 810 can track how busy the processing elements 820 are. For example, considering the scenario where the computing scheduler 810 receives a computing request 805, and assuming each processing element 820 has its own queue 815, the computing scheduler 810 can determine whether processing element 820-1 or 820-2 is capable of processing the computing request. Without information about how busy processing element 820-1 or 820-2 is, the computing scheduler 810 can randomly assign the computing request 805 to the queue associated with processing element 820-1 or 820-2. However, if the computation scheduler 810 receives information via the ready queue 825 that the processing element 820-2 has completed its latest computation request (and is therefore currently idle), the computation scheduler 810 can assign the computation request 805 to the processing element 820-2 without having to guess which of the processing elements 820-1 and 820-2 has a light workload.

[0090] Similarly, although processing element 820-1 or 820-2 may be the more desirable processing element for processing computation request 805, if processing elements 820-1 and 820-2 are currently busy and processing element 820-3 is currently idle, the computation scheduler 810 may schedule processing element 820-3 to process computation request 805 even if it is not desirable for processing element 805-3 to process computation request 805.

[0091] Computation request 805 may include a tag identifying the computation request. Optionally, computation scheduler 810 may assign a tag to computation request 805 to identify the computation request. Processing element 820 may use these tags in ready queue 825 to notify computation scheduler 810 which computation requests have been completed. In this way, computation scheduler 810 can maintain an approximate intention of the workload to be processed by processing element 820 (by comparing what computation requests have been processed by each processing element 820 with what computation requests have been scheduled for each processing element 820).

[0092] Figure 9 The use according to the disclosed embodiments is shown. Figure 1 Multi-process system 135 processing Figure 4 A flowchart of an example program for I / O requests. Figure 9 In the middle, at block 905, Figure 3 The I / O scheduler 305 can receive Figure 4 I / O request 405. At block 910, Figure 4 I / O request 405 can be obtained from Figure 3The I / O scheduler 305 is transmitted to Figure 3 Loading module 325. Finally, at block 915, Figure 3 The loading module 325 can be based on Figure 4 I / O request 405 from Figure 1 The storage device 120 reads data.

[0093] Figure 10 The use according to the disclosed embodiments is shown. Figure 1 Multi-process system processing Figure 4 An alternative flowchart for an example program of I / O requests.

[0094] Figure 10 The use according to the disclosed embodiments is shown. Figure 1 Multi-process system 135 processing Figure 4 An alternative flowchart for the example program that encountered I / O request 405. Figure 10 In, the same sub-icon number can be used to assign to and Figure 9 Similar blocks in [the context]. Figure 10 In the middle, at block 905, Figure 3 The I / O scheduler 305 can receive Figure 4 I / O request 405. At block 1005, Figure 6 Size calculator 605 can determine Figure 4 The I / O request size is 405. At block 1010, Figure 6 The queue selector 620 can be used for Figure 4 I / O request 405 Select Figure 3 Queue 310. At block 1015, Figure 3 The I / O scheduler 305 can Figure 4 The I / O request 405 is placed by Figure 6 The queue selector 620 selects Figure 3 In queue 310. Finally, at block 915, Figure 3 The loading module 325 can be based on Figure 4 I / O request 405 from Figure 1 The storage device 120 reads data.

[0095] Figure 11 The following is illustrated according to the disclosed embodiments. Figure 3 The I / O scheduler 305 in its operation Figure 4 A flowchart of an exemplary procedure for queuing I / O requests when queuing for I / O request 405. Figure 11 In the middle, at block 1105, Figure 3 The I / O scheduler 305 can be based on Figure 4 The size of the I / O request 405 (which can be determined by...) Figure 6Size calculator 605 (determine) to determine Figure 4 The priority label for the I / O request 405. At block 1110, Figure 3 The I / O scheduler 305 can associate priority tags with... Figure 3 In queue 310 Figure 3 This is associated with I / O request 405.

[0096] Figure 12 The following is illustrated according to the disclosed embodiments. Figure 3 Manager 320 will Figure 4 I / O request 405 is assigned to Figure 3 The flowchart of the example program for loading module 325. Figure 12 In the middle, at block 1205, Figure 3 The manager 320 can be accessed from Figure 3 Queue 310 retrieval Figure 4 I / O request 405. At block 1210, Figure 3 The Manager 320 can recognize stored data Figure 1 Storage device 120. As shown in block 1215, this may involve, for example, using... Figure 3 Table 330 will contain the data Figure 4 The identifier 410 is mapped to the storage device storing the data. Figure 5 The identifier is 505. At block 1220, Figure 3 The Manager 320 can recognize Figure 3 Accessible Figure 1 The loading module 325 of the storage device 120. Finally, at block 1225, Figure 3 The Manager 320 can Figure 4 The I / O request 405 was sent to Figure 3 Loading module 325.

[0097] Figure 13 The following is illustrated according to the disclosed embodiments. Figure 3 Loading module 325 from Figure 1 A flowchart of an example program for reading data from a storage device. Figure 13 In the middle, at block 1305, Figure 7 The reader 705 can be from Figure 1 Storage device 120 read Figure 4 Vector 415. At block 1310, Figure 7 The adder 710 can perform... Figure 4 Summing vector 415. Finally, at block 1315, Figure 1 The multi-process system 135 can send data to Figure 1 The computing system 140 Figure 8 The computation scheduler 810 is used for processing Figure 8 The calculation request is 805.

[0098] Figure 14 The following is illustrated according to the disclosed embodiments. Figure 1 The computing system 140 processes Figure 8 A flowchart of an example program for calculating a request. Figure 14 In the middle, at block 1405, Figure 8 The computation scheduler 810 can be used by Figure 1 Multi-process system 135 response Figure 4 The data read from the I / O request 405 is used for scheduling. Figure 4 The computation request 805 is processed. At block 1410, Figure 8 The processing element 820 can be from Figure 8 The computing scheduler 810 receives Figure 8 The computation request is 805. For example, in block 1405, Figure 8 The computing scheduler 810 can Figure 8 The computation request 805 is placed Figure 8 In queue 815, and in block 1410, Figure 8 The processing element 820 can be from Figure 8 Queue 815 retrieval Figure 8 The calculation request is 805.

[0099] At block 1415, Figure 8 The processing element 820 can process Figure 8 The computation request is 805. Finally, at block 1420, Figure 8 The processing element 820 can notify Figure 8 The computation scheduler 810: It has been completed Figure 8 The computation request 805 requires processing. For example, Figure 8 The processing element 820 can place information Figure 8 In the ready queue 825, to notify Figure 8 Computation scheduler 810: Figure 8 The processing element 820 has been completed. Figure 8 The processing of the computation request 805. In some disclosed embodiments, as shown by dashed line 1425, block 1420 may be omitted (e.g., if in Figure 8 The processing element at 820 only schedules Figure 8 One of the computation requests is 805, and Figure 8 The computation scheduler 810 knows Figure 8 How long should the processing element 820 take to process? Figure 8 (Calculation request 805).

[0100] Figures 15A to 15B The following is illustrated according to the disclosed embodiments. Figure 8 The computation scheduler 810 is arranged Figure 8 The processing element 820 is used to process Figure 8 The flowchart of the example program for calculating request 805. Figure 15A In the middle, at block 1505, Figure 8 The compute scheduler 810 is selectable Figure 8 The processing element 820 is used to process Figure 8 The computation request is 805. At block 1510, Figure 8 The computing scheduler 810 can Figure 8 The calculation request 805 was sent Figure 8 The processing element 820. Optionally, at block 1515, Figure 8 The computing scheduler 810 can Figure 8 The computation request 805 was assigned to Figure 8 Queue 815.

[0101] As a replacement for block 1505, in block 1520 ( Figure 15B ) place, Figure 8 The computation scheduler 810 can identify tasks suitable for processing Figure 8 Request 805 Figure 8 The processing element is of type 820. At block 1525, Figure 8 The computing scheduler 810 can Figure 8 The computation request 805 is assigned to the appropriate Figure 8 The processing element of type 820 Figure 8 Queue 815.

[0102] In another alternative to block 1505, at block 1530, Figure 8 The computation scheduler 810 can determine Figure 8 The computation request is for workload 805. At block 1535, Figure 8 The computation scheduler 810 can be based on Figure 8 The computational request 805 workload will Figure 8 The computation request 805 was assigned to Figure 8 Queue 815.

[0103] exist Figures 9 to 15B Some embodiments disclosed are shown in the figures. However, those skilled in the art will recognize that other disclosed embodiments are also possible by changing the order of blocks, by omitting blocks, or by including links not shown in the figures. All such variations of the flowchart, whether explicitly described or not, are considered to be disclosed embodiments.

[0104] The disclosed embodiments include a multi-process system. A multi-process system can load data from an I / O process / memory pool using input / output (I / O) requests, which can be based on data to be processed using computational requests. Data can be retrieved using a loading module associated with a storage device in the I / O process / memory pool. When reading data, the use of multiple storage devices achieves low latency by leveraging parallel data access to multiple storage devices, providing a technological advantage over storing data in a single storage device.

[0105] Different I / O requests can be queued in different queues. The use of multiple queues offers technical advantages because I / O requests that may involve large or small amounts of data are not delayed by multiple I / O requests of different data sizes.

[0106] Data retrieved by a multi-process system can be provided to a computing system. The computing system can use the data to schedule computing requests. The computing system can use different queues based on the workload of computing requests, thereby providing the technical advantage of meeting the queries per second promised by the service level agreement.

[0107] Deep learning recommendation model (DLRM) workloads can be input / output (I / O) intensive. To meet service level agreement (SLA) requirements, dynamic random access memory (DRAM) may be needed to store large embedding tables (up to 100GB or more). Such a large amount of DRAM can be expensive.

[0108] For small query sizes, solid-state drives (SSDs) with user-space drives can be used to store embedded tables to meet the SLA. However, for reasonably large query sizes (>=256), a single SSD may struggle to meet the SLA. Furthermore, with a single SSD, queries may not be able to be executed in parallel to achieve high queries per second (QPS) due to potential input / output (I / O) bottlenecks within the SSD.

[0109] Query schedulers based on computation with multiple SSDs can be inefficient and prone to load balancing issues (where some SSDs handle a large percentage of queries, while others handle a smaller percentage). SSDs handling a large number of queries may not have sufficient I / O to process them (very similar to a single SSD model), while SSDs handling a small number of queries may be underutilized.

[0110] Disclosed embodiments may include multi-process and multi-SSD systems with an I / O scheduler for achieving QPS and low latency. Scheduled embedded table I / O can be used to schedule queries to different I / O queues.

[0111] The I / O process / SSD pool can retrieve I / O requests from the I / O queue based on I / O requests and load status for various SSDs.

[0112] Within the IO process / SSD pool, multiple SSDs can be accessed by a multi-process userspace non-volatile memory fast (UNVME) driver / application programming interface (API) with varying numbers of active threads to satisfy I / O requests.

[0113] The second-level computation scheduler can be used to further optimize computation latency and QPS based on computational intensity.

[0114] The following discussion aims to provide a brief, general description of one or more suitable machines capable of implementing specific aspects of the disclosure. One or more machines can be controlled at least in part by input from conventional input devices (such as keyboards, mice, etc.) and by instructions received from another machine, interaction with a virtual reality (VR) environment, biometric feedback, or other input signals. As used herein, the term "machine" is intended to broadly include a single machine, a virtual machine, or a system of machines, virtual machines, or devices operating together communicatively connected to each other. Exemplary machines include computing devices (such as personal computers, workstations, servers, portable computers, handheld devices, telephones, tablet computers, etc.) and transportation devices (such as private or public transportation vehicles (e.g., cars, trains, taxis, etc.)).

[0115] One or more machines may include embedded controllers (such as programmable or non-programmable logic devices or arrays, application-specific integrated circuits (ASICs), embedded computers, smart cards, etc.). One or more machines may utilize one or more connections to one or more remote machines (such as via network interfaces, modems, or other communicative couplings). Machines may be interconnected via physical and / or logical networks (such as intranets, the Internet, local area networks, wide area networks, etc.). Those skilled in the art will understand that network communications may utilize various wired and / or wireless short-range or long-range carriers and protocols (including radio frequency (RF), satellite, microwave, IEEE 802.11, etc.). (Optics, infrared, cables, lasers, etc.)

[0116] Embodiments of this disclosure can be described by referring to or in conjunction with associated data including functions, programs, data structures, applications, etc., which, when accessed by a machine, cause the machine to perform tasks or define abstract data types or low-level hardware contexts. The associated data may be stored, for example, in volatile and / or non-volatile memory (e.g., RAM, ROM, etc.), or in other storage devices and their associated storage media (including hard disk drives, floppy disks, optical storage devices, magnetic tapes, flash memory, memory sticks, digital video disks, bio-storage devices, etc.). The associated data may be transmitted in the form of packets, serial data, parallel data, propagated signals, etc., through a transmission environment including physical and / or logical networks, and may be used in compressed or encrypted formats. The associated data may be used in a distributed environment and stored locally and / or remotely for machine access.

[0117] Disclosed embodiments may include a tangible, non-transitory machine-readable medium comprising instructions executable by one or more processors, including instructions for performing the disclosed elements as described herein.

[0118] The various operations of the methods described above can be performed by any suitable means capable of performing the operations (such as various hardware and / or one or more software components, circuits and / or one or more modules). The software may include an ordered list of executable instructions for implementing logical functions and may be implemented in any processor-readable medium for use by or in conjunction with an instruction execution system, device, or apparatus (such as a single-core or multi-core processor or a system containing a processor).

[0119] The blocks or steps of methods, algorithms, and functions described in conjunction with the embodiments disclosed herein can be implemented directly in hardware, as software modules executed by a processor, or a combination of both. If implemented in software, the functions may be stored as one or more instructions or code on or transmitted through a tangible, non-transitory computer-readable medium. The software modules may reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0120] Having described and illustrated the principles of the disclosure with reference to the illustrated embodiments, it will be appreciated that the illustrated embodiments may be modified in arrangement and detail without departing from such principles, and that the illustrated embodiments may be combined in any desired manner. Furthermore, although the foregoing discussion focuses on particular embodiments, other configurations are contemplated. Specifically, even though expressions such as "according to the disclosed embodiments" are used herein, these phrases are intended to refer generally to the possibilities of embodiments and not to limit the disclosure to particular embodiment configurations. As used herein, these terms may refer to the same or different embodiments that can be combined into other embodiments.

[0121] The foregoing illustrative embodiments are not to be construed as limiting the scope of the disclosure. Although some embodiments have been described, those skilled in the art will readily understand that many modifications may be made to those embodiments without substantially departing from the novel teachings and advantages of this disclosure. Therefore, all such modifications are intended to be included within the scope of the disclosure as defined in the claims.

[0122] The disclosed embodiments are extendable to, but not limited to, the following statements:

[0123] Statement 1: The disclosed embodiments include a system comprising:

[0124] Storage device, used to store data;

[0125] A loading module for reading the data from the storage device, at least in part, based on input / output (I / O) requests; and

[0126] A scheduler is used to receive the I / O request and transmit the I / O request to the loading module based at least in part on the size of the I / O request.

[0127] Statement 2. The disclosed embodiments include the system described in Statement 1, wherein the scheduler is configured to deliver I / O requests to a queue based at least in part on the size of the I / O request.

[0128] Statement 3. The disclosed embodiments include the system described in Statement 2, the system further comprising: a manager for retrieving the I / O request from the queue and assigning the I / O request to a loading module.

[0129] Statement 4. The disclosed embodiments include the system described in Statement 3, wherein:

[0130] The system also includes:

[0131] A second storage device for storing second data; and

[0132] The second loading module is used to read second data from the second storage device; and

[0133] The manager is configured to assign I / O requests to the loading module based at least in part on the I / O request requesting the data.

[0134] Statement 5. One disclosed embodiment includes the system described in Statement 4, wherein:

[0135] The I / O request includes a first identifier of the data; and

[0136] The manager includes a table that maps a first identifier of the data to a second identifier of the storage device.

[0137] Statement 6. The disclosed embodiments include the system described in Statement 3, wherein the system further includes: a second loading module for reading the data from a storage device.

[0138] Statement 7. The disclosed embodiments include the system described in Statement 6, wherein the manager is configured to select a loading module to handle the I / O request.

[0139] Statement 8. The disclosed embodiments include the system described in Statement 3, wherein:

[0140] The queue includes a first-in, first-out (FIFO) queue; and

[0141] The manager is configured to access I / O requests from the head of the queue.

[0142] Statement 9. The disclosed embodiments include the system described in Statement 3, wherein:

[0143] The queue includes a priority queue;

[0144] The scheduler is configured to associate priority tags with I / O requests in the queue based at least in part on the size of the I / O request; and

[0145] The manager is configured to access the I / O requests from the queue based at least in part on the priority label.

[0146] Statement 10. The disclosed embodiments include the system described in Statement 3, wherein:

[0147] The scheduler includes a threshold; and

[0148] The scheduler is configured to place the I / O request in the queue based at least in part on the fact that the size of the I / O request is less than the threshold.

[0149] Statement 11. The disclosed embodiments include the system described in Statement 10, wherein:

[0150] The system also includes a second queue; and

[0151] The scheduler is configured to place the second I / O request in the second queue at least in part based on the size of the second I / O request exceeding a threshold.

[0152] Statement 12. The disclosed embodiments include the system described in Statement 11, wherein the manager is configured to retrieve the I / O request from the queue and retrieve a second I / O request from a second queue using polling access.

[0153] Statement 13. The disclosed embodiments include the system described in Statement 1, wherein the loading module includes:

[0154] A reader is used to read the first vector and the second vector from the data in the storage device;

[0155] An adder is used to add the first vector and the second vector.

[0156] Statement 14. The disclosed embodiments include the system described in Statement 1, wherein the loading module is configured to send the data to a second scheduler.

[0157] Statement 15. The disclosed embodiments include the system described in Statement 14, the system further comprising:

[0158] Processing elements; and

[0159] The second scheduler is configured to schedule the processing of computation requests that use the data by the processing elements.

[0160] Statement 16. The disclosed embodiments include the system described in Statement 14, wherein a second scheduler is configured to assign the computation request to a queue.

[0161] Statement 17. The disclosed embodiments include the system described in Statement 16, wherein a second scheduler is configured to allocate the computing requests to the queue based at least in part on the workload of the computing requests.

[0162] Statement 18. The disclosed embodiments include the system described in Statement 16, wherein a processing element is configured to retrieve the computation request from the queue.

[0163] Statement 19. The disclosed embodiments include the system described in Statement 16, wherein:

[0164] The queue is associated with a processing element; and

[0165] The second scheduler is configured to assign the second computation request to the second queue associated with the second processing element.

[0166] Statement 20: The disclosed embodiments include the system described in Statement 16, wherein the queue is associated with a type of processing element.

[0167] Statement 21: The disclosed embodiments include the system described in Statement 16, wherein the processing element is configured to return a completion status to a second scheduler.

[0168] Statement 22. The disclosed embodiments include the system described in Statement 21, wherein the processing element is configured to place a completion status in a second queue.

[0169] Statement 23. The disclosed embodiments include the system described in Statement 22, wherein a second scheduler is configured to retrieve the completion status from a second queue.

[0170] Statement 24. The disclosed embodiments include the system described in Statement 22, wherein the second processing element is configured to place a second completion state in a second queue.

[0171] Statement 25. The disclosed embodiments include a method comprising:

[0172] Receive input / output (I / O) requests at the scheduler;

[0173] The I / O request is transmitted from the scheduler to the loading module based at least in part on the size of the I / O request; and

[0174] The loading module reads data from the storage device, at least in part, based on the I / O request.

[0175] Statement 26, the disclosed embodiments include the method according to Statement 25, wherein the step of transferring the I / O request from the scheduler to the loading module based at least in part on the size of the I / O request includes:

[0176] The size of the I / O request is determined by the scheduler;

[0177] The queue is identified by the scheduler based at least in part on the size of the I / O request; and

[0178] The scheduler places the I / O request in the queue.

[0179] Statement 27. The disclosed embodiments include the method according to Statement 26, wherein the step of reading the data from the storage device by the loading module includes:

[0180] The manager retrieves the I / O request from the queue; and

[0181] The I / O request is sent from the manager to the loading module.

[0182] Statement 28, the disclosed embodiments include the method according to Statement 27, wherein the step of sending the I / O request to the loading module includes:

[0183] The manager identifies the storage device storing the data; and

[0184] The manager identifies the loading module at least in part based on the loading module of the accessible storage device.

[0185] Statement 29, the disclosed embodiments include the method according to Statement 28, wherein the step of identifying the storage device storing the data by the manager includes: mapping a first identifier of the I / O request to a second identifier of the storage device.

[0186] Statement 30, the disclosed embodiments include the method according to statement 28, wherein the step of the manager identifying the storage device storing the data includes: the manager identifying the storage device storing the data from the storage device and a second storage device.

[0187] Statement 31: The disclosed embodiments include the method according to Statement 27, wherein:

[0188] The queue includes a first-in, first-out (FIFO) queue;

[0189] The step of the scheduler placing I / O requests in the queue includes:

[0190] The priority label of the I / O request is determined at least in part based on the size of the I / O request; and

[0191] Associate the priority label with the I / O request in the queue.

[0192] Statement 32, the disclosed embodiments include the method according to Statement 31, wherein the step of retrieving the I / O request from the queue by the manager includes: retrieving the I / O request from the queue by the manager at least in part based on a priority label.

[0193] Statement 33, the disclosed embodiments include the method according to statement 27, wherein the step of identifying the queue by the scheduler at least in part based on the size of the I / O request includes: identifying the queue by the scheduler at least in part based on the size of the I / O request being less than a threshold.

[0194] Statement 34, the disclosed embodiments include the method according to statement 33, wherein the step of identifying the queue by the scheduler at least in part based on the size of the I / O request further includes: identifying the second queue by the scheduler at least in part based on the size of the second I / O request being greater than a threshold.

[0195] Statement 35, the disclosed embodiments include the method according to Statement 34, wherein the step of retrieving the I / O request from the queue by the manager includes: retrieving the I / O request from the queue using polling access and retrieving a second I / O request from a second queue.

[0196] Statement 36, the disclosed embodiments include the method according to Statement 25, wherein the step of reading the data from the storage device by the loading module includes:

[0197] Read the first vector from the storage device;

[0198] Read the second vector from the storage device; and

[0199] The data is generated by adding the first vector and the second vector together.

[0200] Statement 37. The disclosed embodiments include the method according to Statement 25, the method further comprising: sending the data from the loading module to a second scheduler.

[0201] Statement 38, the disclosed embodiments include the method according to Statement 37, the method further comprising: scheduling processing elements to use the data to process computation requests.

[0202] Statement 39, the disclosed embodiments include the method according to statement 38, wherein the step of scheduling processing elements to process computation requests using the data includes: allocating computation requests to queues.

[0203] Statement 40, the disclosed embodiments include the method according to Statement 39, wherein the step of allocating computing requests to the queue includes: allocating computing requests to the queue based at least in part on the workload of the computing requests.

[0204] Statement 41: The disclosed embodiments include the method according to Statement 40, the method further comprising:

[0205] The processing element retrieves the computation request from the queue; and

[0206] The processing element uses the data to process the computation request.

[0207] Statement 42, the disclosed embodiments include the method according to statement 41, the method further comprising returning a completion status from the processing element to a second scheduler in a second queue.

[0208] Statement 43. The disclosed embodiments include the method according to Statement 39, the method further comprising retrieving the completion status from the second queue by a second scheduler.

[0209] Statement 44, the disclosed embodiments include the method according to statement 43, the method further comprising: retrieving a second completion state from a second queue by a second scheduler, the second completion state being placed in the second queue by a second processing element.

[0210] Statement 45, the disclosed embodiments include the method according to Statement 39, wherein the step of allocating a computation request to the queue includes: allocating the computation request to the queue associated with the processing element.

[0211] Statement 46. The disclosed embodiments include the method according to Statement 39, wherein:

[0212] The step of scheduling processing elements to use the data to process computation requests further includes: identifying the type of processing element based at least in part on the computation request; and

[0213] The step of assigning a computation request to the queue includes: assigning the computation request to the queue associated with the type of the processing element.

[0214] Statement 47. The disclosed embodiments include an article of manufacture comprising a non-transitory storage medium storing instructions that, when executed by a machine, cause:

[0215] Receive input / output (I / O) requests at the scheduler;

[0216] The I / O request is transmitted from the scheduler to the loading module based at least in part on the size of the I / O request; and

[0217] The loading module reads data from the storage device, at least in part, based on the I / O request.

[0218] Statement 48, the disclosed embodiments include the article of manufacture according to Statement 47, wherein the step of transmitting the I / O request from the scheduler to the loading module based at least in part on the size of the I / O request includes:

[0219] The size of the I / O request is determined by the scheduler;

[0220] The queue is identified by the scheduler based at least in part on the size of the I / O request; and

[0221] The scheduler places the I / O request in the queue.

[0222] Statement 49. The disclosed embodiments include the article of manufacture according to Statement 48, wherein the step of reading the data from the storage device by the loading module includes:

[0223] The manager retrieves the I / O request from the queue; and

[0224] The I / O request is sent from the manager to the loading module.

[0225] Statement 50, the disclosed embodiments include the article of manufacture according to Statement 49, wherein the step of sending the I / O request to the loading module includes:

[0226] The manager identifies the storage device storing the data; and

[0227] The manager identifies the loading module at least in part based on the loading module of the accessible storage device.

[0228] Statement 51: The disclosed embodiments include the article of manufacture according to Statement 50, wherein the step of identifying the storage device storing the data by the manager includes: mapping a first identifier of the I / O request to a second identifier of the storage device.

[0229] Statement 52, the disclosed embodiments include the article of manufacture according to Statement 50, wherein the step of the manager identifying the storage device storing the data includes: the manager identifying the storage device storing the data from the storage device and a second storage device.

[0230] Statement 53: The disclosed embodiments include the article of manufacture according to Statement 49, wherein:

[0231] The queue includes a first-in, first-out (FIFO) queue;

[0232] The step of the scheduler placing the I / O request in the queue includes:

[0233] The priority label of the I / O request is determined at least in part based on the size of the I / O request; and

[0234] Associate the priority label with the I / O request in the queue.

[0235] Statement 54, the disclosed embodiments include the article of manufacture according to Statement 53, wherein the step of retrieving the I / O request from the queue by the manager includes: retrieving the I / O request from the queue by the manager at least in part based on a priority tag.

[0236] Statement 55, the disclosed embodiments include the article of manufacture according to Statement 49, wherein the step of identifying the queue by the scheduler at least in part based on the size of the I / O request includes: identifying the queue by the scheduler at least in part based on the size of the I / O request being less than a threshold.

[0237] Statement 56. The disclosed embodiments include the article of manufacture according to Statement 55, wherein the step of identifying the queue by the scheduler at least in part based on the size of the I / O request further includes: identifying the second queue by the scheduler at least in part based on the size of the second I / O request being greater than a threshold.

[0238] Statement 57. The disclosed embodiments include the article of manufacture according to Statement 56, wherein the step of retrieving the I / O request from the queue by the manager includes: retrieving the I / O request from the queue using polling access and retrieving a second I / O request from a second queue.

[0239] Statement 58, the disclosed embodiments include the article of manufacture according to Statement 47, wherein the step of reading the data from the storage device by the loading module includes:

[0240] Read the first vector from the storage device;

[0241] Read the second vector from the storage device; and

[0242] The data is generated by adding the first vector and the second vector together.

[0243] Statement 59. The disclosed embodiments include the article of manufacture according to Statement 47, wherein additional instructions are stored on a non-transitory storage medium, which, when executed by the machine, cause the data to be sent from the loading module to a second scheduler.

[0244] Statement 60, the disclosed embodiments include the article of manufacture according to Statement 59, wherein additional instructions are stored on a non-transitory storage medium, which, when executed by the machine, cause a scheduling processing element to use the data to process a computation request.

[0245] Statement 61. The disclosed embodiments include the article of manufacture according to Statement 60, wherein the step of scheduling the processing element to process a computation request using the data includes assigning the computation request to a queue.

[0246] Statement 62. The disclosed embodiments include the article of manufacture according to Statement 61, wherein the step of allocating a computing request to the queue includes: allocating the computing request to the queue at least in part based on the workload of the computing request.

[0247] Statement 63. The disclosed embodiments include the article of manufacture according to Statement 62, wherein additional instructions are stored on a non-transitory storage medium, which, when executed by the machine, cause:

[0248] The processing element retrieves the computation request from the queue; and

[0249] The processing element uses the data to process the computation request.

[0250] Statement 64. The disclosed embodiments include the article of manufacture according to Statement 63, wherein the non-transitory storage medium stores additional instructions that, when executed by the machine, cause a completion status to be returned from the processing element to the second scheduler in a second queue.

[0251] Statement 65. The disclosed embodiments include the article of manufacture according to Statement 61, wherein the non-transitory storage medium stores additional instructions that, when executed by the machine, cause the completion status to be retrieved from the second queue by a second scheduler.

[0252] Statement 66. The disclosed embodiments include the article of manufacture according to Statement 65, wherein the non-transitory storage medium stores additional instructions that, when executed by the machine, cause a second scheduler to retrieve a second completion state from a second queue, the second completion state being placed in the second queue by a second processing element.

[0253] Statement 67. The disclosed embodiments include the article of manufacture according to Statement 61, wherein the step of allocating a computation request to the queue includes: allocating the computation request to the queue associated with the processing element.

[0254] Statement 68. The disclosed embodiments include the article of manufacture according to Statement 61, wherein:

[0255] The step of scheduling processing elements to use the data to process computation requests further includes: identifying the type of processing element based at least in part on the computation request; and

[0256] The step of assigning a computation request to the queue includes: assigning the computation request to the queue associated with the type of the processing element.

[0257] Therefore, given the wide variety of arrangements of the embodiments described herein, this specific implementation and the appended materials are intended to be illustrative only and should not be considered as limiting the scope of the disclosure. Thus, the disclosure claims protection for all such modifications that may fall within the scope and spirit of the appended claims and their equivalents.

Claims

1. A system including a storage device, the system comprising: Storage devices used to store data; A loading module for reading the data from the storage device, at least in part, based on an input / output request; A scheduler is configured to receive the input / output request and place the input / output request in a queue, at least in part, based on the fact that the size of the input / output request is less than a threshold, for transmission to the loading module; A second storage device stores the second data; as well as The second loading module is used to read second data from the second storage device. The input / output request includes a first identifier of the data. The system further includes a manager for retrieving the input / output request from the queue and determining, via an access table, whether the data requested in the input / output request is stored in a storage device, in order to allocate the input / output request to the loading module. The table maps the first identifier of the data to the second identifier of the storage device, and maps the third identifier of the second data to the fourth identifier of the second storage device. The second identifier uniquely identifies the storage device, and the fourth identifier uniquely identifies the second storage device.

2. The system according to claim 1, wherein: The manager is configured to assign the input / output request to the loading module based at least in part on the input / output request requesting the data.

3. The system according to claim 2, wherein: The manager includes a table that maps a first identifier of the data to a second identifier of the storage device.

4. The system according to claim 1, wherein, The system also includes a third loading module for reading the data from a storage device.

5. The system according to claim 4, wherein, The manager is configured to select the loading module to handle the input / output request.

6. The system according to claim 1, wherein: The system also includes a second queue; and The scheduler is configured to place the second input / output request in the second queue at least in part based on the size of the second input / output request exceeding a threshold.

7. The system according to claim 6, wherein, The manager is configured to retrieve the input / output request from the queue and the second input / output request from the second queue using polling access.

8. The system according to any one of claims 1 to 7, wherein, The loaded modules include: A reader is used to read the first vector and the second vector from the data in the storage device; An adder is used to add the first vector and the second vector.

9. The system according to any one of claims 1 to 7, wherein, The loading module is configured to send the data to the second scheduler.

10. A method for a system including a storage device, the method comprising: Receive input / output requests at the scheduler; The size of the input / output request is determined by the scheduler; The queue is identified by the scheduler based at least in part on the fact that the size of the input / output request is less than a threshold; The scheduler places the input / output requests in the queue for transmission to the loading module; The loading module reads data from the storage device, at least in part, based on the input / output request; as well as The second loading module reads the second data from the second storage device. The input / output request includes a first identifier of the data. The step of reading the data from the storage device includes: the manager retrieving the input / output request from the queue, and determining, through an access table, whether the storage device stores the requested data in the input / output request, so as to allocate the input / output request to the loading module. The table maps the first identifier of the data to the second identifier of the storage device, and maps the third identifier of the second data to the fourth identifier of the second storage device. The second identifier uniquely identifies the storage device, and the fourth identifier uniquely identifies the second storage device.

11. The method according to claim 10, wherein, The step of assigning the input / output request to the loading module includes: The manager identifies the storage device storing the data; and The manager identifies the loading module at least in part based on the loading module that has access to the storage device.

12. The method according to claim 11, wherein, The step of the manager identifying the storage device storing the data includes: mapping a first identifier of the input / output request to a second identifier of the storage device.

13. The method according to claim 10, further comprising: The scheduler identifies the second queue based at least in part on the fact that the size of the second input / output request is greater than a threshold; as well as The scheduler places the second input / output request in the second queue for delivery to the loading module.

14. The method according to claim 13, wherein, The step of retrieving the input / output request from the queue by the manager includes: retrieving the input / output request from the queue using polling access and retrieving a second input / output request from a second queue.

15. The method according to any one of claims 10 to 14, wherein, The steps for the loading module to read the data from the storage device include: Read the first vector from the storage device; Read the second vector from the storage device; and The data is generated by adding the first vector and the second vector together.

16. The method according to claim 10, further comprising: The data is sent from the loading module to the second scheduler.

17. An article of manufacture comprising a non-transitory storage medium storing instructions that, when executed by a machine, cause: Receive input / output requests at the scheduler; The size of the input / output request is determined by the scheduler; The queue is identified by the scheduler based at least in part on the fact that the size of the input / output request is less than a threshold; The scheduler places the input / output requests in the queue for transmission to the loading module; The loading module reads data from the storage device, at least in part, based on the input / output request; as well as The second loading module reads the second data from the second storage device. The input / output request includes a first identifier of the data. The step of reading the data from the storage device includes: the manager retrieving the input / output request from the queue, and determining, through an access table, whether the storage device stores the requested data in the input / output request, so as to allocate the input / output request to the loading module. The table maps the first identifier of the data to the second identifier of the storage device, and maps the third identifier of the second data to the fourth identifier of the second storage device. The second identifier uniquely identifies the storage device, and the fourth identifier uniquely identifies the second storage device.

Images