System for machine learning and method for key-value access in machine learning system

Abstract

A system for machine learning and a method for key-value access in a machine learning system are provided. The system includes a graphics processing unit having a graphics processing unit memory and a key-value storage device connected to the graphics processing unit memory. The method includes writing, with the graphics processing unit, a key-value request to a key-value request queue in an input-output region of the graphics processing unit memory, the key-value request including a key. The method also includes reading, with the key-value storage device, the key-value request from the key-value request queue and writing, with the key-value storage device, a value to the input-output region of the graphics processing unit memory in response to the key-value request, the value corresponding to the key of the key-value request.

Description

[0001] This invention is a divisional application of the invention patent application filed on January 21, 2019, with application number 201910054704.X and title "System and Method for Machine Learning". Technical Field

[0002] One or more aspects of embodiments of this disclosure relate to machine learning, and more specifically, to a system and method for avoiding serialized access to key values ​​in a machine learning system. Background Technology

[0003] In some existing solid-state drives (SSDs) with block interfaces, key-value access to data stored on the SSD requires the central processing unit (CPU) to provide the key-value interface during stochastic machine learning training, which randomly samples a subset of the entire training data. The host CPU performs file index lookups and file system accesses to identify the location of the data that leads to serialized key-value access. This type of serialized key-value access can limit performance.

[0004] Therefore, there is a need for an improved system and method for performing machine learning involving access to key values ​​of data. Summary of the Invention

[0005] According to an embodiment of the present invention, a method for machine learning is provided, the method comprising: writing a first key value request into a key value request queue in a first input-output area of ​​a first memory using a first graphics processing unit, the first memory being connected to the first graphics processing unit, the first key value request containing a key; reading the first key value request from the key value request queue using a first key value storage device connected to the first memory; and writing a first value into the first input-output area of ​​the first memory using the first key value storage device in response to the first key value request, the first value corresponding to the key of the first key value request.

[0006] In one embodiment, the method includes: using a first key value storage device to perform a key lookup in the first key value storage device to retrieve a first value.

[0007] In one embodiment, the first key value request includes a return value area, which is the area assigned to the first value.

[0008] In one embodiment, writing the first value to the first input-output area of ​​the first memory includes writing the first value to the return value area.

[0009] In one embodiment, writing a first value to a first input-output area of ​​a first memory includes a queue of return values ​​for writing the first value to the first input-output area of ​​the first memory.

[0010] In one embodiment, the method includes: configuring a host connected to a first key value storage device and a first graphics processing unit to: the first key value storage device access a first input-output area of ​​a first memory to receive a key value request and write a value in response to the key value request; and the first graphics processing unit storing the key value request in the first input-output area of ​​the first memory and reading a value from the first input-output area of ​​the first memory.

[0011] In one embodiment, the method includes: writing a second key value request to a key value request queue in an input-output area of ​​a second memory connected to the second graphics processing unit using a second graphics processing unit connected to a host, the second key value request containing a key; reading the second key value request from the key value request queue using a second key value storage device connected to the host and connected to the second memory; and writing a second value to the input-output area of ​​the second memory using the second key value storage device in response to the second key value request, the second value corresponding to the key of the second key value request.

[0012] In one embodiment, the method includes: performing a key lookup in the first key value storage device to retrieve a first value using a first key value storage device; and performing a key lookup in the second key value storage device to retrieve a second value while performing the key lookup using the first key value storage device.

[0013] In one embodiment, using a first key value storage device to read a first key value request includes reading the first key value request via peer-to-peer direct memory access.

[0014] In one embodiment, writing a first value using a first key value storage device includes writing the first value via peer-to-peer direct memory access.

[0015] In one embodiment, the first key value storage device is connected to the first graphics processing unit via a peripheral component interconnection connection.

[0016] In one embodiment, the method includes: after writing a first key value request using a first graphics processing unit and before writing a first value using a first key value storage device, writing a second key value request to a key value request queue using the first graphics processing unit.

[0017] In one embodiment, the method includes: using a first graphics processing unit to write a second key value request to a key value request queue in a second input-output region of a first memory, the second key value request containing a key; using a second key value storage device connected to the first memory to read the second key value request from the key value request queue in the second input-output region of the first memory; and in response to the second key value request, using the second key value storage device to write a second value to the second input-output region of the first memory, the second value corresponding to the key of the second key value request.

[0018] In one embodiment, the method includes: performing a key lookup in the first key value storage device to retrieve a first value using a first key value storage device; and performing a key lookup in the second key value storage device to retrieve a second value while performing the key lookup using the first key value storage device.

[0019] According to an embodiment of the present invention, a system for machine learning is provided, the system comprising: a graphics processing unit; a memory connected to the graphics processing unit; and a key value storage device; the key value storage device being connected to the graphics processing unit via peripheral component interconnection; the graphics processing unit being configured to perform memory-mapped input and output operations in an input-output region of the memory, and configured to write one or more key value requests to a key value request queue in the input-output region; the key value storage device being configured to: perform memory-mapped input and output operations in the input-output region; read one or more key value requests from the key value request queue; and write values ​​corresponding to the keys of the key value requests in response to the key value requests of the one or more key value requests to the input-output region of the memory.

[0020] In one embodiment, the key value request includes a return value area, which is the area assigned to the value.

[0021] In one embodiment, writing the value to the input-output area of ​​the memory includes writing the value to the return value area.

[0022] In one embodiment, writing the value to the input-output area of ​​the memory includes a queue of return values ​​in the input-output area of ​​the memory.

[0023] According to an embodiment of the present invention, a system for machine learning is provided, the system comprising: a graphics processing unit; a key value storage device; and a shared memory component for communicating between the graphics processing unit and the key value storage device; the graphics processing unit being configured to send one or more key value requests to the key value storage device via the shared memory component for communication, the key value storage device being configured to: receive one or more key value requests; and, in response to the key value requests of the one or more key value requests, send values ​​to the graphics processing unit via the shared memory component for communication, the values ​​corresponding to the keys of the key value requests.

[0024] In one embodiment, the shared memory component for communication includes a memory connected to the graphics processing unit and configured to be accessed by a key value storage device via a peripheral component interconnection connection through peer-to-peer direct memory access. Attached Figure Description

[0025] These and other features and advantages of this disclosure will be understood and appreciated with reference to this specification, claims, and drawings, wherein:

[0026] Figure 1 This is a functional block diagram of a system used for machine learning.

[0027] Figure 2 This is a block diagram of a graphics card equipped with an onboard SSD according to an embodiment of the present disclosure.

[0028] Figure 3 This is a data flow diagram according to an embodiment of the present disclosure.

[0029] Figure 4 This is a timing diagram according to an embodiment of the present disclosure.

[0030] Figure 5 This is a timing diagram according to an embodiment of the present disclosure.

[0031] Explanation of icon numbers

[0032] 105: Host application;

[0033] 110: Software key value storage;

[0034] 115: File system;

[0035] 120: Block Interface;

[0036] 125: Graphics processing unit memory;

[0037] 205: Onboard key value solid-state driver;

[0038] 210: Graphics processing unit;

[0039] 305: Graphics Processing Unit Application;

[0040] 310: Key value request queue. Detailed Implementation

[0041] The detailed description set forth below with reference to the accompanying drawings is intended as a description of exemplary embodiments of systems and methods, and is not intended to represent only the forms in which this disclosure can be constructed or utilized for performing machine learning involving key value access to data provided according to this disclosure. The description is illustrated in conjunction with the shown embodiments to illustrate features of this disclosure. However, it will be understood that the same or equivalent functionality and structure may be achieved using different embodiments, which are also intended to be covered within the scope of this disclosure. As indicated elsewhere herein, the same element numbers are intended to indicate the same element or feature.

[0042] Existing machine learning platforms suffer from drawbacks when used in stochastic machine learning training methods that randomly sample a subset of the entire training data. Because the CPU is involved to provide the key value interface and data transfer across the peripheral component interconnect express (PCIe) bus, such platforms can suffer from low graphics processing unit (GPU) utilization due to key value access during stochastic machine learning training. As mentioned above, in some prior art systems, the host CPU performs file index lookups and file system accesses to identify the location of data that would lead to serialized key value access. In contrast, in some embodiments, performance is improved because the CPU is not involved in key value access to data stored on an onboard SSD. The GPU sends key value commands directly to the onboard key value storage device (e.g., the onboard key value SSD) on the graphics card, which contains both the GPU and the onboard key value SSD. This enables asynchronous key value access, thereby reducing the impact of access latency. As used herein, a “key value storage device” is a persistent storage device (such as an SSD) configured to respond to a key value request (each containing a key) by returning a value corresponding to the key contained in the request.

[0043] Figure 1This diagram illustrates the overall process of key value access to data stored on an SSD with a block interface using a software key value store during GPU machine learning training. First, the host application 105 initiates key value access by sending a "retrieve" request to the software key value store 110. To identify the location of data for a given key, the software key value store 110 accesses an index table that stores the file offset of the data corresponding to the specified key. Subsequently, the software key value store 110 accesses a file system 115 with the file offset, and the file system 115 accesses the SSD with the block interface 120 and retrieves the data corresponding to the specified key. Once the value becomes available to the host application, the host application transfers the value to the GPU memory 125 for GPU computation and launches the GPU kernel. All these operations are performed sequentially for a software key value store with a prior art SSD having a block interface.

[0044] like Figure 1 As shown, key value access to an SSD with a block interface involves several computational steps running on the host to identify the location of data corresponding to a specified key. The GPU can only perform the corresponding computation after the computation of the software key value storage is completed. Because the GPU needs to wait for the completion of key value access from other GPUs, the latency of key value access increases when multiple GPUs are used in the system, leading to serialization of GPU computation. As a result, key value access to existing SSDs with block interfaces limits the number of GPUs that can be used efficiently in the system.

[0045] In some embodiments, graphics cards with an onboard SSD (or “key value SSD”) having a key value interface are used to overcome some of the drawbacks of existing technology systems. Figure 2An exemplary system having such a device is illustrated. Unlike prior art systems where data movement between storage and the GPU requires data transfer across the global PCIe bus (thus increasing latency for data access from the GPU), a graphics card with an onboard key value SSD 205 can reduce the additional load by utilizing peer-to-peer (P2P) direct memory access (DMA) between the onboard key value SSD 205 and the GPU 210 and giving the GPU 210 full P2P DMA control. In some embodiments, the onboard key value SSD 205 provides key value commands as non-standard commands. For example, a key value request queue (discussed in further detail below) can be implemented as a nonvolatile memory express (NVMe) command queue, where NVMe commands are vendor-proprietary commands defined in the firmware of the SSD 205 and in the driver software on the GPU, corresponding to key value commands (i.e., commands used to request values ​​from the SSD 205 corresponding to the keys obtained as part of the commands).

[0046] In some embodiments, such systems can be used to provide asynchronous key value access in an onboard key value SSD, and some embodiments utilize a key value SSD within a graphics card for random sampling of training data. Figure 3The flow of key value access during machine learning training in some embodiments is illustrated. A significant difference between these embodiments and some prior art systems is that in these embodiments, the GPU sends key value commands directly to the key value SSD 205. First, during the initial phase of executing the host application, the host application maps the memory of the special GPU device onto the peripheral component interconnect (PCI) base address register (BAR) storage area, thereby establishing direct communication between the key value SSD 205 and the GPU. The area of ​​GPU memory allocated for communication between the key value SSD 205 and the GPU (e.g., using memory-mapped input-output) using this process is referred to herein as the “input-output area” of the GPU memory. The GPU memory input-output area, which can be directly accessed by both the GPU and the key value SSD 205, can functionally operate as shared memory. The GPU application 305 issues an access request to the key value SSD 205 by performing memory-mapped input-output on the key value SSD 205 and supplying the bus address of the exposed GPU memory. After the firmware within the key value SSD 205 performs a key lookup to retrieve the value corresponding to the key, the firmware writes the value to the mapped GPU device memory (i.e., to the input-output area of ​​the GPU memory) without the need for the host application 105 to relay the data.

[0047] In some embodiments, a key value request queue (KVRQ) 310 is used, and key value access is unobstructed in the sense that the GPU does not need to wait for a response to the first request before generating a second subsequent request. Alternatively, the GPU places a key value request in the key value request queue 310, and the request is then processed by the key value SSD 205. Thus, when the GPU application places a request in the key value request queue 310, the request operation is complete. The key value request queue 310 holds incomplete requests such that the number of entries in the key value request queue 310 is equal to the number of key value requests. When the value is transferred to GPU memory, the firmware in the SSD 205 releases the key value request queue entry corresponding to the specified key.

[0048] Individual key value access for each GPU allows key value access from multiple GPUs to overlap. For example, in a system with two GPUs each connected to a corresponding key value SSD, the two GPUs can issue requests simultaneously, and their respective key value SSDs can respond simultaneously. Figure 4 This illustrates the operations involved in such an instance, where each of the two GPUs is connected to an SSD with a corresponding key value. Figure 4 The time saved by the system is also shown compared to prior art methods in which GPU computation is serialized, in such embodiments where two GPUs are used to perform overlapping key value access. In some embodiments, three or more (e.g., any number) GPUs may each be connected to a corresponding key value SSD and perform overlapping (e.g., simultaneous) key value operations.

[0049] In some embodiments, the separation of key value access requests and responses enables asynchronous key value access, for example, enabling batch processing of multiple requests from the GPU. Figure 5 An example of asynchronous key value access when batching two key value commands is illustrated. Compared to synchronous key value access, where GPU computation and SSD device access are serialized, asynchronous key value access in some embodiments makes it possible to leverage GPU computation to overlap multiple key value commands. In this example, the GPU continuously issues fetch requests instead of waiting for each previous request to complete. In some embodiments, three or more (e.g., any number) GPUs may each be connected to a corresponding key value SSD and perform overlapping (e.g., simultaneous) key value operations.

[0050] In some embodiments, when a key value SSD retrieves a value in response to a key value request, the key value SSD writes the retrieved value back to the key value request queue, i.e., back to a region of memory (or "return value region") allocated within the key value request for this purpose. In other embodiments, the key value SSD instead writes the retrieved value to a separate queue (or "return value queue") allocated in the input-output region of the GPU memory. In some embodiments, instead of having each GPU have a single dedicated key value SSD (each GPU sending a key value request to a single dedicated key value SSD), a single GPU may have several key value SSDs. In such embodiments, several key value request queues, each for a respective key value SSD, may be allocated in the GPU memory. In other embodiments, several GPUs may be connected to a single key value SSD, which may (e.g.) serve key value requests in a corresponding key value request queue in the GPU in a round-robin manner.

[0051] In some embodiments, the task performed by the host application involves only establishing a communication path between the GPU and the SSD. This improves the scalability of these embodiments by avoiding, or potentially causing, serialization of GPU computations due to key value access operations performed by the host application on the CPU. Thus, these embodiments enable scaling out multiple GPUs to accelerate machine learning training. By replacing complex key value software with a simpler device interface, some embodiments also reduce, or potentially reduce, the resource requirements imposed on the host, including, for example, the number of CPU cores. Avoiding such requirements results in better energy efficiency.

[0052] Some embodiments may be constructed using one or more processing circuits. The term "processing circuit" is used herein to mean any combination of hardware, firmware, and software used to process data or digital signals. Processing circuit hardware may include, for example, application-specific integrated circuits (ASICs), general-purpose or special-purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices such as field-programmable gate arrays (FPGAs). As used herein, in a processing circuit, each function is performed by hardware configured (i.e., hardwired) to perform said function, or by more general-purpose hardware (such as a CPU) configured to execute instructions stored in a non-transitory storage medium. Processing circuitry may be fabricated on a single printed circuit board (PCB) or distributed over several interconnected PCBs. Processing circuitry may include other processing circuitry; for example, processing circuitry may include two processing circuits interconnected on a PCB: an FPGA and a CPU.

[0053] It will be understood that while the terms “first,” “second,” “third,” etc., may be used herein to describe various elements, components, regions, layers, and / or segments, these elements, components, regions, layers, and / or segments should not be limited by these terms. These terms are used only to distinguish one element, component, region, layer, or segment from another. Therefore, the first element, component, region, layer, or segment discussed herein may be referred to as a second element, component, region, layer, or segment without departing from the spirit and scope of the invention.

[0054] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the concept of the invention. As used herein, the terms “substantially,” “about,” and similar terms are used as approximate terms rather than terms of degree and are intended to take into account the inherent biases of measured or calculated values ​​that will be recognized by those skilled in the art. As used herein, the term “major component” means a component present in a composition, polymer, or product in an amount greater than that of any other single component in the composition or product. In contrast, the term “primary component” means a component constituting at least 50% by weight or more of the composition, polymer, or product. As used herein, when applied to multiple articles, the term “major portion” means at least half of said article.

[0055] As used herein, unless the context clearly indicates otherwise, the singular form “a / an” is intended to include the plural form as well. It will be further understood that, when used in this specification, the terms “comprises” and / or “comprising” specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. When preceding a list of elements, expressions such as “at least one of” modify the entire list of elements and not individual elements in the list. Furthermore, when describing embodiments of the inventive concept, the use of “may” means “one or more embodiments of this disclosure.” Additionally, the term “exemplary” means instance or illustration. As used herein, the terms “use,” “using,” and “used” may be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively.

[0056] It will be understood that when an element or layer is referred to as being “on”, “connected to,” “coupled to,” or “adjacent to” another element or layer, it may be directly on, directly connected to, coupled to, or adjacent to the other element or layer, or one or more intervening elements or layers may be present. In contrast, when an element or layer is referred to as being “directly” “above” another element or layer, “directly connected to,” “directly coupled to,” or “immediately adjacent to” the other element or layer, no intervening element or layer is present.

[0057] While exemplary embodiments of systems and methods for performing machine learning involving key-value access to data have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Accordingly, it should be understood that systems and methods for performing machine learning involving key-value access to data, constructed in accordance with the principles of this disclosure, may be implemented in ways other than those specifically described herein. The invention is also defined in the following claims and their equivalents.

Claims

1. A method for key value storage and retrieval in a machine learning system, the method comprising: Input and output operations are performed by a key value storage device connected to the memory in the input-output region of the memory, which is mapped to another device; The key value storage device reads a first value from the key value request queue of the input-output area based on a first key value request issued by the other device, the first key value request being based on the configuration of the host device; as well as The key value storage device writes the first value into the input-output area of ​​the memory according to the first key value request. When the key value storage device writes the first value into the memory, the key value storage device releases the key value request queue entry corresponding to the specified key in the key value request queue.

2. The method of claim 1, further comprising performing a key lookup in the key value storage device to retrieve the first value, for writing the first value to the input-output area of ​​the memory.

3. The method of claim 1, wherein the first key value request includes a return value area, the return value area being an area assigned to the first value.

4. The method of claim 3, wherein writing the first value to the input-output region of the memory includes writing the first value to the return value region.

5. The method according to claim 1, further comprising: The configuration is received from the host device by the other device; as well as Based on the configuration from the host device, the other device requests to store the key value in the input-output area of ​​the memory, and the other device reads the value from the input-output area of ​​the memory.

6. The method of claim 1, wherein the other device mapped to the memory includes a graphics processing unit configured to request writing the first key value to the input-output region of the memory and to read the first value from the input-output region of the memory.

7. The method of claim 1, wherein the first key value request is read from the key value request queue by the key value storage device via peer-to-peer direct memory access.

8. The method of claim 1, wherein the first value is written to the input-output area of ​​the memory by the key value storage device via peer direct memory access.

9. The method of claim 1, wherein the key value storage device is connected to the other device by an interconnection of peripheral components.

10. The method of claim 1, wherein the first key value request is written by the other device to the key value request queue of the input-output area; and The method further includes, after the writing of the first key value request performed by the other device and before the writing of the first value performed by the key value storage device, the other device writes the second key value request to the key value request queue.

11. The method of claim 1, wherein the first key value request includes an address that exposes the input-output region of the memory of the other device based on the configuration of the host device.

12. A system for machine learning, the system comprising: Key value storage device; as well as A memory connected to the key value storage device, the memory being mapped to another device connected to the key value storage device via peripheral component interconnects. The key value storage device is configured to: Input and output operations are performed in the input-output area of ​​the memory; A first value is read from the key value request queue of the input-output area based on a first key value request issued by the other device, the first key value request being based on the configuration of the host device; as well as The request is made to write the first value into the input-output area of ​​the memory based on the first key value. When the key value storage device writes the first value into the memory, the key value storage device releases the key value request queue entry corresponding to the specified key in the key value request queue.

13. The system of claim 12, wherein the key value storage device is further configured to perform a key lookup in the key value storage device to retrieve the first value, so as to write the first value into the input-output area of ​​the memory.

14. The system of claim 12, wherein the first key value request includes a return value area, the return value area being an area assigned to the first value.

15. The system of claim 12, wherein the other device includes a graphics processing unit configured to request writing the first key value into the input-output region of the memory and to read the first value from the input-output region of the memory.

16. The system of claim 12, wherein the key value storage device is configured to read the first key value request from the key value request queue via peer-to-peer direct memory access.

17. The system of claim 12, wherein the key value storage device is configured to write the first value to the input-output area of ​​the memory via peer-to-peer direct memory access.

18. A system for machine learning, the system comprising: Graphics processing unit; Key value storage device; as well as A shared memory, used for communication between the graphics processing unit and the key value storage device, wherein the shared memory is a memory mapped to the graphics processing unit. The key value storage device is configured to: Input and output operations are performed in the shared memory; A first value is read from the key value request queue of the shared memory according to a first key value request issued by the graphics processing unit, wherein the first key value request is based on the configuration of the host device; as well as The request is to write the first value into the shared memory based on the first key value. When the key value storage device writes the first value to the shared memory, the key value storage device releases the key value request queue entry corresponding to the specified key in the key value request queue.

19. The system of claim 18, wherein the graphics processing unit is configured to request writing the first key value to the shared memory and read the first value from the shared memory, and The graphics processing unit is memory-mapped to the shared memory to perform memory-mapped input-output operations in the shared memory.

20. The system of claim 19, wherein the key value storage device is configured to read the first key value request from the key value request queue via peer-to-peer direct memory access, and The key value storage device is configured to write the first value to the shared memory via peer-to-peer direct memory access.

Images