System for machine learning and method of key value access in machine learning system

Abstract

The invention provides a system for machine learning and a method for key value access in a machine learning system. The system includes a graphics processing unit having a graphics processing unit memory, and a key value storage device connected to the graphics processing unit memory. In one embodiment, a method includes writing, with a graphics processing unit, a key value request to a key value request queue in an input-output region of graphics processing unit memory, the key value request including a key. The method also includes reading, with the key value storage device, the key value request from the key value request queue, and writing, with the key value storage device, a value to an input-output region of the graphics processing unit memory in response to the key value request, the value corresponding to a key of the key value request.

Description

[0001] The present invention is a divisional application of the invention patent application with application number 201910054704.X filed on January 21, 2019 and invention name “System and Method for Machine Learning”. Technical Field

[0002] One or more aspects of embodiments according to the present disclosure relate to machine learning, and more particularly, to a system and method for avoiding serialized key value access in a machine learning system. Background Art

[0003] In some prior art solid state drives (SSDs) with a block interface, key value access to data stored in the SSD requires involvement of a central processing unit (CPU) to provide a key value interface during random machine learning training that randomly samples a subset of the entire training data. The host CPU performs file index lookups and file system accesses to identify the location of the data resulting in serialized key value accesses. Such serialized key value accesses may limit performance.

[0004] Therefore, there is a need for an improved system and method for performing machine learning involving key-value access to data. Summary of the invention

[0005] According to one embodiment of the present invention, a method for machine learning is provided, the method comprising: using a first graphics processing unit to write a first key value request to a key value request queue in a first input-output area of ​​a first memory, the first memory being connected to the first graphics processing unit, the first key value request comprising a key; using a first key value storage device connected to the first memory to read the first key value request from the key value request queue; and in response to the first key value request, using the first key value storage device to write a first value to the first input-output area of ​​the first memory, the first value corresponding to the key of the first key value request.

[0006] In one embodiment, a method includes utilizing a first key value storage device to perform a key lookup in the first key value storage device to retrieve a first value.

[0007] In one embodiment, the first key value request includes a return value area, which is an area allocated to the first value.

[0008] In one embodiment, writing the first value to the first input-output area of ​​the first memory includes writing the first value to a return value area.

[0009] In one embodiment, writing the first value to the first input-output area of ​​the first memory includes writing the first value to a return value queue in the first input-output area of ​​the first memory.

[0010] In one embodiment, a method includes: configuring, using a host connected to a first key value storage device and connected to a first graphics processing unit: the first key value storage device accesses a first input-output area of ​​a first memory to receive a key value request, and writes a value in response to the key value request; and the first graphics processing unit stores the key value request in the first input-output area of ​​the first memory, and reads the value from the first input-output area of ​​the first memory.

[0011] In one embodiment, a method includes: using a second graphics processing unit connected to a host to write a second key value request to a key value request queue in an input-output area of ​​a second memory, the second memory being connected to the second graphics processing unit, the second key value request including a key; using a second key value storage device connected to the host and connected to the second memory to read the second key value request from the key value request queue; and in response to the second key value request, using the second key value storage device to write a second value to the input-output area of ​​the second memory, the second value corresponding to the key of the second key value request.

[0012] In one embodiment, the method includes: utilizing a first key value storage device to perform a key lookup in the first key value storage device to retrieve a first value; and utilizing a second key value storage device to perform a key lookup in the second key value storage device to retrieve a second value while utilizing the first key value storage device to perform the key lookup.

[0013] In one embodiment, utilizing the first key value storage device to read the first key value request includes reading the first key value request via peer to peer direct memory access.

[0014] In one embodiment, writing the first value utilizing the first key value storage device includes writing the first value via peer to peer direct memory access.

[0015] In one embodiment, the first key value storage device is connected to the first graphics processing unit using a peripheral component interconnect connection.

[0016] In one embodiment, a method includes writing a second key value request to a key value request queue with the first graphics processing unit after writing the first key value request with the first graphics processing unit and before writing the first value with the first key value storage device.

[0017] In one embodiment, the method includes: using a first graphics processing unit to write a second key value request to a key value request queue in a second input-output area of ​​a first memory, the second key value request including a key; using a second key value storage device connected to the first memory to read the second key value request from the key value request queue in the second input-output area of ​​the first memory; and in response to the second key value request, using the second key value storage device to write a second value to the second input-output area of ​​the first memory, the second value corresponding to the key of the second key value request.

[0018] In one embodiment, the method includes: utilizing a first key value storage device to perform a key lookup in the first key value storage device to retrieve a first value; and utilizing a second key value storage device to perform a key lookup in the second key value storage device to retrieve a second value while utilizing the first key value storage device to perform the key lookup.

[0019] According to one embodiment of the present invention, a system for machine learning is provided, the system comprising: a graphics processing unit; a memory connected to the graphics processing unit; and a key value storage device; the key value storage device is connected to the graphics processing unit using a peripheral component interconnect connection; the graphics processing unit is configured to perform memory mapping input and output operations in an input-output area of ​​the memory, and is configured to write one or more key value requests to a key value request queue within the input-output area; the key value storage device is configured to: perform memory mapping input and output operations in the input-output area; read one or more key value requests from the key value request queue; and write a value to the input-output area of ​​the memory in response to a key value request of the one or more key value requests, the value corresponding to the key of the key value request.

[0020] In one embodiment, the key value request contains a return value area, which is an area allocated to the value.

[0021] In one embodiment, writing the value to an input-output area of ​​the memory includes writing the value to a return value area.

[0022] In one embodiment, writing the value to an input-output area of ​​memory includes writing the value to a return value queue in the input-output area of ​​memory.

[0023] According to one embodiment of the present invention, a system for machine learning is provided, the system comprising: a graphics processing unit; a key value storage device; and a shared memory component for communicating between the graphics processing unit and the key value storage device; the graphics processing unit is configured to send one or more key value requests to the key value storage device via the shared memory component for communication, and the key value storage device is configured to: receive the one or more key value requests; and in response to the key value request of the one or more key value requests, send a value to the graphics processing unit via the shared memory component for communication, the value corresponding to the key of the key value request.

[0024] In one embodiment, a shared memory means for communicating includes a memory connected to a graphics processing unit and configured to be accessed by a key value storage device through a peripheral component interconnect connection via peer-to-peer direct memory access. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] These and other features and advantages of the present disclosure will be appreciated and understood with reference to the specification, claims, and drawings, in which:

[0026] Figure 1 is a functional block diagram of a system for machine learning.

[0027] Figure 2 is a block diagram of a graphics card equipped with an onboard SSD according to an embodiment of the present disclosure.

[0028] Figure 3 is a data flow diagram according to an embodiment of the present disclosure.

[0029] Figure 4 is a timing diagram according to an embodiment of the present disclosure.

[0030] Figure 5 is a timing diagram according to an embodiment of the present disclosure.

[0031] Description of Figure Numbers

[0032] 105: Host application;

[0033] 110: software key value storage;

[0034] 115: file system;

[0035] 120: block interface;

[0036] 125: Graphics processing unit memory;

[0037] 205: onboard key value solid state drive;

[0038] 210: Graphics processing unit;

[0039] 305: Graphics processing unit application;

[0040] 310: Key value request queue. DETAILED DESCRIPTION

[0041] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of an exemplary embodiment of a system and method, and is not intended to represent the only form that the present disclosure can be constructed or utilized, and the system and method are used to perform machine learning involving key value access to data provided according to the present disclosure. The description sets forth the features of the present disclosure in conjunction with the illustrated embodiments. However, it will be understood that different embodiments may be utilized to implement the same or equivalent functions and structures, and the different embodiments are also intended to be encompassed within the scope of the present disclosure. As indicated elsewhere herein, the same element numbers are intended to indicate the same elements or features.

[0042] Prior art machine learning platforms have disadvantages when used in stochastic machine learning training methods that randomly sample subsets of the entire training data. Because the CPU needs to be involved to provide a key value interface and data transmission across a peripheral component interconnect express (PCIe) bus, such machine learning platforms may suffer from low graphics processing unit (GPU) utilization due to key value access during stochastic machine learning training. As mentioned above, in some prior art systems, the host central processing unit (CPU) performs file index lookups and file system accesses to identify the location of data that results in serialized key value accesses. In contrast, in some embodiments, performance is improved because the CPU is not involved in key value accesses to data stored in an onboard SSD. The GPU sends key value commands directly to an onboard key value storage device (e.g., an onboard key value SSD) on a graphics card (e.g., a GPU) that includes a GPU and an onboard key value SSD, which enables asynchronous key value access, thereby reducing the impact of access latency. As used herein, a "key value storage device" is a persistent storage device (such as an SSD) that is configured to respond to key value requests (each containing a key) by returning a value in response to each such request, the value corresponding to the key contained in the request.

[0043] Figure 1The overall flow of key value access to data stored on an SSD with a block interface using a software key value store during GPU machine learning training is depicted. First, the host application 105 initiates key value access by sending a "get" request to the software key value store 110. In order to identify the location of the data for a certain key, the software key value store 110 accesses an index table that stores the file offset of the data corresponding to the specified key. The software key value store 110 then accesses a file system 115 with the file offset, and the file system 115 accesses an SSD with a block interface 120 and extracts the data corresponding to the specified key. Once the value becomes available to the host application, the host application transfers the value to the GPU memory 125 for GPU computation and launches the GPU kernel. For a software key value store with a prior art SSD with a block interface, all of these operations are performed sequentially.

[0044] like Figure 1 As shown in , key value access to an SSD with a block interface involves several computational steps running on the host to identify the location of data corresponding to a specified key. The GPU can only perform the corresponding calculation after the calculation of the software key value storage is completed. Because the GPU needs to wait for the completion of key value access from other GPUs, the latency of key value access increases when more GPUs are used in the system, resulting in serialization of GPU calculations. As a result, key value access to prior art SSDs with block interfaces limits the number of GPUs that can be efficiently used in the system.

[0045] In some embodiments, a graphics card having an onboard SSD with a key-value interface (or "key-value SSD") is used to overcome some of the shortcomings of prior art systems. Figure 2An exemplary system with such a device is depicted. Unlike prior art systems where moving data between storage and the GPU requires data transfer across a global PCIe bus (thus increasing latency for data access from the GPU), a graphics card with an onboard key value SSD 205 can reduce overhead by utilizing peer to peer (P2P) direct memory access (DMA) between the onboard key value SSD 205 and the GPU 210 and giving the GPU 210 full P2P DMA control. In some embodiments, the onboard key value SSD 205 provides key value commands as non-standard commands. For example, the key value request queue (discussed in further detail below) can be implemented as a nonvolatile memory express (NVMe) command queue, where NVMe commands are vendor-proprietary commands defined in the firmware of the SSD 205 and in the driver software on the GPU to correspond to key value commands (i.e., commands to request a value corresponding to a key obtained as part of the command from the SSD 205).

[0046] In some embodiments, such a system may be used to provide asynchronous key value access in an onboard key value SSD, and some embodiments utilize the key value SSD within a graphics card to perform random sampling of training data. Figure 3The flow of key value access during machine learning training in some embodiments is depicted. A significant difference between such embodiments and some prior art systems is that in such embodiments, the GPU sends key value commands directly to the key value SSD 205. First, during the initial stage of executing the host application, the host application maps the memory of the special GPU device on the peripheral component interconnect (PCI) base address register (BAR) storage area, thereby establishing direct communication between the key value SSD 205 and the GPU. The area of ​​GPU memory allocated for communication between the key value SSD 205 and the GPU (e.g., using memory mapped input-output) using this process may be referred to herein as the "input-output area" of the GPU memory. The input-output area of ​​the GPU memory that can be directly accessed by both the GPU and the key value SSD 205 can be functionally operated as a shared memory. The GPU application 305 issues a get request to the key value SSD 205 by performing a memory mapped input-output on the key value SSD 205 and supplying the bus address of the exposed GPU memory. After the firmware within the key value SSD 205 performs a key lookup to retrieve the value corresponding to the key, the firmware writes the value to the mapped GPU device memory (ie, to the input-output area of ​​the GPU memory) without the mediation of the host application 105.

[0047] In some embodiments, a key value request queue (KVRQ) 310 is used, and key value access is non-blocking in the sense that the GPU does not have to wait for a response to a first request before making a second subsequent request. Instead, the GPU places a key value request into the key value request queue 310, and the request is in turn processed by the key value SSD 205. Thus, when the GPU application places a request into the key value request queue 310, the request operation is completed. The key value request queue 310 holds outstanding requests so that the number of entries in the key value request queue 310 is the number of key value requests. When the value is transferred to the GPU memory, the firmware in the SSD 205 releases the key value request queue entry corresponding to the specified key.

[0048] Separate key value accesses for each GPU allow key value accesses from multiple GPUs to overlap. For example, in a system with two GPUs each connected to a respective key value SSD, the two GPUs may issue requests simultaneously, and their respective key value SSDs may respond simultaneously. Figure 4 Operation of such an example is shown involving two GPUs, each connected to a corresponding key value SSD. Figure 4 Also shown is the time saved by the system, compared to the prior art method in which GPU computations are serialized, in such embodiments where two GPUs are utilized to perform overlapping key value accesses. In some embodiments, three or more (e.g., any number) GPUs may each be connected to a respective key value SSD and perform overlapping (e.g., simultaneous) key value operations.

[0049] In some embodiments, the separation of requests and responses for key value access enables asynchronous key value access, such as enabling batch processing of multiple requests from a GPU. Figure 5 An example of asynchronous key value access when batching two key value commands is shown. Compared to synchronous key value access in which GPU computing and SSD device access are serialized, the asynchronous key value access of some embodiments makes it possible to overlap multiple key value commands using GPU computing. In this example, the GPU continuously issues acquisition requests instead of waiting for the completion of the previous request each time. In some embodiments, three or more (e.g., any number) GPUs can each connect to a corresponding key value SSD and perform overlapping (e.g., simultaneous) key value operations.

[0050] In some embodiments, when a key value SSD retrieves a value in response to a key value request, the key value SSD writes the retrieved value back to the key value request queue, i.e., writes back to an area of ​​memory (or "return value area") allocated within the key value request for this purpose. In other embodiments, the key value SSD instead writes the retrieved value to a separate queue (or "return value queue") allocated in an input-output area of ​​GPU memory. In some embodiments, instead of having each GPU have a single dedicated key value SSD to which each GPU sends a key value request, a single GPU may have several key value SSDs. In such embodiments, several key value request queues, each for a corresponding key value SSD, may be allocated in GPU memory. In other embodiments, several GPUs may be connected to a single key value SSD, which may, for example, service key value requests in corresponding key value request queues in the GPUs in a round-robin manner.

[0051] In some embodiments, the tasks performed by the host application involve only establishing a path for communication between the GPU and the SSD, which improves the scalability of these embodiments by avoiding or serializing GPU computations caused by key value access operations performed by the host application on the CPU. In this way, these embodiments can achieve outward expansion of multiple GPUs to accelerate machine learning training. By replacing complex key value software with a simple device interface, some embodiments also reduce or may impose resource requirements on the host, including (for example) requirements for the number of CPU cores. Avoiding such requirements can result in better energy efficiency.

[0052] Some embodiments may be constructed using one or more processing circuits. The term "processing circuit" is used herein to mean any combination of hardware, firmware, and software for processing data or digital signals. The processing circuit hardware may include, for example, an application specific integrated circuit (ASIC), a general or dedicated central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), and a programmable logic device such as a field programmable gate array (FPGA). As used herein, in a processing circuit, each function is performed by hardware configured (i.e., hardwired) to perform the function, or by more general hardware (such as a CPU) configured to execute instructions stored in a non-temporary storage medium. The processing circuit may be manufactured on a single printed circuit board (PCB) or distributed over several interconnected PCBs. The processing circuit may include other processing circuits; for example, a processing circuit may include two processing circuits FPGA and CPU interconnected on a PCB.

[0053] It will be understood that although the terms "first," "second," "third," etc., may be used herein to describe various elements, components, regions, layers, and / or sections, these elements, components, regions, layers, and / or sections should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer, or section from another element, component, region, layer, or section. Thus, a first element, component, region, layer, or section discussed herein may be referred to as a second element, component, region, layer, or section without departing from the spirit and scope of the inventive concept.

[0054] The terms used herein are for the purpose of describing specific embodiments only, and are not intended to limit the inventive concept. As used herein, the terms "substantially", "about" and similar terms are used as approximate terms rather than as terms of degree, and are intended to take into account the inherent deviations of the measured or calculated values ​​that will be recognized by those of ordinary skill in the art. As used herein, the term "major component" refers to a component present in a composition, polymer or product in an amount greater than any other single component in the composition or product. In contrast, the term "primary component" refers to a component that constitutes at least 50% by weight or more than 50% by weight of a composition, polymer or product. As used herein, when applied to multiple objects, the term "major portion" means at least half of the object.

[0055] As used herein, unless the context clearly indicates otherwise, the singular form "a / an" is intended to also include the plural form. It will be further understood that when used in this specification, the terms "comprises" and / or "comprising" specify the presence of stated features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. When before a list of elements, a statement such as "at least one of" modifies the entire list of elements and does not modify a single element in the list. In addition, when describing an embodiment of the inventive concept, the use of "may" refers to "one or more embodiments of the present disclosure". In addition, the term "exemplary" means an example or illustration. As used herein, the terms "use", "using", and "used" may be considered synonymous with the terms "utilize", "utilizing", and "utilized", respectively.

[0056] It will be understood that when an element or layer is referred to as being “on,” “connected to,” “coupled to,” or “adjacent to” another element or layer, it can be directly on, directly connected to, coupled to, or adjacent to the other element or layer, or one or more intervening elements or layers may be present. In contrast, when an element or layer is referred to as being “directly over,” “directly connected to,” “directly coupled to,” or “immediately adjacent to” another element or layer, there are no intervening elements or layers present.

[0057] Although exemplary embodiments of systems and methods for performing machine learning involving key-value access to data have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Accordingly, it should be understood that systems and methods for performing machine learning involving key-value access to data constructed in accordance with the principles of the present disclosure may not be implemented as specifically described herein. The invention is also defined in the following claims and their equivalents.

Claims

1. A method for accessing a key value in a machine learning system, the method comprising: performing input and output operations by a key value storage device connected to a memory in an input-output region of the memory that is memory mapped to another device; reading, by the key value storage device, a first value from a key value request queue of the input-output region according to a first key value request issued by the other device, wherein the first key value request is based on a configuration of a host device; as well as The first value is written into the input-output area of ​​the memory by the key value storage device according to the first key value request.

2. The method of claim 1 further comprising performing, by the key value storage device, a key lookup in the key value storage device to retrieve the first value for the writing of the first value into the input-output area of ​​the memory. 3 . The method according to claim 1 , wherein the first key value request includes a return value area, and the return value area is an area allocated to the first value. 4 . The method of claim 3 , wherein the writing of the first value into the input-output area of ​​the memory comprises writing the first value to the return value area.

5. The method according to claim 1, further comprising: receiving, by the other device, the configuration from the host device; as well as Based on the configuration from the host device, a key value is requested to be stored in the input-output area of ​​the memory by the other device, and a value is read from the input-output area of ​​the memory by the other device.

6. The method of claim 1 , wherein the other device memory-mapped to the memory comprises a graphics processing unit configured to write the first key value request to the input-output region of the memory and to read the first value from the input-output region of the memory.

7. The method of claim 1, wherein the first key value request is read by the key value storage device from the key value request queue via peer direct memory access.

8. The method of claim 1, wherein the first value is written by the key value storage device to the input-output area of ​​the memory via peer-to-peer direct memory access.

9. The method of claim 1, wherein the key value storage device is connected to the other device by a peripheral component interconnect connection.

10. The method of claim 1, wherein the first key value request is written by the other device into the key value request queue of the input-output area; and The method further includes writing a second key value request to the key value request queue by the other device after the writing of the first key value request by the other device and before the writing of the first value by the key value storage device.

11. A system for machine learning, the system comprising: A key value storage device; as well as a memory connected to the key value storage device, the memory being a memory mapped to another device connected to the key value storage device by a peripheral component interconnect connection, The key value storage device is configured as follows: performing input and output operations in an input-output region of said memory; reading a first value from a key value request queue of the input-output region according to a first key value request issued by the other device, the first key value request being based on a configuration of a host device; as well as The first value is requested to be written to the input-output area of ​​the memory according to the first key value.

12. The system of claim 11, wherein the key value storage device is further configured to perform a key lookup in the key value storage device to retrieve the first value to write the first value to the input-output area of ​​the memory.

13. The system of claim 11, wherein the first key value request includes a return value area, the return value area being an area allocated to the first value.

14. The system of claim 11, wherein the other device comprises a graphics processing unit configured to write the first key value request to the input-output area of ​​the memory and read the first value from the input-output area of ​​the memory.

15. The system of claim 11, wherein the key value storage device is configured to read the first key value request from the key value request queue via peer-to-peer direct memory access.

16. The system of claim 11, wherein the key value storage device is configured to write the first value to the input-output area of ​​the memory via peer-to-peer direct memory access.

17. A system for machine learning, the system comprising: Graphics processing unit; A key value storage device; as well as a shared memory for communicating between the graphics processing unit and the key value storage device, the shared memory being a memory mapped to the graphics processing unit, The key value storage device is configured as follows: performing input and output operations in the shared memory; reading a first value from a key value request queue of the shared memory according to a first key value request issued by the graphics processing unit, wherein the first key value request is based on a configuration of a host device; as well as The first value is requested to be written into the shared memory according to the first key value.

18. The system of claim 17, wherein the graphics processing unit is configured to write the first key value request to the shared memory and read the first value from the shared memory, and The graphics processing unit is memory mapped to the shared memory to perform memory mapped input-output operations in the shared memory.

19. The system of claim 18, wherein the key value storage device is configured to read the first key value request from the key value request queue via peer-to-peer direct memory access, and Wherein the key value storage device is configured to write the first value to the shared memory via peer direct memory access.

20. The method of claim 1, wherein the first key value request comprises an address that causes memory mapping to be exposed to the input-output region of the memory of the other device based on the configuration of the host device.

Images