Method for processing data by a data processing accelerator and data processing accelerator

By introducing operating system modules and kernel scheduling mechanisms into the data processing accelerator, the problems of complexity and low resource management efficiency of DPA and host communication in the existing technology are solved, and efficient data processing and resource utilization are achieved, suitable for high-performance computing and machine learning models inferred tasks.

CN112948070BActive Publication Date: 2024-11-22BAIDU USA LLC +1
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202011455568.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-12-10
Filing Date
2020-12-10
Publication Date
2024-11-22
Estimated Expiration
2040-12-10

AI Technical Summary

Technical Problem

Existing data processing accelerators (DPAs) often require vendor-specific programs and cores, resulting in complex communication with host computing devices, low resource management and scheduling efficiency, making it difficult to achieve high-performance parallel data computing.

Method used

By introducing operating system modules into the data processing accelerator, commands to execute data processing tasks are scheduled based on one or more cores, and completing signals are generated to confirm the task is completed, simplifying communication between the host and DPA, and improving resource management and task scheduling efficiency. .

Benefits of technology

Implements an efficient data processing process that receives requests from the host, schedules the kernel to execute commands and sends results, improves data processing throughput and resource utilization, simplifies communication between the host and DPA, and is suitable for high-performance computing and machine learning models. inference task.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112948070B_ABST
    Figure CN112948070B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a method for processing data by a data processing (DP) accelerator, a data processing accelerator, and a non-transitory machine-readable medium. The method includes: receiving a request from a host by a data processing accelerator, the host having a first operating system that hosts an application requesting to perform a data processing task; scheduling one or more commands for completing the data processing task by an operating system module of the data processing accelerator based on one or more kernels, wherein the operating system module hosts a second operating system in the data processing accelerator to call the one or more kernels to perform the data processing task; generating a completion signal by the operating system module of the data processing accelerator when completing the one or more scheduled commands; and sending a result back to the application to confirm the completion of the one or more scheduled commands for the data processing task based on the completion signal.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention generally relate to machine learning model training and inference. More specifically, embodiments of the present invention relate to methods for processing data by a data processing (DP) accelerator (DPA), a data processing accelerator, and a non-transitory machine-readable medium. Background Art

[0002] Most DPAs require vendor-specific, dedicated programs and kernels. As a result, DPAs are typically configured to communicate with a host computing device (e.g., a CPU system) for the host computing device to manage resources and schedule jobs for the DPA. As data processing accelerators (DPAs) continue to evolve into high-performance accelerated computing devices, more and more applications are being written to perform data-parallel computations using DPAs similar to general-purpose computing devices. Summary of the invention

[0003] The present disclosure is directed to providing a method for processing data by a data processing (DP) accelerator, a data processing accelerator, and a non-transitory machine-readable medium.

[0004] According to one aspect of the present disclosure, a method for processing data by a data processing accelerator is provided, the method comprising: receiving a request by the data processing accelerator from a host, the host having a first operating system that hosts an application requesting to perform a data processing task; scheduling, by an operating system module of the data processing accelerator, one or more commands for completing the data processing task based on one or more kernels, wherein the operating system module hosts a second operating system within the data processing accelerator to call the one or more kernels to perform the data processing task; generating a completion signal by the operating system module of the data processing accelerator when the one or more scheduled commands are completed; and sending a result back to the application to confirm the completion of the one or more scheduled commands for the data processing task based on the completion signal.

[0005] According to another aspect of the present disclosure, a data processing (DP) accelerator is provided, comprising: a host interface coupled to a host, the host executing a first operating system in which an application is hosted, wherein the host interface receives a request from the host and initiated by the application to perform a data processing task; one or more kernels; and an operating system module having a second operating system executed therein, the operating system module being configured to perform operations, the operations comprising scheduling one or more commands for executing the data processing task based on the one or more kernels; generating a completion signal when the one or more scheduled commands are completed, and sending a result back to the application via the host interface to confirm the completion of the one or more scheduled commands for the data processing task based on the completion signal.

[0006] According to yet another aspect of the present disclosure, a non-transitory machine-readable medium is provided, having instructions stored therein, which, when executed by a processor, cause the processor to perform the method for processing data by a data processing accelerator as described above or below. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Embodiments of the invention are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0008] Figure 1 is a block diagram illustrating an example of a system configuration for communication between a host and a data processing (DP) accelerator according to one embodiment.

[0009] Figure 2 is a block diagram illustrating an example of a host server in communication with a data processing (DP) accelerator according to one embodiment.

[0010] Figure 3A is a block diagram illustrating a host in communication with a data processing accelerator according to one embodiment.

[0011] Figure 3B is a block diagram illustrating operating system modules according to one embodiment.

[0012] Figure 4 is a block diagram illustrating example operations of an operating system module according to one embodiment.

[0013] Figure 5 is a block diagram illustrating an example of a kernel-type command according to one embodiment.

[0014] Figure 6 is a block diagram illustrating an example of a task-type command according to one embodiment.

[0015] Figure 7is a flow chart illustrating a method performed by a DP accelerator according to one embodiment. DETAILED DESCRIPTION

[0016] Various embodiments and aspects of the present disclosure will be described with reference to the details discussed below, and the accompanying drawings will illustrate various embodiments. The following description and the accompanying drawings are illustrative of the present disclosure and should not be construed as limiting the present disclosure. Many specific details are described to provide a comprehensive understanding of the various embodiments of the present disclosure. However, in some cases, in order to provide a brief discussion of the embodiments of the present disclosure, known or conventional details are not described.

[0017] References in the specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present disclosure. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment.

[0018] The following embodiments relate to using a data processing (DP) accelerator to increase the processing throughput of certain types of operations that can be offloaded (or delegated) from a host device to a DP accelerator. The DP accelerator may be a graphics processing unit (GPU), an artificial intelligence (AI) accelerator, a math coprocessor, a digital signal processor (DSP), or other type of processor. The DP accelerator may be a proprietary design, such as Baidu's AI accelerators, GPU, etc. Although the embodiments are shown and described using a host device securely coupled to one or more DP accelerators, the concepts described herein may be more generally implemented as a distributed processing system.

[0019] The host device and the DP accelerator may be interconnected via a high-speed bus such as a peripheral component express (PCIe), or other high-speed buses such as Ethernet. The host device and the DP accelerator may exchange keys and initiate a secure channel on the PCIe bus before performing the operations of the various aspects of the present invention described below. Some of the operations include the DP accelerator using an artificial intelligence (AI) model (or a machine learning model) to perform inference using data provided by the host device.

[0020] Embodiments disclose systems and methods for operating system modules for data processing accelerators (DPAs). The operating system module can host an operating system (OS) (e.g., an embedded or real-time OS) that can schedule jobs for the DPA and manage resources (e.g., computing units, memory, IO bandwidth) for the DPA. For one embodiment, the system receives, through the DPA, a request for executing or performing a data processing task sent by an application executed by a host. The host includes a host OS executed therein, and the DP accelerator has its own OS (referred to herein as an accelerator OS or client OS) executed therein. The system, through the operating system module of the DPA, schedules one or more commands for completing the data processing task based on one or more kernels. When the one or more scheduled commands are completed, the system generates a completion signal through the operating system module of the DPA. The system sends the result back to the application based on the completion signal to confirm the completion of the one or more commands for the data processing task.

[0021] Any of the above functions may be programmed as executable instructions onto one or more non-transitory computer-readable media. When the executable instructions are executed by a processing system having at least one hardware processor (e.g., GPU, ASIC, FPGA), the processing system causes the functions to be implemented. Any of the above functions may be implemented by a processing system having at least one hardware processor, the hardware processor being coupled to a memory programmed with the executable instructions, which, when executed, causes the processing system to implement the functions.

[0022] With respect to any of the above aspects, in some embodiments, the host processor may be a central processing unit (CPU) and the DP accelerator may be a general purpose processing unit (GPU) coupled to the CPU via a bus or interconnect. The DP accelerator may be implemented in the form of an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) device, or other form of integrated circuit (IC). Alternatively, the host processor may be part of a main data processing system and the DP accelerator may be one of many distributed systems as a secondary system, the main system being able to offload its data processing tasks remotely via a link or network (e.g., a cloud computing system such as a software as a service, or SaaS system, or a platform as a service, or PaaS system). The link between the host processor and the DP accelerator may be a peripheral component interconnect express (PCIe) link or a network connection such as an Ethernet connection.

[0023] Figure 1 is a block diagram showing an example of a system configuration for communication between a host and a data processing (DP) accelerator according to one embodiment. Figure 1, system configuration 100 includes, but is not limited to, one or more client devices 101-102 communicatively coupled to a data processing server (e.g., a host) 104 via a network 103. Client devices 101-102 can be any type of client device, such as a personal computer (e.g., a desktop, a laptop, and a tablet computer), a "thin" client, a personal digital assistant (PDA), a web-enabled appliance, a smart watch, or a mobile phone (e.g., a smartphone), etc. Alternatively, client devices 101-102 can be other servers. Network 103 can be any type of network, such as a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination thereof, wired or wireless. Communications between client devices 101-102 and data processing server 104 via network 103 can be protected, for example, by TLS / SSL.

[0024] The data processing server (e.g., host) 104 can be any kind of server or server cluster, such as a web or cloud server, an application server, a back-end server, or a combination thereof. The server 104 may include an interface that allows a client, such as a client device 101-102, to access resources or services provided by the server 104. For example, the server 104 can be a cloud server or a server in a data center, providing various cloud services to clients, such as, for example, cloud storage, cloud computing services, big data services, modeling services, machine learning training services, data mining services, etc. The server 104 can be configured as part of a software as a service (SaaS) or platform as a service (PaaS) system on a cloud, which can be a private cloud, a public cloud, or a hybrid cloud. The interface may include a web interface, an application programming interface (API), and / or a command line interface (CLI).

[0025] For example, a client, in this example a user application of client device 101 (e.g., a web browser, an application) may send or transmit instructions for execution (e.g., artificial intelligence (AI) training, inference instructions, etc.) to server 104, and the instructions are received by server 104 via an interface on network 103. In response to the instructions, server 104 communicates with DP accelerators 105-107 to complete the execution of the instructions. In another embodiment, the client may run the application remotely from server 104 (via a web browser or web application), and the client may remotely request the application to execute the instructions. In some embodiments, the instructions are machine learning type instructions, where a DP accelerator, as a dedicated machine or processor, can execute the instructions many times faster than the execution of server 104. Therefore, server 104 can control / manage execution jobs for one or more DP accelerators in a distributed manner. Server 104 then returns the execution results to client devices 101-102. A DP accelerator or AI accelerator may include one or more dedicated processors, such as those that can be accessed from a server. Obtained Artificial Intelligence (AI) chipset, or alternatively, the DP accelerator can be from Or some other AI chipset vendor's AI chipset.

[0026] According to one embodiment, each of the applications accessing any of the DP accelerators 105-107 and hosted by the DP server 104, also referred to as the host, may be verified to be provided by a trusted source or vendor. Each of the applications may be launched and executed in a trusted execution environment (TEE) specifically configured and executed by a central processing unit (CPU) of the host 104. When the application is configured to access any of the DP accelerators 105-107, a secure connection is established between the host 104 and a corresponding one of the DP accelerators 105-107, thereby protecting data exchanged between the host 104 and each of the DP accelerators 105-107 from malware attacks.

[0027] Figure 2 2 is a block diagram illustrating an example of a host communicating with a data processing (DP) accelerator according to one embodiment. The system 200 can provide a communication scheme between the host and the DP accelerator with or without hardware modification to the DP accelerator. Figure 2, the host machine or server 104 can be depicted as a system with one or more layers to protect against intrusion, such as user applications 203, runtime libraries 205, drivers 209, operating systems 211, and hardware 213 (e.g., security modules (Trusted Platform Module (TPM)) / central processing units (CPUs)). The host machine 104 is typically a CPU system that can control and manage execution jobs on the host system or DP accelerators 105-107. In order to protect the communication channel between the DP accelerator and the host machine, different components may be required to protect different layers in the host system that are vulnerable to data intrusion or attack. For example, a trusted execution environment (TEE) can protect the user application layer and the runtime library layer from data intrusion.

[0028] refer to Figure 2 In one embodiment, the host system 104 includes hardware 213 having one or more CPUs and equipped with a security module (such as a trusted platform module (TPM)) within the host machine 104. The TPM is a dedicated chip on an endpoint device that is used to store encryption keys (e.g., RSA encryption keys) specific to the host system for hardware authentication. Each TPM chip may include one or more RSA key pairs (e.g., public and private key pairs), called endorsement keys (EKs) or endorsement certificates (ECs), i.e., root keys. The key pairs are maintained inside the TPM chip and cannot be accessed by software. Then, the key parts of the firmware and software can be hashed by the EK or EC before being executed to protect the system from unauthorized firmware and software modifications. Therefore, the TPM chip on the host machine can be used as a trusted root for secure boot.

[0029] The TPM chip also protects the driver 209 and the operating system (OS) 211 in the working kernel space to communicate with the DP accelerator. Here, the driver 209 is provided by the DP accelerator vendor and can be used as one or more drivers for user applications to control the communication channel between the host and the DP accelerator. Because the TPM chip and secure boot protect the OS and drivers in their kernel space, the TPM also effectively protects the driver 209 and the operating system 211.

[0030] Since the communication channel for the DP accelerators 105 - 107 may be exclusively occupied by the OS and the driver, the communication channel is also protected by the TPM chip.

[0031] In one embodiment, the host machine 104 includes a trusted execution environment (TEE) 201 that is forced to be protected by TPM / CPU213. TEE is a secure environment. TEE can ensure that the code and data loaded into the TEE are protected in terms of confidentiality and integrity. An example of TEE can be Intel Software Guard Extensions (SGX) or AMD Secure Encrypted Virtualization (SEV). Intel SGX and / or AMD SEV may include a set of central processing unit (CPU) instruction codes that allow user-level code to allocate a private area of ​​the CPU's memory, which is protected from the impact of processes running at a higher privilege level. Here, TEE 201 can protect user applications 203 and runtime libraries 205, wherein user applications 203 and runtime libraries 205 can be provided by end users (end user) and DP accelerator vendors, respectively. Here, runtime library 205 can convert API calls into commands for the execution, configuration and / or control of DP accelerators. In one embodiment, runtime library 205 provides a set of predetermined (e.g., predefined) kernels for user application execution.

[0032] In another embodiment, the host machine 104 includes a memory security application 207 implemented using a memory-safe language such as Rust and Golang. These memory security applications running on a memory-safe Linux version such as MesaLock Linux can further protect the system 200 from data confidentiality and integrity attacks. However, the operating system can be any Linux distribution, UNIX, Windows OS, or Mac OS.

[0033] In one embodiment, the system can be set up as follows: Install the Storage Secure Linux distribution onto a system equipped with TPM secure boot (such as Figure 2 The installation can be performed offline during the manufacturing or preparation phase. The installation can also ensure that the user space applications of the host system are programmed using a memory-safe programming language. Ensuring that other applications running on the host system 104 are memory-safe applications can further mitigate potential confidentiality and integrity attacks on the host system 104.

[0034] After installation, the system can then be started through a TPM-based secure boot. TPM secure boot ensures that only signed / authenticated operating systems and accelerator drivers are started in the kernel space that provides accelerator services. In one embodiment, the operating system can be loaded through a virtual machine monitor (hypervisor). Note that a virtual machine monitor or virtual machine manager is computer software, firmware, or hardware that creates and runs virtual machines. Note that kernel space is a declarative region or scope in which a kernel (i.e., a predetermined set of (e.g., predefined) functions to be executed) is identified to provide functions and services to user applications. In the event that system integrity is compromised, TPM secure boot may not be able to start, but instead shut down the system.

[0035] After secure boot, runtime library 205 runs and creates TEE 201, and TEE 201 places runtime library 205 in a trusted storage space associated with CPU 213. Next, user application 203 is started in TEE 201. In one embodiment, user application 203 and runtime library 205 are statically linked and started together. In another embodiment, runtime library 205 is first started in TEE, and then user application 203 is dynamically loaded in TEE 201. In another embodiment, user application 203 is first started in TEE, and then runtime library 205 is dynamically loaded in TEE 201. Note that statically linked libraries are libraries linked to applications at compile time. Dynamic loading can be performed by a dynamic linker. The dynamic linker loads and links shared libraries for running user applications at runtime. Here, user application 203 and runtime library 205 in TEE 201 are visible to each other at runtime, for example, all process data are visible to each other. However, external access to TEE is denied.

[0036] In another embodiment, the user application can only call kernels from a set of kernels predetermined by the runtime library 205. In another embodiment, the user application 203 and the runtime library 205 are reinforced using a side channel free algorithm to defend against side channel attacks such as cache-based side channel attacks. A side channel attack is any attack based on information obtained from the implementation of a computer system, rather than based on weaknesses in the implemented algorithm itself (e.g., cryptanalysis and software errors). Examples of side channel attacks include cache attacks, which are attacks based on the ability of an attacker to monitor the cache of a shared physical system in a virtualized environment or cloud environment. Reinforcement may include shielding of the cache, outputs generated by the algorithm to be placed on the cache. Then, when the user application completes execution, the user application terminates its execution and exits from the TEE.

[0037] In summary, system 200 provides multiple layers of protection for the DP accelerator (e.g., communication of data such as machine learning models, training data, and inference outputs) to prevent loss of data confidentiality and integrity. System 200 may include a TPM-based secure boot protection layer, a TEE protection layer, and a kernel authentication / verification layer. Additionally, system 200 may provide a memory-safe user space by ensuring that other applications on the host machine are implemented using memory-safe programming languages, which may further eliminate attacks by eliminating potential memory corruptions / vulnerabilities. Additionally, system 200 may include applications using side-channel-free algorithms to defend against side-channel attacks, such as cache-based side-channel attacks.

[0038] Figure 3A is a block diagram illustrating a host computer in communication with a data processing accelerator according to one embodiment. System 300 may be used for Figure 23 , in one embodiment, the host system 104 includes a runtime library 205, which includes a host channel manager (HCM) 259. In one embodiment, the HCM 259 includes an authentication module 251, a termination module 252, a key manager 253, a key storage 254, and an encryption engine 255. The authentication module 251 can authenticate user applications running on the host server 104 to obtain permission to access or use resources of the DP accelerator. The termination module 252 can terminate the connection (e.g., the channel associated with the connection will be terminated). The key manager 253 can manage (e.g., create or destroy) asymmetric key pairs or symmetric keys, which are used to encrypt / decrypt one or more data packets used for different secure data exchange channels. Here, each user application (as a Figure 2 The user application 203 of the embodiment of the present invention may correspond to or be mapped to different secure data exchange channels in a one-to-many relationship, and each data exchange channel may correspond to a DP accelerator. The key storage 254 may store an encrypted asymmetric key pair or a symmetric key. The encryption engine 255 may encrypt or decrypt data packets of data exchanged via any of the secure channels.

[0039] refer to Figure 3A In one embodiment, the DP accelerator 105 includes an ACM 270 (similarly, the DP accelerator 107 includes 290) and a security unit (SU) 275. The security unit 275 may include a key manager 271, a key storage 272, a true random number generator 273, and an encryption engine 274. The key manager 271 may manage (e.g., generate, securely maintain, and / or destroy) asymmetric key pairs or symmetric keys. The key storage 272 may store encrypted asymmetric key pairs or symmetric keys. The true random number generator 273 may generate true random numbers for encryption. The encryption engine 274 may encrypt or decrypt key information or data packets for data exchange. The security unit 275 may contain one or more RSA key pairs (e.g., public and private key pairs), referred to as endorsement keys (EKs) or endorsement credentials (ECs), i.e., root keys. The key pairs are maintained inside the security unit 275 and cannot be accessed by software. The security unit can then use the EK or EC to encrypt data or generate other security keys (which in turn can be used to encrypt data). Basically, the root certificate (EK / EC) is hardened to the security unit 275 for encryption purposes. In some embodiments, the ACM 270 and SU 275 are integrated modules.

[0040] In one embodiment, the DP accelerator (DPA) 105 includes an operating system module 282. The operating system module 282 may include an operating system. The operating system may be an embedded operating system or a real-time operating system. The operating system may schedule tasks / commands and manage resources (computing units, memory) for the DPA. In one embodiment, the operating system module 282 may be implemented as a software-based module or a hardware-based module based on an FPGA, an ASIC, and / or a microcontroller unit.

[0041] As mentioned above, reference Figure 3A , the runtime library 205 may provide a set of predetermined (e.g., predefined) kernels for execution by the user application. The runtime library 205 may convert API calls into kernel executions, and convert kernel executions into data, configuration, and / or control commands for execution by the DPA. The DPA may then receive commands (e.g., data, control, and configuration commands) and initiate commands for execution by the DPA. A typical task / job may require hundreds of commands to be transmitted from the host 104 to the DPA 105, in which case the cost for transmitting the commands over the communication channel 215 is high.

[0042] Although the runtime library 205 and the driver 209 of the host 104 include information of the hardware resources of the accelerator 105, the accelerator 105 may include a real-time operating system to provide a set of kernels, drivers, resource management, and / or a task scheduler to schedule tasks for execution, similar to the scheduling performed by the runtime library 205 of the host 104. In this case, the runtime library 205 and the driver 209 are modified, and the tasks previously performed by the runtime library 205 and the driver 209 are delegated to the operating system module 282 of the DPA 105. For example, the operating system module 282 of the DPA 105 may decompose a task (such as an inference or training task) into many subtasks (e.g., kernels), and further decompose the subtasks into atomic commands (e.g., data, control, and configuration commands) to be executed by the DPA. Therefore, the complexity of the task is abstracted from the runtime library of the host, or the granularity of the commands initiated by the host 104 is at a high level of abstraction.

[0043] Figure 3B30 is a block diagram illustrating an operating system module according to one embodiment. The operating system module 282 may include a receiver unit 301, a session creator 303, a task scheduler 305, a resource allocator 307, a kernel executor 309, a command executor 311, a transmitter unit 313, and a driver 315. These components represent at least a portion of an operating system, i.e., an accelerator or guest OS, such as an embedded or real-time OS that typically has a small memory footprint in an operating environment suitable for integrated circuit (IC) devices. The receiver unit 301 receives data from a host server. The data may include driver files for DPA, kernel data, AI model data, etc.

[0044] The session creator 303 may create a session context for a data processing task. The session context or context may access / read / write resources or data of the commands and tasks of the requesting application for the session. The task scheduler 305 may schedule one or more tasks for execution by the DPA. The task scheduler 305 may determine a plurality of kernels to be executed to complete a task and schedule the plurality of kernels for execution. The resource allocator 307 may allocate / deallocate the computing resources and storage resources of the DPA to execute a plurality of kernels. The kernel executor 309 may launch and execute a kernel. Examples of kernels may include kernels that preprocess data objects or apply functions to data objects.

[0045] The command executor 311 may initiate and execute data, control, kernel, and configuration commands to complete execution for multiple kernels. Control commands include commands that control which hardware resources of the DPA are used to execute kernels. Data commands may include commands to manipulate data, such as placing data at a specific memory location or applying arithmetic operations to data. Configuration commands may include commands to configure the context (or settings / environment) in which the data commands are to be executed. The transmitter unit 313 may send completion confirmations and / or execution results to the host. Drivers 315 include one or more driver files for enabling the operating system module 282 to access, use, and manage the hardware resources of the DPA 105.

[0046] Figure 4 is a block diagram illustrating an example operation of an operating system module according to one embodiment. Figure 4In one embodiment, in operation S401, a user application requests resources and tasks to be performed by the DPA via communication between the host server 104 and the DPA 105. Operations S402-S406 may be performed by the operating system module 282 in response to the request. The OS module 282 may include at least a portion of a typical OS such as an accelerator or guest OS that is responsible for managing resources and performing data processing tasks (e.g., AI modeling or AI inference). The OS module 282 is configured to determine the processing resources (e.g., processing cores, memory) required to perform the task requested by the host, load the processing cores and the data to be processed in the appropriate memory locations, and execute certain commands or instructions to perform the requested task.

[0047] In one embodiment, the application request includes a request for a high-level task / operation (e.g., a training or inference operation to be applied to some input data based on an AI algorithm / model). In this case, rather than determining and scheduling a kernel or a single atomic command through the runtime library of the host 104 to complete the requested operation, the host 104 directly sends / forwards the request to the DPA 105 for the high-level operation to be performed by the DPA. In one embodiment, the request includes a request for a high-level task / operation (e.g., a training or inference operation to be applied to some input data based on an AI algorithm / model). In this case, rather than determining and scheduling a kernel or a single atomic command through the runtime library of the host 104 to complete the requested operation, the host 104 directly sends / forwards the request to the DPA 105 for the high-level operation to be performed by the DPA. Figure 6 The high-level command format shown is sent to the DPA for high-level operations.

[0048] refer to Figure 6 In one embodiment, the command format may include formatting fields such as the type of command, AI algorithm / model identifier (name / version), input parameters, and / or output parameters (output parameters are optional). In the absence of an operating system module, the host typically communicates with the system via a command such as Figure 5 The low-level command format shown requests sub-commands / granular tasks to complete the requested operation. For example, without an accelerator OS, the host OS must maintain resource usage (e.g., memory usage) of what resources, where, and how much resources have been allocated by the accelerator.

[0049] For the accelerator OS, the accelerator OS is responsible for most or all of the resource management of the accelerator. The accelerator OS is responsible for allocating processing resources (e.g., the number of cores, the amount of storage allocated for the task), including where the core should be loaded into the memory (e.g., a specific memory location or memory area), and the data area where the input and output data should be loaded and stored, as well as the timing of executing instructions when processing data. In this way, the host OS does not need to track what resources are used and how to use the processing resources. All the host OS needs to do is send simple instructions to the accelerator OS, and the accelerator OS will take over and do whatever it needs to do to complete the requested task without interacting with the host OS or only minimally interacting with the host OS.

[0050] refer to Figure 5 In one embodiment, the command format may include formatting fields such as the type of command (e.g., data, control, or configuration), command / kernel name / identifier, input parameters, and / or output parameters (output parameters are optional).

[0051] In operation S402, the operating system module 282 of the DPA 105 receives the request and parses the request to determine the type of operation, input data, and one or more algorithms / models to be executed. Upon receiving the request, in operation S403, the module 282 creates a session (as part of session 411) to provide a context / environment for performing the requested operation. In one embodiment, the session is identified by a session identifier associated with the requesting application. In one embodiment, the session can create a sandbox environment to isolate the session execution from other sessions. In one embodiment, the created session can be a virtual session of a virtual environment to virtualize one or more data processing resources of the DPA.

[0052] In operation S404, module 282 allocates resources 413 to perform the requested operation. Examples of resources include processing / computing units and memory associated with the DPA. Module 282 may determine a portion of available resources as a portion of total available resources. In one embodiment, module 282 may allocate resources based on the determined available resources and the resources required to complete the requested operation.

[0053] In operation S405, the module 282 may schedule multiple kernel / atomic commands to be executed (as part of the data processing task 415) to complete the requested operation. In one embodiment, the module 282 determines and instantiates one or more kernel executors based on the processing task 415. The kernel executor specifies one or more commands to be executed by the DPA 105. In one embodiment, the module 282 determines and schedules one or more commands for each of the kernel executors to be executed by the DPA 105.

[0054] In operation S406, module 282 obtains a completion confirmation indicating that the operation requested by the user application has been completed. In operation S407, module 282 sends the completion confirmation and any data processing results to the requesting user application via host 104.

[0055] Figure 7 700 is a flow chart illustrating a method performed by a DP accelerator according to one embodiment. Process 700 may be performed by processing logic that may include software, hardware, or a combination thereof. For example, process 700 may be performed by a processor such as Figure 3A The DP accelerator 105 and other DP accelerators are executed. Figure 7At block 701, the processing logic receives a request from a data processing accelerator, which is sent by an application to perform a data processing task. At block 702, the processing logic schedules one or more commands for completing the data processing task based on one or more cores through an operating system module of the data processing accelerator. At block 703, upon completion of the one or more scheduled commands, the processing logic generates a completion signal through the operating system module of the data processing accelerator. At block 704, the processing logic sends a result back to the application based on the completion signal to confirm the completion of the one or more scheduled commands for the data processing task.

[0056] In one embodiment, the processing logic further creates an application session context for the data processing task, wherein the application session context is isolated from contexts of application sessions of other data processing tasks of the data processing accelerator. In another embodiment, the isolation isolates resources for the data processing task from resources for other data processing tasks of the data processing accelerator. In one embodiment, the processing logic further allocates resources for the data processing task through an operating system module of the data processing accelerator, wherein the resources include computing resources and storage resources.

[0057] In one embodiment, the request is sent by the application through the host, where the host assigns one or more data processing tasks to the one or more data processing accelerators. In another embodiment, the host is relieved of scheduling and managing one or more commands required to complete the data processing tasks. In one embodiment, the processing logic further receives one or more kernels from the host, the one or more kernels including instructions for executing the one or more commands for completing the data processing tasks.

[0058] Note that some or all of the components shown and described above may be implemented in software, hardware, or a combination thereof. For example, these components may be implemented as software installed and stored in a permanent storage device, which may be loaded and executed in a memory by a processor (not shown) to perform the processes or operations described throughout the present application. Alternatively, these components may be implemented as executable code programmed or embedded into dedicated hardware, such as an integrated circuit (e.g., a dedicated IC or ASIC), a digital signal processor (DSP), or a field programmable gate array (FPGA), which may be accessed via a corresponding driver and / or an operating system from an application. In addition, these components may be implemented as specific hardware logic in a processor or processor core as part of an instruction set accessible to software components via one or more specific instructions.

[0059] Some portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities.

[0060] It should be remembered, however, that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise stated, as will be apparent from the above discussion, it should be understood that throughout the specification, discussions using terms such as those set forth in the appended claims refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.

[0061] The techniques shown in the figure can be implemented using code and data stored and executed on one or more electronic devices. Such electronic devices use computer-readable media (such as non-transitory computer-readable storage media (e.g., magnetic disks, optical disks, random access memory, read-only memory, flash memory devices, phase change memory)) and temporary computer-readable transmission media (e.g., electrical, optical, acoustic or other forms of propagated signals - such as carrier waves, infrared signals, digital signals) to store and communicate (internally and / or over a network with other electronic devices) code and data.

[0062] The processes or methods depicted in the foregoing figures may be performed by processing logic including hardware (e.g., circuits, dedicated logic, etc.), firmware, software (e.g., embodied on a non-transitory computer-readable medium), or a combination of both. Although the processes or methods are described above in terms of some sequential operations, it should be understood that some of the operations described may be performed in a different order. In addition, some operations may be performed in parallel rather than sequentially.

[0063] In the foregoing description, embodiments of the present disclosure have been described with reference to specific exemplary embodiments of the present disclosure. Obviously, various modifications may be made thereto without departing from the broader spirit and scope of the present disclosure as set forth in the appended claims. Therefore, the description and the accompanying drawings should be considered illustrative rather than restrictive.

Claims

1. A method for processing data by a data processing accelerator, the method comprising: A request is received by a data processing accelerator from a host having a first operating system that hosts an application requesting to perform a data processing task; The request uses a request command format, which includes a command type and an algorithm / model identifier, and does not include a kernel command type and a command / kernel identifier, and the data processing accelerator parses the request and determines an operation type and one or more algorithms / models to be executed; Allocating resources for the data processing task by an operating system module of the data processing accelerator, wherein the resources include computing resources and storage resources; The operating system module of the data processing accelerator schedules one or more commands for completing the data processing task based on one or more kernels, wherein the operating system module hosts a second operating system in the data processing accelerator to call the one or more kernels to perform the data processing task; generating, by the operating system module of the data processing accelerator, a completion signal upon completion of the one or more commands; as well as A result is sent back to the application to confirm completion of the one or more commands for the data processing task based on the completion signal.

2. The method of claim 1, further comprising creating an application session context for the data processing task, wherein the application session context is isolated from application session contexts of other data processing tasks of the data processing accelerator. 3 . The method of claim 2 , wherein the isolation is to isolate resources used for the data processing task from resources used for the other data processing tasks of the data processing accelerator. 4 . The method of claim 1 , wherein the request is sent by the application via the host, wherein the host assigns one or more data processing tasks to one or more data processing accelerators.

5. The method of claim 4, wherein the host is relieved of scheduling and managing the one or more commands required to complete the data processing task.

6. A data processing DP accelerator, comprising: A host interface coupled to a host executing a first operating system in which an application is hosted, wherein the host interface receives a request from the host and initiated by the application to request execution of a data processing task; the request uses a request command format, the request command format includes a type of command, an algorithm / model identifier, and does not include a type of kernel command, a command / kernel identifier, and the data processing accelerator parses the request and determines an operation type and one or more algorithms / models to be executed; One or more kernels; as well as An operating system module having a second operating system executed therein, the operating system module being configured to perform operations including Allocating resources for the data processing task, wherein the resources include computing resources and storage resources; scheduling one or more commands for executing the data processing task based on the one or more cores; upon completion of the one or more commands, generating a completion signal, and A result is sent back to the application via the host interface to confirm completion of the one or more commands for the data processing task based on the completion signal.

7. The DP accelerator of claim 6, wherein the operating system module is further used to create an application session context for the data processing task, wherein the application session context is isolated from application session contexts of other data processing tasks of the data processing accelerator.

8. The DP accelerator of claim 7, wherein the isolation isolates resources used for the data processing task from resources used for the other data processing tasks of the data processing accelerator.

9. The DP accelerator of claim 6, wherein the request is sent by the application via the host, wherein the host assigns one or more data processing tasks to one or more data processing accelerators.

10. The DP accelerator of claim 9, wherein the host is relieved of scheduling and managing the one or more commands required to complete a data processing task.

11. A non-transitory machine-readable medium having instructions stored therein, which, when executed by a processor, cause the processor to perform the method for processing data by a data processing accelerator according to any one of claims 1 to 5.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for processing data by a data processing accelerator according to any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • Flash-based accelerator and computing device including same

    CN109460369A

  • Processing system for task scheduling and distribution and acceleration method thereof

    CN110032453A

  • Method and apparatus for tracking computing system, and computer readable medium

    CN110058987A

  • Processor accelerator interface virtualization

    US20140007098A1

  • Data processing systems

    US20150293774A1