GPU-oriented data stream runtime system

Abstract

The invention discloses a GPU-oriented data stream runtime system, and relates to the technical field of task scheduling, the method comprises a device end and a host end, and the device end comprises a first management unit, a computing core module and a first task queue module; the first task queue module stores tasks meeting the task dependency relationship through a task queue; the first management unit obtains a ready task from the first task queue module as a target task, determines computing resources required by the target task and sends the computing resources to the computing core module; the computing core module runs the target task after receiving the target task and computing resources required by the target task; the host side sends task information to the first management unit and receives task result data fed back by the calculation core module; the host end comprises a user interface, and the user interface transmits a memory area specified by a user, computing resources required during task execution and an execution function to the first task queue module when a task is created. The flexibility of task scheduling can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of task scheduling, and in particular to a GPU-oriented data flow runtime system. Background Art

[0002] The core idea of ​​the data flow model is to abstract the entire computing task into a data flow graph. The processing and direction of the data flow are divided into a series of fine-grained computing units, which can be described by a directed acyclic graph. In the data flow graph, nodes represent computing units, and edges represent data dependencies between nodes. When all the pre-dependencies of a computing unit are met, execution can begin. The data flow model can tap into the potential parallelism in the task and express the program more naturally with data as the center.

[0003] The codelet program execution model is a fine-grained, task-driven parallel program execution model based on the data flow model. It supports the use of data flow models for calculations on control flow machines. Its most basic concept is called Codelet, which consists of a code snippet and is the smallest unit that can be scheduled. It is atomic and uninterruptible. Codelet contains a dependency count that describes the dependent tasks that need to be completed before the task is executed. Codelet tasks and dependencies can form a data flow graph called CDG (Codelet Graph). In order to better achieve data locality, the model divides CDG into several sub-CDGs and introduces TP (Thread Procedure) to save the data required by Codelet.

[0004] There are programming frameworks and runtime systems designed based on the codelet program execution model in current work. DARTS is a data flow runtime system developed by the University of Delaware that is completely based on the codelet model, but it only runs on general-purpose platforms. If the backend of DARTS is simply mapped to a GPU (Graphics Processing Unit), high performance cannot be achieved due to low GPU utilization and high synchronization overhead.

[0005] The advantage of the dataflow programming model is that it can describe computational codes that can decompose more tasks and exploit the parallelism therein. However, the existing dataflow runtime system is designed for general platforms. The GPU platform is suitable for executing tasks with high concurrency, and the platform has the following characteristics: rich computing resources, with a large number of SIMT (Single Instruction Multiple Threads) computing cores managed by several stream processors; the computing cores have a fast context switching mechanism to achieve mutual masking of computing and memory access; the GPU memory has a rich hierarchy, consisting of low-speed global memory and high-speed on-chip storage, and can achieve aggregated memory access through hardware. The relevant dataflow runtime system has not made effective designs for multi-level memory and load balancing, and cannot directly utilize the architectural characteristics of the GPU. Therefore, the existing dataflow runtime system cannot be directly ported to the GPU platform, and needs to be designed to achieve higher performance.

[0006] Currently, there are several works that implement data flow-like task-parallel runtime systems on GPUs, which are mainly divided into two modes: non-persistent kernel mode dominated by the CPU (Central Processing Unit) and persistent kernel mode with CPU-GPU collaboration. For task scenarios with high parallelism, coarse task granularity, and the need for distributed node collaboration, the non-persistent kernel mode can often achieve higher performance; while for scenarios with low parallelism or fine task granularity, the persistent kernel mode performs better.

[0007] However, the above two modes have high requirements on the lower limit of task granularity, and it is easy to have the problem that the scheduling overhead of tasks is much higher than the computing time; when a large number of identical tasks are sent at the same time, it will also lead to extremely high queue operation overhead. At the same time, since the threads that execute the scheduling logic may be located in different stream processors, their local queues are located in different on-chip storage (such as GPU shared memory) and cannot directly access each other, it is impossible to directly implement queue-based load balancing strategies.

[0008] When the task granularity is large, task scheduling-based runtime systems often need to perform synchronization between thread blocks. However, this function can only be achieved through cooperative groups, which is often unacceptable in runtime systems that are sensitive to overhead.

[0009] The current task scheduling runtime system is only for static graphs and cannot implement the scheduling of dynamic graphs. There is also no persistent kernel mode framework to support inter-task dependencies.

[0010] In summary, there are three main problems with the current data flow framework: (1) The codelet-based data flow runtime system is only implemented for general platforms, and there is no relevant design and support for GPU platforms; (2) The existing GPU-based task scheduling system cannot fully utilize the memory hierarchy and architectural characteristics of the GPU, the task scheduling and dependency management overhead is high, and dynamic load balancing cannot be achieved. (3) The existing GPU task scheduling system cannot use both persistent kernel and non-persistent kernel modes at the same time, cannot switch between tasks of different granularity, and has poor versatility. Summary of the invention

[0011] The purpose of this application is to provide a GPU-oriented data flow runtime system that can improve the flexibility of task scheduling.

[0012] To achieve the above objectives, this application provides the following solutions:

[0013] In a first aspect, the present application provides a GPU-oriented data flow runtime system, comprising: a device end and a host end, wherein the device end comprises a first management unit, a computing core module, and a first task queue module;

[0014] The first task queue module is used to save tasks that meet the task dependency relationship through a task queue;

[0015] The first management unit is used to poll the tasks in the first task queue module, obtain the ready task from the first task queue module as the target task, determine the computing resources required for the target task, and send the target task and the computing resources required for the target task to the computing core module;

[0016] The computing core module is used to receive the target task and the computing resources required by the target task, then run the target task, and send the task result data obtained after running the target task to the host end;

[0017] The host end is used to send task information to the first management unit and receive the task result data fed back by the computing core module; the task information includes required computing resources;

[0018] The host end includes a user interface, and the user interface is used to transmit the user-specified memory area, the required computing resources and the execution function when the task is executed to the first task queue module when creating a task.

[0019] Optionally, the first management unit includes a scheduler and an evaluator;

[0020] The scheduler is used to poll the tasks in the first task queue module in real time, and obtain the ready tasks from the first task queue module as target tasks;

[0021] The evaluator is used to obtain the computing resources required by the target task, and send the target task and the computing resources required by the target task to the computing core module.

[0022] Optionally, the first task queue module includes an evaluator submodule and a task queue submodule;

[0023] The task queue submodule is used to allocate task queues of different task types to different memory levels;

[0024] The evaluator submodule is used for selecting, when there is a task to be inserted, a task queue with the smallest delay or the least load after the task to be inserted is inserted into the task queue as a target task queue, and inserting the task to be inserted into the target task queue.

[0025] Optionally, the task queue in the first task queue module includes a shared memory queue and a global memory queue;

[0026] The shared memory queue is a lock-free queue based on compare-and-swap, and the global memory queue is a queue based on unified computing device architecture warp primitive operations;

[0027] The evaluator submodule is also used to insert the task to be inserted into the global memory queue when the computing resources required by the task to be inserted are greater than the computing resources contained in a stream processor; when the computing resources required by the task to be inserted are less than or equal to the computing resources contained in a stream processor, insert the task to be inserted into the shared memory queue.

[0028] Optionally, the task types include traditional tasks and dynamic resource expansion tasks;

[0029] The evaluator is also used to obtain the task type of the target task. When the acquired task type is the traditional task, the target task and the computing resources required for the target task are sent to the computing core module; when the acquired task type is the dynamic resource expansion task, additional hardware resources are applied for through a dynamic parallel method, and the target task is sent to the additional hardware resources for execution in the form of a kernel function startup.

[0030] Optionally, the computing core module includes multiple computing threads, and each computing thread is physically mapped to a thread bundle or a stream processor of the GPU.

[0031] Optionally, the host end is further used to release the downstream task of the target task after the computing core module runs the target task;

[0032] The first task queue module is used to save the downstream task when the released downstream task meets the corresponding task dependency relationship.

[0033] Optionally, when the first management unit receives the task information, it uses a multi-level task dispatching strategy to dispatch the task information to a corresponding computing thread.

[0034] Optionally, a multi-level task dispatching strategy is used to dispatch the task information to corresponding computing threads, specifically including:

[0035] When the number of computing resources is less than the number of threads in a warp, the first management unit is used to broadcast the task information to all computing threads in the warp by register broadcast and directly start the calculation;

[0036] When the number of computing resources is greater than the number of threads in a thread bundle, the first management unit is used to maintain a first target memory in the shared memory, and each computing core module polls the first target memory until a task corresponding to the task information is obtained;

[0037] When the computing resources are multiple thread blocks, the first management unit is used to maintain a second target memory in the global memory, and dispatch the task information to each stream processor through the second target memory; after receiving the task information, the leader thread of each stream processor will dispatch the task information to the computing thread in the same stream processor through the second target memory.

[0038] Optionally, in terms of sending task information to the first management unit and receiving the task result data fed back by the computing core module, the host end is specifically used for:

[0039] Sending task information to the first management unit through an asynchronous data loading flow;

[0040] The task result data fed back by the computing core module is received through an asynchronous data write-back flow, and the dependency relationship of the downstream tasks of the target task corresponding to the task result data is released according to a preset task dependency flow graph.

[0041] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0042] The present application provides a GPU-oriented data flow runtime system, wherein the host side includes a user interface, and the user interface is used to transmit a user-specified memory area, required computing resources for task execution, and execution functions to the first task queue module when creating a task, thereby achieving flexible configuration of the memory area, required computing resources for task execution, and execution functions, reducing the impact of restrictions on fixed task scheduling modes, and thus improving the flexibility and versatility of task scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A schematic diagram of the structure of a GPU-oriented data stream runtime system provided in one embodiment of the present application;

[0045] Figure 2 A schematic diagram of the structure of a first management unit provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the structure of a first task queue module provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the structure of an evaluator submodule provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of the scheduling relationship between a traditional task and a dynamic resource expansion task (DRET) provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of the implementation process of a GPU-oriented data stream runtime system provided in one embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0052] The technical ideas of this application are as follows: 1. This application is aimed at the GPU platform and proposes a CPU-GPU collaborative scheduling runtime system design to solve the problems of low flexibility and poor versatility of existing task systems and realize flexible scheduling of multi-granularity tasks. 2. This application also proposes a load balancing optimization method based on hardware scheduling to solve the problem that the existing runtime system cannot achieve load balancing and make full use of computing resources. 3. This application proposes a multi-level task dispatching strategy for management nodes to solve the high overhead problem of management node task scheduling, thereby advancing the computing process faster and achieving higher performance.

[0053] This application provides a GPU-oriented data flow runtime system, such as Figure 1 As shown, the GPU-oriented data flow runtime system includes a device side and a host side, and the device side includes a first management unit, a computing core module and a first task queue module.

[0054] The first task queue module is used to save tasks that satisfy task dependencies through a task queue.

[0055] The first management unit is used to poll the tasks in the first task queue module, obtain the ready tasks from the first task queue module as target tasks, determine the computing resources required for the target tasks, and send the target tasks and the computing resources required for the target tasks to the computing core module.

[0056] The computing core module is used to receive the target task and the computing resources required by the target task, then run the target task, and send the task result data obtained after running the target task to the host end.

[0057] The host end is used to send task information to the first management unit and receive the task result data fed back by the computing core module; the task information includes required computing resources.

[0058] The host end includes a user interface, and the user interface is used to transmit the user-specified memory area, the required computing resources and the execution function when the task is executed to the first task queue module when creating a task.

[0059] In this application, the first management unit is mainly responsible for task scheduling, creation of subsequent tasks, notification and resource recovery, etc.; the computing core module is responsible for calculating the assigned tasks and releasing the subsequent dependencies after the task is completed. The first task queue module is responsible for saving tasks that meet the dependencies and can be executed. It has multiple levels of memory and different queue types. The host side includes a second management unit, which is used to asynchronously transmit data and schedule the execution of host-side tasks.

[0060] Due to the structural characteristics of the GPU, the computing core module will achieve higher computing performance when facing branchless code and is not suitable for running complex task scheduling instructions. Therefore, this application schedules ready tasks to the computing core module, such as Figure 1 As shown in ① and ②.

[0061] In an exemplary embodiment, Figure 2 As shown, the first management unit includes a scheduler and an evaluator.

[0062] The scheduler is used for polling the tasks in the first task queue module in real time, obtaining the ready tasks from the first task queue module as target tasks, and sending the target tasks to the evaluator.

[0063] The evaluator is used to obtain the computing resources required for the target task, and send the target task and the computing resources required for the target task to the computing core module. Specifically, the evaluator is used to parse the task type and the required computing resources of the target task, so as to adopt different task startup methods and dynamic resource allocation, and send them to the downstream computing core module to start calculation.

[0064] The computing core module is used to run the target task sent from the first management unit.

[0065] In an exemplary embodiment, the computing core module logically includes a large number of computing threads, and each computing thread is physically mapped to a thread bundle or a stream processor of the GPU. The thread bundle or stream processor can be adjusted by the user or flexibly selected by the runtime system. When the computing core calculates and runs the target task, the dependencies of the downstream tasks of the target task will be released. When the dependencies of the downstream tasks are all satisfied after the release, the downstream tasks will be inserted into the first task queue module, such as Figure 1 As shown in ④.

[0066] When the task dependencies are met and the computing resources are insufficient, the task requires a container to save the task. Therefore, the present application designs a first task queue module (ready first task queue module) to complete the task saving function. Figure 3 shown.

[0067] In an exemplary embodiment, the first task queue module includes an evaluator submodule and a task queue submodule.

[0068] The task queue submodule is used to allocate task queues of different task types to different memory levels to reduce the overhead of queue operations.

[0069] The evaluator submodule is used to select the task queue with the smallest delay or the least load after the task to be inserted into the task queue as the target task queue when there is a task to be inserted, and insert the task to be inserted into the target task queue, such as Figure 4 shown.

[0070] In an exemplary embodiment, the task queue adopts a multi-memory level design, and the task queue in the first task queue module includes a shared memory queue and a global memory queue.

[0071] In the shared memory queue hierarchy, since there are fewer concurrent queue operations and they are limited to the same stream processor, the shared memory queue described in the present application is a lock-free queue based on Compare And Swap (CAS).

[0072] The global memory queue is a queue based on the Compute Unified Device Architecture (CUDA) thread warp primitive operation. All threads in the thread warp can operate the queue independently without deadlock, and the mutual exclusion overhead of queue access can also be greatly reduced.

[0073] Due to the SIMT limitation of the thread warp, a deadlock problem may occur when the threads in the thread warp perform CAS operations at the same time. Therefore, the threads in the same thread warp will assign the tasks to be inserted to the same thread for processing.

[0074] The evaluator submodule is used to evaluate the task queue that the task to be inserted should be inserted into. The evaluator submodule evaluates the task type and determines how the task is executed.

[0075] The evaluator submodule is also used to evaluate the required resource type. When the computing resources required by the task to be inserted are greater than the computing resources contained in a stream processor, the task to be inserted is inserted into the global memory queue; when the computing resources required by the task to be inserted are less than or equal to the computing resources contained in a stream processor, the task to be inserted is inserted into the shared memory queue to achieve lower memory access latency and reduce mutual exclusion operation overhead.

[0076] In an exemplary embodiment, in terms of sending task information to the first management unit and receiving the task result data fed back by the computing core module, the host end is specifically used to: send task information to the first management unit through an asynchronous data loading flow; receive the task result data fed back by the computing core module through an asynchronous data write back flow, and release the dependency of the downstream tasks of the target task corresponding to the task result data according to a preset task dependency flow graph.

[0077] The host-side task needs to go through four stages: loading data stage, execution stage, unloading data stage and callback stage. The loading data stage is used to complete the data loading required for the task, that is, transferring data from the host side to the device side, which is completed by the asynchronous data loading flow; the execution stage is performed by the device side, and the required computing resources are specified by the user and automatically allocated by the hardware; the unloading data stage is used to write the calculated data from the device side back to the host side memory, which is completed by the asynchronous data write-back flow; the callback stage is used to release the dependencies of downstream tasks according to the task dependency flow graph after the task is executed.

[0078] The second management unit includes a transmission module and a second task queue module. In the second task queue module, it is relatively simple for the host to implement a mutually exclusive queue. At present, there is a lot of work to implement high-performance mutually exclusive queues in different scenarios. This application adopts the existing work to implement it. The transmission module mainly includes two functions: 1. Get executable tasks from the mutually exclusive queue; 2. Complete the data transmission and startup of the task. Figure 1 As shown in ⑤ and ⑥.

[0079] The specific steps of obtaining an executable task from a mutex queue are as follows: There is usually only one or a limited number of mutex queues on the host side, and the locations of these mutex queues are visible to all transmission modules or scheduling modules. Therefore, obtaining a task only requires dequeuing the mutex queue (because it is a mutex queue, the dequeuing operation is thread-safe). If a null pointer is obtained, it means that the queue is empty; otherwise, an executable task is obtained.

[0080] The user interface includes interfaces for each stage according to the user's computing needs. The specific user interface allows the user to explicitly specify the memory area to be transferred, the resources required for task execution, and the execution function and additional processing required for callbacks when creating a task, which has high flexibility.

[0081] After adding the host-side computing module, users can directly create host-side tasks for tasks with high parallelism, coarse granularity and simpler dependencies. For tasks with low parallelism and more complex task dependencies, users can create device-side tasks to start persistent kernel mode to reduce kernel function startup overhead and dynamic load balancing, thereby achieving higher performance.

[0082] The host-side computing module is a runtime system on the host side, which is an extended runtime system of the device-side runtime system of this application. Host-side tasks are generally scheduled in the host-side queue, and require loading, execution, and unloading data processes. Therefore, the entire scheduling process requires the host-side computing module to be responsible.

[0083] The user can directly create a host-side task, which specifically includes: the user interface inherits the host-side task base class, and implements the virtual functions required by the host-side task to achieve the creation of the host-side task.

[0084] In the existing work types, tasks are defined as the smallest unit of scheduling. In persistent kernel mode, tasks are acquired by the management unit of the runtime system and sent to the computing unit allocated when the runtime system is started. However, there are two problems with the traditional task scheduling method: 1. When an application requires a large number of resources, the application will be divided into a large number of tasks, generating a large amount of queue operation overhead, which seriously affects performance. 2. It is difficult to perform efficient load balancing of tasks. Since scheduling threads cannot access each other's on-chip high-speed storage, it is very easy for some management threads to occupy a large number of tasks while other threads are idle.

[0085] To address the above issues, this application proposes DRET and expands the runtime system in combination with a dynamic parallel library. When executing a DRET task, it does not use the hardware resources pre-applied by the runtime system, but dynamically applies for additional hardware resources through dynamic parallelism before execution to further improve the utilization of hardware resources and load balancing effects.

[0086] DRET inherits from the traditional task type and sets additional parameters, including the number of stream processors required to execute the task, the number of threads required for each stream processor, and the identification bit. When creating a DRET, the user explicitly specifies the number of computing cores required for the task and the computing resources of each computing core, and within the execution logic, the user can obtain the identification id of the grid and thread block where the task is located at runtime.

[0087] In an exemplary embodiment, the task types include traditional tasks and dynamic resource expansion tasks. During the execution process, the evaluator of the first management unit of the runtime system detects the task type.

[0088] The evaluator is also used to obtain the task type of the target task. When the obtained task type is the traditional task, the target task and the computing resources required by the target task are sent to the computing core module, such as Figure 5 As shown in ④; when the task type obtained is the dynamic resource expansion task, additional hardware resources are applied for through a dynamic parallel method, and the target task is sent to the additional hardware resources for execution in the form of a kernel function startup, such as Figure 5 As shown in ③. Due to the GPU's fast context switching mechanism and computational memory access masking mechanism, the additional hardware resources applied can improve the hardware resource utilization rate and implement a load balancing strategy based on hardware scheduling, thereby further improving program performance.

[0089] In an exemplary embodiment, the host end is also used to release the downstream tasks of the target task after the computing core module runs the target task.

[0090] The first task queue module is used to save the downstream task when the released downstream task meets the corresponding task dependency relationship.

[0091] In an exemplary embodiment, the first management unit is configured to, upon receiving the task information, dispatch the task information to a corresponding computing thread using a multi-level task dispatching strategy.

[0092] GPU has a large number of computing threads, which makes it possible to use an efficient context switching mechanism to hide the latency of memory access. The management node is responsible for obtaining the information of tasks to be executed through the task queue and distributing the information to a large number of computing threads. Therefore, the management thread allocation pressure is high and it is easy to generate extremely high scheduling overhead. In order to solve the above problems, this application designs an efficient multi-level task dispatching strategy for the first management unit to efficiently dispatch the information to all corresponding computing threads after obtaining the task information.

[0093] The task information also includes the parameters required for task execution, the task context information, and the task execution function.

[0094] In implementation here, the first management unit obtains the task pointer and obtains the above information through indirect access.

[0095] A multi-level task dispatching strategy is adopted to dispatch the task information to the corresponding computing thread, specifically including: when the number of computing resources is less than the number of threads in a thread bundle, the first management unit is used to broadcast the task information to all computing threads in the thread bundle through register broadcast and directly start the calculation, that is, execute the execution function of this task.

[0096] When the number of computing resources is greater than the number of threads in a thread bundle, the first management unit is used to maintain a first target memory in the shared memory, and each computing core module polls the first target memory until a task corresponding to the task information is obtained.

[0097] When the computing resource is a plurality of thread blocks, the first management unit is used to maintain a second target memory in the global memory, and dispatch the task information to each stream processor through the second target memory; after receiving the task information, the leader thread of each stream processor dispatches the task information to the computing thread in the same stream processor through the second target memory. In order to prevent failed polling from continuously occupying hardware resources, the present application adds a clock() function to explicitly switch threads after failure to give up computing resources.

[0098] The amount of computing resources is defined by the user through the user interface.

[0099] The process of the runtime system of this application is that the user defines several tasks (including the execution method of the task, the resources required for the task, and the dependencies between the tasks) through the user interface to generate a dependency graph, and the runtime system of this application can schedule the execution of the task according to this dependency graph. The purpose of this application is to reduce the scheduling overhead as much as possible and provide more abundant functions for users to use, so that programming at the user level is simpler.

[0100] Through the multi-level task dispatching strategy, task information will be dispatched through multiple memory levels to reduce runtime system scheduling overhead and efficiently promote task execution.

[0101] As an efficient and scalable data flow runtime system, the unique design and implementation method of this application shows significant advantages and progress in data flow programming implementation for GPU architecture. Compared with the existing technology, the advantages and positive effects of this application are mainly reflected in the following aspects:

[0102] (1) Efficient support for GPU platforms: This application designs an efficient data flow programming framework and runtime system for the GPU architecture, which effectively solves the problems of low hardware utilization and difficult programming when the GPU platform has complex dependencies.

[0103] (2) Effective cross-platform adaptability: The programming interface provided by this application has the same or similar interfaces as other platforms, including general-purpose platforms and multi-core processor platforms. Users do not need to consider the backend parallel programming model, but only need to implement the computing task logic and build a data flow graph. The runtime system will automatically schedule the data flow graph and adopt different scheduling implementations for different backends, so that it can quickly adapt to and show efficient performance on a variety of hardware platforms.

[0104] (3) Efficient load balancing strategy: This application implements an efficient load balancing solution through dynamic parallelism. Efficient resource utilization and load balancing are achieved through dynamic scheduling at the software level and dynamic parallel resource reallocation at the hardware level, thereby efficiently promoting the execution of data flow tasks to achieve higher performance.

[0105] (4) An efficient scheduling scheme that integrates persistent kernels and non-persistent kernels: The integrated scheduling scheme designed in this application can flexibly switch between tasks of different granularities and scales, thereby minimizing the task scheduling overhead of the runtime system and allowing users to describe tasks in a more fine-grained manner to fully utilize the performance improvement brought by the data flow concept.

[0106] In general, this application provides a programming framework and corresponding runtime system for GPU architecture in terms of data flow programming. Through a unified programming interface, efficient scheduling strategies and functional support, it achieves efficient adaptation of the runtime system and the back-end architecture, significantly reducing the user's programming difficulty.

[0107] In an exemplary embodiment, the implementation process of a GPU-oriented data flow runtime system of the present application is as follows: Figure 6 shown.

[0108] Step 1: Front-end user task construction

[0109] When using the framework on the GPU platform, the user needs to complete the construction of the front-end task. For the application to be streamed, the user uses the unified programming interface provided by this application to abstract the application into several context structures, static computing tasks, and dynamic computing task types and specify the data dependencies between tasks. After completing the task construction, the dependencies between tasks constitute a dependency graph, and the user sends the dependency graph to the runtime system through the specified user interface.

[0110] Step 2: Runtime system scheduling operation

[0111] Based on the tasks provided by the user, the runtime system first abstracts the tasks into a fine-grained data flow graph, and waits for the scheduling unit to execute the scheduled tasks. On the device side, the runtime system runs multiple first management units, each of which holds a private queue in the task queue module, and tries to obtain tasks in the global queue and the private queue. When the first management unit obtains a user-defined static task, the task will be evaluated and scheduled to run on the computing core module; when a dynamic resource expansion task is detected, additional resources will be applied for and task execution will be started through the method sent by the kernel function, so as to achieve load balancing of GPU computing resources. After the computing core module is finished running, the dependencies of downstream tasks will be released, and tasks with all dependencies met will be inserted into the task queue until all tasks are executed.

[0112] On the host side, after the runtime system completes task abstraction, the task scheduling module also starts to execute tasks. The second management unit will take the tasks defined by the user on the host side from the task queue and execute the three stages of data upload, task execution, and data writeback. After execution, the execution module of the second management unit will call the callback function of the task and release the dependencies of subsequent tasks. After all tasks are executed, the second management unit releases the occupancy and executes other logic.

[0113] Step 3: Runtime system underlying hardware mapping

[0114] This application divides the runtime system into two parts: device-side core resources and host-side core resources. On the device side, each stream processor will run several computing thread blocks at the same time. Each computing thread block corresponds to a computing core module, and its size is determined by the computing core module. Thread No. 0 of each computing thread block is mapped to the first management unit, which is responsible for acquiring tasks and broadcasting tasks to all threads in the thread block. The task queue module adopts multi-level queue storage. The global queue is located in the global memory for access by all first management units; the first management unit private queue is located in the shared memory of the stream processor where each first management unit is located.

[0115] On the host side, the present application starts several second management units according to the number of CPU cores, and each second management unit is executed by a thread. The task queue is allocated in the memory, and all second management units are visible.

[0116] Through the above steps, users can use the interface of this application to describe the task execution logic and the dependencies between tasks, easily stream data applications and obtain good performance.

[0117] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A GPU-oriented data flow runtime system, characterized in that: The GPU-oriented data flow runtime system includes a device end and a host end, and the device end includes a first management unit, a computing core module and a first task queue module; The first task queue module is used to save tasks that meet the task dependency relationship through a task queue; The first management unit is used to poll the tasks in the first task queue module, obtain the ready task from the first task queue module as the target task, determine the computing resources required for the target task, and send the target task and the computing resources required for the target task to the computing core module; The computing core module is used to receive the target task and the computing resources required by the target task, then run the target task, and send the task result data obtained after running the target task to the host end; The host end is used to send task information to the first management unit and receive the task result data fed back by the computing core module; the task information includes required computing resources; The host end includes a user interface, and the user interface is used to transmit the user-specified memory area, the required computing resources and the execution function when the task is executed to the first task queue module when creating a task.

2. The GPU-oriented data flow runtime system according to claim 1, characterized in that: The first management unit includes a scheduler and an evaluator; The scheduler is used to poll the tasks in the first task queue module in real time, and obtain the ready tasks from the first task queue module as target tasks; The evaluator is used to obtain the computing resources required by the target task, and send the target task and the computing resources required by the target task to the computing core module.

3. The GPU-oriented data flow runtime system according to claim 1, characterized in that: The first task queue module includes an evaluator submodule and a task queue submodule; The task queue submodule is used to allocate task queues of different task types to different memory levels; The evaluator submodule is used for selecting, when there is a task to be inserted, a task queue with the smallest delay or the least load after the task to be inserted is inserted into the task queue as a target task queue, and inserting the task to be inserted into the target task queue.

4. The GPU-oriented data flow runtime system according to claim 3, characterized in that: The task queue in the first task queue module includes a shared memory queue and a global memory queue; The shared memory queue is a lock-free queue based on compare-and-swap, and the global memory queue is a queue based on unified computing device architecture warp primitive operations; The evaluator submodule is also used to insert the task to be inserted into the global memory queue when the computing resources required by the task to be inserted are greater than the computing resources contained in a stream processor; when the computing resources required by the task to be inserted are less than or equal to the computing resources contained in a stream processor, insert the task to be inserted into the shared memory queue.

5. The GPU-oriented data flow runtime system according to claim 2, characterized in that: The task types include traditional tasks and dynamic resource expansion tasks; The evaluator is also used to obtain the task type of the target task. When the acquired task type is the traditional task, the target task and the computing resources required for the target task are sent to the computing core module; when the acquired task type is the dynamic resource expansion task, additional hardware resources are applied for through a dynamic parallel method, and the target task is sent to the additional hardware resources for execution in the form of a kernel function startup.

6. The GPU-oriented data flow runtime system according to claim 1, characterized in that: The computing core module includes multiple computing threads, and each computing thread is physically mapped to a thread bundle or a stream processor of the GPU.

7. The GPU-oriented data flow runtime system according to claim 1, characterized in that: The host end is also used to release the downstream tasks of the target task after the computing core module runs the target task; The first task queue module is used to save the downstream task when the released downstream task meets the corresponding task dependency relationship.

8. The GPU-oriented data flow runtime system according to claim 1, characterized in that: The first management unit is used to dispatch the task information to the corresponding computing thread by adopting a multi-level task dispatching strategy when receiving the task information.

9. The GPU-oriented data flow runtime system according to claim 8, characterized in that: A multi-level task dispatching strategy is used to dispatch the task information to the corresponding computing thread, specifically including: When the number of computing resources is less than the number of threads in a warp, the first management unit is used to broadcast the task information to all computing threads in the warp by register broadcast and directly start the calculation; When the number of computing resources is greater than the number of threads in a thread bundle, the first management unit is used to maintain a first target memory in the shared memory, and each computing core module polls the first target memory until a task corresponding to the task information is obtained; When the computing resources are multiple thread blocks, the first management unit is used to maintain a second target memory in the global memory, and dispatch the task information to each stream processor through the second target memory; after receiving the task information, the leader thread of each stream processor will dispatch the task information to the computing thread in the same stream processor through the second target memory.

10. The GPU-oriented data flow runtime system according to claim 1, characterized in that: In terms of sending task information to the first management unit and receiving the task result data fed back by the computing core module, the host end is specifically used for: Sending task information to the first management unit through an asynchronous data loading flow; The task result data fed back by the computing core module is received through an asynchronous data write-back flow, and the dependency relationship of the downstream tasks of the target task corresponding to the task result data is released according to a preset task dependency flow graph.

Images