A GPU resource scheduling method and device

Abstract

This invention discloses a GPU resource scheduling method and apparatus, relating to the field of heterogeneous computing technology. The method includes: acquiring multiple first feature parameters and multiple second feature parameters of multiple GPUs; inputting the multiple first feature parameters into a first time prediction model to obtain load values ​​on the multiple GPUs; determining the first predicted runtime of a process to be processed in descending order across the multiple GPUs based on the load values; inputting the multiple first feature parameters and multiple second feature parameters into a second time prediction model to obtain the second predicted runtime of the process to be processed in descending order across the multiple GPUs; and selecting the GPU with the smallest second predicted runtime among the GPUs with the lowest first predicted runtime to run the process to be processed. This method can avoid the load imbalance phenomenon where some GPUs are idle while others are fully loaded, reducing program runtime and maximizing the utilization of limited computing resources in single-node scenarios with limited computing resources.

Description

Technical Field

[0001] This invention relates to the field of heterogeneous computing technology, and more specifically to a GPU resource scheduling method and apparatus. Background Technology

[0002] In recent years, with the booming development of industries such as artificial intelligence, virtual reality, and financial simulation, the demand for computing power has been increasing. CPUs are finding it increasingly difficult to support large-scale parallel computing, and the contradiction between CPU hardware resources and computing power requirements has gradually become apparent. The widespread adoption of GPUs has alleviated this contradiction. Because GPUs are suitable for scenarios involving simple, repetitive calculations on large-scale, independent data, they are widely used in computationally intensive applications, and GPU parallel computing technology has become a mainstream research direction.

[0003] In the embedded systems field, next-generation intelligent drones, unmanned vehicles, and other unmanned systems are widely used. Therefore, GPU computing platforms for embedded systems play a crucial role in common areas such as unmanned reconnaissance, image detection, and target recognition. Embedded system platforms have limited computing resources and weaker performance than conventional mainframes, making performance evaluation and resource management of embedded GPU platforms particularly important.

[0004] Currently, the main application area for GPU resource management is cloud computing services, and the research focus is often on distributed GPU clusters. The core of resource management in this field lies in load balancing of computing resources among different nodes. Generally, since NVIDIA itself does not provide an adaptive scheduling interface, users often need to manually select appropriate GPU resources when submitting tasks. Reasonable resource selection can avoid the load imbalance phenomenon where some GPUs are idle while others are fully loaded. In addition to the device resource selection problem in multi-GPU environments, the rational utilization of GPU resources is also a core issue of resource management. Therefore, in the field of embedded GPUs, how to reduce program runtime and maximize the utilization of limited computing resources in single-node scenarios, thereby designing a reasonable task scheduling strategy, is an urgent technical problem to be solved. Summary of the Invention

[0005] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, a first aspect of this invention proposes a GPU resource scheduling method, applied in the scheduler of a GPU resource management system, comprising:

[0006] Obtain multiple first feature parameters and multiple second feature parameters from multiple GPUs;

[0007] Multiple first feature parameters are input into the first-time prediction model to obtain the load values ​​on multiple GPUs;

[0008] Based on multiple load values, determine the first predicted runtime of the process to be processed in descending order across multiple GPUs;

[0009] By inputting multiple first feature parameters and multiple second feature parameters into the second time prediction model, the second predicted runtime of the process to be processed is obtained in descending order on multiple GPUs;

[0010] The GPU with the shortest second predicted runtime among the GPUs ranked last in the first predicted runtime is selected to run the pending process.

[0011] Optionally, multiple first feature parameters are input into the first-time prediction model to obtain load values ​​on multiple GPUs, including:

[0012] Input multiple state feature parameters from each GPU into the corresponding nodes in the first layer of the first-time prediction model;

[0013] The multiple state feature parameters input from each node are fuzzified to obtain the fuzzy linguistic variable value of each state feature parameter. The fuzzy linguistic variable value includes a fuzzy set, and the fuzzy set contains multiple fuzzy subsets.

[0014] Multiple fuzzy subsets of each state feature parameter are input into the nodes of the second layer, and the membership value of each fuzzy subset is output by performing a membership function operation. The membership function includes a Gaussian membership function.

[0015] Input multiple membership values ​​into multiple nodes in the third layer representing fuzzy rules, and output multiple matching values;

[0016] A weighted average method is used to defuzzify multiple matching values ​​to obtain the load value of each GPU.

[0017] Optionally, it also includes training a first-time prediction model, which includes:

[0018] The membership values ​​output by the nodes of the second layer of the first-time prediction model are input into multiple nodes of the third layer, and the matching degree values ​​are output and the maximum value of the matching degree function is calculated.

[0019] During the training of the parameters of the first-time prediction model, as the number of training samples increases, the weight values ​​of each node of the first-time prediction model are continuously adjusted based on feedback.

[0020] Set a first threshold. If the maximum value of the matching degree function is less than the first threshold, then decrease the weight value of each node; if the maximum value of the matching degree function is greater than the first threshold, then increase the weight value of each node.

[0021] When training the structure of the first-time prediction model, a second threshold is set. If the maximum value of the matching degree function is less than the second threshold, a fuzzy rule is added; otherwise, no fuzzy rule is added.

[0022] A third threshold is set. If the absolute value of the difference between the GPU load value output by the prediction model in the first time and the GPU load value is less than the third threshold, the fuzzy rule added in this training is deleted; otherwise, the fuzzy rule added in this training is not deleted.

[0023] Optionally, based on multiple load values, the first predicted runtime of the process to be processed, arranged in descending order across multiple GPUs, is determined, including:

[0024] Get the load values ​​of multiple GPUs;

[0025] The first predicted runtime is obtained by multiplying the load value of each GPU by the idle runtime and sorting them in descending order. This yields the first predicted runtime in descending order for each GPU.

[0026] Optionally, multiple first feature parameters and multiple second feature parameters are input into the second time prediction model to obtain the second predicted runtime of the process to be processed in descending order on multiple GPUs, including:

[0027] The nvml library is used to obtain the values ​​of various state characteristic parameters and kernel function characteristic parameters of the GPU under different loads, which are used as test values. The actual execution time of the corresponding tasks under different loads is obtained as a test set.

[0028] The test values ​​are input into the second-time prediction model for training to obtain the predicted values.

[0029] The predicted values ​​are compared with the test set using a loss function. The loss function is continuously minimized during the model iteration process until the error between the predicted values ​​and the test set is within the range of MSE and R2 set by the system, thus obtaining an accurate second-time prediction model.

[0030] The values ​​of each state feature parameter and kernel function feature parameter of each GPU are input into the trained second time prediction model to obtain the second prediction runtime.

[0031] Optionally, the multiple first characteristic parameters include multiple state characteristic parameters of the GPU. The multiple state characteristic parameters of the GPU include at least one of GPU utilization, memory utilization, performance status, power consumption, temperature, core frequency, memory frequency, and running process. The values ​​of each state characteristic parameter of the GPU are obtained through the scheduler's NVML library.

[0032] Optionally, the multiple second characteristic parameters include multiple kernel function characteristic parameters of the GPU. The multiple kernel function characteristic parameters of the GPU include at least one of the following: number of thread bundles, shared memory transactions, texture / L1 joint cache transactions, L2 cache transactions, and global memory transactions. The values ​​of the multiple kernel function characteristic parameters of the GPU are obtained through a fixed parameter set of the scheduler.

[0033] The present invention also provides a GPU resource scheduling device, comprising:

[0034] The feature parameter acquisition module is used to acquire multiple first feature parameters and multiple second feature parameters from multiple GPUs;

[0035] The load value acquisition module is used to input multiple first feature parameters into the first-time prediction model to obtain the load values ​​on multiple GPUs.

[0036] The first-time acquisition module is used to determine the first predicted runtime of the process to be processed in descending order on multiple GPUs based on multiple load values;

[0037] The second time acquisition module is used to input multiple first feature parameters and multiple second feature parameters into the second time prediction model to obtain the second predicted running time of the process to be processed in descending order on multiple GPUs;

[0038] The GPU determination module is used to obtain the GPU with the smallest second predicted runtime among multiple GPUs ranked lower in the first predicted runtime order and to run the pending process.

[0039] The present invention also provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the GPU resource scheduling method as described in the first aspect.

[0040] The present invention also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the GPU resource scheduling method as described in the first aspect.

[0041] This invention provides a GPU resource scheduling method and apparatus, which have the following advantages compared with the prior art:

[0042] This invention proposes a GPU task execution time prediction method based on runtime load assessment and provides a specific implementation scheme. This method can allocate appropriate GPUs to tasks during task scheduling, avoiding load imbalances where some GPUs are idle while others are fully loaded. Furthermore, in cases of resource contention, it selects GPUs with lower resource contention for task scheduling, thereby reducing the time consumption caused by GPU resource preemption. This approach reduces program runtime and maximizes the utilization of limited computing resources in single-node scenarios with limited computing resources. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0044] Figure 1 A flowchart of a GPU resource scheduling method provided in an embodiment of the present invention;

[0045] Figure 2 This invention provides a system architecture for a GPU resource scheduling method.

[0046] Figure 3 This invention provides a front-end to back-end data transmission format for a GPU resource scheduling method according to an embodiment of the invention.

[0047] Figure 4 This invention provides a backend-to-frontend data transmission format for a GPU resource scheduling method according to an embodiment of the invention.

[0048] Figure 5 The working principle of the controller of a GPU resource scheduling method provided in this embodiment of the invention;

[0049] Figure 6 The working principle of the scheduler in a GPU resource scheduling method provided by an embodiment of the present invention;

[0050] Figure 7 The SOFNN network structure of a GPU resource scheduling method provided in this embodiment of the invention;

[0051] Figure 8 This invention provides an embodiment of a GPU resource scheduling method that includes a gradient boosting tree iterative process.

[0052] Figure 9 This is a structural block diagram of a GPU resource scheduling device provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but may include more or fewer operational steps based on conventional or non-inventive labor. In actual system or server product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0055] Figure 1 This is a flowchart of a GPU resource scheduling method provided in an embodiment of the present invention.

[0056] like Figure 1 As shown, this invention is applied to a scheduler, and the performance evaluation method includes:

[0057] Step 101: Obtain multiple first feature parameters and multiple second feature parameters from multiple GPUs;

[0058] Specifically, multiple first-characteristic parameters and multiple second-characteristic parameters are obtained through the NVML library and nvprof, respectively. NVML is a dynamic library provided by NVIDIA that can monitor parameters such as GPU temperature and usage. It's used to monitor and manage various states of NVIDIA GPUs (graphics processing units) and is included in the GPUDeployment Kit. The latest version of this tool is released as part of the CUDA Toolkit, and the latest nvml.lib and nvml.h are both in CUDA (Compute Unified Device Architecture). nvprof is a command-line profiler available for Linux, Windows, and OS X. Running an application using `nvprof . / myApp` quickly displays a summary of all kernels and memory copies used by the application. The summary combines all calls to the same kernel, showing the total time for each kernel as a percentage of the total application time. In addition to summary mode, nvprof also supports GPU tracing and API tracing modes. It allows you to see a complete list of all kernel startups and memory copies, and in API tracing mode, you can also see a complete list of all CUDA API calls.

[0059] This invention uses remote API (Application Program Interface) forwarding technology to implement GPU virtualization.

[0060] The front-end is configured with CUDA interceptors and communicator components to intercept related processes.

[0061] The interceptor component is a runtime environment compatible with the CUDA computing platform, primarily simulating interfaces such as the CUDA runtime and CUDA drivers. When a CUDA application uses the GPU for computation, it automatically invokes the CUDA interceptor component. Because the CUDA interceptor component provides a CUDA interface compatible with NVIDIA GPUs, programs can run in a virtual GPU environment without modification at runtime.

[0062] The front-end communicator component is used by the front-end to intercept requests and make RPC (Remote Procedure Call) requests. This component serializes the function name and parameters and forwards them to the back-end through a TCP connection, and receives the results after the back-end has completed execution.

[0063] The backend sets up communicator, controller, and scheduler components.

[0064] The backend communicator and the frontend communicator interact to form a typical client / server (C / S) architecture communication system, where the frontend communicator is the client and the backend communicator is the server. After receiving a request from the frontend, the backend communicator deserializes the data to obtain the function name and parameters.

[0065] The controller component controls the overall backend architecture. Its main functions include function mapping for resolving tasks, managing process waiting queues for queuing tasks, and creating task processes.

[0066] The scheduler is a core functional component of the system, running the GPU resource scheduling method proposed in this invention.

[0067] Figure 2 This invention provides a system architecture for a GPU resource scheduling method.

[0068] like Figure 2 As shown, the system architecture includes a CUDA interceptor and communicator component set at the front end, and a communicator, controller and scheduler component set at the back end, as well as a controller component and a scheduler.

[0069] Figure 3 This invention provides a front-end to back-end data transmission format for a GPU resource scheduling method according to an embodiment of the present invention.

[0070] like Figure 3As shown, the data transmission format from the front end to the back end includes sequentially connecting the third interface identifier area, the buffer size area, and the buffer area.

[0071] Figure 4 This invention provides a backend-to-frontend data transmission format for a GPU resource scheduling method according to an embodiment of the present invention.

[0072] like Figure 4 As shown, the data transmission format from the backend to the frontend includes the sequentially connected result status code area, the buffer size area, and the buffer area.

[0073] Figure 5 The working principle of the controller of a GPU resource scheduling method provided in this embodiment of the invention.

[0074] like Figure 5 As shown, after the controller receives the connection request from the front end transmitted by the backend communicator, it performs function interface mapping, manages the process waiting queue, and creates task processes.

[0075] Figure 6 The working principle of the scheduler in a GPU resource scheduling method provided in this embodiment of the invention.

[0076] like Figure 6 As shown, after obtaining multiple state parameters of the GPU, the scheduler uses an evaluation model to evaluate the GPU's resources, selects a scheduling strategy based on the GPU's resources, and schedules the appropriate GPU to process the task process.

[0077] In one possible implementation, the plurality of first characteristic parameters include a plurality of state characteristic parameters of the GPU, which include at least one of GPU utilization, memory utilization, performance status, power consumption, temperature, core frequency, memory frequency, and running process.

[0078] In the embodiments provided by the present invention, the plurality of first feature parameters include GPU state features, which include:

[0079] (1) GPU utilization:

[0080] GPU utilization is the percentage of time a kernel function actually runs on the GPU stream processor within a past sampling period. GPU utilization is a time-based metric; a higher utilization rate indicates that the kernel function actually utilizes the stream processor for a longer period.

[0081] (2) Video memory utilization:

[0082] Memory utilization is the percentage of time a kernel function spends reading from or writing to global memory within a past sampling period. Like GPU utilization, memory utilization is a time-based metric; the higher the memory utilization, the longer the kernel function actually spends reading from or writing to global memory. However, a high frequency of global memory read / write operations means a lower hit rate for L1 cache, L2 cache, and texture cache, so memory utilization is essentially an inverse metric.

[0083] (3) Performance status:

[0084] Performance status is a metric defined by NVML to measure the current performance of a GPU, represented by an integer value. According to the official documentation, this metric ranges from 0 to 15, with 32 indicating that the current GPU performance status cannot be obtained. A higher value indicates stronger GPU performance, while a lower value indicates weaker performance. Based on actual parameter data collected from a GTX 1060, the performance status value is 8 in idle state, gradually decreasing as device load increases.

[0085] (4) Power consumption:

[0086] Power consumption is a metric that measures the energy consumption of a GPU and related circuitry (such as global memory). Different GPU models have different power consumption, and the power consumption of the same GPU also varies depending on the workload. Actual testing of this metric showed that GPU power consumption increases significantly with increasing device load. Once the power consumption reaches its peak, further increases in workload do not result in a noticeable change in power consumption.

[0087] (5) Temperature:

[0088] GPU temperature changes with increasing load; as the load increases, the GPU temperature rises accordingly. To protect the GPU, NVIDIA automatically slows down or shuts it down when the GPU temperature reaches a certain threshold. For example, the default slowdown threshold for the GTX 1060 is 99 degrees Celsius, and the default shutdown threshold is 102 degrees Celsius. Therefore, temperature also has a certain impact on load assessment.

[0089] (6) Core frequency:

[0090] Core frequency, measured in MHz, refers to the operating frequency of the GPU core. This metric can reflect the performance of the GPU core to some extent. Experiments conducted on a GTX 1060 showed that for the same task, the execution time decreased as the core frequency increased.

[0091] (7) Memory frequency:

[0092] Memory frequency, or GPU global memory operating frequency, is measured in MHz and is the reciprocal of the memory clock cycle. Similarly, in experiments conducted on the GTX 1060, the execution time of the same task decreased as the memory frequency increased.

[0093] (8) Number of running processes:

[0094] The number of running processes is the most direct indicator of GPU load. Processes running on the GPU consume computing and storage resources, leading to resource contention between them. The more running processes, the higher the actual GPU load. NVML provides an interface to retrieve two types of processes: image task processes and compute task processes.

[0095] In one possible implementation, the multiple second characteristic parameters include multiple kernel function characteristic parameters of the GPU, which include at least one of thread bundle number, shared memory transaction, texture / L1 joint cache transaction, L2 cache transaction, and global memory transaction.

[0096] In the embodiments provided by the present invention, the second feature parameter includes the kernel function feature parameter of the GPU, and the kernel function feature parameters of multiple GPUs include:

[0097] (1) Number of thread bundles:

[0098] A thread bundle on a GPU is an official scheduling unit for thread scheduling; it's a collection of threads. Because thread bundles are a critical resource for thread scheduling, they are considered representative of computing resources. The number of thread bundles is determined at compile time. One method is to use nvprof or CUPTI to obtain the `warps_launched` event from the kernel function; another is to calculate it using a formula:

[0099]

[0100] (2) Shared memory transactions:

[0101] Shared memory is a memory region within a stream multiprocessor (SM) that boasts extremely low access latency. Thread blocks can request shared memory for inter-thread communication. When the total shared memory demand of all thread blocks in the SM is less than the total shared memory capacity, thread blocks can execute alternately. When the total shared memory demand of all thread blocks exceeds the total shared memory capacity, thread blocks can only execute serially. If a single thread block requests more shared memory than the total shared memory capacity, a compilation error will occur. Shared memory operations are mainly divided into two processes: load and store, representing the processes of reading and writing data from shared memory, respectively. The metrics `shared_load_transactions` and `shared_store_transactions` can be obtained using `nvprof` or `CUPTI`. These metrics represent the number of shared memory load and store transactions, respectively. Each time a thread executes shared memory-related instructions, one or more transactions are generated. `shared_load_transactions` and `shared_store_transactions` can measure the specific usage of shared memory by kernel functions.

[0102] (3) Texture / L1 co-cached transactions:

[0103] This patent uses the Pascal GPU architecture, which integrates the L1 cache and texture cache into a single storage area to form a joint cache. The GPU's L1 cache operates exactly like the CPU's L1 cache; resources that a Service Provider (SP) needs immediately are stored in the L1 cache, hence the L1 cache can be called a hardware-controlled cache. When an SP needs certain data but the L1 cache misses, it is read from the L2 cache and stored in the L1 cache. In other words, the L1 cache is inconsistent; the contents of the L1 caches of different Service Providers (SMs) are unrelated. The `tex_cache_transactions` metric, representing the number of transactions read from the joint cache, can be obtained using `nvprof` or `CUPTI`.

[0104] (4) L2 cache transactions:

[0105] L2 cache is a storage area used for high-speed caching of data read from global memory. L2 read / write speeds fall between L1 cache and global memory. It is shared by multiple SMs within the GPU, and all SMs can read and write data at the same memory address in the L2 cache. L2 cache allows direct communication between SMs without going through global memory, therefore this cache portion has a larger capacity than the cache within each SM. The `l2_read_transactions` and `l2_write_transactions` metrics can be obtained using `nvprof` or `CUPTI`, representing the number of L2 cache read / write request transactions, respectively. L2 cache transactions can be divided into L2-global transactions and L2-local transactions, which access global and local memory, respectively. Local memory is essentially a part of global memory, characterized by being accessible only to specific threads.

[0106] (5) Global memory transactions:

[0107] Global memory is one of the biggest factors limiting GPU access speed because it has the largest storage space, making it the slowest memory to access. Current global memory can support direct communication between GPUs, and its transfer performance has been greatly improved. The `gld_transactions` and `gst_transactions` metrics can be obtained using `nvprof` or `CUPTI`, representing the global memory load and store processes, respectively.

[0108] Table 1 shows the set of GPU load metrics.

[0109] Table 1

[0110]

[0111] As shown in Table 1, the second column is the English description of each load metric parameter of the GPU, and the third column is the Chinese description of each load metric parameter of the GPU.

[0112] Step 102: Input multiple first feature parameters into the first-time prediction model to obtain the load values ​​on multiple GPUs;

[0113] Specifically, the first-time prediction model evaluates GPU status parameters, comprehensively considers factors that may increase load, and allocates GPUs with lighter loads to tasks.

[0114] In one possible implementation, multiple first feature parameters are input into a first-time prediction model to obtain load values ​​on multiple GPUs, including:

[0115] Input multiple state feature parameters from each GPU into the corresponding nodes in the first layer of the first-time prediction model;

[0116] The multiple state feature parameters input from each node are fuzzified to obtain the fuzzy linguistic variable value of each state feature parameter. The fuzzy linguistic variable value includes a fuzzy set, and the fuzzy set contains multiple fuzzy subsets.

[0117] Multiple fuzzy subsets of each state feature parameter are input into the nodes of the second layer, and the membership value of each fuzzy subset is output by performing a membership function operation. The membership function includes a Gaussian membership function.

[0118] Input multiple membership values ​​into multiple nodes in the third layer representing fuzzy rules, and output multiple matching values;

[0119] A weighted average method is used to defuzzify multiple matching values ​​to obtain the load value of each GPU.

[0120] In the embodiments provided by this invention, the first-time prediction model uses SOFNN (Self-Organizing Fuzzy Neural Network), and an SOFNN network consists of a total of 4 layers of nodes.

[0121] The first layer is the input layer: the nodes in this layer are input nodes, represented by x. i (i = 1, 2, ..., 8) indicates that there are a total of 8 nodes, corresponding to the 8 parameters of the GPU load metric set: GU, MU, PERF, PWR, TEMP, SCLK, MCLK, and PNUM. This layer inputs the 8 parameters into the next membership function layer.

[0122] The second layer is the membership function layer: This layer first performs fuzzification, converting the eight fixed input variables obtained from the upper layer into fuzzy linguistic variable values, where each fuzzy linguistic variable value is a fuzzy set. The nodes in this layer are composed of each input variable x. i The fuzzy subset is composed of elements on the universe of discourse, including positive large (PB), positive small (PS), zero (ZO), negative large (NB), and negative small (NS). Then, a membership function operation is performed on each fuzzy subset, yielding the result μ. ij The output node of this layer is shown in formula (2). The membership functions of the nodes include triangular, trapezoidal, S-shaped, Z-shaped, and Gaussian shapes. This patent adopts the Gaussian membership function, where x... i This represents the 8 input nodes of the first layer, m ij and σ ij N represents the center and width of the membership function, respectively. i This represents the number of fuzzy subsets corresponding to each input variable. The total number of output nodes in this layer is represented by N(2), as shown in formula (3):

[0123]

[0124]

[0125] The third layer is the rule layer: each node in this layer represents a fuzzy control rule, corresponding to Ni fuzzy sets in the previous layer, and is used to complete the precondition matching operation of the fuzzy logic rules. Its input is the membership degree μ obtained from the previous layer. ij The output is the matching degree φj. This layer uniformly uses a fuzzy AND operation, multiplying the 8 membership values ​​belonging to the same fuzzy set j to obtain a matching degree φ. j This operation is performed on each fuzzy set, resulting in a total of N. i Each matching degree is used as an output node. The specific calculation formula is shown in formula (4), and N is the same as above. i This indicates the number of fuzzy subsets corresponding to the input variable.

[0126]

[0127] The final layer is the output layer: the output variable y is the current GPU load value, obtained using a weighted average method for defuzzification. N obtained from the rule layer... i Matching degree φ j As input, respectively, and with the corresponding weights w j Multiply and sum, then multiply by N. i Matching degree φ j Divide the sum of the results to obtain the final output variable y. The specific calculation is shown in formula (5), where w j These are the weighting coefficients.

[0128]

[0129] In one possible implementation, training a first-time prediction model is also included, which includes:

[0130] The membership values ​​output by the nodes of the second layer of the first-time prediction model are input into multiple nodes of the third layer, and the matching degree values ​​are output and the maximum value of the matching degree function is calculated.

[0131] During the training of the parameters of the first-time prediction model, as the number of training samples increases, the weight values ​​of each node of the first-time prediction model are continuously adjusted based on feedback.

[0132] Set a first threshold. If the maximum value of the matching degree function is less than the first threshold, then decrease the weight value of each node; if the maximum value of the matching degree function is greater than the first threshold, then increase the weight value of each node.

[0133] When training the structure of the first-time prediction model, a second threshold is set. If the maximum value of the matching degree function is less than the second threshold, a fuzzy rule is added; otherwise, no fuzzy rule is added.

[0134] A third threshold is set. If the absolute value of the difference between the GPU load value output by the prediction model in the first time and the GPU load value is less than the third threshold, the fuzzy rule added in this training is deleted; otherwise, the fuzzy rule added in this training is not deleted.

[0135] In the embodiments provided by this invention, the training of SOFNN includes two parts: structure training and parameter training. Initially, there is only one node. First, the node is fuzzified through a membership function layer to obtain the membership function value; then, the matching degree function value is obtained through a rule layer, and the maximum value β of the matching degree function is calculated. max The definition is as shown in formula (6).

[0136]

[0137] During parameter training, as the number of training samples increases, the weight values ​​of each node are continuously adjusted based on feedback to complete the model parameter training. A threshold ε is set; if β... max If β < ε, then reduce the node's weight value; if β < ε, then reduce the node's weight value. max If the value is greater than ε, then the weight of the node is increased.

[0138] The structure training completes the addition / deletion of fuzzy rules. A fixed value B is set. th ,β max th If the weight parameters are adjusted, a fuzzy rule is added; otherwise, no rule is added. Furthermore, for cases where the weight parameters have been adjusted, the final GPU load value is calculated using the four layers of the SOFNN network. The absolute value of the difference between this value and the actual GPU load value is denoted as C. r Set a threshold C. th If C r <C th If it does, then delete the previously added fuzzy rule; otherwise, do not delete it.

[0139] Figure 7 This invention provides an SOFNN network structure for a GPU resource scheduling method.

[0140] like Figure 7 As shown, the solid line corresponds to the SOFNN network structure, and the dashed line corresponds to the training process of the SOFNN network structure.

[0141] Step 103: Based on multiple load values, determine the first predicted runtime of the process to be processed in descending order across multiple GPUs;

[0142] ​After obtaining the first predicted runtime, arrange the first predicted runtime in either ascending or descending order.

[0143] In one possible implementation, determining the first predicted runtime of the process to be processed across multiple GPUs in descending order based on multiple load values ​​includes:

[0144] Get the load values ​​of multiple GPUs;

[0145] The first predicted runtime is obtained by multiplying the load value of each GPU by the idle runtime and sorting them in descending order. This yields the first predicted runtime in descending order for each GPU.

[0146] In the embodiments provided by this invention, the GPU load value y is represented by Load Index (LI), where a larger LI value indicates a heavier GPU load. For GPU load testing, a standard CUDA test program, matrix multiplication (MM), is used for calculation. It is assumed that MM takes T seconds to run under GPU idle conditions. e The runtime is T when the GPU load is LI. LI Then LI can be calculated using the following formula (7), and the product of the load and the no-load running time is the first predicted running time.

[0147]

[0148] Step 104: Input multiple first feature parameters and multiple second feature parameters into the second time prediction model to obtain the second predicted running time of the process to be processed in descending order on multiple GPUs;

[0149] Based on the first-time prediction model, the second-time prediction model combines kernel function characteristics to analyze the kernel function resource contention and selects GPUs with less resource contention for task scheduling, thereby reducing the time consumption problem caused by GPU resource contention.

[0150] In one possible implementation, multiple first feature parameters and multiple second feature parameters are input into a second time prediction model to obtain the second predicted runtime of the process to be processed, arranged in descending order across multiple GPUs, including:

[0151] The nvml library is used to obtain the values ​​of various state characteristic parameters and kernel function characteristic parameters of the GPU under different loads, which are used as test values. The actual execution time of the corresponding tasks under different loads is obtained as a test set.

[0152] The test values ​​are input into the second-time prediction model for training to obtain the predicted values.

[0153] The predicted values ​​are compared with the test set using a loss function. The loss function is continuously minimized during the model iteration process until the error between the predicted values ​​and the test set is within the range of MSE and R2 set by the system, thus obtaining an accurate second-time prediction model.

[0154] The values ​​of each state feature parameter and kernel function feature parameter of each GPU are input into the trained second time prediction model to obtain the second prediction runtime.

[0155] In the embodiments provided by the present invention, the second prediction runtime is XGBoost. XGBoost is an ensemble algorithm based on gradient boosting trees. Its core idea is to build several weak evaluators (here, tree models) for modeling to obtain better performance than a single model. Here, a weak evaluator refers to any model with a prediction accuracy greater than 50%.

[0156] When using XGBoost for prediction, it is not necessary to care whether the feature parameters of the kernel function have a significant impact on the results. It is only necessary to take all the collected features (GU, MU, PERF, PWR, TEMP, SCLK, MCLK, PNUM, warps_num, shared_load_transactions, shared_store_transactions, tex_cache_transactions, l2_read_transactions, l2_write_transactions, gld_transactions, gst_transactions) as valid inputs, and then split the dataset into training set and test set.

[0157] The specific process is as follows: different tasks are placed on a 2080ti GPU device, and the load is gradually increased. At the same time, the NVML library is used to obtain multiple sets of real-time GPU state feature values ​​and kernel function feature values ​​under different loads, and these are recorded as the training set. The actual execution time of the tasks under different load conditions is also obtained as the test set.

[0158] First, the parameters from the training set are fed into the XGBoost model for training to obtain predicted values. Then, the predicted values ​​are compared with the true values ​​in the test set using a loss function. During model iteration, the loss function is continuously minimized until the error between the predicted values ​​and the true values ​​in the test set is within the system's set MSE and R0. 2 Within the range.

[0159] The iterative process of gradient boosting trees is as follows: Gradient boosting regression trees start training from a single tree, and the number of trees is continuously increased during the training iterations until a pre-set value is reached, thus forming a strong evaluator composed of several tree models. Samples can be obtained by weighted summation of the tree results, as shown in formula (8). Where K represents the number of trees, k represents the k-th tree, and γ k h represents the weight of the k-th tree. k This represents the predicted value on the k-th tree.

[0160]

[0161] XGBoost, an improved version of gradient boosting trees, offers better predictions than ordinary gradient boosting trees. The weight of each leaf node in XGBoost is the regression value of the sample at that leaf node within the tree, denoted as f. k (x i f k Let x represent the k-th tree. i Let i represent the feature vector of sample i. When there are K gradient boosting trees in total on XGBoost, the predicted value of sample i, i.e., the expected execution time of the task taking resource competition into account, is calculated as shown in formula (9):

[0162]

[0163] The loss function is a metric that measures the difference between the true value and the predicted value, used to evaluate the performance of the model during iteration. Traditional gradient boosting algorithms use 0-1 loss functions or log loss functions for classification prediction, and squared error for regression prediction. To achieve a balance between model performance and computational efficiency, XGBoost considers that the size of the trees may affect the training speed; therefore, its objective function is expressed as follows: where i represents the i-th sample in the dataset, m represents the amount of data in the k-th tree, and K represents the number of trees. The left side of the objective function represents the traditional loss function used to measure the difference between the true value y. i Compared with the predicted value The difference between the two values ​​is represented by the right-hand side of the objective function, which indicates the model complexity. By minimizing the model error rate and complexity, the objective function value can be minimized, as shown in formula (10).

[0164]

[0165] Model evaluation is an important part of machine learning. This patent selects two indicators: mean squared error and coefficient of determination, and their formulas are shown in (11) and (12):

[0166]

[0167]

[0168] In the above formula, the mean square error ranges from [0, +∞), and the true value y i Compared with the predicted value The greater the difference between them, the greater the mean squared error. The coefficient of determination ranges from -∞ to 1; the closer the coefficient of determination is to 1, the greater the true value y. i Compared with the predicted value The smaller the difference, the better. When the coefficient of determination is 0, it means that the model's predicted value is consistent with the mean. When the coefficient of determination is less than 0, it means that the model is worse than the mean.

[0169] The second prediction runtime is obtained by inputting all the collected features into the trained XGBoost model.

[0170] Figure 8 The iterative process of gradient boosting trees is demonstrated.

[0171] like Figure 8 As shown, there is only one tree at the beginning of the iteration, and the number of trees increases as the iteration progresses, thus forming a strong evaluator composed of several tree models.

[0172] Step 105: Obtain the GPU with the shortest second predicted runtime from among the GPUs ranked last in the first predicted runtime order and start the pending process.

[0173] Specifically, the second prediction time is sorted according to the first prediction running time, either from largest to smallest or smallest to largest.

[0174] Step 105: Obtain the GPU with the shortest second predicted runtime from among the GPUs ranked last in the first predicted runtime order and start the pending process.

[0175] Specifically, task scheduling is handled by a scheduling thread created by the scheduler. The scheduler is responsible for detecting status parameters. During task scheduling, the scheduling thread periodically retrieves processes from the task waiting queue and allocates GPU execution devices to them. First, tasks are scheduled according to the first time prediction model. During execution, the approximate execution time of the task is determined, and then it is checked whether it exceeds the first predicted execution time. If it exceeds the first predicted execution time, the device with the lowest load is selected. Then, tasks are scheduled according to the second time prediction model, selecting the device with the shortest predicted execution time based on the predicted execution time. The GPU with the shortest expected execution time is allocated to the kernel function. If it is less than the first predicted execution time, the scheduling fails.

[0176] The present invention also provides a GPU resource scheduling device 2, comprising:

[0177] The feature parameter acquisition module 201 is used to acquire multiple first feature parameters and multiple second feature parameters of multiple GPUs;

[0178] The load value acquisition module 202 is used to input multiple first feature parameters into the first-time prediction model to obtain the load values ​​on multiple GPUs;

[0179] The first-time acquisition module 203 is used to determine the first predicted runtime of the process to be processed in descending order on multiple GPUs based on multiple load values;

[0180] The second time acquisition module 204 is used to input multiple first feature parameters and multiple second feature parameters into the second time prediction model to obtain the second predicted running time of the process to be processed in descending order on multiple GPUs.

[0181] GPU determination module 205 is used to obtain the GPU with the smallest second predicted runtime among multiple GPUs ranked lower in the first predicted runtime order and to run the pending process.

[0182] Figure 9 A structural diagram of the performance evaluation device provided by the present invention.

[0183] like Figure 9 As shown, the structure diagram includes: feature parameter acquisition module 201, load value acquisition module 202, first time acquisition module 203, second time acquisition module 204, and GPU determination module 205.

[0184] In one possible implementation, the load value acquisition module 202 includes:

[0185] Input unit 2021 inputs multiple state feature parameters from each GPU into the corresponding nodes in the first layer of the first-time prediction model;

[0186] The fuzzification unit 2022 performs a fuzzification operation on multiple state feature parameters input from each node to obtain the fuzzy linguistic variable value of each state feature parameter. The fuzzy linguistic variable value includes a fuzzy set, and the fuzzy set contains multiple fuzzy subsets.

[0187] Membership value determination unit 2022 inputs multiple fuzzy subsets of each state feature parameter into the nodes of the second layer, and outputs the membership value result of the membership function operation on each fuzzy subset. The membership function includes the Gaussian membership function.

[0188] Matching unit 2023 inputs multiple membership values ​​into multiple nodes of the third layer representing fuzzy rules and outputs multiple matching values;

[0189] The defuzzing unit 2024 uses a weighted average method to defuzzify multiple matching degree values ​​to obtain the load value of each GPU.

[0190] In one possible implementation, a first-time prediction model training unit 2025 is also included, which is specifically used for:

[0191] The membership values ​​output by the nodes of the second layer of the first-time prediction model are input into multiple nodes of the third layer, and the matching degree values ​​are output and the maximum value of the matching degree function is calculated.

[0192] During the training of the parameters of the first-time prediction model, as the number of training samples increases, the weight values ​​of each node of the first-time prediction model are continuously adjusted based on feedback.

[0193] Set a first threshold. If the maximum value of the matching degree function is less than the first threshold, then decrease the weight value of each node; if the maximum value of the matching degree function is greater than the first threshold, then increase the weight value of each node.

[0194] When training the structure of the first-time prediction model, a second threshold is set. If the maximum value of the matching degree function is less than the second threshold, a fuzzy rule is added; otherwise, no fuzzy rule is added.

[0195] A third threshold is set. If the absolute value of the difference between the GPU load value output by the prediction model in the first time and the GPU load value is less than the third threshold, the fuzzy rule added in this training is deleted; otherwise, the fuzzy rule added in this training is not deleted.

[0196] In one possible implementation, the first-time acquisition module 203 is specifically used for:

[0197] Get the load values ​​of multiple GPUs;

[0198] The first predicted runtime is obtained by multiplying the load value of each GPU by the idle runtime and sorting them in descending order. This yields the first predicted runtime in descending order for each GPU.

[0199] In one possible implementation, the second time acquisition module 204 is specifically used for:

[0200] The nvml library is used to obtain the values ​​of various state characteristic parameters and kernel function characteristic parameters of the GPU under different loads, which are used as test values. The actual execution time of the corresponding tasks under different loads is obtained as a test set.

[0201] The test values ​​are input into the second-time prediction model for training to obtain the predicted values.

[0202] The predicted values ​​are compared with the test set using a loss function. The loss function is continuously minimized during the model iteration process until the error between the predicted values ​​and the test set is within the range of MSE and R2 set by the system, thus obtaining an accurate second-time prediction model.

[0203] The values ​​of each state feature parameter and kernel function feature parameter of each GPU are input into the trained second time prediction model to obtain the second prediction runtime.

[0204] In one possible implementation, the feature parameter acquisition module 201 includes a first parameter acquisition unit 2011. The first feature parameter of the first parameter acquisition unit 2011 includes multiple state feature parameters of the GPU. The multiple state feature parameters of the GPU include at least one of GPU utilization, memory utilization, performance status, power consumption, temperature, core frequency, memory frequency and running process. The values ​​of each state feature parameter of the GPU are obtained through the scheduler's NVML library.

[0205] In one possible implementation, the feature parameter acquisition module 201 includes a second parameter acquisition unit 2012. The second parameter acquisition unit 2012 has multiple second feature parameters, including multiple kernel function feature parameters of the GPU. The multiple kernel function feature parameters of the GPU include at least one of thread bundle number, shared memory transaction, texture / L1 joint cache transaction, L2 cache transaction, and global memory transaction. The values ​​of the multiple kernel function feature parameters of the GPU are obtained through a fixed parameter set of the scheduler.

[0206] In one possible implementation, an apparatus is also provided, the apparatus including a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the GPU resource scheduling method described in the embodiments of the present invention.

[0207] In one possible implementation, a computer-readable storage medium is also provided, which stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the GPU resource scheduling method described in the embodiments of the present invention.

[0208] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0209] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0210] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0211] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A GPU resource scheduling method, characterized in that, Schedulers used in GPU resource management systems include: Obtain multiple first feature parameters and multiple second feature parameters of multiple GPUs, wherein the multiple first feature parameters include multiple state feature parameters of the GPUs, and the multiple second feature parameters include multiple kernel function feature parameters of the GPUs; Multiple first feature parameters are input into a first time prediction model to obtain multiple load values ​​on the GPUs. The first time prediction model is a self-organizing fuzzy neural network. Obtain the load values ​​of multiple GPUs, multiply the load value of each GPU by the idle runtime to obtain the first predicted runtime, and sort them in descending order to obtain the first predicted runtime in descending order on each GPU. The nvml library is used to obtain the values ​​of various state feature parameters and kernel function feature parameters of the GPU under multiple sets of different loads as test values. The actual execution time of the corresponding tasks under different loads is obtained as a test set. The test values ​​are input into the second time prediction model for training to obtain predicted values. The predicted values ​​are compared with the test set through a loss function. During the model iteration process, the loss function is continuously minimized until the error between the predicted value and the test set is within the MSE and R2 range set by the system, thus obtaining the trained second time prediction model. The values ​​of various state feature parameters and kernel function feature parameters of each GPU are input into the trained second time prediction model to obtain the second predicted runtime. The second time prediction model is an XGBoost model. The GPU with the shortest second predicted runtime among the GPUs ranked last in the first predicted runtime is selected to run the pending process.

2. The method of claim 1, wherein, The step of inputting multiple first feature parameters into a first-time prediction model to obtain multiple load values ​​on the GPU includes: Multiple state feature parameters on each GPU are input into the corresponding nodes in the first layer of the first time prediction model; The multiple state feature parameters input from each node are fuzzified to obtain the fuzzy linguistic variable value of each state feature parameter. The fuzzy linguistic variable value includes a fuzzy set, which contains multiple fuzzy subsets. Multiple fuzzy subsets of each state feature parameter are input into the nodes of the second layer, and the membership degree value of each fuzzy subset is output by performing a membership function operation, wherein the membership function includes a Gaussian membership function. Input multiple membership values ​​into multiple nodes in the third layer representing fuzzy rules, and output multiple matching values; The weighted average method is used to defuzzify the multiple matching degree values ​​to obtain the load value of each GPU.

3. The method of claim 2, wherein, It also includes training the first time prediction model, which includes: The membership values ​​output by the nodes of the second layer of the first time prediction model are input into the multiple nodes of the third layer, and a matching degree value is output and the maximum value of the matching degree function is calculated. During the training of the parameters of the first time prediction model, as the number of training samples increases, the weight values ​​of each node of the first time prediction model are continuously adjusted. A first threshold is set. If the maximum value of the matching degree function is less than the first threshold, the weight value of each node is reduced; if the maximum value of the matching degree function is greater than the first threshold, the weight value of each node is increased. When training the structure of the first time prediction model, a second threshold is set. If the maximum value of the matching degree function is less than the second threshold, the fuzzy rule is added; otherwise, the fuzzy rule is not added. A third threshold is set. If the absolute value of the difference between the GPU load value output by the first time prediction model and the GPU load value is less than the third threshold, the fuzzy rule added in this training is deleted; otherwise, the fuzzy rule added in this training is not deleted.

4. The method according to any one of claims 1 to 3, characterized in that, The GPU's various state characteristic parameters include at least one of GPU utilization, memory utilization, performance status, power consumption, temperature, core frequency, memory frequency, and running process. The values ​​of each of the GPU's state characteristic parameters are obtained through the scheduler's NVML library.

5. The method according to any one of claims 1-3, characterized in that, The GPU's kernel function characteristic parameters include at least one of the following: thread bundle number, shared memory transaction, texture / L1 joint cache transaction, L2 cache transaction, and global memory transaction. The values ​​of the GPU's kernel function characteristic parameters are obtained through a fixed parameter set of the scheduler.

6. A GPU resource scheduling apparatus, characterized by comprising: include: The feature parameter acquisition module is used to acquire multiple first feature parameters and multiple second feature parameters of multiple GPUs. The multiple first feature parameters include multiple state feature parameters of the GPUs, and the multiple second feature parameters include multiple kernel function feature parameters of the GPUs. The load value acquisition module is used to input multiple first feature parameters into a first time prediction model to obtain load values ​​on multiple GPUs, wherein the first time prediction model is a self-organizing fuzzy neural network. The first-time acquisition module is used to acquire the load values ​​of multiple GPUs, multiply the load value of each GPU by the idle runtime to obtain the first predicted runtime, and sort them in descending order to obtain the first predicted runtime in descending order on each GPU. The second time acquisition module is used to obtain the values ​​of various state feature parameters and kernel function feature parameters of the GPU under multiple sets of different loads using the NVML library, as test values, and to obtain the actual execution time of the corresponding tasks under different loads, as a test set; the test values ​​are input into the second time prediction model for training to obtain predicted values; the predicted values ​​are compared with the test set through a loss function, and the loss function is continuously minimized during model iteration until the error between the predicted value and the test set is within the MSE and R2 range set by the system, thus obtaining the trained second time prediction model; the values ​​of various state feature parameters and kernel function feature parameters of each GPU are input into the trained second time prediction model to obtain the second predicted runtime, wherein the second time prediction model is an XGBoost model; The GPU determination module is used to obtain the GPU with the smallest second predicted runtime among multiple GPUs ranked lower in the first predicted runtime order and to run the pending process.

7. An electronic device, comprising: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the GPU resource scheduling method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the GPU resource scheduling method as described in any one of claims 1-5.

Images