A GPU resource scheduling method and system based on dynamic weight calculation

Abstract

The application discloses a GPU resource scheduling method and system based on dynamic weight calculation, relates to the technical field of algorithm services, and comprises the following steps: pre-running an algorithm according to an initial weight of the algorithm and an algorithm process designed according to an analysis task; acquiring algorithm computing power resource data, and pre-evaluating the algorithm computing power resource; setting a priority order and a dynamic weight of an algorithm process according to the algorithm computing power resource data, the pre-evaluated algorithm computing power resource and algorithm task analysis, calculating the algorithm computing power resource and generating a dynamic scheduling scheme; verifying and adjusting the algorithm computing power resource; loading full data to run the algorithm, executing the algorithm process, and monitoring the execution process of the algorithm process. The application can effectively solve the problem that, in the case of limited servers and GPUs, more users can execute analysis tasks as much as possible without queuing and waiting or task failure caused by resource competition, and ensure efficient and stable operation of an algorithm platform.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of algorithm service, and more particularly to a GPU resource scheduling method and system based on dynamic weight calculation. BACKGROUND

[0002] Under the background of the big data era, systems related to artificial intelligence and data analysis need to execute a large number of algorithms, thereby putting forward higher requirements for the computing power and memory capacity of GPUs. The current mainstream algorithm execution process is based on the modification of a workflow engine, and its execution characteristics are consistent with the workflow execution, and the entire workflow node is executed one by one. Since the workflow is business-oriented, each audit only produces audit logs, so it does not need to pay too much attention to the performance of the workflow. However, the algorithm process is different, some data preprocessing nodes can be executed in seconds, while some machine learning algorithms need to be executed for several hours, and the execution time and system resources occupied by each node are different.

[0003] On the one hand, the high cost of hardware makes it a big challenge to obtain sufficient computing resources, which leads to the inability to run multiple high-consumption resource algorithms at the same time; on the other hand, in a multi-user environment, due to the lack of effective resource management mechanism, it is impossible to estimate the resources required by the next algorithm, and when a large number of algorithm requests flow in at the same time, the algorithm server resources quickly reach saturation, and eventually lead to algorithm execution failure, memory overflow and other problems, affecting the normal operation of the business process.

[0004] In addition, in the existing machine learning and deep learning computing framework, resource scheduling mainly relies on manual setting or simple code preset, and the user must manually evaluate whether the required computing resources can meet the needs of algorithm running, and in the execution process, once the failure caused by insufficient resources is encountered, it is necessary to constantly adjust the parameters and repeatedly try until successful running. The existing technology lacks accurate estimation and dynamic adjustment mechanism for system resource usage, resulting in low resource utilization, especially in multi-task parallel processing and GPU cluster environment, the management and allocation of computing power resources need to be optimized. In the multi-user high-concurrency scenario, due to the lack of effective resource scheduling mechanism, a large number of concurrent algorithm tasks will cause the server resources to quickly reach saturation. In this case, even if the user makes personal adjustments, the problem of resource shortage and execution failure will still be frequently encountered.

[0005] Therefore, how to provide a GPU resource scheduling method and system based on dynamic weight calculation, effectively master the execution order of the algorithm, and ensure the priority execution of the key algorithm. At the same time, it is necessary to implement appropriate flow control measures to effectively manage the traffic of algorithm requests in high-concurrency situations, prevent resource over-expansion, and improve the stability and performance of the algorithm server, which is a problem that needs to be solved by those skilled in the art. SUMMARY

[0006] Therefore, the application provides a GPU resource scheduling method and system based on dynamic weight calculation, which effectively grasps the order of algorithm execution and ensures the priority execution of key algorithms.

[0007] To achieve the above purpose, the application adopts the following technical scheme: a GPU resource scheduling method based on dynamic weight calculation, comprising:

[0008] Pre-running the algorithm according to the initial weight of the algorithm and the algorithm process designed by analyzing the task;

[0009] Obtaining algorithm computing resource data and pre-evaluating the algorithm computing resource;

[0010] According to the algorithm computing resource data, the pre-evaluated algorithm computing resource and the algorithm task analysis, setting the priority order and dynamic weight of the algorithm process execution, calculating the algorithm computing resource and generating a dynamic scheduling scheme;

[0011] Verifying and adjusting the algorithm computing resource;

[0012] Based on the algorithm process and the dynamic scheduling scheme, loading full data to run the algorithm, executing the algorithm process, and monitoring the execution process of the algorithm process.

[0013] Preferably, after executing the algorithm process, the result is returned and the resource performance evaluation of this run is performed, and the optimal execution record is stored.

[0014] Preferably, the pre-evaluation of the algorithm computing resource comprises: pre-setting the memory consumption formula and the corresponding execution weight of the machine learning and deep learning algorithm and dynamically obtaining the GPU information. In this way, the required resources in the algorithm execution process can be accurately predicted and estimated according to the specific requirements of the task.

[0015] Preferably, based on the execution time of the pre-running, the total time length of the algorithm process is obtained.

[0016] The dynamic weight of the algorithm is calculated by the total time length of the algorithm process, the proportion of the time length of a single algorithm, and the GPU resource consumption.

[0017] Preferably, an algorithm process engine scheduling scene is created to perform multi-user concurrent algorithm execution operation.

[0018] The first execution algorithm of the algorithm process engine is added to the task scheduling queue, and the execution algorithm task is verified.

[0019] Preferably, the algorithm computing resource is verified and adjusted, including:

[0020] According to the execution weight in the algorithm information and the execution weight of the existing task in the task scheduling queue, the task with high execution weight is verified and executed in priority;

[0021] In the task scheduling queue, it is judged whether the remaining resources of the current server are sufficient to execute the current algorithm;

[0022] If the resources are sufficient, the current algorithm is executed; if the resources are insufficient, the next algorithm task in the task scheduling queue is executed, and the current algorithm is reordered and queued as a new task according to the execution weight and waits for verification and execution.

[0023] Preferably, the execution process of the algorithm flow is monitored, including:

[0024] If the algorithm execution fails, the algorithm flow execution ends and returns an error message;

[0025] If the algorithm execution succeeds, the next algorithm is obtained according to the mark of the algorithm and is added to the priority queue;

[0026] After the algorithm execution succeeds, if there is no next component, it is judged that the algorithm flow engine is completely executed.

[0027] Preferably, the execution weight of the algorithm is calculated according to the current GPU real-time resource consumption, the current actual consumption of the algorithm, the consumption of other algorithms in parallel with the current algorithm, and the estimated consumption of the next algorithm of the current algorithm.

[0028] Preferably, after the algorithm flow is executed, the result is returned and the performance of the current running resource is evaluated, and the optimal execution record is stored, including:

[0029] After each algorithm execution, the execution record is stored; when the algorithm is used again, if the factors affecting the priority do not change, the existing algorithm is directly called for priority setting; if the factors affecting the priority change, dynamic adjustment is performed;

[0030] The performance optimization execution record is marked by evaluating each execution and the historically stored record.

[0031] Preferably, a GPU resource scheduling system based on dynamic weight calculation, including:

[0032] The receiving module is used for receiving an algorithm task;

[0033] The calculation module is used for calculating the GPU video memory consumption of the algorithm task according to the preset video memory consumption formula and the algorithm execution weight;

[0034] A monitoring module is configured to acquire GPU information of a current server, wherein the GPU information comprises total GPU memory, GPU memory utilization rate and remaining GPU memory amount;

[0035] A scheduling module is configured to add a task to a task scheduling queue according to the GPU information (remaining GPU memory amount) of the current server and an algorithm execution weight;

[0036] An execution module is configured to execute an algorithm task according to the task scheduling queue.

[0037] According to the technical solution described above, compared with the prior art, the application provides a GPU resource scheduling method and system based on dynamic weight calculation, which comprises the following steps: pre-running an algorithm according to an initial weight of the algorithm and an algorithm process designed by analyzing a task; acquiring algorithm computing resource data and pre-evaluating the algorithm computing resource; setting a priority order and a dynamic weight of an execution algorithm process according to the algorithm computing resource data, the pre-evaluated algorithm computing resource and algorithm task analysis, calculating the algorithm computing resource and generating a dynamic scheduling scheme; verifying and adjusting the algorithm computing resource; loading full data to run the algorithm based on the algorithm process and the dynamic scheduling scheme, executing the algorithm process and monitoring the execution process of the algorithm process. The application can effectively solve the problem that, in the case of limited servers and GPUs, more users can execute analysis tasks as much as possible without queuing or task failure caused by resource competition, and ensure efficient and stable operation of an algorithm platform. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of the provided drawings.

[0039] Figure 1 A dynamic scheduling process schematic diagram provided by the application.

[0040] Figure 2 An algorithm process engine structure schematic diagram provided by the application.

[0041] Figure 3 A GPU resource scheduling method flowchart schematic diagram based on dynamic weight calculation provided by the application. DETAILED DESCRIPTION

[0042] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0043] The embodiments of the present application disclose a GPU resource scheduling method and system based on dynamic weight calculation, which can effectively solve the problem that in the case of limited servers and GPUs, as many users as possible can execute analysis tasks without queuing or task failure caused by resource competition, and ensure efficient and stable operation of the algorithm platform.

[0044] The embodiments of the present application disclose a GPU resource scheduling method based on dynamic weight calculation, as shown in the figure, which comprises the following steps. Figure 1

[0045] S100, pre-running the algorithm according to the initial weight of the algorithm and the algorithm process designed for the analysis task; the algorithm pre-running is mainly the implementation of the data process before the machine learning or deep learning task combined with a relatively small amount of data, and then the information of the algorithm using computing power can be extracted from the pre-running for pre-evaluation and calculation, and when multiple user concurrent algorithm tasks or multiple algorithm tasks are involved, the consumption of GPU computing power resources can be dynamically optimized according to the specific conditions.

[0046] The dynamic optimization process is as follows: when there are multiple user concurrent algorithm tasks or multiple algorithm tasks, the system first sorts the algorithm weights passed by the application end and the task weights in the task queue, and the tasks with higher weights will be executed first. In the task scheduling queue, the execution of the current algorithm task is based on the comparison between the estimated GPU memory and the remaining GPU memory of the server. The system judges whether the remaining GPU memory of the current server is sufficient to execute the current algorithm task. If the resources are sufficient, the task is executed; if the resources are insufficient, the system executes the next algorithm task in the scheduling queue. The new task will enter the algorithm queue and be re-ordered and queued for execution according to the weight.

[0047] S200, obtaining algorithm computing power resource data and pre-evaluating the algorithm computing power resources; pre-estimating whether the algorithm computing power and memory demand can meet the running, and if not, pre-scheduling management is performed;

[0048] The algorithm uses GPU computing power information acquisition and resource consumption evaluation, which involves GPU memory calculation, GPU information, etc.

[0049] S201, algorithm occupies GPU memory calculation

[0050] ​Firstly, the embodiment of the present application sets a series of memory consumption formulas of machine learning and deep learning algorithms and corresponding initial weights of the algorithms in advance, so that the required resources in the execution process can be accurately predicted and estimated according to the specific requirements of the task.

[0051] Based on the machine learning and deep learning algorithm model, in the model training process, the use of memory not only involves model parameters, but also includes model weights, optimizers, gradients and several aspects inside the calculation graph:

[0052] (1) Model parameters: the structure of each model is different, and the parameter quantity calculation method is also different. The following will take a text classification model based on CNN (Convolutional Neural Network) as an example:

[0053] Total parameter quantity calculation formula = word vector mapping parameter quantity + convolution layer parameter quantity + fully connected layer 1 parameter quantity + fully connected layer 2 parameter quantity.

[0054] Word vector mapping (EmbeddingLayer) layer parameter quantity = vocab_size*embedding_dim.

[0055] Vocabulary size (vocab_size) is 5000.

[0056] Word vector dimension (embedding_dim) is 64.

[0057] Parameter quantity: 5000*64 = 320,000.

[0058] The parameter quantity calculation formula of the convolution layer (ConvolutionalLayer) is represented as:

[0059] Conv = (kernel_size*embedding_dim+1)*num_filters.

[0060] Convolution kernel size (kernel_size) is 5.

[0061] Input channel number (embedding_dim) is 64.

[0062] The number of convolution kernels (num_filters) is 256.

[0063] Parameter quantity: (5*64+1)*256 = (320+1)*256 = 321*256 = 8,192.

[0064] The parameter quantity calculation formula of the fully connected layer 1 (FullyConnectedLayer1) is represented as:

[0065] F1 = num_filters * hidden_dim + hidden_dim.

[0066] The number of input neurons (num_filters) is 256.

[0067] The number of output neurons (hidden_dim) is 128.

[0068] Parameter quantity: 256 * 128 + 128 = 32,768 + 128 = 32,896.

[0069] The formula for calculating the parameter quantity of the fully connected layer 2 (FullyConnectedLayer2, i.e., the classifier) is:

[0070] F2 = hidden_dim * num_classes + num_classes.

[0071] The number of input neurons (hidden_dim) is 128.

[0072] The number of output neurons (num_classes) is 10.

[0073] Parameter quantity: 128 * 10 + 10 = 1,280 + 10 = 1,290.

[0074] Total parameter quantity = 320,000 + 8,192 + 32,896 + 1,290 = 352,376.

[0075] Therefore, the CNN-based text classification model has a total of 352,376 trainable model parameters.

[0076] (2) Model weights (execution weights): The calculation method of this part is the same as that of the model parameters.

[0077] (3) Optimizer: If the AdamW optimizer is selected, each parameter needs to occupy 8 bytes because two states need to be maintained. Therefore, the memory used by the optimizer is twice that of the full-precision (float32) model weight. The AdamW optimized by bitsandbytes is used in the embodiment of the present application, and each parameter only needs to occupy 2 bytes, i.e., half of the full-precision (float32) model weight.

[0078] (4) Gradient: The memory occupied by the gradient is the same as that of the full-precision (float32) model weight.

[0079] (5) Calculation of internal variables in the graph: frameworks such as Pytorch or Tensorflow use graph structures for calculation, and graph nodes need to be stored during forward and backward propagation, so they also occupy GPU memory. This part is related to the specific implementation of the model and batch_size, and the larger the batch_size, the more GPU memory is occupied.

[0080] Therefore, only considering the first 3 parts, the GPU memory required for model training is at least 3-4 times that of inference. Combined with actual experience, the GPU memory required for training is usually more than 10 times the actual calculation.

[0081] The rough formula for GPU memory usage is as follows:

[0082] GPU memory usage (MB) = 10 * (number of model parameters + input data size * batch) * data type size / (1024 bytes^2)

[0083] Note: In general, the data type is float32 (single-precision floating-point) or int32 (integer), which occupies 4B.

[0084] S202, dynamically obtaining GPU information

[0085] By dynamically obtaining the GPU-related information of the server, such as memory usage, the scheduling of tasks can be adjusted in real time to optimize the use of GPU memory resources. In the embodiments of the present application, the following information is obtained through the Python pynvml module: the number of GPUs, the total memory of each GPU, the used and remaining memory (the proportion of space occupied when processing tasks), the current usage rate (the proportion of time occupied when processing tasks), the total power consumption / current power consumption, the GPU driver version, etc.

[0086] S300, according to the algorithm computing resource data, the pre-evaluated algorithm computing resource, and the algorithm task analysis, setting the priority order and dynamic weight of the algorithm execution process, calculating the algorithm computing resource and generating a dynamic scheduling scheme.

[0087] Among them, (1) algorithm priority definition rule

[0088] The algorithm priority needs to be set in combination with multiple factors, including but not limited to user demand, algorithm importance, resource utilization, algorithm real-time requirement, etc.

[0089] ① User demand: Users may have different priority requirements for certain tasks. The system can adjust the execution order of the algorithm according to the user's demand.

[0090] ② Algorithm importance: Some algorithms have urgent requirements and need to get the processing result more quickly. The urgency of the task is considered when setting the priority of the algorithm.

[0091] ③Resource utilization: dynamically adjust the priority according to the resource utilization of the algorithm server system. When the system resources are relatively idle, the priority of the algorithm can be adjusted to improve the throughput of the overall system.

[0092] ④Algorithm real-time requirement: set the corresponding priority for tasks that require real-time response. For tasks that require real-time response, a higher priority can be assigned.

[0093] (2) Factors affecting priority

[0094] The factors affecting the priority of the algorithm include the following:

[0095] ①Data volume: the amount of data processed by the algorithm during execution is a key factor. Different algorithms have different sensitivities to data volume, and some algorithms may require more resources and time when facing large-scale data. The size of the data volume directly affects the execution efficiency of the algorithm, thereby affecting the priority of the algorithm.

[0096] ②Algorithm complexity: the complexity of different algorithms varies. Algorithm complexity describes the computational resources and time required for algorithm execution, usually represented by big O notation. The level of algorithm complexity directly affects the execution efficiency of the algorithm, and algorithm complexity is an important priority factor.

[0097] ③Algorithm server remaining resources: during execution, the remaining resources of the algorithm server have an important impact on the execution priority of the algorithm. If the current algorithm server resources cannot meet the needs of a certain algorithm, the algorithm needs to wait for a judgment, and only when there are enough resources can it begin execution. The remaining resources of the server are a key decision factor.

[0098] ④Required resources: different algorithms have different resource requirements. Some algorithms may require more computing resources, memory or storage resources. Algorithms with higher resource requirements can be assigned higher priority to enable them to complete execution in a shorter time.

[0099] ⑤GPU resource allocation: in GPU algorithm execution, dynamically obtain the GPU algorithm server resource situation, judge whether the GPU algorithm service video card has enough video memory required by the algorithm, and use the video memory that meets the algorithm's resource requirements to complete execution.

[0100] S400, verify and adjust the algorithm computing power resources; before algorithm execution, perform algorithm server resource verification to evaluate whether the execution conditions are met. If the conditions are insufficient, the algorithm will be re-added to the priority queue to control effective resource utilization.

[0101] S500, based on the algorithm flow and the dynamic scheduling scheme, loading full data to run the algorithm according to the priority and the execution weight, executing the algorithm flow, and monitoring the execution process of the algorithm flow. A closed loop of algorithm execution management is realized, and information and results in the calculation process are fed back. The algorithm flow engine provided in the embodiment of the application can efficiently manage and allocate computing resources, can estimate the required memory and display memory resources according to the current algorithm model and data, and further judge whether the current system resources can support the running of the next algorithm.

[0102] Specifically, the method further comprises: S600, returning a result and performing resource performance evaluation of this running after the algorithm flow is executed, and storing an optimal execution record.

[0103] After the algorithm flow is executed, the result is returned, the resource performance of this running is evaluated, and the optimal execution record is stored.

[0104] After each algorithm execution is completed, the execution record is stored; when the algorithm is used again, if the factors affecting the priority do not change greatly, the existing algorithm is directly called for priority setting; if the factors affecting the priority change, dynamic adjustment is performed.

[0105] The execution record of performance optimization is marked by evaluating each execution and the historically stored record.

[0106] The embodiment of the application designs a method for calculating the GPU resource occupation of an algorithm running time model and realizing dynamic scheduling of an algorithm task according to next-step resource consumption prediction. Unlike the existing scheduling optimization of a GPU during algorithm model running, the embodiment of the application first designs a multi-index weight method for scheduling:

[0107] (1) Algorithm initial (static) weight: the initial weight is artificially set according to the importance of the algorithm in the system and the resource control demand, and is used as an index for dynamic calculation and weighting during actual running;

[0108] (2) Algorithm dynamic weight: a pre-running mechanism is designed, a small amount of test data is loaded and quickly executed according to the actual parameters of the algorithm before the task is formally run, and the dynamic weight of the algorithm is calculated according to the total time length of the entire task execution, the proportion of the single algorithm time length, and the GPU resource consumption.

[0109] (3) Algorithm formal execution weight: the algorithm is loaded with full data to run formally, and the execution weight of the algorithm is calculated according to the current GPU real-time resource consumption, the current actual consumption of the algorithm, the consumption of other algorithms in parallel with the algorithm, and the estimated consumption conditions of the next-level algorithm of the algorithm.

[0110] According to the calculation method, the system has precise control over the running weight of the algorithm to be run, the running algorithm, the next level algorithm to be associated and started after the running ends, and the GPU resource occupation. According to whether the weight and resources are sufficient, the high-weight algorithm is scheduled to enter the GPU execution, the low-weight algorithm is suspended to wait for execution, and the remaining GPU resources are allocated in time to form a perfect and efficient algorithm task scheduling engine, which meets the demand of more efficient completion of calculation under limited computing power.

[0111] Specifically, the total duration of the algorithm flow is obtained based on the execution time of the pre-run;

[0112] The dynamic weight of the algorithm is calculated by the total duration of the algorithm flow, the single algorithm duration ratio, and the GPU resource consumption.

[0113] Specifically, the calculation method of the dynamic weight is as follows:

[0114] Weighted Score=Default Weight +(Max Weight -Default Weight )×Sigmoid(T)×WG;

[0115] WG: The ratio of GPU resources required for trial operation to GPU resources currently available. This is a value between 0 and 1, 0 indicating no GPU resources required, and 1 indicating that the required GPU resources are equal to the total available GPU resources.

[0116] Default Weight : Default weight. This is the basic score of the algorithm without considering the execution duration and resource consumption.

[0117] Max Weight : Maximum weight, which is the upper limit of the highest score that the algorithm can obtain.

[0118] T: The proportion of algorithm execution duration to total duration, which is a value between 0 and 1.

[0119] Sigmoid(T): Sigmoid function, which maps T to a value between 0 and 1, used to smoothly adjust the score.

[0120] In machine learning and deep learning algorithm tasks, it is likely that there are not single algorithm tasks, but workflow execution tasks containing multiple components. Each component of the workflow is an algorithm task.

[0121] The initial weight of all algorithms is set to 1, whether it is a single algorithm or an algorithm task mode of workflow assembly. The administrator can manually set the weight according to the importance and necessity of the algorithm on the management end to control the priority order of algorithm execution. The greater the default weight value, the higher the execution priority. The administrator can customize the algorithm execution order as needed.

[0122] The dynamic weight of the algorithm is calculated according to the execution time consumption of the system trial run. The overall length of the algorithm process is calculated by the length of the single algorithm, and the dynamic weight is recalculated using a linear formula.

[0123] As shown in Figure 3 In one specific embodiment of the present application, the algorithm needs to find the first component required by the algorithm when executing. The system will add the algorithm information to be executed to the priority queue according to the Web service client FeignClient mode request priority queue. The priority queue will be sorted according to the execution weight size in the algorithm information, and the algorithm with the largest weight will be operated by the algorithm "task". The algorithm "task" will query the algorithm address, required resources (memory, video memory) and other information from the algorithm library. The algorithm server obtains whether the current server resource meets the resource required by the "task" algorithm task. If it does not meet the requirement, it will add the algorithm information to the priority queue server again after waiting for a certain time. Wait for the next "task" operation. If the algorithm server meets the requirements of the algorithm, execute the operation. If the algorithm is executed in the execution operation, the system will query the next algorithm information and weight according to the association between the algorithms and add it to the priority queue. The priority queue is sorted and the "task" operation is completed.

[0124] Specifically, create an algorithm process engine scheduling scenario for multi-user concurrent algorithm execution operation;

[0125] Get the first execution algorithm of the algorithm process engine and add it to the task scheduling queue to verify the execution algorithm task.

[0126] Specifically, the priority queue is implemented based on the following two data types:

[0127] ①LinkedTransferQueue: It is a bounded blocking transfer queue, whose underlying structure is a linked list. This data type is suitable for programs with producer and consumer structure. In practical use of the system, one or more producers are responsible for generating data, while one or more consumers are responsible for processing these data. This data structure is shared by producers and consumers. Producers add new data to this data structure, and then consumers take data from the data structure for processing. If the data structure is empty, i.e. there is no data available for consumption, the consumer will be blocked until new data is available for consumption. Conversely, if the data structure is full, i.e. there is no empty space to accommodate new data, the producer will be blocked until there is an empty space in the data structure to place new data. This mechanism ensures smooth flow and synchronous processing of data, avoiding data loss or resource waste.

[0128] ②PriorityBlockingQueue: It is an unbounded blocking queue provided by Java concurrency package, which is thread-safe. The elements in the queue are sorted according to their natural order, or according to the Comparator provided when constructing. The elements implement the Comparable interface with the compareTo() method. When the system inserts algorithm data into the structure, it compares the data element one by one until it finds its position.

[0129] In the embodiment of the application, the elements of LinkedTransferQueue are stored according to the arrival order, so the earlier the arrival, the earlier the consumption. The system implements the compareTo() method of PriorityBlockingQueue structure, so that the consumption order is determined by the priority rather than the arrival time. These elements will be sorted according to their priority, and the higher the level, the earlier the consumption.

[0130] The algorithm identifier is a unique identifier assigned by the algorithm flow engine. When the element is consumed, the algorithm flow engine schedules and starts executing the task. According to the algorithm identifier, the algorithm library service obtains its algorithm details, including the server information where the algorithm is located, the resource conditions required by the algorithm, etc. The algorithm is obtained by the algorithm library service. The server information is obtained according to the server information and the FeignClient mode to obtain the resource utilization of the specified server. If the remaining resources are sufficient for the current algorithm execution, the execution operation is performed. If the resources cannot meet the algorithm execution, a prompt information is returned, and the algorithm is re-joined in the priority queue after sleeping for a specified time. The algorithm execution operation is started, and if the algorithm execution fails, the algorithm flow execution ends and an error information is returned. If the algorithm execution is successful, the next algorithm is obtained according to the mark of the algorithm and is added to the priority queue. If the algorithm execution is successful, the algorithm flow engine is judged whether all the execution is completed.

[0131] Specifically, the algorithm computing resource is verified and adjusted, including:

[0132] According to the execution weight in the algorithm information and the execution weight of the existing task in the task scheduling queue, the task with high execution weight is verified and executed in priority;

[0133] In the task scheduling queue, it is judged whether the remaining resources of the current server are sufficient to execute the current algorithm;

[0134] If the resources are sufficient, the current algorithm is executed; if the resources are insufficient, the next algorithm task in the task scheduling queue is executed, and the current algorithm is reordered and queued according to the execution weight as a new task and waits for verification and execution.

[0135] Due to limited system resources, for algorithm task nodes that cannot be executed immediately, a waiting queue is needed. When an algorithm flow execution is completed, the system will first recycle the resources, then take an algorithm task from the queue, and judge whether the current system remaining resources meet the running conditions. If not, it continues to wait, if yes, it is executed immediately. Through the above scheduling, it is ensured that all algorithm flows will not fail due to system resource problems during execution.

[0136] Specifically, the execution process of the monitoring algorithm flow includes:

[0137] If the algorithm execution fails, the algorithm flow execution ends and an error information is returned;

[0138] If the algorithm execution is successful, the next algorithm is obtained according to the mark of the algorithm and is added to the priority queue;

[0139] After the algorithm execution is successful, if there is no subordinate component, it is judged whether the algorithm flow engine is completely executed.

[0140] Specifically, the execution weight of the algorithm is calculated according to the current GPU real-time resource consumption, the current actual consumption of the algorithm, the consumption of other algorithms in parallel with the current algorithm and the estimated consumption of the next algorithm of the current algorithm.

[0141] The embodiment of the application discloses a GPU resource scheduling method based on dynamic weight calculation, sets an estimated default weight according to GPU computing power and display memory resources required by an algorithm during running, dynamically calculates and determines a real-time weight of the algorithm according to matching conditions of GPU computing power and display memory actually required by the algorithm and current available resources, correlation of the algorithm in a task flow with front and rear flows and relative importance between different tasks and other factors during running, and realizes scheduling and optimization of an algorithm execution sequence. In the embodiment of the application, the initial weight is a default execution order value of the system to the algorithm (set according to algorithm complexity and time consumption); the dynamic weight is calculated by the system according to actual resource utilization ratio and time consumption of the algorithm during a trial running process of the algorithm. The execution weight is a general term of the initial weight and the dynamic weight.

[0142] Specifically, a GPU resource scheduling system based on dynamic weight calculation, as shown in Figure 2 , comprises:

[0143] A receiving module is used to receive an algorithm task; the algorithm task is generated by a client, and algorithm calling and data parameters of the algorithm task are sent to the receiving module;

[0144] A calculation module is used to calculate GPU display memory consumption of the algorithm task according to a preset display memory consumption formula and an algorithm execution weight;

[0145] A monitoring module is used to obtain GPU information of a current server; the GPU information comprises total display memory, utilization and remaining display memory;

[0146] A scheduling module is used to add the task to a task scheduling queue according to the remaining display memory and the algorithm execution weight;

[0147] An execution module is used to execute the algorithm task according to the task scheduling queue.

[0148] In the embodiment of the present application, in order to improve the efficiency of task execution, a task scheduling queue is introduced. When a new algorithm task is generated, the system will first sort the algorithm weight passed by the application end and the task weight in the existing task scheduling queue, and the task with higher weight will be executed first. In the task scheduling queue, the execution of the current algorithm task will be based on the comparison between the estimated GPU memory and the remaining GPU memory of the server. The system will judge whether the remaining GPU memory of the current server is sufficient to execute the current algorithm task. If the resources are sufficient, the task will be executed; if the resources are insufficient, the next algorithm task in the scheduling queue will be executed. The new task will enter the algorithm queue and be reordered and queued according to the weight for execution. Through this logic, the embodiment of the present application ensures that the tasks can be reasonably and orderly executed, thereby improving the overall task execution efficiency.

[0149] The technical points in the embodiment of the present application are about the scheduling processing logic of the task scheduling queue, which specifically includes the following points:

[0150] Algorithm elements:

[0151] (1) Algorithm weight: passed by the application end;

[0152] (2) GPU memory required by the algorithm: calculated by the algorithm GPU memory calculation formula;

[0153] (3) Current GPU remaining memory: obtained by real-time monitoring of the GPU related information of the server.

[0154] The processing logic of the embodiment of the present application is as follows:

[0155] (1) When a new algorithm task is generated, the system will first sort the algorithm in the task queue according to the weight value passed by the task application end. The task with higher weight will be executed first.

[0156] (2) Calculate the GPU memory required by all algorithms in the task queue.

[0157] (3) When executing the algorithm in the algorithm queue, the current GPU remaining memory is obtained in real time.

[0158] (4) Compare the GPU memory required by the algorithm to be executed and the current GPU remaining memory.

[0159] (5) Judge whether the GPU remaining memory of the current server is sufficient to execute the current algorithm task. If the resources are sufficient, the task will be executed; if the resources are insufficient, the next algorithm task in the scheduling queue will be executed.

[0160] (6) Reinsert the algorithm task that does not meet the execution into the task queue. Re-enter step (1).

[0161] Overall, the technical scheme of the embodiment of the application realizes optimization of resource use in the task execution process by using the scheduling processing logic of the task scheduling queue through three elements of algorithm weight, GPU memory amount required by the algorithm and current GPU remaining memory amount, and improves the use efficiency of GPU memory resources.

[0162] In one specific embodiment of the application, as shown in Figure 3 The algorithm flow engine scheduling scenario is performed by multiple users concurrently, and the first execution algorithm of the algorithm flow engine is obtained.

[0163] The obtained first execution algorithm is added to the priority queue; the priority queue service includes an in-queue position and an out-of-queue position; the algorithms are sorted according to the size of the dynamic weight, and the larger the dynamic weight, the closer to the out-of-queue position, and the smaller the dynamic weight, the closer to the in-queue position; when the algorithm data is inserted, the dynamic weight of the current algorithm is compared with all weight data elements in the priority queue service one by one until the position of the algorithm is found. According to their priority order, the algorithm with a higher dynamic weight will be consumed first; when the element is consumed, the execution algorithm task is verified and executed.

[0164] The algorithm details are obtained from the algorithm library service according to the algorithm identifier; the algorithm library service includes an algorithm library of machine learning or deep learning; the algorithm details include server information of the algorithm, resource conditions required by the algorithm, etc. The algorithm obtains the server information from the algorithm library service, and then obtains the resource utilization condition of the specified server according to the server information and the FeignClient mode.

[0165] If the remaining resources cannot meet the algorithm execution, a prompt information is returned, and the algorithm (for example, algorithm m, and the corresponding weight value is n) is re-added to the priority queue after sleeping for a specified time length.

[0166] If the remaining resources are sufficient for the current algorithm execution, the algorithm execution operation is performed; after the algorithm execution operation is started, if the algorithm execution fails, the algorithm flow execution ends and an error information is returned; if the algorithm execution is successful, the next-level algorithm is obtained according to the algorithm mark and added to the priority queue (at this time, the dynamic weight of the next-level algorithm is the highest, and the algorithm execution operation is performed after the priority queue). If the algorithm execution is successful, the algorithm flow engine is determined whether all execution is completed. If the algorithm is executed completely, the algorithm flow engine execution ends.

[0167] In one specific embodiment of the present application, some of the work flow algorithms built in the system need to use GPU resources, because compared with CPU resources, GPU can provide faster processing speed. Especially for machine learning model training and inference, and image processing algorithms, the effect of GPU is obvious. For example, for resources using CNN (Convolutional Neural Network), when processing the same data and logic, the computing power of GPU is obviously higher than that of CPU, so it can process data faster.

[0168] In the embodiment of the present application, the CNN (Convolutional Neural Network) algorithm is trained according to the different computing power resources of CPU and GPU. For the same set of CIFAR-10 training data set, 5 complete training iterations are performed. As can be seen from Table 1, in the training of the CNN algorithm, using Tesla P100-PCIE-16GB GPU to perform 5 iterations on the entire data set takes 66.98 seconds. Using 88-thread CPU to perform 5 iterations on the entire data set takes 86.18 seconds. It can be seen that under different resource computing power conditions, the GPU computing power resource improves the execution speed of the algorithm by nearly 20%.

[0169] Table 1

[0170] Type of computing power Number of iterations Time consumed (seconds) CPU 5 86.18 GPU 5 66.98 CPU 20 329.34 GPU 20 247.70 CPU 100 1698.67 GPU 100 1212.08

[0171] The various embodiments described in the specification are described in progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0172] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A GPU resource scheduling method based on dynamic weight calculation, characterized in that, The method comprises the following steps: Pre-running the algorithm according to the initial weight of the algorithm and the algorithm flow designed according to the analysis task; Obtaining algorithm computing resource data and pre-evaluating the algorithm computing resource; Setting the priority order and dynamic weight of the algorithm flow according to the algorithm computing resource data, the pre-evaluated algorithm computing resource and the algorithm task analysis, calculating the algorithm computing resource and generating a dynamic scheduling scheme; Verifying and adjusting the algorithm computing resource; Based on the algorithm flow and the dynamic scheduling scheme, loading full data to run the algorithm, executing the algorithm flow and monitoring the execution process of the algorithm flow; The total time length of the algorithm flow is obtained based on the execution time length of the pre-running; The dynamic weight of the algorithm is calculated based on the total time length of the algorithm flow, the single algorithm time length proportion and the GPU resource consumption; Creating an algorithm flow engine scheduling scene for multi-user concurrent algorithm execution operation; The first execution algorithm of the algorithm flow engine is added to the task scheduling queue, and the execution algorithm task is verified; The algorithm is formally run by loading full data, and the execution weight of the algorithm is calculated based on the current GPU real-time resource consumption, the current actual consumption of the algorithm, the consumption of other algorithms in parallel with the algorithm and the estimated consumption of the next algorithm of the algorithm; Verifying and adjusting the algorithm computing resource, including: The tasks in the task scheduling queue are sorted according to the execution weight in the algorithm information and the execution weight of the existing tasks in the task scheduling queue, and the tasks with high execution weight are verified and executed in priority; In the task scheduling queue, it is judged whether the remaining resources of the current server are sufficient to execute the current algorithm; If the resources are sufficient, the current algorithm is executed; if the resources are insufficient, the next algorithm task in the task scheduling queue is verified and executed, and the current algorithm is reordered and queued as a new task according to the execution weight; The dynamic weight is calculated as follows: Weighted Score = Default Weight + (Max Weight - Default Weight ) x Sigmoid(T) x WG; wherein, WG represents the ratio of GPU resources required for trial operation to GPU resources at current execution; Default Weight represents a default weight; Max Weight represents a maximum weight; T represents the proportion of the execution time of the algorithm to the total time; and Sigmoid(T) represents a Sigmoid function for smoothly adjusting the score.

2. The GPU resource scheduling method based on dynamic weight calculation according to claim 1, characterized in that, Further comprising: After the algorithm flow is executed, the result is returned and the resource performance of this run is evaluated, and the optimal execution record is stored.

3. The GPU resource scheduling method based on dynamic weight calculation according to claim 1, characterized in that, The pre-evaluation of the algorithm computing resource comprises: pre-setting the memory consumption formula and the corresponding execution weight of the machine learning and deep learning algorithm and dynamically obtaining the GPU information.

4. The GPU resource scheduling method based on dynamic weight calculation according to claim 1, characterized in that, The monitoring of the execution process of the algorithm flow comprises: If the algorithm execution fails, the algorithm flow execution ends and error information is returned; If the algorithm execution succeeds, the next algorithm is obtained according to the algorithm mark and added to the priority queue; After the algorithm execution succeeds, if there is no next component, it is judged whether the algorithm flow engine is completely executed.

5. The GPU resource scheduling method based on dynamic weight calculation according to claim 2, characterized in that, After the algorithm flow is executed, the result is returned and the resource performance of this run is evaluated, and the optimal execution record is stored, including: After each algorithm execution, the execution record is stored; when the algorithm is used again, if the factors affecting the priority do not change, the existing algorithm is directly called for priority setting; if the factors affecting the priority change, dynamic adjustment is performed; The execution record of performance optimization is marked by evaluating each execution and the historically stored record.

6. A GPU resource scheduling system based on dynamic weight calculation, applying the GPU resource scheduling method based on dynamic weight calculation of any one of claims 1-5, characterized in that, The method comprises the following steps: A receiving module is configured to receive an algorithm task; A calculation module is configured to calculate the GPU memory consumption of the algorithm task according to the pre-set memory consumption formula and the algorithm execution weight. A monitoring module is configured to acquire GPU information of a current server; A scheduling module is configured to add a task to a task scheduling queue according to the GPU information of the current server and an algorithm execution weight; An execution module is configured to execute an algorithm task according to the task scheduling queue.

Images