GPU resource scheduling method and system based on dynamic weight calculation

Abstract

The invention discloses a GPU resource scheduling method and system based on dynamic weight calculation, and relates to the technical field of algorithm services, and the method comprises the steps: carrying out the pre-operation of an algorithm according to the initial weight of the algorithm and the algorithm flow of analysis task design; obtaining algorithm computing power resource data, and pre-evaluating algorithm computing power resources; according to the algorithm computing power resource data, the pre-evaluated algorithm computing power resource and algorithm task analysis, setting a priority sequence and a dynamic weight for executing an algorithm process, computing the algorithm computing power resource and generating a dynamic scheduling scheme; verifying and adjusting the computing power resource of the algorithm; and loading a full data operation algorithm, executing an algorithm process, and monitoring an execution process of the algorithm process. According to the method, under the condition that servers and GPUs are limited, more users can execute analysis tasks as much as possible, the situation of task failure caused by queuing waiting or resource competition does not occur, and efficient and stable operation of an algorithm platform is ensured.

Description

Technical Field

[0001] The present invention relates to the field of algorithm service technology, and more specifically to a GPU resource scheduling method and system based on dynamic weight calculation. Background Art

[0002] In the era of big data, systems related to artificial intelligence and data analysis need to execute many algorithms, which in turn places higher demands on the computing power and memory capacity of GPUs. At present, the mainstream algorithm execution processes are based on the transformation of workflow engines. Their execution characteristics are consistent with workflow execution, and the entire workflow nodes are executed one by one. Since workflows are business-oriented, each audit only generates an audit log, so there is no need to pay too much attention to workflow performance. However, the algorithm processes are different. Some data preprocessing nodes can be executed in seconds, while some machine learning algorithms take hours to execute. The execution time and system resources occupied by each node are different.

[0003] On the one hand, the expensive hardware costs make it a major challenge to obtain sufficient computing resources, resulting in the inability to run multiple high-resource-consuming algorithms at the same time; on the other hand, in a multi-user environment, due to the lack of an effective resource management mechanism, it is impossible to estimate the resources required for the next algorithm. When a large number of algorithm requests pour in at the same time, the algorithm server resources quickly become saturated, and ultimately lead to problems such as algorithm execution failure and memory overflow, affecting the normal progress of business processes.

[0004] In addition, in the existing machine learning and deep learning computing frameworks, resource scheduling mainly relies on manual settings or simple code presets. Users must manually evaluate whether the required computing resources can meet the needs of algorithm operation, and during the execution process, once a failure caused by insufficient resources is encountered, it is necessary to continuously adjust the parameters and repeat the attempt until it runs successfully. The existing technology lacks an accurate prediction and dynamic adjustment mechanism for system resource usage, resulting in low resource utilization, especially in multi-task parallel processing and GPU cluster environments. The management and allocation of computing resources urgently need to be optimized. In the scenario of high concurrency of multiple users, due to the lack of an effective resource scheduling mechanism, a large number of concurrent algorithm tasks will cause server resources to quickly saturate. In this case, even if the user makes personal adjustments, they will still frequently encounter problems of resource shortages and execution failures.

[0005] Therefore, how to provide a GPU resource scheduling method and system based on dynamic weight calculation to effectively grasp the order of algorithm execution and ensure the priority execution of key algorithms. At the same time, it is necessary to implement appropriate flow control measures to effectively manage the flow of algorithm requests under high concurrency conditions, prevent excessive expansion of resources, and improve the stability and performance of algorithm servers. This is a problem that technicians in this field need to solve urgently. Summary of the invention

[0006] In view of this, the present invention provides a GPU resource scheduling method and system based on dynamic weight calculation, which effectively grasps the order of algorithm execution and ensures that key algorithms are executed first. At the same time, it is necessary to implement appropriate flow control measures to effectively manage the flow of algorithm requests under high concurrency conditions, prevent excessive expansion of resources, and improve the stability and performance of the algorithm server.

[0007] In order to achieve the above object, the present invention adopts the following technical solution: a GPU resource scheduling method based on dynamic weight calculation, comprising:

[0008] Pre-run the algorithm according to the algorithm initial weights and the algorithm flow designed for the analysis task;

[0009] Obtain algorithm computing power resource data and conduct preliminary assessment of algorithm computing power resources;

[0010] According to the algorithm computing power resource data, pre-assessed algorithm computing power resources and algorithm task analysis, set the priority order and dynamic weight of the algorithm execution process, calculate the algorithm computing power resources and generate a dynamic scheduling plan;

[0011] Verify and adjust algorithm computing resources;

[0012] Based on the algorithm flow and dynamic scheduling plan, load the full amount of data to run the algorithm, execute the algorithm flow, and monitor the execution process of the algorithm flow.

[0013] Preferably, it also includes: after executing the algorithm process, returning the result and performing a resource performance evaluation for this operation, and storing the optimal execution record.

[0014] Preferably, the algorithm computing resources are pre-evaluated, including: pre-setting the graphics memory consumption formula and corresponding execution weights of machine learning and deep learning algorithms and dynamically obtaining GPU information. In this way, the resources required during the algorithm execution process can be accurately predicted and estimated according to the specific needs of the task.

[0015] Preferably, the total duration of the algorithm process is obtained based on the execution time of the pre-run;

[0016] The dynamic weight of the algorithm is calculated by the total duration of the algorithm process, the proportion of the duration of a single algorithm, and the GPU resource consumption.

[0017] Preferably, an algorithm process engine scheduling scenario is created to perform concurrent algorithm execution operations by multiple users;

[0018] Get the first execution algorithm of the algorithm process engine and add it to the task scheduling queue to verify the execution algorithm task.

[0019] Preferably, the algorithm computing resources are verified and adjusted, including:

[0020] According to the execution weight in the algorithm information and the execution weight of the existing tasks in the task scheduling queue, the tasks with high execution weight are prioritized for verification and execution;

[0021] In the task scheduling queue, determine whether the remaining resources of the current server are sufficient to execute the current algorithm;

[0022] If resources are sufficient, the current algorithm is executed; if resources are insufficient, the next algorithm task in the task scheduling queue is executed, and the current algorithm is reordered as a new task according to the execution weight and queued for verification execution.

[0023] Preferably, the execution process of the monitoring algorithm flow includes:

[0024] If the algorithm fails to execute, the algorithm process execution ends and returns an error message;

[0025] If the algorithm is executed successfully, the subordinate algorithm is obtained according to the algorithm's tag and added to the priority queue;

[0026] After the algorithm is successfully executed, if there are no lower-level components, it is determined that the algorithm process engine has been fully executed.

[0027] Preferably, the execution weight of the algorithm is calculated based on the current GPU real-time resource consumption, the current actual consumption of the algorithm, the consumption of other algorithms running in parallel with the current algorithm, and the estimated consumption conditions of the next level algorithm of the current algorithm.

[0028] Preferably, after executing the algorithm process, the result is returned and the resource performance of this operation is evaluated, and the optimal execution record is stored, including:

[0029] After each algorithm is executed, the execution record is stored; when the algorithm is used again, if the factors affecting the priority do not change, the existing algorithm priority settings are directly called; if the factors affecting the priority change, dynamic adjustments are made;

[0030] Evaluate each execution and historically stored records, and mark execution records for performance optimization.

[0031] Preferably, a GPU resource scheduling system based on dynamic weight calculation includes:

[0032] A receiving module, used for receiving algorithm tasks;

[0033] A calculation module is used to calculate the GPU memory consumption estimated by the algorithm task according to the preset memory consumption formula and algorithm execution weight;

[0034] The monitoring module is used to obtain the GPU information of the current server; the GPU information includes the total amount of video memory, utilization rate and remaining video memory;

[0035] The scheduling module is used to add tasks to the task scheduling queue according to the GPU information (remaining video memory) of the current server and the algorithm execution weight;

[0036] The execution module is used to execute algorithm tasks according to the task scheduling queue.

[0037] Through the above technical solutions, it can be known that compared with the prior art, the present invention discloses a GPU resource scheduling method and system based on dynamic weight calculation, including: pre-running the algorithm according to the algorithm initial weight and the algorithm flow designed for the analysis task; obtaining the algorithm computing power resource data, and pre-evaluating the algorithm computing power resources; setting the priority order and dynamic weight of the algorithm flow according to the algorithm computing power resource data, the pre-evaluated algorithm computing power resources and the algorithm task analysis, calculating the algorithm computing power resources and generating a dynamic scheduling scheme; verifying and adjusting the algorithm computing power resources; based on the algorithm flow and the dynamic scheduling scheme, loading the full amount of data to run the algorithm, executing the algorithm flow, and monitoring the execution process of the algorithm flow. The present invention can effectively solve the problem of satisfying as many users as possible to perform analysis tasks when the server and GPU are limited, without queuing or task failure due to resource competition, and ensure the efficient and stable operation of the algorithm platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0039] Figure 1 A schematic diagram of the dynamic scheduling process provided by the present invention.

[0040] Figure 2 This is a schematic diagram of the algorithm flow engine structure provided by the present invention.

[0041] Figure 3 A schematic diagram of a flow chart of a GPU resource scheduling method based on dynamic weight calculation provided by the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The embodiment of the present invention discloses a GPU resource scheduling method and system based on dynamic weight calculation, which can effectively solve the problem of satisfying as many users as possible to perform analysis tasks without queuing or task failure due to resource competition when the server and GPU are limited, thereby ensuring the efficient and stable operation of the algorithm platform.

[0044] The embodiment of the present invention discloses a GPU resource scheduling method based on dynamic weight calculation, such as Figure 1 As shown, including:

[0045] S100. Pre-run the algorithm according to the algorithm flow designed according to the initial weights of the algorithm and the analysis tasks; the algorithm pre-run mainly implements the data flow with a relatively small amount of data before the machine learning or deep learning tasks, and then the information on the algorithm's computing power usage can be extracted from the pre-run for preliminary evaluation and calculation. When it involves multi-user concurrent algorithm tasks or multi-algorithm tasks, the consumption of GPU computing power resources can be dynamically optimized based on the specific situation.

[0046] Among them, the dynamic optimization process: when there are multiple users concurrent algorithm tasks or multiple algorithm tasks, the system will first sort according to the algorithm weights transmitted by the application end and the existing task weights in the task queue, and tasks with higher weights will be executed first. In the task scheduling queue, the execution of the current algorithm task is based on the comparison between the estimated GPU video memory and the remaining GPU video memory of the server. The system determines whether the remaining GPU video memory of the current server is sufficient to execute the current algorithm task. If the resources are sufficient, the task will be executed; if the resources are insufficient, the system will schedule the next algorithm task in the queue for execution. The new task will enter the algorithm queue, and will be re-sorted according to the weight and queued for execution.

[0047] S200, obtaining algorithm computing power resource data, and pre-evaluating the algorithm computing power resources; estimating whether the algorithm computing power and video memory requirements can meet the operation, and if not, pre-scheduling management;

[0048] The algorithm uses GPU computing power information acquisition and resource consumption evaluation, involving GPU memory calculation, GPU information, etc.

[0049] S201, algorithm occupies GPU memory calculation

[0050] First, the embodiment of the present invention pre-sets a series of graphics memory consumption formulas and corresponding algorithm initial weights for machine learning and deep learning algorithms, so that the resources required during the execution process can be accurately predicted and estimated according to the specific requirements of the task.

[0051] Based on the machine learning and deep learning algorithms mentioned above, the use of video memory during model training involves not only model parameters, but also model weights, optimizers, gradients, and internal aspects of the computational graph:

[0052] (1) Model parameters: The structures of different models are different, and the parameter calculation methods are also different. The following will take the text classification model based on CNN (convolutional neural network) as an example:

[0053] The total parameter calculation formula = word vector mapping parameters + convolutional layer parameters + fully connected layer 1 parameters + fully connected layer 2 parameters.

[0054] The word vector mapping (EmbeddingLayer) layer parameter quantity = vocab_size*embedding_dim.

[0055] The vocabulary size (vocab_size) is 5000.

[0056] The word embedding dimension (embedding_dim) is 64.

[0057] Number of parameters: 5000*64=320,000.

[0058] The calculation formula of the convolutional layer parameters is expressed as:

[0059] Conv=(kernel_size*embedding_dim+1)*num_filters.

[0060] The convolution kernel size (kernel_size) is 5.

[0061] The number of input channels (embedding_dim) is 64.

[0062] The number of convolution kernels (num_filters) is 256.

[0063] Parameter quantity: (5*64+1)*256=(320+1)*256=321*256=8,192.

[0064] The parameter calculation formula of Fully Connected Layer 1 is expressed as:

[0065] F1=num_filters*hidden_dim+hidden_dim.

[0066] The number of input neurons (num_filters) is 256.

[0067] The number of output neurons (hidden_dim) is 128.

[0068] Parameter quantity: 256*128+128=32,768+128=32,896.

[0069] The parameter calculation formula of Fully Connected Layer 2 (i.e. classifier) ​​is expressed as:

[0070] F2=hidden_dim*num_classes+num_classes.

[0071] The number of input neurons (hidden_dim) is 128.

[0072] The number of output neurons (num_classes) is 10.

[0073] Parameter quantity: 128*10+10=1,280+10=1,290.

[0074] Total number of parameters = 320,000 + 8,192 + 32,896 + 1,290 = 352,376.

[0075] Therefore, this CNN-based text classification model has a total of 352,376 trainable model parameters.

[0076] (2) Model weight (execution weight): The calculation method of this part is the same as the model parameters.

[0077] (3) Optimizer: If the AdamW optimizer is selected, each parameter needs to occupy 8 bytes because two states need to be maintained. Therefore, the video memory used by the optimizer is twice the full-precision (float32) model weight. The embodiment of the present invention uses bitsandbytes optimized AdamW, and each parameter only needs to occupy 2 bytes, which is half of the full-precision (float32) model weight.

[0078] (4) Gradients: Gradients occupy the same amount of video memory as full-precision (float32) model weights.

[0079] (5) Variables inside the computational graph: Frameworks such as Pytorch or Tensorflow use graph structures for computation. Graph nodes need to be stored during forward and backward propagation, so they also need to occupy video memory. This part is related to the specific implementation of the model and batch_size. The larger the batch_size, the more video memory is occupied.

[0080] Therefore, considering only the first three parts, the graphics memory required for model training is at least 3-4 times that for inference. Combined with practical experience, the graphics memory required for training is usually more than 10 times that for actual computing.

[0081] The rough formula for video memory usage is as follows:

[0082] Video memory usage (MB) = 10*(number of model parameters + input data size*batch)*data type size / (1024bytes^2)

[0083] Note: Usually, the data type is float32 (single-precision floating point) or int32 (integer), which occupies 4B.

[0084] S202. Dynamically obtain GPU information

[0085] By dynamically acquiring GPU-related information of the server, such as video memory usage, the scheduling of tasks can be adjusted in real time to achieve the purpose of optimizing the use of video memory resources. In the embodiment of the present invention, the following information is mainly obtained through Python's pynvml module: the number of GPUs, the total video memory of each GPU, the used and remaining video memory (the proportion of space occupied when processing tasks), the current usage rate (the proportion of time occupied when processing tasks), the total power consumption / current power consumption, the GPU driver version, etc.

[0086] S300. According to the algorithm computing power resource data, the pre-assessed algorithm computing power resources and the algorithm task analysis, the priority order and dynamic weight of the execution algorithm process are set, the algorithm computing power resources are calculated and a dynamic scheduling plan is generated.

[0087] Among them, (1) algorithm priority definition rules

[0088] Algorithm priority needs to be set based on many factors, including but not limited to user needs, algorithm importance, resource utilization, algorithm real-time requirements, etc.

[0089] ① User requirements: Users may have different priority requirements for certain tasks. The system can adjust the execution order of the algorithm according to user requirements.

[0090] ② Algorithm importance: Some algorithms have urgent requirements and need to get processing results more quickly. Consider the urgency of the task when setting the algorithm priority.

[0091] ③ Resource utilization: Dynamically adjust the priority based on the resource utilization of the algorithm server system. When system resources are relatively idle, the priority of the algorithm can be adjusted to improve the overall system throughput.

[0092] ④ Algorithm real-time requirements: Some tasks have real-time requirements and set corresponding priorities. For tasks that require real-time response, higher priorities can be given.

[0093] (2) Factors affecting priority

[0094] Factors that affect algorithm priority include the following:

[0095] ① Data volume: The amount of data processed by the algorithm during execution is a key factor. Different algorithms have different sensitivities to data volume, and some algorithms may require more resources and time when facing large-scale data. The amount of data directly affects the execution efficiency of the algorithm, thereby affecting the priority of the algorithm.

[0096] ② Algorithm complexity: The complexity of different algorithms varies. Algorithm complexity describes the computing resources and time required for algorithm execution, usually expressed in Big O notation. The level of algorithm complexity directly affects the execution efficiency of the algorithm, and algorithm complexity is an important priority factor.

[0097] ③ Remaining resources of the algorithm server: During the execution process, the remaining resources of the algorithm server have an important impact on the execution priority of the algorithm. If the resources of the current algorithm server cannot meet the needs of a certain algorithm, the algorithm needs to wait and make a judgment until there are enough resources to start execution. The remaining resource status of the server is a key decision-making factor.

[0098] ④ Required resources: Different algorithms have different requirements for system resources. Some algorithms may require more computing resources, memory, or storage resources. Algorithms with higher resource requirements can be given higher priority so that they can complete execution in a shorter time.

[0099] ⑤GPU resource allocation: During GPU algorithm execution, dynamically obtain the GPU algorithm server resource status, determine whether the GPU algorithm service graphics cards have the video memory required by the algorithm, and use the video memory that meets the resources required by the algorithm to complete the execution.

[0100] S400, verify and adjust the algorithm computing resources; before executing the algorithm, check the algorithm server resources to evaluate whether the execution conditions are met. If the conditions are not met, add the algorithm back to the priority queue to control the effective use of resources.

[0101] S500, based on the algorithm flow and dynamic scheduling scheme, according to the priority and execution weight, load the full amount of data to run the algorithm, execute the algorithm flow, and monitor the execution process of the algorithm flow. A closed loop of algorithm execution management is implemented, and information and results in the calculation process are fed back. The algorithm flow engine that can efficiently manage and allocate computing resources provided by the embodiment of the present invention can estimate the required memory and video memory resources based on the current algorithm model and data, and then determine whether the current system resources support the operation of the next algorithm.

[0102] Specifically, it also includes: S600, after executing the algorithm process, returning the result and performing the resource performance evaluation for this operation, and storing the optimal execution record.

[0103] After the algorithm process is executed, the result is returned and the resource performance of this operation is evaluated, and the optimal execution record is stored, including:

[0104] After each algorithm is executed, the execution record is stored; when the algorithm is used again, if the factors affecting the priority do not change significantly, the existing algorithm priority settings are directly called; if the factors affecting the priority change, dynamic adjustments are made;

[0105] Evaluate each execution and historically stored records, and mark execution records for performance optimization.

[0106] The embodiment of the present invention designs a method for dynamically scheduling algorithm tasks based on the GPU resource usage of the calculation model during algorithm execution and the prediction of resource consumption in the next step. Different from the existing optimization of GPU scheduling during algorithm model execution, the embodiment of the present invention first designs a multi-index weighting method for scheduling:

[0107] (1) Initial (static) weight of the algorithm: The initial weight is set manually based on the importance of the algorithm in the system and the resource management requirements, and serves as an indicator for dynamic weighting during actual runtime;

[0108] (2) Dynamic weight of algorithm: A pre-run mechanism is designed. Before the task is officially run, a small amount of test data is loaded and quickly executed according to the actual parameters set by the algorithm. The dynamic weight of the algorithm is calculated based on the ratio of the total execution time of the entire task to the execution time of a single algorithm and the GPU resource consumption.

[0109] (3) Algorithm formal execution weight: The algorithm is loaded with full data and formally runs. The execution weight of the algorithm is calculated based on the current GPU real-time resource consumption, the actual consumption of the algorithm, the consumption of other algorithms running in parallel with the algorithm, and the estimated consumption conditions of the next level algorithm of the algorithm.

[0110] Based on this set of calculation methods, the system has precise control over the running weights and GPU resource usage of the algorithms to be run, the algorithms currently being run, and the next-level algorithms to be started after the end of the run. Based on the sufficiency of the weights and resources, the system schedules high-weight algorithms to enter the GPU for execution, suspends low-weight algorithms to wait for execution, and allocates the remaining GPU resources in a timely manner, forming a complete and efficient algorithm task scheduling engine that meets the needs of completing more and more efficient calculations under limited computing power.

[0111] Specifically, based on the execution time of the pre-run, the total duration of the algorithm process is obtained;

[0112] The dynamic weight of the algorithm is calculated by the total duration of the algorithm process, the proportion of the duration of a single algorithm, and the GPU resource consumption.

[0113] Specifically, the calculation method of dynamic weight is as follows:

[0114] Weighted Score=Default Weight +(Max Weight -Default Weight )×Sigmoid(T)×WG;

[0115] Among them, WG: the ratio of GPU resources required for the trial run to the GPU resources during the current execution. This is a value between 0 and 1, where 0 means no GPU resources are required and 1 means the required GPU resources are equal to all currently available GPU resources.

[0116] Default Weight : Default weight. This is the basic score of the algorithm without considering execution time and resource consumption.

[0117] Max Weight : Maximum weight, which is the upper limit of the highest score that the algorithm can obtain.

[0118] T: The ratio of algorithm execution time to total execution time, a value between 0 and 1.

[0119] Sigmoid(T): Sigmoid function, which maps T to a value between 0 and 1 and is used to smoothly adjust the score.

[0120] In machine learning and deep learning algorithm tasks, it is likely not a single algorithm task, but a workflow execution task that includes multiple components. Each component of the workflow is an algorithm task.

[0121] Regardless of whether it is a single algorithm or a workflow-based algorithm task mode, the initial weight of all algorithms is set to 1. Administrators can manually set weights based on the importance and necessity of the algorithms on the management side to control the priority order of algorithm execution. The larger the default weight value, the higher the execution priority. Administrators can customize the algorithm execution order as needed.

[0122] The dynamic weight of the algorithm is calculated based on the execution time of the system trial run. The overall duration of the algorithm process and the duration of a single algorithm are calculated as a ratio. The dynamic weight is recalculated for the default algorithm using a linear formula.

[0123] like Figure 3 As shown, in a specific embodiment of the present invention, when the algorithm is executed, it is first necessary to find the first component required by the algorithm. The system will request the priority queue according to the Web service client FeignClient mode, and add the algorithm information to be executed to the priority queue. The priority queue will be sorted according to the execution weight in the algorithm information, and the queue will perform the algorithm "task" operation on the algorithm with the largest weight. The algorithm "task" will query the service address of the algorithm, the required resources (memory, video memory) and other information according to the algorithm library. The algorithm server obtains whether the current server resource situation meets the resources required by the "task" algorithm task. If not, wait for a certain period of time and add the algorithm information to the priority queue server again. Wait for the next "task" operation. If the algorithm server situation meets the requirements of the algorithm, the execution operation is performed. If the algorithm is executed during the execution operation, the system will query the subordinate algorithm information and weight according to the association between the algorithms and add it to the priority queue. After sorting by the priority queue, the "task" operation is completed.

[0124] Specifically, create an algorithm process engine scheduling scenario to perform concurrent algorithm execution operations for multiple users;

[0125] Get the first execution algorithm of the algorithm process engine and add it to the task scheduling queue to verify the execution algorithm task.

[0126] Specifically, the priority queue is implemented based on the following two data types:

[0127] ①LinkedTransferQueue: It is an unbounded blocking transfer queue, and its basic structure is a linked list structure. This data type is suitable for programs with producer and consumer structures. In the actual use of the system, there can be one or more producers responsible for generating data, and one or more consumers responsible for processing this data. And this data structure is shared by producers and consumers. The producer adds new data to this data structure, and then the consumer takes the data out of the data structure for processing. If the data structure is empty, that is, there is no data available for consumption, the consumer will be blocked until new data is available for consumption. On the contrary, if the data structure is full, that is, there is no space to accommodate new data, the producer will be blocked until there is space in the data structure to place new data. This mechanism ensures the smooth flow and synchronous processing of data, avoiding data loss or waste of resources.

[0128] ②PriorityBlockingQueue: It is an unbounded blocking queue provided by the Java concurrent package. It is thread-safe. The elements in this queue are sorted according to their natural order or according to the Comparator provided during construction. The elements implement the Comparable interface with the compareTo() method. When the system inserts algorithmic data into the structure, it compares it with the data elements one by one until it finds its position.

[0129] In the embodiment of the present invention, the elements of LinkedTransferQueue are stored in the order of arrival, so the earlier the elements are, the earlier they are consumed. The system implements the compareTo() method of the PriorityBlockingQueue structure so that its consumption order is determined by priority rather than arrival time. The elements will be sorted according to their priority, and the higher level ones will be consumed first.

[0130] The algorithm identifier is a unique identifier assigned by the algorithm flow engine. When an element is consumed, the algorithm flow engine schedules and starts to execute the task. The algorithm library service obtains the algorithm details according to the algorithm identifier, including the server information where the algorithm is located and the resources required by the algorithm. The algorithm obtains the server information from the algorithm library service, and then obtains the resource utilization of the specified server according to its server information and FeignClient mode. If the remaining resources are sufficient for the current algorithm execution, the execution operation is performed. If the resources cannot meet the algorithm execution, a prompt message is returned, and the algorithm is re-added to the priority queue after sleeping for a specified period of time. The algorithm execution operation is started. If the algorithm execution fails, the algorithm flow execution ends and an error message is returned. If the algorithm execution succeeds, the subordinate algorithm is obtained according to the algorithm tag and added to the priority queue. If the algorithm execution succeeds, if there is no subordinate component, the algorithm flow engine will determine whether the execution is complete. If the algorithm is fully executed, the algorithm flow engine execution ends.

[0131] Specifically, the algorithm computing resources are verified and adjusted, including:

[0132] According to the execution weight in the algorithm information and the execution weight of the existing tasks in the task scheduling queue, the tasks with high execution weight are prioritized for verification and execution;

[0133] In the task scheduling queue, determine whether the remaining resources of the current server are sufficient to execute the current algorithm;

[0134] If resources are sufficient, the current algorithm is executed; if resources are insufficient, the next algorithm task in the task scheduling queue is executed, and the current algorithm is reordered as a new task according to the execution weight and queued for verification execution.

[0135] Due to limited system resources, algorithm task nodes that cannot be executed immediately need to be added to the waiting queue. When an algorithm process is completed, the system will first recycle resources, then take out an algorithm task from the queue, and determine whether the remaining resources of the current system meet the running conditions. If not, continue to wait, and if satisfied, execute immediately. The above scheduling ensures that all algorithm processes will not fail to execute due to system resource issues during execution.

[0136] Specifically, the execution process of the monitoring algorithm flow includes:

[0137] If the algorithm fails to execute, the algorithm process execution ends and returns an error message;

[0138] If the algorithm is executed successfully, the subordinate algorithm is obtained according to the algorithm's tag and added to the priority queue;

[0139] After the algorithm is successfully executed, if there are no lower-level components, it is determined that the algorithm process engine has been fully executed.

[0140] Specifically, the execution weight of the algorithm is calculated based on the current GPU real-time resource consumption, the current actual consumption of the algorithm, the consumption of other algorithms running in parallel with the current algorithm, and the estimated consumption conditions of the next level algorithm of the current algorithm.

[0141] The embodiment of the present invention discloses a GPU resource scheduling method based on dynamic weight calculation, which sets an estimated default weight according to the GPU computing power and video memory resources required when the algorithm is running, and dynamically calculates and determines the real-time weight of the algorithm according to the matching of the GPU computing power and video memory actually required by the algorithm with the currently available resources, the correlation between the algorithm and the previous and next processes in the task flow, and the relative importance of different tasks, so as to achieve scheduling and optimization of the algorithm execution order. In the embodiment of the present invention, the initial weight is the system's default execution ranking value for the algorithm (set according to the algorithm complexity and time consumption); the dynamic weight is calculated by the system according to the actual resource utilization ratio and the time consumption of the algorithm during the trial operation of the algorithm. The execution weight is a general term for the initial weight and the dynamic weight.

[0142] Specifically, a GPU resource scheduling system based on dynamic weight calculation, such as Figure 2 As shown, including:

[0143] A receiving module is used to receive an algorithm task; the algorithm task is generated by a client, and the algorithm call and data parameters of the algorithm task are sent to the receiving module;

[0144] A calculation module is used to calculate the GPU memory consumption estimated by the algorithm task according to the preset memory consumption formula and algorithm execution weight;

[0145] The monitoring module is used to obtain the GPU information of the current server; the GPU information includes the total amount of video memory, utilization rate and remaining video memory;

[0146] The scheduling module is used to add tasks to the task scheduling queue according to the remaining video memory and the algorithm execution weight;

[0147] The execution module is used to execute algorithm tasks according to the task scheduling queue.

[0148] In an embodiment of the present invention, in order to improve the efficiency of task execution, a task scheduling queue is introduced. When a new algorithm task is generated, the system will first sort it according to the algorithm weight transmitted by the application end and the existing task weights in the task scheduling queue, and tasks with higher weights will be executed first. In the task scheduling queue, the execution of the current algorithm task will be based on the comparison between the estimated GPU video memory and the remaining GPU video memory of the server. The system will determine whether the remaining GPU video memory of the current server is sufficient to execute the current algorithm task. If the resources are sufficient, the task will be executed; if the resources are insufficient, the system will schedule the next algorithm task in the queue for execution. The new task will enter the algorithm queue, and will be reordered according to the weight and queued for execution. Through this logic, the embodiment of the present invention ensures that the tasks can be executed reasonably and orderly, thereby improving the overall task execution efficiency.

[0149] The technical key points in the embodiment of the present invention are about the scheduling processing logic of the task scheduling queue, which specifically includes the following key points:

[0150] Algorithm elements:

[0151] (1) Algorithm weight: transmitted through the application end;

[0152] (2) The amount of GPU memory required by the algorithm: calculated using the algorithm GPU memory calculation formula;

[0153] (3) Current remaining GPU memory: obtained by monitoring the GPU-related information of the server in real time.

[0154] The processing logic of the embodiment of the present invention is as follows:

[0155] (1) When a new algorithm task is generated, the system will first sort it with the existing algorithms in the task queue according to the weight value transmitted by the task application. Tasks with higher weights will be executed first.

[0156] (2) Calculate the amount of GPU memory required by the algorithms in all task queues.

[0157] (3) When executing an algorithm in the algorithm queue, obtain the current remaining GPU memory in real time.

[0158] (4) Compare the amount of GPU memory required by the algorithm to be executed and the amount of GPU memory remaining.

[0159] (5) Determine whether the remaining GPU memory of the current server is sufficient to execute the current algorithm task. If the resources are sufficient, the task is executed; if the resources are insufficient, the system will schedule the next algorithm task in the queue for execution.

[0160] (6) Re-insert the algorithm tasks that do not meet the execution requirements into the task queue and re-enter step (1).

[0161] In general, the technical solution of the embodiment of the present invention uses the scheduling processing logic of the task scheduling queue through three factors: algorithm weight, GPU video memory required by the algorithm, and the current remaining GPU video memory, to optimize the use of resources during task execution and improve the efficiency of GPU video memory resource utilization.

[0162] In a specific embodiment of the present invention, Figure 3 As shown, in the algorithm process engine scheduling scenario, multiple users concurrently perform algorithm execution operations to obtain the first execution algorithm of the algorithm process engine.

[0163] Add the first execution algorithm obtained to the priority queue; in the priority queue service, it includes the entry position and the exit position; sort by dynamic weight, the larger the dynamic weight, the closer to the exit position, the smaller the dynamic weight, the closer to the entry position; when inserting algorithm data, compare the dynamic weight of the current algorithm with all weight data elements in the priority queue service one by one until its position is found. According to their priority sorting, the one with high dynamic weight will be consumed first; when the element is consumed, start verifying the execution algorithm task.

[0164] The algorithm library service obtains the algorithm details according to the algorithm identifier; the algorithm library service includes the algorithm library for machine learning or deep learning; the algorithm details include the server information where the algorithm is located, the resources required by the algorithm, etc. The algorithm obtains the server information from the algorithm library service, and then obtains the resource utilization of the specified server according to its server information and FeignClient mode.

[0165] If the remaining resources cannot satisfy the algorithm execution, a prompt message is returned, and this algorithm (for example, algorithm m, whose corresponding weight value is n) is re-added to the priority queue after sleeping for a specified period of time.

[0166] If the remaining resources are sufficient for the current algorithm to execute, the algorithm execution operation is performed; after the algorithm execution operation is started, if the algorithm execution fails, the algorithm process execution ends and an error message is returned; if the algorithm execution succeeds, the subordinate algorithm is obtained according to the algorithm tag and added to the priority queue (at this time, the subordinate algorithm has the highest dynamic weight, and it directly passes through the priority queue and performs the algorithm execution operation). If the algorithm execution succeeds and there are no subordinate components, the algorithm process engine will determine whether the execution is complete. If the algorithm is fully executed, the algorithm process engine execution ends.

[0167] In a specific embodiment of the present invention, some of the workflow algorithms built into the system require the use of GPU resources, because GPUs can provide faster processing speeds than CPU resources. In particular, the effect of GPUs is obvious for the training and reasoning of machine learning models, as well as algorithms such as image processing. For example, for resources using CNN (convolutional neural network), when processing the same data and logic, the computing power of GPUs is significantly higher than that of CPUs, so data can be processed faster.

[0168] The embodiment of the present invention trains the CNN (convolutional neural network) algorithm according to different computing power resources of the CPU and GPU, and performs 5 complete training iterations on the same set of CIFAR-10 training data sets. It can be seen from Table 1 that in the training of the CNN algorithm, it takes 66.98 seconds to perform 5 iterations on the entire data set using the Tesla P100-PCIE-16GB GPU. It takes 86.18 seconds to perform 5 iterations on the entire data set using an 88-thread CPU. It can be seen that under different resource computing power conditions, the GPU computing power resources have improved the execution speed of the algorithm by nearly 20%.

[0169] Table 1

[0170] Hash Type Iteration times Time consumed (seconds) CPU 5 86.18 GPU 5 66.98 CPU 20 329.34 GPU 20 247.70 CPU 100 1698.67 GPU 100 1212.08

[0171] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0172] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A GPU resource scheduling method based on dynamic weight calculation, characterized in that: include: Pre-run the algorithm according to the algorithm initial weights and the algorithm flow designed for the analysis task; Obtain algorithm computing power resource data and conduct preliminary assessment of algorithm computing power resources; According to the algorithm computing power resource data, pre-assessed algorithm computing power resources and algorithm task analysis, set the priority order and dynamic weight of the algorithm execution process, calculate the algorithm computing power resources and generate a dynamic scheduling plan; Verify and adjust algorithm computing resources; Based on the algorithm flow and dynamic scheduling plan, load the full amount of data to run the algorithm, execute the algorithm flow, and monitor the execution process of the algorithm flow.

2. The GPU resource scheduling method based on dynamic weight calculation according to claim 1, characterized in that: Also includes: After the algorithm process is executed, the results are returned and the performance of the resources for this operation is evaluated, and the optimal execution record is stored.

3. The GPU resource scheduling method based on dynamic weight calculation according to claim 1, characterized in that: Pre-evaluate the algorithm computing resources, including: pre-setting the memory consumption formula and corresponding execution weights of machine learning and deep learning algorithms and dynamically obtaining GPU information.

4. The GPU resource scheduling method based on dynamic weight calculation according to claim 1, characterized in that: Based on the execution time of the pre-run, the total duration of the algorithm process is obtained; The dynamic weight of the algorithm is calculated by the total duration of the algorithm process, the proportion of the duration of a single algorithm, and the GPU resource consumption.

5. The GPU resource scheduling method based on dynamic weight calculation according to claim 1, characterized in that: Create algorithm process engine scheduling scenarios to perform concurrent algorithm execution operations for multiple users; Get the first execution algorithm of the algorithm process engine and add it to the task scheduling queue to verify the execution algorithm task.

6. The GPU resource scheduling method based on dynamic weight calculation according to claim 5, characterized in that: Verify and adjust the algorithm computing resources, including: According to the execution weight in the algorithm information and the execution weight of the existing tasks in the task scheduling queue, the tasks with high execution weight are prioritized for verification and execution; In the task scheduling queue, determine whether the remaining resources of the current server are sufficient to execute the current algorithm; If resources are sufficient, the current algorithm is executed; if resources are insufficient, the next algorithm task in the task scheduling queue is verified and the current algorithm is reordered as a new task according to the execution weight and queued for verification execution.

7. The GPU resource scheduling method based on dynamic weight calculation according to claim 6, characterized in that: The execution process of the monitoring algorithm flow includes: If the algorithm fails to execute, the algorithm process execution ends and returns an error message; If the algorithm is executed successfully, the subordinate algorithm is obtained according to the algorithm's tag and added to the priority queue; After the algorithm is successfully executed, if there are no lower-level components, it is determined that the algorithm process engine has been fully executed.

8. The GPU resource scheduling method based on dynamic weight calculation according to claim 6, characterized in that: The execution weight of the algorithm is calculated based on the current GPU real-time resource consumption, the current actual consumption of the algorithm, the consumption of other algorithms running in parallel with the current algorithm, and the estimated consumption of the next-level algorithm of the current algorithm.

9. The GPU resource scheduling method based on dynamic weight calculation according to claim 2, characterized in that: After the algorithm process is executed, the results are returned and the resource performance of this operation is evaluated, and the optimal execution record is stored, including: After each algorithm is executed, the execution record is stored; when the algorithm is used again, if the factors affecting the priority do not change, the existing algorithm priority settings are directly called; if the factors affecting the priority change, dynamic adjustments are made; Evaluate each execution and historically stored records, and mark execution records for performance optimization.

10. A GPU resource scheduling system based on dynamic weight calculation, characterized in that: include: A receiving module, used for receiving algorithm tasks; A calculation module is used to calculate the GPU memory consumption estimated by the algorithm task according to the preset memory consumption formula and algorithm execution weight; Monitoring module, used to obtain GPU information of the current server; The scheduling module is used to add tasks to the task scheduling queue according to the GPU information and algorithm execution weight of the current server; The execution module is used to execute algorithm tasks according to the task scheduling queue.

Images