A computing resource allocation method for parallel processing of large models

Abstract

The present application relates to the field of data processing technology, and specifically to a method for allocating computing power resources for parallel processing of large models, the method comprising: obtaining the execution time, calculation results, required storage capacity and storage capacity of each historical computing task in each computing device; determining the execution efficiency of each historical computing task in each computing device; determining the long-term overload operation capability of each computing device; determining the computing power allocation priority of each computing task, so as to allocate computing power resources to each computing task. The present application allocates computing power resources to computing tasks based on the probability of computing power allocation between computing tasks and arbitrary computing power devices, combined with a probability selection model, which can reduce the possibility of computing power equipment paralysis when computing power resources are allocated, and improve the overall efficiency of large model training.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a computing resource allocation method for parallel processing of large models. Background Art

[0002] With the development of Internet and Internet of Things technologies, the amount of data that can be collected is growing exponentially. At the same time, in order to solve complex problems in the real world, more advanced and larger-scale model architectures are often required. The expansion of the scale of large models will increase the difficulty of model training. The traditional single-machine single-card mode can no longer meet the requirements for large model training. The limitation of computing power resources has become a major bottleneck for large model training. Large model parallel processing is achieved by allocating the architecture of large models to different computing power devices. It is necessary to consider the matching of computing power resources and training task volume, match computing power resources, and realize distributed training of large models.

[0003] Existing computing resource allocation algorithms include random allocation algorithm and polling allocation algorithm. The advantage of random allocation algorithm is that it is easy to implement and suitable for small model training; however, when applied to large model training, since the random allocation algorithm randomly allocates tasks, there is a problem of not being able to balance the load tasks, which can easily cause computing equipment to be overloaded or even paralyzed. When using the polling allocation algorithm, although the risk of computing equipment paralysis is low, there is a problem that the computing resources of high-performance computing nodes cannot be fully utilized. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a computing resource allocation method for parallel processing of large models to solve the existing problems.

[0005] A computing resource allocation method for large model parallel processing in this application adopts the following technical solution:

[0006] An embodiment of the present application provides a computing resource allocation method for large model parallel processing, the method comprising the following steps:

[0007] Obtain the execution time, calculation results, required storage capacity and available storage capacity of each historical computing task in each computing device;

[0008] Determining the execution efficiency of each historical computing task in each computing device based on the required storage capacity of each historical computing task in each computing device and the storable capacity;

[0009] The storage capacity required in each computing device is greater than the segmentation threshold, and the historical computing tasks that are continuous in time constitute the long-term heavy load group of each computing device;

[0010] Determine the long-term heavy-load operation capability of each computing device based on the change trend of the execution efficiency, execution time, required storage capacity and calculation results of all historical computing tasks in each long-term heavy-load group in each computing device;

[0011] Obtain the release time, processing start time, current calculation time and required storage capacity of each computing task in each stage of the large model;

[0012] Based on the release time, processing start time, current computing time and required storage capacity of each computing task, the computing power allocation priority of each computing task is determined, and the computing power resources are allocated to each computing task in combination with the storage capacity and the long-term heavy-load operation capability.

[0013] Preferably, the method for determining the execution efficiency of each historical computing task in each computing device is:

[0014] Analyze the ratio of the required storage capacity of each historical computing task in each computing device to the cumulative result of the required storage capacity of all historical computing tasks, and record it as the workload ratio of each historical computing task in each computing device;

[0015] Analyze the ratio of the required storage capacity of each historical computing task in each computing device to the available storage capacity of the corresponding computing device, and record it as the resource utilization rate of each historical computing task in each computing device;

[0016] The expression of the execution efficiency of each historical computing task in each computing device is: ; In the formula, is the execution efficiency of the jth historical computing task in the i-th computing device; is the resource usage rate of the jth historical computing task in the i-th computing device; It is the workload ratio of the jth historical computing task in the i-th computing device.

[0017] Preferably, the process of determining the long-term heavy-load operation capability of each computing device is as follows:

[0018] Determine the paralysis possibility of each computing device based on the change trend of the execution efficiency of all historical computing tasks in all long-term overload groups in each computing device and the execution time of all historical computing tasks;

[0019] Determine the long-term overload reliability of each computing device based on the required storage capacity and calculation results of all historical computing tasks in all long-term overload groups in each computing device;

[0020] The long-term heavy-load operation capability of each computing device is the ratio of the long-term heavy-load reliability of each computing device to the possibility of paralysis.

[0021] Preferably, the method for determining the paralysis possibility of each computing power device is:

[0022] Based on the changing trend of the execution efficiency of all historical computing tasks in each long-term overload group in each computing device, determine the efficiency loss of each long-term overload group in each computing device;

[0023] Analyze the sum of the execution time of all historical computing tasks whose resource usage rate in each long-term overload group in each computing device is greater than the preset threshold, and record it as the overload time of each long-term overload group in each computing device;

[0024] The probability of paralysis of each computing device is the result of the fusion of the efficiency loss of all long-term overload groups in each computing device and the overload duration.

[0025] Preferably, the method for determining the efficiency loss of each long-term heavy-load group in each computing power device is:

[0026] The execution efficiency of all historical computing tasks in each long-term overload group in each computing device is used as the input of the slope estimation algorithm to obtain the slope estimation value of each long-term overload group in each computing device;

[0027] The expression for the efficiency loss of each long-term overload group in each computing power device is: ; Indicates the efficiency loss of the kth long-term overload group in the i-th computing device; Represents the slope estimate of the k-th long-term overload group in the i-th computing device.

[0028] Preferably, the method for determining the long-term overload reliability of each computing device is:

[0029] If the calculation results of each historical calculation task in each computing power device are correct, the correctness of each historical calculation task in each computing power device is 1, otherwise, the correctness is -1;

[0030] Analyze the product of the workload ratio and correctness of each historical computing task in each computing device, and record it as the computing contribution value of each historical computing task in each computing device;

[0031] Analyze the ratio of the cumulative sum of the calculation contribution values ​​of all historical calculation tasks in each long-term overload group in each computing power device to the cumulative sum of the required storage capacity, and record it as the contribution efficiency ratio of each long-term overload group in each computing power device;

[0032] The long-term overload reliability of each computing device is the average of the contribution efficiency ratios of all long-term overload groups in each computing device.

[0033] Preferably, the method for determining the computing power allocation priority of each computing task is:

[0034] Analyze the difference between the processing start time and the release time of each computing task, and record it as the waiting time of each computing task;

[0035] Analyze the difference between the current calculation time and the processing start time of each calculation task, and record it as the processing time of each calculation task;

[0036] Analyze the ratio of the processing time of each computing task to the cumulative processing time of all computing tasks, and record it as the task completion degree of each computing task;

[0037] Based on the waiting time, task completion and required storage capacity of each computing task, the computing power allocation priority of each computing task is determined.

[0038] Preferably, determining the priority of computing power allocation for each computing task based on the waiting time, task completion degree and required storage capacity of each computing task further includes:

[0039] Analyze the product of the waiting time of each computing task and the task completion degree, and record it as the execution urgency of each computing task;

[0040] The computing power allocation priority of each computing task is the result of the fusion of the execution urgency of each computing task and the required storage capacity.

[0041] Preferably, the allocating computing resources to each computing task includes:

[0042] Based on the required storage capacity of each computing task and the storage capacity of each computing device, determine the volume adaptation between each computing task and any computing device, and use the product of the volume adaptation and the long-term heavy-load operation capability of the corresponding computing device as the input of the activation function to obtain the computing power allocation probability between each computing task and any computing device;

[0043] According to the order of the computing power allocation priority of each computing task from large to small, the computing power allocation probability between each computing task and all computing power devices is used as the input of the probability selection model, and one computing power device is output at a time. If the storage capacity of the computing power device is greater than the required storage capacity of the computing task, stop allocating additional computing power devices to the computing task. Otherwise, allocate additional computing power devices to the computing task.

[0044] Preferably, the expression of the volume adaptability between each computing task and any computing power device is: ; In the formula, Indicates the volume adaptability between the nth computing task and the i-th computing device; Indicates the required storage capacity of the nth computing task; Represents the storage capacity of the i-th computing device; Represents a second preset value, where the second preset value is a positive number less than 1.

[0045] This application has at least the following beneficial effects:

[0046] This application is based on the long-term overload paralysis possibility and long-term overload reliability of any computing power device, and can obtain the long-term overload operation capability of the computing power device. Its beneficial effect is to comprehensively consider the technical accuracy of the computing power device, avoid considering only the paralysis possibility of the computing power device as the basis for the allocation of computing power resources, which may easily lead to the long-term output of erroneous calculation results by computing power devices with low calculation accuracy, resulting in a huge risk of error propagation, and improving the accuracy of subsequent computing power resource allocation.

[0047] This application is based on the volume adaptability between computing tasks and arbitrary computing power equipment and the long-term overload reliability of computing power equipment, and can obtain the computing power allocation probability between computing tasks and arbitrary computing power equipment. The beneficial effect is to avoid using computing power equipment with larger storage capacity for computing tasks with smaller workloads, resulting in waste of computing power equipment resources, and improve the overall efficiency of large model training.

[0048] This application allocates computing power resources through a probabilistic selection model based on the probability of computing power allocation between computing tasks and arbitrary computing power devices. The beneficial effect is that the probabilistic selection model is used to avoid long-term use of computing power devices and reduce the possibility of computing power devices being paralyzed when computing power resources are allocated. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 A flowchart of the steps of a computing resource allocation method for parallel processing of large models;

[0051] Figure 2 A schematic diagram of the paralysis possibility extraction process;

[0052] Figure 3 Flowchart of steps for paralysis potential analysis;

[0053] Figure 4 This is a schematic diagram of the long-term heavy-load operation capacity extraction process;

[0054] Figure 5 Schematic diagram of the large model computing resource allocation process. DETAILED DESCRIPTION

[0055] In order to further explain the technical means and effects adopted by this application to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a computing resource allocation method for large model parallel processing proposed in this application in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0056] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0057] The following is a detailed description of a specific solution of a computing resource allocation method for large model parallel processing provided by the present application in conjunction with the accompanying drawings.

[0058] An embodiment of the present application provides a computing resource allocation method for large model parallel processing. Specifically, the following computing resource allocation method for large model parallel processing is provided. Figure 1 , the method comprises the following steps:

[0059] Step S1: Obtain relevant data of computing tasks and computing power equipment through the monitoring platform.

[0060] This embodiment uses a pipeline parallel algorithm to divide the large model into several stages, and allocates different computing power devices to each stage for calculation.

[0061] A data parallel algorithm is used to divide the training tasks in each stage of the large model into several computing tasks, and computing power resources are allocated to all computing tasks in each stage.

[0062] Among them, the pipeline parallel algorithm and data parallel technology are both well-known technologies, and the specific process will not be repeated here.

[0063] This embodiment sets up a monitoring platform during the large-model distributed training process to perform data monitoring and computing resource allocation for computing tasks and computing power equipment.

[0064] Furthermore, the execution time, calculation results and required storage capacity of a preset number of historical computing tasks in each computing device are obtained, as well as the storage capacity of each computing device.

[0065] Among them, the difference between the processing end time and the processing start time of each historical computing task in each computing power device is recorded as the execution time of each historical computing task in each computing power device.

[0066] Obtain the release time, processing start time, current calculation time and required storage capacity of each computing task in each stage of the large model.

[0067] It should be noted that the value of the preset number is artificially set. In this embodiment, the value of the preset number is 100. The implementer can also set it according to the specific situation. This embodiment does not impose any special restrictions.

[0068] Step S2: Determine the paralysis possibility of each computing device based on the change trend of the execution efficiency and execution time of all historical computing tasks in each long-term overload group in each computing device.

[0069] During the distributed training of large models, the collaborative work between a large number of computing devices can reasonably allocate computing resources and improve computing resource utilization to reduce the training time of large models.

[0070] However, due to the differences in processor heat dissipation and usage time of different computing devices, the processing performance of different computing devices for computing tasks is also different. If only the storage capacity of the computing device is used as an indicator of the performance of the computing device, it is easy to cause computing devices with high storage capacity but low processing performance to have low processing efficiency when dealing with tasks with large workloads, making the parallel processing of large models inefficient.

[0071] The required storage capacity of all historical computing tasks in each computing device is used as the input of the threshold segmentation algorithm to obtain the segmentation threshold;

[0072] The required storage capacity of each computing device is greater than the segmentation threshold, and the historical computing tasks that are continuous in time form each long-term overload group of each computing device. If there is no long-term overload group in the computing device, the computing tasks in the device will not be analyzed.

[0073] It should be noted that there are many commonly used threshold segmentation algorithms. In this embodiment, the Otsu threshold segmentation algorithm is used to obtain the segmentation threshold to classify historical computing tasks. The implementer may also use other threshold segmentation algorithms to classify historical computing tasks according to specific circumstances. This embodiment does not impose any special restrictions.

[0074] S201: Determine the execution efficiency of each historical computing task in each computing device based on the required storage capacity of each historical computing task in each computing device and the storable capacity.

[0075] In the process of large model parallel processing, in order to improve the efficiency of large model training, computing devices will be continuously assigned large workload computing tasks. As low-performance computing devices process computing tasks for a long time, their execution efficiency will drop significantly. Therefore, when allocating computing resources, the execution efficiency of computing devices under long-term heavy-load operation conditions should be considered to avoid long-term assignment of large workload computing tasks that may cause computing devices to be paralyzed.

[0076] Therefore, based on the required storage capacity of each historical computing task in each computing device and the storable capacity, the execution efficiency of each historical computing task in each computing device is determined to judge the execution efficiency of the computing device under long-term heavy-load operation conditions, specifically:

[0077] Analyze the ratio of the required storage capacity of each historical computing task in each computing device to the cumulative result of the required storage capacity of all historical computing tasks, and record it as the workload ratio of each historical computing task in each computing device;

[0078] Analyze the ratio of the required storage capacity of each historical computing task in each computing device to the available storage capacity of the corresponding computing device, and record it as the resource utilization rate of each historical computing task in each computing device;

[0079] The expression of the execution efficiency of each historical computing task in each computing device is: ; In the formula, is the execution efficiency of the jth historical computing task in the i-th computing device; is the resource usage rate of the jth historical computing task in the i-th computing device; It is the workload ratio of the jth historical computing task in the i-th computing device.

[0080] When the workload ratio is higher and the resource utilization rate is lower, it means that the computing power equipment uses less resource consumption to complete more workload, and the efficiency of completing historical computing tasks is higher; conversely, when the workload ratio is lower and the resource utilization rate is higher, it means that the efficiency of the historical computing tasks of the computing power equipment is lower.

[0081] S202: Determine the efficiency loss of each long-term overload group in each computing power device based on the change trend of the execution efficiency of all historical computing tasks in each long-term overload group in each computing power device.

[0082] In order to quantify the impact of long-term overload on the performance of computing devices, the efficiency loss of each long-term overload group in each computing device is determined by analyzing the change trend of the execution efficiency of all historical computing tasks in each long-term overload group in each computing device, so as to evaluate the health of the computing devices under long-term overload, identify the devices with excessive load, and then adjust the task allocation strategy to avoid allocating too many tasks to the devices that are already in an overloaded state, reduce resource waste and performance loss, specifically:

[0083] The execution efficiency of all historical computing tasks in each long-term overload group in each computing power device is arranged in ascending order of the processing start time, and the arrangement result is used as the input of the Sen slope estimation algorithm to output the slope estimation value of each long-term overload group in each computing power device.

[0084] Furthermore, based on the changing trend of the execution efficiency of all historical computing tasks in each long-term overload group in each computing device, the efficiency loss of the long-term overload group in each computing device is determined to optimize resource allocation, specifically:

[0085] The expression for the efficiency loss of each long-term overload group in each computing power device is: ; Indicates the efficiency loss of the kth long-term overload group in the i-th computing device; Represents the slope estimate of the k-th long-term overload group in the i-th computing device.

[0086] When the slope estimate is greater than or equal to 0, it means that the efficiency loss of the long-term overload group of computing power equipment is not significant, and the efficiency loss degree is equal to the first preset value. When the slope estimate is less than 0, it means that the task execution efficiency of the long-term overload group of computing power equipment is getting lower and lower with the long-term calculation of large workload, and the efficiency loss of the long-term overload group is becoming more and more obvious.

[0087] S203: Based on the execution time of all historical computing tasks in each long-term overload group in each computing device, combined with the efficiency loss, determine the paralysis possibility of each computing device.

[0088] In order to assess the risk of performance failure of each computing device under long-term overload conditions, the execution efficiency change trend and execution time of all historical computing tasks in each long-term overload group of each computing device are analyzed to determine the possibility of paralysis of each computing device, so as to predict and deal with possible equipment paralysis problems in advance, optimize resource allocation, avoid over-reliance on high-risk devices, and ensure the overall stability and reliability of the system. Specifically:

[0089] A preset threshold is set, and the sum of the execution durations of all historical computing tasks whose resource usage rate in each long-term overload group in each computing device is greater than the preset threshold is counted, and recorded as the overload duration of each long-term overload group in each computing device.

[0090] It should be noted that the value of the preset threshold is artificially set. In this embodiment, the value of the preset threshold is 0.9. The implementer can set it according to the specific situation. This embodiment does not impose any special restrictions.

[0091] If the sum of the execution times of all historical computing tasks in any long-term overload group whose resource utilization is greater than the preset threshold is 0, the overload duration takes the first preset value. The longer the overload duration, the higher the resource utilization of the computing device, and the greater the possibility of paralysis caused by long-term overload.

[0092] It should be understood that the value of the first preset value is artificially set, and the first preset value is a real number greater than 0. In this embodiment, the value of the first preset value is 1, and the implementer can also set it according to the specific situation.

[0093] Based on the execution time of all historical computing tasks in each long-term overload group in each computing device, combined with the efficiency loss, the possibility of paralysis of each computing device is determined to discover devices with problems due to long-term overload in advance, thereby optimizing resource allocation to reduce the risk of device failure, specifically:

[0094] The probability of paralysis of each computing device is the result of the fusion of the efficiency loss and overload duration of all long-term overload groups in each computing device.

[0095] It should be understood that fusion refers to combining two or more indicators by adding or multiplying them together in order to obtain a comprehensive indicator, thereby more comprehensively and accurately evaluating a phenomenon or problem. This fusion method is not limited to simple arithmetic operations, but can also include more complex statistical models and analysis methods. The implementer can choose according to the specific situation, and this embodiment does not impose any special restrictions.

[0096] Preferably, as an embodiment of the present application, the method for determining the possibility of paralysis of each computing device is:

[0097] Analyze the product of the efficiency loss and overload duration of each long-term overload group in each computing device, and record it as the first product of each long-term overload group in each computing device;

[0098] The probability of paralysis of each computing device is the cumulative sum of the first products of all long-term overload groups in each computing device.

[0099] The greater the efficiency loss, the more resources the computing device needs to consume to complete the workload under long-term heavy load. Allocating a large workload to the computing device for a long time can easily cause the computing device to be in an overloaded state for a long time, causing the computing device to be paralyzed. The possibility of paralysis due to long-term heavy load is greater.

[0100] Preferably, as an embodiment of the present application, the schematic diagram of the paralysis possibility extraction process is as follows: Figure 2 shown.

[0101] Preferably, as an embodiment of the present application, the paralysis possibility analysis step flow chart is as follows: Figure 3 shown.

[0102] Step S3: Determine the long-term overload operation capability of each computing device based on the change trend of the execution efficiency, execution time, required storage capacity and calculation results of all historical computing tasks in each long-term overload group in each computing device.

[0103] During the parallel processing of large models, the calculation error of one computing device may affect the calculation results of the computing devices in the next stage. The calculation accuracy of the computing device is directly related to the quality of the large model training results. If only the possibility of computing device failure is considered as the basis for computing resource allocation, it is easy to cause computing devices with low calculation accuracy to output wrong calculation results for a long time, resulting in a great risk of error propagation. Even if the probability of computing device failure is low and the training effect is fast, the final large model training results may be unreliable.

[0104] S301: Determine the long-term overload reliability of each computing device based on the required storage capacity and calculation results of all historical computing tasks in each long-term overload group in each computing device.

[0105] In order to evaluate and monitor the performance stability and accuracy of computing equipment under long-term high-load working conditions, the workload proportion and accuracy of all historical computing tasks in each long-term heavy-load group of each computing equipment are analyzed, and the calculation results are combined with the required storage capacity of each historical computing task in each computing equipment to determine the long-term heavy-load reliability of each computing equipment, so as to monitor the performance changes of computing equipment under continuous high-load working, ensure that the performance of computing equipment will not be reduced due to long-term working, and maintain and upgrade computing equipment in advance, optimize resource allocation to reduce errors in the calculation process, specifically:

[0106] If the calculation results of each historical calculation task in each computing power device are correct, the correctness of each historical calculation task in each computing power device is 1, otherwise, the correctness is -1;

[0107] Analyze the product of the workload ratio and correctness of each historical computing task in each computing device, and record it as the computing contribution value of each historical computing task in each computing device;

[0108] When the calculation contribution value is greater than 0, it means that the calculation history tasks of the computing power equipment have a higher positive promoting effect on the parallel processing of large models. When the calculation contribution value is less than 0, it means that the calculation history tasks of the computing power equipment have a higher error propagation effect on the parallel processing of large models.

[0109] Furthermore, based on the required storage capacity and calculation results of all historical computing tasks in each long-term overload group in each computing device, the long-term overload reliability of each computing device is determined to optimize resource allocation and reduce errors in the computing process, specifically:

[0110] Analyze the ratio of the cumulative sum of the calculation contribution values ​​of all historical calculation tasks in each long-term overload group in each computing power device to the cumulative sum of the required storage capacity, and record it as the contribution efficiency ratio of each long-term overload group in each computing power device;

[0111] The long-term overload reliability of each computing device is the average of the contribution efficiency ratios of all long-term overload groups in each computing device.

[0112] The larger the contribution efficiency ratio, the higher the accuracy of the computing power equipment in completing the workload under long-term overload, the more reliable the computing power equipment is in performing tasks under long-term overload, and the greater the long-term overload reliability of the computing power equipment; conversely, the smaller the contribution efficiency ratio, the lower the efficiency of the computing power equipment in completing the workload under long-term overload, the lower the reliability of the computing power equipment in performing tasks under long-term overload, and the smaller the long-term overload reliability of the computing power equipment.

[0113] S302: Determine the long-term heavy-load operation capability of each computing device based on the paralysis possibility and the long-term heavy-load reliability.

[0114] Based on the paralysis possibility and the long-term overload reliability, the long-term overload operation capacity of each computing device is determined to comprehensively evaluate the risk of failure of the computing device under long-term overload, so as to take corresponding preventive measures to maintain the computing device and improve the long-term working capacity of the computing device, specifically:

[0115] The long-term heavy-load operation capability of each computing device is the ratio of the long-term heavy-load reliability of each computing device to the possibility of paralysis.

[0116] When the long-term overload reliability is higher and the possibility of paralysis is lower, it means that the calculation results of the computing power equipment when a large workload is assigned for a long time are more accurate, and the possibility of paralysis when completing the computing task is lower, and the long-term overload operation capability of the computing power equipment is stronger; conversely, when the long-term overload reliability is lower and the possibility of paralysis is greater, it means that the calculation results of the computing power equipment when a large workload is assigned for a long time are more incorrect, and the possibility of paralysis when completing the computing task is higher, and the long-term overload operation capability of the computing power equipment is worse.

[0117] Preferably, as an embodiment of the present application, the schematic diagram of the long-term heavy-load operation capacity extraction process is as follows: Figure 4 shown.

[0118] Step S4: Determine the computing power allocation priority of each computing task based on the release time, processing start time, execution time and required storage capacity of each computing task.

[0119] In the data parallel processing of large models, the training task is divided into several computing tasks by data parallelism. Each computing task is an independent part of the training model. The entire model is completed only after all computing tasks in the training task are completed. In the data parallel training process of large models, if a computing task is responsible for a particularly large workload, then this task will take longer to complete. In order to avoid the entire training process waiting too long due to a large-scale computing task, computing resources should be allocated first to those tasks with large workloads, which can reduce the overall training time and improve training efficiency.

[0120] Therefore, based on the release time, execution time and required storage capacity of each computing task in each computing device, the computing power allocation priority of each computing task is determined to ensure that all computing tasks can get corresponding resource allocation according to their workload and waiting time, while reducing the waiting time of tasks and improving the training efficiency of the model. Specifically:

[0121] Analyze the difference between the current calculation time and the processing start time of each calculation task, and record it as the processing time of each calculation task;

[0122] Analyze the ratio of the processing time of each computing task to the cumulative processing time of all computing tasks, and record it as the task completion degree of each computing task;

[0123] Analyze the product of the waiting time of each computing task and the task completion degree, and record it as the execution urgency of each computing task;

[0124] The longer the waiting time, the higher the execution urgency, in order to avoid the computing task waiting for computing power resources to be allocated. The higher the task completion rate, the higher the completion progress of the training task to which the computing task belongs. In order to avoid the long waiting time in the large model stage, the higher the execution urgency, the computing task should be given priority in the allocation of computing power resources.

[0125] The computing power allocation priority of each computing task is the result of the fusion of the execution urgency and workload of each computing task.

[0126] Preferably, as an embodiment of the present application, the computing power allocation priority of each computing task is the product of the execution urgency of each computing task and the required storage capacity.

[0127] Preferably, as another embodiment of the present application, the computing power allocation priority of each computing task is an exponential function value with a natural constant as the base and the sum of the execution urgency of each computing task and the required storage capacity as the independent variable.

[0128] The longer the waiting time, the higher the execution urgency, in order to avoid the computing task waiting too long for the allocation of computing resources; the higher the completion degree of the training task, the higher the completion progress of the training task to which the computing task belongs, in order to avoid the long waiting time in the large model stage, the higher the execution urgency, and the computing task should be given priority in the allocation of computing resources.

[0129] Step S5: Based on the storage capacity of each computing power device and the required storage capacity of each computing task, determine the volume adaptation between each computing task and any computing power device, and use the product of the volume adaptation and the long-term heavy-load operation capability of the corresponding computing power device as the input of the activation function to obtain the computing power allocation probability between each computing task and any computing power device.

[0130] During the parallel processing of large models, if heavy computing tasks are distributed to multiple large computing devices, the communication overhead of the computing devices will be significantly increased.

[0131] In order to determine whether the computing tasks match the working capacity of the computing devices, the storage capacity of each computing device and the required storage capacity of each computing task are analyzed to determine the volume adaptability between each computing task and any computing device, so as to adjust the resource allocation strategy as needed to avoid device overload. Specifically:

[0132] Based on the storage capacity of each computing device and the required storage capacity of each computing task, determine the volume compatibility between each computing task and any computing device, specifically: ; In the formula, Indicates the volume adaptability between the nth computing task and the i-th computing device; Indicates the required storage capacity of the nth computing task; Represents the storage capacity of the i-th computing device; Represents a second preset value, where the second preset value is a positive number less than 1.

[0133] It should be noted that the value of the second preset value is artificially set, and the second preset value is a positive number far less than 1. In this embodiment, the value of the second preset value is 0.001. The implementer can also set it according to the specific situation. This embodiment does not impose any special restrictions.

[0134] when When , it means that the computing power equipment can handle less workload, and the volume adaptation between each computing task and any computing power equipment is smaller; when When the workload of the computing task is closer to the processable workload of the computing power device, it is more appropriate to allocate the computing task to the computing power device. The greater the volume adaptability between each computing task and any computing power device, the higher the probability of computing power allocation.

[0135] Furthermore, the product of the volume fit between each computing task and any computing power device and the long-term heavy-load operation capacity of the device is used as the input of the Softmax function to obtain the computing power allocation probability of the computing task and any computing power device, where the function of the Softmax function is to make the sum of all computing power allocation probabilities equal to 1.

[0136] Among them, the Softmax function is a well-known technology, and its specific calculation process is not repeated here.

[0137] The higher the long-term overload operation capacity, the lower the efficiency loss of the computing power equipment and the higher the accuracy. In order to ensure that the computing tasks are completed first and reduce the risk of error propagation, the computing power equipment should be allocated to the computing tasks, and the higher the probability of computing power allocation.

[0138] Step S6: Based on the computing power allocation priority of each computing task, combined with the storage capacity and the long-term heavy-load operation capability, computing power resources are allocated to each computing task.

[0139] In order to avoid allocating each task to a device with high processing performance, the low-processing performance device will not run for a long time, while the high-performance computing device will be continuously overloaded.

[0140] According to the order of the computing power allocation priority of each computing task from large to small, the computing power allocation probability between each computing task and any computing power device is used as the input of the probability selection model, and one computing power device is randomly output at a time. If the storage capacity of the computing power device is greater than the required storage capacity of the computing task, stop allocating additional computing power devices to the computing task;

[0141] On the contrary, when the available storage capacity of the computing power device is less than or equal to the required storage capacity of the computing task, continue with the next round of probabilistic selection to allocate additional computing power devices to the computing task until the cumulative sum of the available storage capacity of all computing power devices allocated to the computing task is greater than the required storage capacity of the computing task, and then stop allocating additional computing power devices to the computing task.

[0142] At this point, based on the computing power allocation priority of each computing task, combined with the storage capacity and the long-term heavy-load operation capability, the computing power resource allocation for the computing tasks is completed.

[0143] Preferably, as an embodiment of the present application, a schematic diagram of a large model computing power resource allocation process is shown in Figure 5 shown.

[0144] It should be noted that the above sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above is a description of a specific embodiment of this specification. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0145] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0146] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Modifications to the technical solutions recorded in the aforementioned embodiments, or equivalent replacement of some of the technical features therein, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A computing resource allocation method for large model parallel processing, characterized in that: The method comprises the following steps: Obtain the execution time, calculation results, required storage capacity and available storage capacity of each historical computing task in each computing device; Determining the execution efficiency of each historical computing task in each computing device based on the required storage capacity of each historical computing task in each computing device and the storable capacity; The storage capacity required in each computing device is greater than the segmentation threshold, and the historical computing tasks that are continuous in time constitute the long-term heavy load group of each computing device; Determine the long-term heavy-load operation capability of each computing device based on the change trend of the execution efficiency, execution time, required storage capacity and calculation results of all historical computing tasks in each long-term heavy-load group in each computing device; Obtain the release time, processing start time, current calculation time and required storage capacity of each computing task in each stage of the large model; Based on the release time, processing start time, current computing time and required storage capacity of each computing task, determine the computing power allocation priority of each computing task, and combine the storage capacity and the long-term heavy-load operation capability to allocate computing power resources to each computing task; The method for determining the execution efficiency of each historical computing task in each computing device is as follows: Analyze the ratio of the required storage capacity of each historical computing task in each computing device to the cumulative result of the required storage capacity of all historical computing tasks, and record it as the workload ratio of each historical computing task in each computing device; Analyze the ratio of the required storage capacity of each historical computing task in each computing device to the available storage capacity of the corresponding computing device, and record it as the resource utilization rate of each historical computing task in each computing device; The expression of the execution efficiency of each historical computing task in each computing device is: ; In the formula, is the execution efficiency of the jth historical computing task in the i-th computing device; is the resource usage rate of the jth historical computing task in the i-th computing device; is the workload ratio of the jth historical computing task in the i-th computing device; The process of determining the long-term heavy-load operation capability of each computing device is as follows: Determine the paralysis possibility of each computing device based on the change trend of the execution efficiency of all historical computing tasks in all long-term overload groups in each computing device and the execution time of all historical computing tasks; Determine the long-term overload reliability of each computing device based on the required storage capacity and calculation results of all historical computing tasks in all long-term overload groups in each computing device; The long-term heavy-load operation capability of each computing device is the ratio of the long-term heavy-load reliability of each computing device to the possibility of paralysis; The method for determining the paralysis possibility of each computing power device is as follows: Based on the changing trend of the execution efficiency of all historical computing tasks in each long-term overload group in each computing device, determine the efficiency loss of each long-term overload group in each computing device; Analyze the sum of the execution time of all historical computing tasks whose resource usage rate in each long-term overload group in each computing device is greater than the preset threshold, and record it as the overload time of each long-term overload group in each computing device; The probability of paralysis of each computing device is the result of the fusion of the efficiency loss of all long-term overload groups in each computing device and the overload duration; The method for determining the efficiency loss of each long-term heavy-load group in each computing power device is as follows: The execution efficiency of all historical computing tasks in each long-term overload group in each computing device is used as the input of the slope estimation algorithm to obtain the slope estimation value of each long-term overload group in each computing device; The expression for the efficiency loss of each long-term overload group in each computing device is: ; Indicates the efficiency loss of the kth long-term overload group in the i-th computing device; represents the slope estimate of the k-th long-term overload group in the i-th computing device; The method for determining the long-term overload reliability of each computing device is as follows: If the calculation results of each historical calculation task in each computing power device are correct, the correctness of each historical calculation task in each computing power device is 1, otherwise, the correctness is -1; Analyze the product of the workload ratio and correctness of each historical computing task in each computing device, and record it as the computing contribution value of each historical computing task in each computing device; Analyze the ratio of the cumulative sum of the calculation contribution values ​​of all historical calculation tasks in each long-term overload group in each computing power device to the cumulative sum of the required storage capacity, and record it as the contribution efficiency ratio of each long-term overload group in each computing power device; The long-term overload reliability of each computing device is the average of the contribution efficiency ratios of all long-term overload groups in each computing device.

2. A computing resource allocation method for large model parallel processing as claimed in claim 1, characterized in that: The method for determining the computing power allocation priority of each computing task is as follows: Analyze the difference between the processing start time and the release time of each computing task, and record it as the waiting time of each computing task; Analyze the difference between the current calculation time and the processing start time of each calculation task, and record it as the processing time of each calculation task; Analyze the ratio of the processing time of each computing task to the cumulative processing time of all computing tasks, and record it as the task completion degree of each computing task; Based on the waiting time, task completion and required storage capacity of each computing task, the computing power allocation priority of each computing task is determined.

3. A computing resource allocation method for large model parallel processing as described in claim 2, characterized in that: The step of determining the priority of computing power allocation for each computing task based on the waiting time, task completion degree and required storage capacity of each computing task further includes: Analyze the product of the waiting time of each computing task and the task completion degree, and record it as the execution urgency of each computing task; The computing power allocation priority of each computing task is the result of the fusion of the execution urgency of each computing task and the required storage capacity.

4. A computing resource allocation method for large model parallel processing as claimed in claim 1, characterized in that: The allocating computing resources to each computing task includes: Based on the required storage capacity of each computing task and the storage capacity of each computing device, determine the volume adaptation between each computing task and any computing device, and use the product of the volume adaptation and the long-term heavy-load operation capability of the corresponding computing device as the input of the activation function to obtain the computing power allocation probability between each computing task and any computing device; According to the order of the computing power allocation priority of each computing task from large to small, the computing power allocation probability between each computing task and all computing power devices is used as the input of the probability selection model, and one computing power device is output at a time. If the storage capacity of the computing power device is greater than the required storage capacity of the computing task, stop allocating additional computing power devices to the computing task. Otherwise, allocate additional computing power devices to the computing task.

5. A computing resource allocation method for large model parallel processing as claimed in claim 4, characterized in that: The expression of the volume adaptability between each computing task and any computing power device is: ; In the formula, Indicates the volume adaptability between the nth computing task and the i-th computing device; Indicates the required storage capacity of the nth computing task; Represents the storage capacity of the i-th computing device; Represents a second preset value, where the second preset value is a positive number less than 1.

Images