Heterogeneous accelerator load balancing method, device, equipment, medium and program product
By dynamically adjusting the subtask allocation in pipeline parallelism in heterogeneous hardware environment, the problem of unbalanced resource utilization in heterogeneous hardware environment is solved, training efficiency and model performance are improved, and efficient on different computing infrastructures is achieved Large-scale deep learning model training.
Patent Information
- Application Number
- CN202411391799.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-02-11
AI Technical Summary
In heterogeneous hardware environments, especially in large-scale training clusters, due to the lack of the ability to effectively handle hardware heterogeneity, the optimal utilization of resources is often impossible, resulting in an imbalance between computing load and communication requirements, and Impacts training efficiency and model performance.
In the case of pipeline parallelism, the target task is divided into several subtasks based on the pipeline stage, and the subtasks are assigned to each heterogeneous accelerator based on the allocation strategy, and the first of each heterogeneous accelerator in the execution of the subtask is obtained. The processing speed is then dynamically adjusts the subtasks corresponding to at least one heterogeneous accelerator to achieve load balancing.
By dynamically adjusting sub-tasks, the load of each heterogeneous accelerator can be more detailed, training efficiency and model performance can be improved, and large-scale deep learning models can be trained efficiently on various computing infrastructures.
Smart Images
Figure CN119415247A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, medium and program product for heterogeneous accelerator load balancing. Background Art
[0002] With the rapid development of artificial intelligence technology, large-scale deep learning models, such as ChatGPT, have shown excellent performance in natural language processing tasks such as dialogue, text summarization, and translation. However, the training of these models faces huge challenges, mainly because the number of their model parameters exceeds the storage capacity of typical computing devices and usually requires years of training on TB-level datasets. This large-scale training demand has driven research and industry to adopt distributed training techniques to manually design parallel strategies such as data parallelism (DP), tensor parallelism (TP), and pipeline parallelism (PP) to fully utilize the aggregate computing power of all available devices.
[0003] In traditional technologies, in heterogeneous hardware environments, especially in large-scale training clusters, the lack of the ability to effectively handle hardware heterogeneity often makes it impossible to achieve optimal resource utilization, resulting in an imbalance between computing load and communication requirements, which in turn affects training efficiency and model performance. Summary of the invention
[0004] Based on this, it is necessary to provide a heterogeneous accelerator load balancing method, device, equipment, medium and program product that can balance the load of each heterogeneous accelerator and improve training efficiency and model performance in response to the above technical problems.
[0005] In a first aspect, the present application provides a heterogeneous accelerator load balancing method, the method comprising:
[0006] In the presence of pipeline parallelism, the target task is divided into several subtasks based on the pipeline stages;
[0007] Allocating each of the subtasks to each heterogeneous accelerator based on an allocation strategy;
[0008] Acquire a first processing speed of each of the heterogeneous accelerators during the execution of the subtask;
[0009] The subtask corresponding to at least one of the heterogeneous accelerators is dynamically adjusted based on the first processing speed.
[0010] In one embodiment, obtaining the first processing speed of each of the heterogeneous accelerators during the execution of the subtask includes:
[0011] A first processing speed is obtained based on an epoch of each of the heterogeneous accelerators during the execution of the subtask.
[0012] In one embodiment, dynamically adjusting the subtask corresponding to at least one of the heterogeneous accelerators based on the first processing speed includes:
[0013] When the first processing speeds of the heterogeneous accelerators are different, the amount of subtasks of the heterogeneous accelerator with a slow first processing speed is reduced, and / or the amount of subtasks of the heterogeneous accelerator with a fast first processing speed is increased.
[0014] In one embodiment, dynamically adjusting the subtask corresponding to at least one of the heterogeneous accelerators based on the first processing speed includes:
[0015] When the first processing speeds of the heterogeneous accelerators are different, the subtasks corresponding to the heterogeneous accelerator with a slow first processing speed are allocated to the heterogeneous accelerator with a fast first processing speed, and the types of the subtasks corresponding to the heterogeneous accelerator with the slow first processing speed are different from the types of the subtasks corresponding to the heterogeneous accelerator with the fast first processing speed.
[0016] In one embodiment, the method further comprises:
[0017] In the absence of pipeline parallelism, allocating the target task to each of the heterogeneous accelerators based on a micro-batch of each of the heterogeneous accelerators;
[0018] Acquire a second processing speed of each of the heterogeneous accelerators during the execution of the target task;
[0019] A micro-batch corresponding to at least one of the heterogeneous accelerators is dynamically adjusted based on the second processing speed.
[0020] In one embodiment, obtaining the second processing speed of each of the heterogeneous accelerators during the execution of the target task includes:
[0021] A second processing speed is obtained based on a target time taken by each of the heterogeneous accelerators to execute a target task corresponding to the micro-batch.
[0022] In one embodiment, dynamically adjusting the micro-batch corresponding to at least one of the heterogeneous accelerators based on the second processing speed includes:
[0023] When the second processing speeds of the heterogeneous accelerators are different, the micro-batch of the heterogeneous accelerator with a slow second processing speed is reduced, and / or the micro-batch of the heterogeneous accelerator with a fast second processing speed is increased.
[0024] In a second aspect, the present application further provides a heterogeneous accelerator load balancing device, the device comprising:
[0025] The task division module is used to divide the target task into several subtasks based on the pipeline stage when there is pipeline parallelism;
[0026] A first allocation module, configured to allocate each of the subtasks to each heterogeneous accelerator based on an allocation strategy;
[0027] A first processing speed acquisition module, used for acquiring a first processing speed of each of the heterogeneous accelerators during the execution of the subtask;
[0028] The first balancing module is used to dynamically adjust the subtask corresponding to at least one of the heterogeneous accelerators based on the first processing speed.
[0029] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any one of the above-mentioned embodiments when executing the computer program.
[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method in any one of the above-mentioned embodiments.
[0031] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the method in any one of the above embodiments when executed by a processor.
[0032] The above-mentioned heterogeneous accelerator load balancing method, device, equipment, medium and program product, in the presence of pipeline parallelism, divides the target task into several subtasks based on the pipeline stage; allocates each subtask to each heterogeneous accelerator based on the allocation strategy, and obtains the first processing speed of each heterogeneous accelerator during the execution of the subtask, and then balances the subtasks corresponding to each heterogeneous accelerator based on the first processing speed. In this way, the load balancing of pipeline parallelism is achieved through the first processing speed of each heterogeneous accelerator when the pipeline is parallel, with finer granularity, which can balance the load of each heterogeneous accelerator and improve training efficiency and model performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0034] FIG1 is a diagram of an application environment of a heterogeneous accelerator load balancing method according to an embodiment;
[0035] FIG2 is a schematic diagram of a flow chart of a method for load balancing of heterogeneous accelerators in one embodiment;
[0036] FIG3 is a schematic diagram of a flow chart of a method for load balancing of heterogeneous accelerators in another embodiment;
[0037] FIG4 is a structural block diagram of a heterogeneous accelerator load balancing device in one embodiment;
[0038] FIG. 5 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0040] The heterogeneous accelerator load balancing method provided in the embodiment of the present application can be applied to the application environment as shown in Figure 1. Wherein multiple heterogeneous accelerators are used for automatic parallel training, and different accelerators are heterogeneous, for example, different accelerators have at least one different communication bandwidth, video memory size, and computing power peak. In order to balance the heterogeneity of each accelerator and balance the load between each accelerator, the present application provides a heterogeneous accelerator load balancing method, which is applied to a load balancing module, which can load balance each heterogeneous accelerator, allowing each heterogeneous accelerator in a heterogeneous computing environment to work efficiently according to its actual capabilities, thereby improving the computing efficiency of the overall system and the performance of model training, and ensuring that large-scale deep learning models can be efficiently trained on various computing infrastructures.
[0041] Specifically, when there is pipeline parallelism, the load balancing module divides the target task into several subtasks based on the pipeline stage; allocates each subtask to each heterogeneous accelerator based on the allocation strategy; obtains the first processing speed of each heterogeneous accelerator during the execution of the subtask; and dynamically adjusts the subtask corresponding to at least one heterogeneous accelerator based on the first processing speed. In this way, the load balancing of pipeline parallelism is achieved through the first processing speed of each heterogeneous accelerator when the pipeline is parallel, with finer granularity, which can balance the load of each heterogeneous accelerator and improve training efficiency and model performance.
[0042] Among them, the above-mentioned heterogeneous accelerator may include data parallelism (DP), tensor parallelism (TP) and pipeline parallelism (PP) configurations. Within each heterogeneous accelerator node, tasks are further subdivided and assigned to different parallel groups. These groups are sets of computing units formed according to their respective parallel computing functions and purposes. The data parallel group (DP group) is responsible for managing data parallelism, ensuring the uniform distribution and synchronous processing of training data among different nodes, and optimizing data mobility across nodes. The tensor parallel group (TP group) consists of computing units that process different tensor slices in the same model. Tensor parallelism is implemented within the node, and multiple TP groups collaborate to process different tensor slices of the same data to fully utilize the computing resources within each node and improve computing efficiency. The pipeline parallel group (PP group) consists of a series of computing units sorted according to the model processing flow. Tasks are assigned to different nodes based on the processing stage of the model to implement pipeline parallelism. In this way, different levels or operations of the model can be processed in parallel on different nodes, thereby improving the overall training speed and resource utilization.
[0043] Optionally, each heterogeneous accelerator node includes multiple TP groups and PP groups, which are responsible for processing different model layers or operations to ensure that various parallel strategies can work together efficiently.
[0044] In an exemplary embodiment, as shown in FIG2 , a heterogeneous accelerator load balancing method is provided, and the method is applied to the load balancing module in FIG1 as an example for description, including the following steps S202 to S208 . Among them:
[0045] S202: In the case of pipeline parallelism, the target task is divided into a number of subtasks based on pipeline stages.
[0046] Specifically, pipeline parallelism consists of a series of computing units arranged according to the model processing flow. Tasks are assigned to different nodes based on the processing stage of the model to implement pipeline parallelism. In this way, different levels or operations of the model can be processed in parallel on different nodes, thereby improving the overall training speed and resource utilization. Pipeline parallelism includes several processing flows, each of which corresponds to a pipeline stage, so the target task can be divided into multiple subtasks corresponding to the pipeline stage.
[0047] For example, suppose a data processing pipeline has three pipeline stages: data preprocessing, data analysis, and result output. Therefore, based on these three pipeline stages, the target task is divided into several subtasks, such as data preprocessing subtask, data analysis subtask, and result output subtask.
[0048] S204: Allocate each subtask to each heterogeneous accelerator based on the allocation strategy.
[0049] Specifically, the allocation strategy is a strategy for allocating each subtask to a heterogeneous accelerator. For example, the heterogeneous accelerator corresponding to each subtask can be determined based on the processing capabilities of each heterogeneous accelerator and the complexity of each subtask. For example, the processing capabilities of heterogeneous accelerator node A, heterogeneous accelerator node B, and heterogeneous accelerator node C increase in sequence. The complexity of the corresponding subtasks A1, B1, and C1 also increases in sequence. Therefore, subtasks with high task complexity can be allocated to heterogeneous accelerator nodes with strong processing capabilities. For example, in the above example, subtask A1 is allocated to heterogeneous accelerator node A, subtask B1 is allocated to heterogeneous accelerator node B, and subtask C1 is allocated to heterogeneous accelerator node C.
[0050] Continuing with the previous example, each stage is processed by a different hardware node, and the processing capabilities of heterogeneous accelerator node A, heterogeneous accelerator node B, and heterogeneous accelerator node C increase in sequence. Assuming the initial state, based on the allocation strategy, heterogeneous accelerator node A is responsible for data preprocessing and has the slowest processing speed. Heterogeneous accelerator node B is responsible for data analysis and has a moderate speed. Heterogeneous accelerator node C is responsible for result output and has the fastest processing speed.
[0051] S206: Obtain a first processing speed of each heterogeneous accelerator during the execution of the subtask.
[0052] S208: Dynamically adjust a subtask corresponding to at least one heterogeneous accelerator based on the first processing speed.
[0053] The first processing speed is used to measure the speed at which each heterogeneous accelerator processes subtasks. Since each heterogeneous accelerator node processes subtasks in parallel according to the pipeline, if the processing in one pipeline stage is slow, it will affect the processing of other pipeline stages, such as causing the heterogeneous accelerators corresponding to other pipeline stages to be idle. For example, after the system has been running for a period of time, it is monitored that heterogeneous accelerator node A often becomes a bottleneck, and the processing speed lags behind other nodes. In order to solve this problem, the load balancing module obtains the first processing speed of each heterogeneous accelerator during the execution of the subtask. And based on the first processing speed, the subtasks corresponding to each heterogeneous accelerator are adjusted.
[0054] Optionally, the adjustment of the subtasks of each heterogeneous accelerator node includes the adjustment of the number of subtasks and / or the adjustment of the type of subtasks. The first processing speed is used to measure the performance of the heterogeneous accelerator node, and the subtasks of each heterogeneous accelerator node are dynamically adjusted based on the performance of each heterogeneous accelerator node, so as to evenly distribute the task load to different heterogeneous accelerator nodes according to the processing capacity of the heterogeneous accelerator node, so as to give full play to the computing advantages of each heterogeneous accelerator node.
[0055] The above-mentioned heterogeneous accelerator load balancing method, in the presence of pipeline parallelism, divides the target task into several subtasks based on the pipeline stage; allocates each subtask to each heterogeneous accelerator based on the allocation strategy, and obtains the first processing speed of each heterogeneous accelerator during the execution of the subtask, and then balances the subtasks corresponding to each heterogeneous accelerator based on the first processing speed, so that the load balancing of pipeline parallelism is realized by the first processing speed of each heterogeneous accelerator during pipeline parallelism, with finer granularity, which can balance the load of each heterogeneous accelerator and improve training efficiency and model performance. Compared with the manual parallel strategy involved in this application, load balancing is automatically performed with higher efficiency. Compared with directly applying the load balancing scheme of accelerators of the same type to heterogeneous accelerators, this application introduces finer granularity pipeline parallelism, and in the pipeline parallel mode, the first processing speed of each accelerator is comparable, so that it can work efficiently according to the actual first processing speed of each accelerator, and improve the computing efficiency of the overall system and the performance of model training based on the first processing speed, ensuring that large-scale deep learning models can be efficiently trained on various computing infrastructures.
[0056] In one optional embodiment, obtaining a first processing speed of each heterogeneous accelerator during the execution of the subtask includes: obtaining the first processing speed based on an epoch of each heterogeneous accelerator during the execution of the subtask.
[0057] Among them, epoch means the training process of passing all samples in the training data set once (and only once). In an epoch, the training algorithm will input all samples into the model in the set order for forward propagation, loss calculation, back propagation and parameter update. Therefore, in this application, the time corresponding to the epoch of each heterogeneous accelerator node during the execution of the subtask can be obtained. Since it is a pipeline, the amount of data processed by each heterogeneous accelerator node is the same, and the shorter the time corresponding to the epoch, the faster the first processing speed. Therefore, the first processing speed of each heterogeneous accelerator node can be obtained based on the time corresponding to the epoch of each heterogeneous accelerator during the execution of the subtask. The size of the first processing speed is used to characterize the performance of each heterogeneous accelerator node.
[0058] In one of the optional embodiments, dynamically adjusting the subtasks corresponding to at least one heterogeneous accelerator based on the first processing speed includes: when the first processing speeds of the heterogeneous accelerators are different, reducing the amount of subtasks of the heterogeneous accelerator with a slow first processing speed, and / or increasing the amount of subtasks of the heterogeneous accelerator with a fast first processing speed.
[0059] Among them, the dynamic adjustment of the subtasks corresponding to each heterogeneous accelerator may include adjustment of the subtask amount, for example, the subtask amount of the first heterogeneous accelerator with a slow processing speed may be reduced, or the subtask amount of the first heterogeneous accelerator with a fast processing speed may be increased, or the subtask amount of the first heterogeneous accelerator with a slow processing speed may be reduced and the subtask amount of the first heterogeneous accelerator with a fast processing speed may be increased.
[0060] Continuing with the previous example, after the system has been running for a period of time, it is monitored that heterogeneous accelerator node A often becomes a bottleneck, and its processing speed lags behind other nodes. The system decides to reduce the amount of preprocessing tasks for heterogeneous accelerator node A, while increasing the amount of tasks for heterogeneous accelerator nodes B and C.
[0061] In other embodiments, when there are only two heterogeneous accelerators, the task load of the slower accelerator is reduced; or the task load of the faster accelerator is increased; or the task load of the slower accelerator is reduced while the task load of the faster accelerator is increased.
[0062] In other embodiments, when there are more than three heterogeneous accelerators, the task load of the slowest accelerator or the task load of the other accelerators except the fastest accelerator among the three or more heterogeneous accelerators may be reduced, which is not specifically limited here. However, optionally, the task load of the slowest accelerator is reduced first, or the accelerators are sorted according to the first processing speed, and then the accelerators of the first preset ratio are obtained in the order of the first processing speed from small to large, and the task loads of the obtained accelerators are reduced.
[0063] In the case where there are more than three heterogeneous accelerators, the task load of the fastest accelerator or the task load of the other accelerators except the slowest accelerator among the three or more heterogeneous accelerators may be increased, and no specific limitation is made here. However, optionally, the task load of the fastest accelerator is increased first, or the accelerators are sorted according to the first processing speed, and then the accelerators of the second preset ratio are obtained in the order of the first processing speed from large to small, and the task loads of the obtained accelerators are increased.
[0064] In one of the optional embodiments, dynamically adjusting a subtask corresponding to at least one heterogeneous accelerator based on a first processing speed includes: when the first processing speeds of the heterogeneous accelerators are different, allocating a subtask corresponding to a heterogeneous accelerator with a slow first processing speed to a heterogeneous accelerator with a fast first processing speed, and a type of the subtask corresponding to the heterogeneous accelerator with the slow first processing speed is different from a type of the subtask corresponding to the heterogeneous accelerator with the fast first processing speed.
[0065] In this embodiment, the dynamic adjustment of the subtasks corresponding to each heterogeneous accelerator may include adjusting the subtask type, allocating the subtasks corresponding to the first heterogeneous accelerator with a slow processing speed to the first heterogeneous accelerator with a fast processing speed, and the type of the subtasks corresponding to the first heterogeneous accelerator with a slow processing speed is different from the type of the subtasks corresponding to the first heterogeneous accelerator with a fast processing speed.
[0066] Still continuing with the above example, for example, in the present application, a part of the preprocessing work of the heterogeneous accelerator node A can also be handed over to the heterogeneous accelerator node B or the heterogeneous accelerator node C to complete.
[0067] After this adjustment, the pressure on heterogeneous accelerator node A is reduced. Although heterogeneous accelerator nodes B and C have more tasks, their processing capabilities are stronger, which improves the efficiency of the entire pipeline. Finally, the load balancing of the entire process is achieved, avoiding the overload or idle state of any node.
[0068] In other embodiments, when there are only two heterogeneous accelerators, the subtasks corresponding to the first heterogeneous accelerator with a slower processing speed are allocated to the first heterogeneous accelerator with a faster processing speed.
[0069] In other embodiments, when there are more than three heterogeneous accelerators, the tasks of the first accelerator with the slowest processing speed can be assigned to other heterogeneous accelerators, or the tasks of the other accelerators except the fastest accelerator among the three or more heterogeneous accelerators can be assigned to the fastest accelerator, without specific limitation here. However, optionally, the tasks of the slowest accelerator can be assigned to other heterogeneous accelerators first, or the accelerators can be sorted according to the first processing speed, and then the accelerators of the second preset ratio can be obtained in the order of the first processing speed from small to large, and the tasks of the obtained accelerators can be assigned to other accelerators that are not selected, and optionally, the tasks of the first accelerator with the slowest processing speed can be assigned to the first accelerator with the fastest processing speed, and the tasks of the accelerator with the second slowest processing speed can be assigned to the accelerator with the second fastest processing speed, and so on, without specific limitation here. In the above embodiment, through the above dynamic adjustment, the computing resources of each node can be effectively utilized to improve the efficiency and response speed of the overall system.
[0070] In one of the optional embodiments, the method also includes: in the absence of pipeline parallelism, allocating the target task to each heterogeneous accelerator based on the micro-batch of each heterogeneous accelerator; obtaining a second processing speed of each heterogeneous accelerator during the execution of the target task; and dynamically adjusting the micro-batch corresponding to at least one heterogeneous accelerator based on the second processing speed.
[0071] Among them, when pipeline parallelism is not supported due to the limitations of the model structure or the hardware resources of each heterogeneous accelerator, the complex balancing module can achieve responsible balancing by adjusting the micro-batches of the corresponding heterogeneous accelerators for each target task.
[0072] The micro-batch size refers to the size of the data subset processed in each training iteration. Based on the micro-batches of each heterogeneous accelerator, the target task is assigned to each heterogeneous accelerator. For example, a data task of a micro-batch size is assigned to a heterogeneous accelerator each time. The second processing speed is used to measure the performance of each heterogeneous accelerator node. The size of the micro-batch of each heterogeneous accelerator node is subsequently dynamically adjusted based on the performance to achieve load balancing. The present application introduces a finer granularity micro-batch, and in the absence of a pipeline parallel mode, the second processing speed of each accelerator is comparable based on the micro-batch, and can work efficiently according to the actual second processing speed of each accelerator. Based on the second processing speed, the computing efficiency of the overall system and the performance of model training are improved, ensuring that large-scale deep learning models can be efficiently trained on various computing infrastructures.
[0073] In one of the optional embodiments, obtaining a second processing speed of each heterogeneous accelerator during the execution of the target task includes: obtaining the second processing speed based on a target time taken by each heterogeneous accelerator to execute the target task corresponding to the micro-batch.
[0074] Among them, the micro-batch is the amount of data processed by each heterogeneous accelerator node in each iteration. During initialization, the micro-batches of each heterogeneous accelerator node are the same. In this way, the target time spent by each heterogeneous accelerator node to process data of the same micro-batch size can be compared. The less the target time spent, the faster the second processing speed of the heterogeneous accelerator node.
[0075] In one of the optional embodiments, dynamically adjusting a micro-batch corresponding to at least one heterogeneous accelerator based on the second processing speed includes: when the second processing speeds of the heterogeneous accelerators are different, reducing the micro-batch of the heterogeneous accelerator with a slow second processing speed, and / or increasing the micro-batch of the heterogeneous accelerator with a fast second processing speed.
[0076] Specifically, in this embodiment, by adjusting the micro-batch size, the amount of data that each heterogeneous accelerator node needs to process in a single iteration can be controlled, thereby adjusting the load of each heterogeneous accelerator node according to its computing power. This method is particularly suitable for heterogeneous environments with large performance differences between nodes. This usually involves adjusting the gradient accumulation batch to adapt it to the processing speed and computing power of different devices, so that the utilization of computing resources can still be optimized without relying on pipeline parallelism.
[0077] For ease of understanding, an embodiment is given. Assume that there is a distributed system including three heterogeneous accelerator nodes: heterogeneous accelerator node A is a high-performance GPU with fast processing speed, heterogeneous accelerator node B is a medium-performance GPU, and heterogeneous accelerator node C is a low-performance GPU.
[0078] When performing deep learning model training, if a uniform micro-batch size is used, heterogeneous accelerator node C may become a bottleneck, causing heterogeneous accelerator node A and heterogeneous accelerator node B to be idle while waiting for heterogeneous accelerator node C to complete processing.
[0079] To optimize this situation, the following strategies can be adopted:
[0080] Heterogeneous accelerator node A: Increase the micro-batch size so that it can process more data in each iteration.
[0081] Heterogeneous accelerator Node B: Keep the micro-batch size moderate.
[0082] Heterogeneous accelerator node C: Reduces the micro-batch size, lightening its load and enabling it to complete each iteration faster.
[0083] In this way, each heterogeneous accelerator node can complete tasks while approaching its maximum performance, thereby reducing waiting time and improving overall training efficiency.
[0084] In other embodiments, when there are only two heterogeneous accelerators, the micro-batch of the slower accelerator may be reduced; or the micro-batch of the faster accelerator may be increased; or the micro-batch of the slower accelerator may be reduced while the micro-batch of the faster accelerator may be increased.
[0085] In other embodiments, when there are more than three heterogeneous accelerators, the micro-batch of the slowest accelerator or the micro-batch of the other accelerators except the fastest accelerator among the three or more heterogeneous accelerators may be reduced, which is not specifically limited here. However, optionally, the micro-batch of the slowest accelerator is reduced first, or the accelerators are sorted according to the second processing speed, and then the accelerators of the fourth preset ratio are obtained in the order of the second processing speed from small to large, and the micro-batch of the obtained accelerators is reduced.
[0086] In the case where there are more than three heterogeneous accelerators, the micro-batch of the fastest accelerator or the micro-batch of the other accelerators except the slowest accelerator among the three or more heterogeneous accelerators may be increased, without specific limitation herein. However, optionally, the micro-batch of the fastest accelerator may be increased preferentially, or the accelerators may be sorted according to the second processing speed, and then the accelerators of the second preset ratio are obtained in descending order of the second processing speed, and the micro-batch of the obtained accelerators is increased.
[0087] In the above embodiments, it is ensured that different heterogeneous accelerator nodes in a heterogeneous computing environment can work together and efficiently share training tasks according to their respective characteristics, thereby improving the performance of large-scale model training and overall resource utilization.
[0088] Specifically, as shown in Figure 3, in this embodiment, complex natural language processing tasks are processed, including machine translation, sentiment analysis, and the construction of intelligent dialogue systems. Due to various factors, the team did not have enough graphics cards from the same manufacturer for model training, so it was decided to use different types of GPUs to train large models at the same time.
[0089] Use the methods described above to configure the most appropriate data parallelism, tensor parallelism, and pipeline parallelism strategies. This includes selecting the most appropriate nodes to execute specific parts of the model, and adjusting the data flow and processing tasks between the layers of the model to ensure that each node can operate effectively according to its capabilities.
[0090] For example, first determine whether the target task is suitable for pipeline parallelism. If so, allocate each subtask to each heterogeneous accelerator based on the allocation strategy; obtain the first processing speed of each heterogeneous accelerator during the execution of the subtask; and dynamically adjust the subtask corresponding to at least one heterogeneous accelerator based on the first processing speed. Otherwise, allocate the target task to each heterogeneous accelerator based on the micro-batch of each heterogeneous accelerator; obtain the second processing speed of each heterogeneous accelerator during the execution of the target task; and dynamically adjust the micro-batch corresponding to at least one heterogeneous accelerator based on the second processing speed.
[0091] In the above embodiment, not only the distribution of computing load is optimized, but also the strategy can be dynamically adjusted and optimized according to the performance feedback during actual operation to achieve the best performance and load balancing. This method maximizes the use of resources in a diverse hardware environment and reduces the waste of resources caused by performance mismatch, which is particularly critical for large-scale deep learning model training that requires a long time and a large amount of computing resources.
[0092] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0093] Based on the same inventive concept, the embodiment of the present application also provides a heterogeneous accelerator load balancing device for implementing the heterogeneous accelerator load balancing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more heterogeneous accelerator load balancing device embodiments provided below can refer to the limitations of the heterogeneous accelerator load balancing method above, and will not be repeated here.
[0094] In an exemplary embodiment, as shown in FIG4 , a heterogeneous accelerator load balancing device is provided, including: a task division module 401, a first allocation module 402, a first processing speed acquisition module 403 and a first balancing module 404, wherein:
[0095] The task division module 401 is used to divide the target task into several subtasks based on the pipeline stage when there is pipeline parallelism;
[0096] A first allocation module 402, configured to allocate each subtask to each heterogeneous accelerator based on an allocation strategy;
[0097] A first processing speed acquisition module 403, used to acquire a first processing speed of each heterogeneous accelerator during the execution of a subtask;
[0098] The first balancing module 404 is configured to dynamically adjust the subtasks corresponding to the at least one heterogeneous accelerator based on the first processing speed.
[0099] In one of the optional embodiments, the first processing speed acquisition module 403 is specifically used to obtain the first processing speed based on the epochs of each heterogeneous accelerator during the execution of the subtask.
[0100] In one of the optional embodiments, the first balancing module 404 is specifically used to reduce the amount of subtasks of a heterogeneous accelerator with a slow first processing speed and / or increase the amount of subtasks of a heterogeneous accelerator with a fast first processing speed when the first processing speeds of the heterogeneous accelerators are different.
[0101] In one of the optional embodiments, the first balancing module 404 is specifically used to allocate the subtasks corresponding to the heterogeneous accelerator with a slow first processing speed to the heterogeneous accelerator with a fast first processing speed when the first processing speeds of the heterogeneous accelerators are different, and the type of the subtasks corresponding to the heterogeneous accelerator with the slow first processing speed is different from the type of the subtasks corresponding to the heterogeneous accelerator with the fast first processing speed.
[0102] In one of the optional embodiments, the above device further includes:
[0103] A second allocation module is used to allocate the target tasks to each heterogeneous accelerator based on the micro-batch of each heterogeneous accelerator in the absence of pipeline parallelism;
[0104] A second processing speed acquisition module, used to acquire a second processing speed of each heterogeneous accelerator during the execution of a target task;
[0105] The second balancing module is used to dynamically adjust the micro-batch corresponding to the at least one heterogeneous accelerator based on the second processing speed.
[0106] In one of the optional embodiments, the second processing speed acquisition module is specifically used to obtain the second processing speed based on the target time taken by each heterogeneous accelerator to execute the target task corresponding to the micro-batch.
[0107] In one of the optional embodiments, the second balancing module is specifically used to reduce the micro-batch of the heterogeneous accelerator with a slow second processing speed and / or increase the micro-batch of the heterogeneous accelerator with a fast second processing speed when the second processing speeds of the heterogeneous accelerators are different.
[0108] Each module in the above heterogeneous accelerator load balancing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.
[0109] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a heterogeneous accelerator load balancing method is implemented.
[0110] Those skilled in the art will understand that the structure shown in FIG. 5 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0111] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: in the presence of pipeline parallelism, dividing a target task into a plurality of subtasks based on pipeline stages; allocating each subtask to each heterogeneous accelerator based on an allocation strategy; obtaining a first processing speed of each heterogeneous accelerator during the execution of the subtask; and dynamically adjusting the subtask corresponding to at least one heterogeneous accelerator based on the first processing speed.
[0112] In one embodiment, obtaining a first processing speed of each heterogeneous accelerator during the execution of a subtask implemented by a processor when executing a computer program includes: obtaining the first processing speed based on an epoch of each heterogeneous accelerator during the execution of the subtask.
[0113] In one embodiment, the dynamic adjustment of subtasks corresponding to at least one heterogeneous accelerator based on a first processing speed implemented by a processor when executing a computer program includes: when the first processing speeds of the heterogeneous accelerators are different, reducing the amount of subtasks of the heterogeneous accelerator with a slow first processing speed, and / or increasing the amount of subtasks of the heterogeneous accelerator with a fast first processing speed.
[0114] In one embodiment, the dynamically adjusting subtasks corresponding to at least one heterogeneous accelerator based on a first processing speed implemented when a processor executes a computer program includes: when the first processing speeds of the heterogeneous accelerators are different, allocating the subtasks corresponding to the heterogeneous accelerator with a slow first processing speed to the heterogeneous accelerator with a fast first processing speed, and the type of the subtasks corresponding to the heterogeneous accelerator with the slow first processing speed is different from the type of the subtasks corresponding to the heterogeneous accelerator with the fast first processing speed.
[0115] In one embodiment, when the processor executes the computer program, the following steps are also implemented: in the absence of pipeline parallelism, based on the micro-batch of each heterogeneous accelerator, the target task is allocated to each heterogeneous accelerator; the second processing speed of each heterogeneous accelerator during the execution of the target task is obtained; and the micro-batch corresponding to at least one heterogeneous accelerator is dynamically adjusted based on the second processing speed.
[0116] In one embodiment, obtaining a second processing speed of each heterogeneous accelerator during the execution of a target task implemented by a processor when executing a computer program includes: obtaining the second processing speed based on a target time spent by each heterogeneous accelerator to execute the target task corresponding to the micro-batch.
[0117] In one embodiment, dynamically adjusting a micro-batch corresponding to at least one heterogeneous accelerator based on a second processing speed implemented by a processor when executing a computer program includes: when the second processing speeds of the heterogeneous accelerators are different, reducing the micro-batch of a heterogeneous accelerator with a slow second processing speed, and / or increasing the micro-batch of a heterogeneous accelerator with a fast second processing speed.
[0118] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: in the presence of pipeline parallelism, a target task is divided into a plurality of subtasks based on pipeline stages; each subtask is allocated to each heterogeneous accelerator based on an allocation strategy; a first processing speed of each heterogeneous accelerator during the execution of the subtask is obtained; and a subtask corresponding to at least one heterogeneous accelerator is dynamically adjusted based on the first processing speed.
[0119] In one embodiment, the obtaining of the first processing speed of each heterogeneous accelerator during the execution of the subtask implemented by the processor when the computer program is executed includes: obtaining the first processing speed based on the epoch of each heterogeneous accelerator during the execution of the subtask.
[0120] In one embodiment, the computer program implemented when executed by the processor dynamically adjusts the subtasks corresponding to at least one heterogeneous accelerator based on the first processing speed, including: when the first processing speeds of the heterogeneous accelerators are different, reducing the amount of subtasks of the heterogeneous accelerator with a slow first processing speed, and / or increasing the amount of subtasks of the heterogeneous accelerator with a fast first processing speed.
[0121] In one embodiment, the computer program implemented by the processor when executed dynamically adjusts the subtask corresponding to at least one heterogeneous accelerator based on the first processing speed, including: when the first processing speeds of the heterogeneous accelerators are different, assigning the subtask corresponding to the heterogeneous accelerator with a slow first processing speed to the heterogeneous accelerator with a fast first processing speed, and the type of the subtask corresponding to the heterogeneous accelerator with the slow first processing speed is different from the type of the subtask corresponding to the heterogeneous accelerator with the fast first processing speed.
[0122] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in the absence of pipeline parallelism, based on the micro-batch of each heterogeneous accelerator, the target task is allocated to each heterogeneous accelerator; the second processing speed of each heterogeneous accelerator during the execution of the target task is obtained; and the micro-batch corresponding to at least one heterogeneous accelerator is dynamically adjusted based on the second processing speed.
[0123] In one embodiment, obtaining a second processing speed of each heterogeneous accelerator during the execution of a target task implemented by a processor when a computer program is executed includes: obtaining the second processing speed based on a target time spent by each heterogeneous accelerator to execute the target task corresponding to the micro-batch.
[0124] In one embodiment, the computer program implemented by the processor when executed dynamically adjusts the micro-batch corresponding to at least one heterogeneous accelerator based on the second processing speed, including: when the second processing speeds of the heterogeneous accelerators are different, reducing the micro-batch of the heterogeneous accelerator with a slow second processing speed, and / or increasing the micro-batch of the heterogeneous accelerator with a fast second processing speed.
[0125] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the following steps: in the presence of pipeline parallelism, dividing a target task into a plurality of subtasks based on pipeline stages; allocating each subtask to each heterogeneous accelerator based on an allocation strategy; obtaining a first processing speed of each heterogeneous accelerator during the execution of the subtask; and dynamically adjusting the subtask corresponding to at least one heterogeneous accelerator based on the first processing speed.
[0126] In one embodiment, the obtaining of the first processing speed of each heterogeneous accelerator during the execution of the subtask implemented by the processor when the computer program is executed includes: obtaining the first processing speed based on the epoch of each heterogeneous accelerator during the execution of the subtask.
[0127] In one embodiment, the computer program implemented when executed by the processor dynamically adjusts the subtasks corresponding to at least one heterogeneous accelerator based on the first processing speed, including: when the first processing speeds of the heterogeneous accelerators are different, reducing the amount of subtasks of the heterogeneous accelerator with a slow first processing speed, and / or increasing the amount of subtasks of the heterogeneous accelerator with a fast first processing speed.
[0128] In one embodiment, the computer program implemented by the processor when executed dynamically adjusts the subtask corresponding to at least one heterogeneous accelerator based on the first processing speed, including: when the first processing speeds of the heterogeneous accelerators are different, assigning the subtask corresponding to the heterogeneous accelerator with a slow first processing speed to the heterogeneous accelerator with a fast first processing speed, and the type of the subtask corresponding to the heterogeneous accelerator with the slow first processing speed is different from the type of the subtask corresponding to the heterogeneous accelerator with the fast first processing speed.
[0129] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in the absence of pipeline parallelism, based on the micro-batch of each heterogeneous accelerator, the target task is allocated to each heterogeneous accelerator; the second processing speed of each heterogeneous accelerator during the execution of the target task is obtained; and the micro-batch corresponding to at least one heterogeneous accelerator is dynamically adjusted based on the second processing speed.
[0130] In one embodiment, obtaining a second processing speed of each heterogeneous accelerator during the execution of a target task implemented by a processor when a computer program is executed includes: obtaining the second processing speed based on a target time spent by each heterogeneous accelerator to execute the target task corresponding to the micro-batch.
[0131] In one embodiment, the computer program implemented by the processor when executed dynamically adjusts the micro-batch corresponding to at least one heterogeneous accelerator based on the second processing speed, including: when the second processing speeds of the heterogeneous accelerators are different, reducing the micro-batch of the heterogeneous accelerator with a slow second processing speed, and / or increasing the micro-batch of the heterogeneous accelerator with a fast second processing speed.
[0132] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for load balancing of heterogeneous accelerators, characterized in that: The method comprises: In the presence of pipeline parallelism, the target task is divided into several subtasks based on the pipeline stages; Allocating each of the subtasks to each heterogeneous accelerator based on an allocation strategy; Acquire a first processing speed of each of the heterogeneous accelerators during the execution of the subtask; The subtask corresponding to at least one of the heterogeneous accelerators is dynamically adjusted based on the first processing speed.
2. The method according to claim 1, characterized in that The obtaining of the first processing speed of each of the heterogeneous accelerators during the execution of the subtask includes: A first processing speed is obtained based on an epoch of each of the heterogeneous accelerators during the execution of the subtask.
3. The method according to claim 2, characterized in that The dynamically adjusting the subtask corresponding to at least one of the heterogeneous accelerators based on the first processing speed includes: When the first processing speeds of the heterogeneous accelerators are different, the amount of subtasks of the heterogeneous accelerator with a slow first processing speed is reduced, and / or the amount of subtasks of the heterogeneous accelerator with a fast first processing speed is increased.
4. The method according to claim 2, characterized in that: The dynamically adjusting the subtask corresponding to at least one of the heterogeneous accelerators based on the first processing speed includes: When the first processing speeds of the heterogeneous accelerators are different, the subtasks corresponding to the heterogeneous accelerator with a slow first processing speed are allocated to the heterogeneous accelerator with a fast first processing speed, and the types of the subtasks corresponding to the heterogeneous accelerator with the slow first processing speed are different from the types of the subtasks corresponding to the heterogeneous accelerator with the fast first processing speed.
5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In the absence of pipeline parallelism, allocating the target task to each of the heterogeneous accelerators based on a micro-batch of each of the heterogeneous accelerators; Acquire a second processing speed of each of the heterogeneous accelerators during the execution of the target task; A micro-batch corresponding to at least one of the heterogeneous accelerators is dynamically adjusted based on the second processing speed.
6. The method according to claim 5, characterized in that The obtaining of the second processing speed of each of the heterogeneous accelerators during the execution of the target task includes: A second processing speed is obtained based on a target time taken by each of the heterogeneous accelerators to execute a target task corresponding to the micro-batch.
7. The method according to claim 5, characterized in that The dynamically adjusting the micro-batch corresponding to at least one of the heterogeneous accelerators based on the second processing speed includes: When the second processing speeds of the heterogeneous accelerators are different, the micro-batch of the heterogeneous accelerator with a slow second processing speed is reduced, and / or the micro-batch of the heterogeneous accelerator with a fast second processing speed is increased.
8. A heterogeneous accelerator load balancing device, characterized in that: The device comprises: The task division module is used to divide the target task into several subtasks based on the pipeline stage when there is pipeline parallelism; A first allocation module, configured to allocate each of the subtasks to each heterogeneous accelerator based on an allocation strategy; A first processing speed acquisition module, used for acquiring a first processing speed of each of the heterogeneous accelerators during the execution of the subtask; The first balancing module is used to dynamically adjust the subtask corresponding to at least one of the heterogeneous accelerators based on the first processing speed.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.