A distributed training method, device, equipment and medium based on heterogeneous devices

Abstract

The embodiments of this specification disclose a distributed training method, device, equipment and medium based on heterogeneous devices, which relates to the field of computer technology and is used to solve the problem of low efficiency caused by unreasonable existing distributed training allocation. The method includes: dividing the hardware cluster of the current heterogeneous device to obtain multiple logical computing units; determining the existing division method corresponding to each operator in the current model training task to obtain multiple parallel strategies for each operator and divide them into intra-operator parallel strategies and inter-operator parallel strategies; searching based on the calculation delay and communication delay of the parallel strategy to obtain the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy; mapping the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy to the corresponding node computing unit, so that the node computing unit of the heterogeneous device performs distributed training on the current model training task based on the optimal parallel strategy.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a distributed training method, apparatus, device and medium based on heterogeneous devices. Background Art

[0002] Large models, such as large language models, have replaced small models dedicated to tasks and fields in most natural language processing tasks, and have been widely used, and have been quickly implemented and promoted in many fields. Since large model training requires high computing power, which often exceeds the capabilities of a single device, distributed parallel training is required on a cluster. By distributing the model's parameters and computing tasks across multiple devices, distributed training can significantly improve training speed and efficiency. In addition, heterogeneous computing devices are developing rapidly. By performing distributed parallel training on these devices, not only can the dependence on expensive imported hardware be reduced, but the needs of domestic users can also be better met. Therefore, how to perform distributed training on heterogeneous devices is a key issue in improving training efficiency.

[0003] Since existing heterogeneous devices have diverse hardware architectures, the current deep learning frameworks have different computing capabilities in heterogeneous computing environments. If the load balancing strategy is not appropriate, some devices may be overloaded while other devices are idle, thereby reducing the overall training efficiency. In addition, parameter synchronization in the data parallel strategy during the training process requires frequent network communication, especially when the model has many parameters, the communication time may become a bottleneck in the training process. Therefore, how to automatically optimize the distributed parallel strategy is a problem that needs to be solved in the distributed training of models on heterogeneous computing devices. Summary of the invention

[0004] In order to solve the above technical problems, one or more embodiments of this specification provide a distributed training method, apparatus, device and medium based on heterogeneous devices.

[0005] One or more embodiments of this specification adopt the following technical solutions:

[0006] One or more embodiments of this specification provide a distributed training method based on heterogeneous devices, the method comprising:

[0007] The hardware cluster of the current heterogeneous device is divided to obtain a plurality of logical computing units; wherein the logical computing unit comprises: an architecture computing unit; the architecture computing unit comprises a node computing unit;

[0008] Determine an existing segmentation method corresponding to each operator in the current model training task, and determine multiple intra-operator parallel strategies and inter-operator parallel strategies of each operator according to the calculation graph corresponding to the current model training task and the existing segmentation method;

[0009] Calculating the computation delay and the communication delay corresponding to each parallel strategy, so as to search and obtain the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy based on the computation delay and the communication delay;

[0010] According to mapping the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy to corresponding node computing units, the node computing units of the heterogeneous devices perform distributed training on the current model training task based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy.

[0011] Optionally, in one or more embodiments of the present specification, the hardware cluster of the current heterogeneous device is divided to obtain multiple logical computing units, specifically including:

[0012] Obtain the physical structure differences of each heterogeneous device in the current hardware cluster of heterogeneous devices;

[0013] The current hardware cluster is divided according to the physical structure difference to obtain multiple logical computing units.

[0014] Optionally, in one or more embodiments of the present specification, determining the existing segmentation method corresponding to each operator in the current model training task to obtain multiple intra-operator parallel strategies and inter-operator parallel strategies of each operator specifically includes:

[0015] Determine the existing segmentation method corresponding to each operator in the current model training task;

[0016] The computation graph is segmented according to an existing operator segmentation method to obtain an intra-operator parallel computation graph segmentation set and an inter-operator parallel computation graph segmentation set; wherein the intra-operator parallel computation graph segmentation set includes multiple intra-operator parallel strategies of each operator; and the inter-operator parallel computation graph segmentation set includes multiple inter-operator parallel strategies of each operator.

[0017] Optionally, in one or more embodiments of the present specification, the calculation delay and communication delay corresponding to each parallel strategy are calculated, so as to search and obtain the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy based on the calculation delay and communication delay, specifically including:

[0018] Performing mixed segmentation on the intra-operator parallel strategies to obtain intra-operator parallel strategies to be optimized, and determining the total delay of each intra-operator parallel strategy to be optimized according to the first calculation delay and the first communication delay of the intra-operator parallel strategies to be optimized;

[0019] Performing pipeline parallel segmentation on the inter-operator parallel strategies to obtain inter-operator parallel strategies to be optimized, and determining the total delay of each of the inter-operator parallel strategies to be optimized according to the second calculation delay and the second communication delay of the inter-operator parallel strategies to be optimized;

[0020] Minimizing the total delay of the intra-operator parallel strategy to be optimized to search for the optimal intra-operator parallel strategy;

[0021] The total delay of the inter-operator parallel strategy to be optimized is minimized to search for the optimal inter-operator parallel strategy.

[0022] Optionally, in one or more embodiments of the present specification, the intra-operator parallel strategies are mixed and split to obtain the intra-operator parallel strategies to be optimized, and the total delay of each intra-operator parallel strategy to be optimized is determined according to the first calculation delay and the first communication delay of the intra-operator parallel strategies to be optimized, specifically including:

[0023] By randomly searching the intra-operator parallel computing graph partition set, a mixed partition of the data parallel strategy and the model parallel strategy in the intra-operator parallel computing graph partition set is realized to obtain the intra-operator parallel strategy to be optimized;

[0024] Determine a first communication volume of the parallel strategy within the operator to be optimized based on the matrix dimension data of the forward propagation and the backward propagation corresponding to the hybrid segmentation;

[0025] Determining a first communication delay of the parallel strategy within the operator to be optimized according to the first communication volume and an actual hardware bandwidth of the hardware cluster;

[0026] Determining a first computing delay of the parallel strategy within the operator to be optimized according to the computing cost of the parallel strategy within the operator to be optimized and the hardware computing power of the hardware cluster;

[0027] The first computation delay and the first communication delay of each of the parallel strategies within the to-be-optimized operators are added together to determine a total delay of the parallel strategies within the to-be-optimized operators.

[0028] Optionally, in one or more embodiments of the present specification, pipeline parallel segmentation is performed on the inter-operator parallel strategy to obtain the inter-operator parallel strategy to be optimized, and the total delay of each of the inter-operator parallel strategies to be optimized is determined according to the second calculation delay and the second communication delay of the inter-operator parallel strategy to be optimized, specifically including:

[0029] By randomly searching the segmentation set of the inter-operator parallel computing graph, pipeline parallel segmentation is performed on the inter-operator parallel strategy to obtain the inter-operator parallel strategy to be optimized;

[0030] Determining a second communication volume of the parallel strategy between operators to be optimized based on the matrix dimension data of the forward propagation and the backward propagation corresponding to the parallel segmentation of the pipeline;

[0031] Determining a second communication delay of the inter-operator parallel strategy to be optimized based on the operation feedback data of the inter-operator parallel strategy to be optimized;

[0032] The second computation delay and the second communication delay of each of the inter-operator parallel strategies to be optimized are added together to determine a total delay of the inter-operator parallel strategies to be optimized.

[0033] Optionally, in one or more embodiments of the present specification, mapping the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy to corresponding node computing units, so that the node computing units of the heterogeneous devices perform distributed training on the current model training task based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy, specifically includes:

[0034] Based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy, the model training task corresponding to each operator is divided to obtain multiple model training subtasks;

[0035] Allocate each of the model training subtasks to a corresponding node computing unit based on the correspondence between the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy;

[0036] Based on each of the node computing units, distributed training is performed on multiple model training subtasks corresponding to the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy.

[0037] One or more embodiments of the present specification provide a distributed training device based on heterogeneous devices, the device comprising:

[0038] A splitting unit is used to split the hardware cluster of the current heterogeneous device to obtain multiple logical computing units; wherein the logical computing unit includes: an architecture computing unit; the architecture computing unit includes a node computing unit;

[0039] A parallel strategy determination unit, used to determine an existing segmentation method corresponding to each operator in the current model training task, and determine multiple intra-operator parallel strategies and inter-operator parallel strategies of each operator according to the calculation graph corresponding to the current model training task and the existing segmentation method;

[0040] A parallel strategy optimization unit, used to calculate the computation delay and communication delay corresponding to each parallel strategy, so as to search and obtain the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy based on the computation delay and communication delay;

[0041] A training unit is used to map the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy to corresponding node computing units, so that the node computing units of the heterogeneous devices perform distributed training on the current model training task based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy.

[0042] One or more embodiments of this specification provide a distributed training device based on heterogeneous devices, the device comprising:

[0043] at least one processor; and,

[0044] a memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute any of the above-mentioned methods.

[0046] One or more embodiments of the present specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute any of the above-described methods.

[0047] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0048] The hardware cluster is segmented through a preset multi-level segmentation strategy to form a logical computing unit including an architecture computing unit and a node computing unit. This multi-level segmentation method matches the characteristics of heterogeneous devices more precisely, improving resource utilization and training efficiency. The parallel strategy is subdivided into intra-operator parallel strategy and inter-operator parallel strategy, so that the search is performed through computing delay and communication delay to find the optimal parallel strategy. This detailed division and delay-based optimization search strategy helps to more effectively balance computing and communication costs in a heterogeneous environment, and improves the efficiency of distributed training of current model training tasks. The optimized parallel strategy is mapped to specific heterogeneous device nodes, so that each node can perform distributed training based on the optimal parallel strategy. This adaptive mapping mechanism can make full use of the characteristics of heterogeneous devices, further improve training efficiency and effect, and through the optimal parallel strategy obtained for training, it can reduce the error accumulation caused by factors such as data dependence and communication delay during training, thereby improving the accuracy and performance of the training model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0050] Figure 1 A schematic diagram of a distributed training method based on heterogeneous devices provided in an embodiment of this specification;

[0051] Figure 2 A schematic diagram of hierarchical segmentation of a heterogeneous device provided in an embodiment of this specification;

[0052] Figure 3 A hybrid parallel example diagram of matrix multiplication of data and model provided in the embodiments of this specification;

[0053] Figure 4 A schematic diagram of parallel optimization of a parallel strategy provided in an embodiment of this specification;

[0054] Figure 5 A schematic diagram of the structure of a distributed training device based on heterogeneous devices provided in an embodiment of this specification;

[0055] Figure 6 A schematic diagram of the structure of a distributed training device based on heterogeneous devices provided in an embodiment of this specification;

[0056] Figure 7 A schematic diagram of the structure of a non-volatile storage medium provided in an embodiment of this specification. DETAILED DESCRIPTION

[0057] The embodiments of this specification provide a distributed training method, apparatus, device and medium based on heterogeneous devices.

[0058] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0059] like Figure 1 As shown, the embodiment of this specification provides a flow chart of a distributed training method based on heterogeneous devices. Figure 1It can be seen that in one or more embodiments of this specification, a distributed training method based on heterogeneous devices includes:

[0060] S101: Split the hardware cluster of the current heterogeneous device to obtain multiple logical computing units; wherein the logical computing unit includes: an architecture computing unit; and the architecture computing unit includes a node computing unit.

[0061] Large language models have replaced small models dedicated to tasks and fields in most NLP tasks, and have been widely used, and have been quickly implemented and promoted in many fields. Due to the high computing power requirements for large model training, which often exceeds the capabilities of a single device, distributed parallel training is required on a cluster. By distributing the parameters and computing tasks of the model to multiple devices, distributed training can significantly improve the training speed and efficiency. With the rapid development of heterogeneous computing devices, distributed parallel training on heterogeneous devices can improve the efficiency of model training. In order to be able to adaptively perform distributed training of deep learning models, in the embodiment of this specification, the hardware cluster of the current heterogeneous device is first divided to obtain multiple logical computing units. Among them, it should be noted that the logical computing unit includes: an architecture computing unit, and the architecture computing unit includes a node computing unit. The multi-level segmentation strategy preset in the process divides the hardware cluster to form a logical computing unit including an architecture computing unit and a node computing unit. This multi-level segmentation method can more finely match the characteristics of heterogeneous devices and improve resource utilization and training efficiency.

[0062] like Figure 2 As shown, the outer layer is the architecture computing unit, which is composed of different architecture device resources. Each architecture computing unit includes multiple node computing units, and each node computing unit includes multiple accelerator cards. Therefore, it can be understood that the node computing unit can be a plurality of accelerator cards contained in the architecture computing unit. It should also be explained that the preset multi-level splitting strategy can be based on the hardware characteristics of heterogeneous devices according to the nature of the computational graph splitting problem. After fully considering the diversity of device types and differences in device characteristics, different levels of splitting methods are adopted to support heterogeneous devices of different hardware manufacturers and different models of the same hardware manufacturer.

[0063] Specifically, in one or more embodiments of the present specification, the hardware cluster of the current heterogeneous device is divided to obtain multiple logical computing units, which specifically includes the following processes:

[0064] First, obtain the differences in the physical structures of each heterogeneous device in the hardware cluster of the current heterogeneous device. For example, different hardware manufacturers and different models of intelligent computing devices from the same hardware manufacturer have different physical hardware differences. According to the differences in physical structure, the current hardware cluster can be divided to obtain multiple logical computing units with different architectures. In this way, the diversity of device types and differences in device characteristics are fully considered, so that the division method can support different hardware manufacturers and different models of devices from the same hardware manufacturer, which can better support diversified applications and meet the performance and functional requirements of different applications.

[0065] S102: Determine an existing segmentation method corresponding to each operator in the current model training task, and determine multiple intra-operator parallel strategies and inter-operator parallel strategies of each operator according to the calculation graph corresponding to the current model training task and the existing segmentation method.

[0066] When training large-scale models, the memory of a single computing device often cannot accommodate the parameters and intermediate data of the entire model, resulting in the inability to conduct training. Therefore, in order to achieve distributed training and solve the problems of memory limitations, computing efficiency, communication bottlenecks, and load balancing in the prior art, the training data is divided into multiple subsets and processed in parallel on multiple computing devices to solve this problem. The training speed can be significantly improved. Combining multiple strategies such as data parallelism, pipeline parallelism, and tensor parallelism can also achieve a higher degree of parallel computing. Therefore, in the embodiment of this specification, the existing segmentation method corresponding to each operator in the current model training task is determined, thereby obtaining multiple intra-operator parallel strategies and inter-operator parallel strategies for each operator.

[0067] Specifically, in one or more embodiments of the present specification, determining the existing segmentation method corresponding to each operator in the current model training task to obtain multiple intra-operator parallel strategies and inter-operator parallel strategies of each operator specifically includes the following process:

[0068] First, determine the existing segmentation methods corresponding to each operator in the current model training task, and then segment the computational graph according to the existing operator segmentation method to obtain an intra-operator parallel computational graph segmentation set and an inter-operator parallel computational graph segmentation set. Among them, it can be understood that the intra-operator parallel computational graph segmentation set contains multiple intra-operator parallel strategies for each operator, and the inter-operator parallel computational graph segmentation set contains multiple inter-operator parallel strategies for each operator. Since intra-operator parallelism can minimize the transmission overhead of data between different execution units, and inter-operator parallelism allows more efficient resource scheduling and task allocation, by determining intra-operator parallelism and inter-operator parallelism, this helps to allocate computing resources more reasonably. It helps to ensure the stability and accuracy of the model training process and improve the performance and quality of the final model. In addition, by finding multiple segmentation methods for different operators or node computing units, and generating multiple parallel strategies based on these segmentation methods, the speed of model training can be significantly improved.

[0069] S103: Calculate the computation delay and communication delay corresponding to each parallel strategy, and search and obtain an optimal intra-operator parallel strategy and an optimal inter-operator parallel strategy based on the computation delay and communication delay.

[0070] As the complexity and scale of deep learning models continue to increase, the demand for computing resources and communication resources is also increasing. In order to reasonably allocate resources to obtain the optimal parallel strategy that adapts to different hardware environments, the embodiment of this specification accurately calculates the calculation delay and communication delay of each parallel strategy, and the impact of different strategies on the overall performance can be intuitively understood. And selecting the strategy with the smallest sum of calculation delay and communication delay as the optimal strategy can significantly reduce the total time of model training and improve training efficiency. And by optimizing parallel strategies, it is possible to support the training of larger and more complex models to meet the growing computing needs. It can be seen that through the above steps S102-S103, the parallel strategy is subdivided into intra-operator parallel strategy and inter-operator parallel strategy, and search is performed based on calculation delay and communication delay to find the optimal parallel strategy. This detailed division and delay-based optimization search strategy help to more effectively balance computing and communication costs in heterogeneous environments and improve training efficiency.

[0071] The existing mainstream distributed parallel strategies include data parallelism, model parallelism, and pipeline parallelism. In order to consider various possible combinations of parallel strategies and obtain the optimal parallel strategy, this application needs to analyze the communication of different operators in different hybrid parallel modes. Take the common matrix multiplication operator of the Transformer model as an example. Assume that the input is X, the weight is W, and the output is Y, Y=X*W. At the same time, the operator is model-parallel, data-parallel, and pipeline-parallel. Figure 3The figure shows the mixed parallelism of the operator's data and model. After analysis, the communication volume of the operator's parallelism can be calculated as shown in Table 1. In a similar way, the mixed parallelism of various common operators and the corresponding communication volume and communication method can be analyzed.

[0072] Table 1. Table of the amount of communication for common operators in parallel

[0073]

[0074] like Figure 4 As shown in the schematic diagram of parallel optimization of the parallel strategy, it can be seen that based on the hierarchical segmentation method, the inter-operator parallelism can be allocated to the node computing unit with a small communication bandwidth, and the intra-operator parallelism can be allocated to the inside of the node computing unit with a large communication bandwidth. In order to achieve the acquisition of the optimal strategy, specifically, in one or more embodiments of the present specification, the calculation delay and communication delay corresponding to each parallel strategy are calculated, so as to search and obtain the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy based on the calculation delay and the communication delay, which specifically includes:

[0075] Through the mixed splitting method, the intra-operator parallel strategy can be divided more finely, so that the computing tasks can be more evenly distributed on different computing units. Therefore, the intra-operator parallel strategy is mixed and split to obtain the intra-operator parallel strategy to be optimized, and the intra-operator parallel strategy to be optimized is further evaluated according to the first computing delay and the first communication delay of the intra-operator parallel strategy to be optimized, and the total delay of each intra-operator parallel strategy to be optimized is determined. At the same time, pipeline parallel splitting can enable the computing tasks between operators to be carried out continuously, reducing the waiting time and significantly improving the performance of inter-operator parallelism. Therefore, the inter-operator parallel strategy is pipelined and split to obtain the inter-operator parallel strategy to be optimized, and then in order to achieve a comprehensive evaluation of the total delay, the total delay of each inter-operator parallel strategy to be optimized is determined according to the second computing delay and the second communication delay of the inter-operator parallel strategy to be optimized. Then, the total delay of the parallel strategy within the operator to be optimized and the total delay of the parallel strategy between operators to be optimized are minimized respectively to ensure that the parallel strategy finally selected is optimal while comprehensively considering the computing and communication overheads, thereby searching for the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy respectively.

[0076] Further, in one or more embodiments of the present specification, the intra-operator parallel strategies are mixed and divided, the intra-operator parallel strategies are mixed and divided, the intra-operator parallel strategies to be optimized are obtained, and the total delay of each intra-operator parallel strategy to be optimized is determined according to the first calculation delay and the first communication delay of the intra-operator parallel strategy to be optimized, which specifically includes the following process:

[0077] First, by randomly searching the intra-operator parallel computing graph partition set, a mixed partition of the data parallel strategy and the model parallel strategy in the intra-operator parallel computing graph partition set is realized to obtain the intra-operator parallel strategy to be optimized. Then, as shown in Table 1 above, based on the matrix dimension data of the forward propagation and the backward propagation corresponding to the mixed partition, the first communication volume of the intra-operator parallel strategy to be optimized can be determined. Then, according to the first communication volume and the actual hardware bandwidth of the hardware cluster, the first communication delay of the intra-operator parallel strategy to be optimized can be determined. At the same time, according to the computational cost of the intra-operator parallel strategy to be optimized and the hardware computing power of the hardware cluster, the first computational delay of the intra-operator parallel strategy to be optimized can be determined. The first computational delay of each intra-operator parallel strategy to be optimized is added to the first communication delay to determine the total delay of the intra-operator parallel strategy to be optimized, so as to compare the total delay of all intra-operator parallel strategies to be optimized, and select the strategy with the smallest total delay as the optimal parallel strategy.

[0078] Further, in one or more embodiments of the present specification, pipeline parallel segmentation is performed on the inter-operator parallel strategy to obtain the inter-operator parallel strategy to be optimized, and the total delay of each inter-operator parallel strategy to be optimized is determined according to the second calculation delay and the second communication delay of the inter-operator parallel strategy to be optimized, which specifically includes the following process:

[0079] By randomly searching the inter-operator parallel calculation graph segmentation set, the inter-operator parallel strategy is pipelined and parallel segmented to obtain the inter-operator parallel strategy to be optimized. Then, according to the matrix dimension data of the forward propagation and backward propagation corresponding to the pipeline parallel segmentation as shown in Table 1 above, the second communication volume of the inter-operator parallel strategy to be optimized is determined. At the same time, based on the operation feedback data of the inter-operator parallel strategy to be optimized, the second communication delay of the inter-operator parallel strategy to be optimized is determined. The second calculation delay of each determined inter-operator parallel strategy to be optimized is added to the second communication delay to obtain the total delay of the inter-operator parallel strategy to be optimized. The second calculation delay of each inter-operator parallel strategy to be optimized is added to the second communication delay to determine the total delay of each intra-operator parallel strategy to be optimized, which helps to obtain the optimal inter-operator parallel strategy, and then reduces the calculation waiting time and communication overhead by optimizing the inter-operator parallel strategy, which significantly accelerates the model training process.

[0080] S104: Mapping the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy to corresponding node computing units, so that the node computing units of the heterogeneous devices perform distributed training on the current model training task based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy.

[0081] After obtaining the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy based on the above step S103, the embodiment of this specification maps the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy to the corresponding node computing unit, so that the node computing unit of the heterogeneous device performs distributed training on the current model training task based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy. The optimized parallel strategy is mapped to a specific heterogeneous device node so that each node can perform distributed training based on the optimal parallel strategy. This adaptive mapping mechanism can make full use of the characteristics of heterogeneous devices and further improve the training efficiency and effect. And by obtaining the optimal parallel strategy as described above, the error accumulation caused by factors such as data dependence and communication delay during the training process can be reduced, thereby improving the accuracy and performance of the training model.

[0082] Specifically, in one or more embodiments of the present specification, the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy are mapped to corresponding node computing units, so that the node computing units of heterogeneous devices perform distributed training on the current model training task based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy, specifically including the following processes:

[0083] First, the model training tasks corresponding to each operator are divided based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy to obtain multiple model training subtasks. This division method can ensure that each subtask of the model training task is executed independently, which can fully utilize the advantages of parallel computing and improve training efficiency. Then, each model training subtask is assigned to the corresponding node computing unit through the correspondence between the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy. Through this reasonable correspondence, it can be ensured that each node computing unit can obtain an appropriate amount of computing tasks, thereby avoiding the waste and idleness of computing resources. Then, according to each node computing unit, the multiple model training subtasks corresponding to the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy are distributedly trained. The whole process comprehensively considers multiple aspects such as task division, node computing unit correspondence, and distributed training to form a complete parallel computing framework. This framework can effectively utilize the computing resources of heterogeneous devices and improve the efficiency and quality of model training by optimizing parallel strategies. At the same time, the process also has good scalability and flexibility, and can adapt to model training tasks of different scales and complexities.

[0084] like Figure 5 As shown, the present specification embodiment provides a schematic diagram of the structure of a distributed training device based on heterogeneous devices. Figure 5 It can be seen that in one or more embodiments of this specification, a distributed training device based on heterogeneous devices includes:

[0085] The splitting unit 501 is used to split the hardware cluster of the current heterogeneous device to obtain multiple logical computing units; wherein the logical computing unit includes: an architecture computing unit; the architecture computing unit includes a node computing unit;

[0086] A parallel strategy determination unit 502 is used to determine an existing segmentation method corresponding to each operator in the current model training task, and determine multiple intra-operator parallel strategies and inter-operator parallel strategies of each operator according to the calculation graph corresponding to the current model training task and the existing segmentation method;

[0087] A parallel strategy optimization unit 503 is used to calculate the computation delay and communication delay corresponding to each parallel strategy, so as to search and obtain the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy based on the computation delay and communication delay;

[0088] The training unit 504 is used to map the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy to corresponding node computing units, so that the node computing units of the heterogeneous devices perform distributed training on the current model training task based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy.

[0089] like Figure 6 As shown, the present specification embodiment provides a structural diagram of a distributed training device based on heterogeneous devices. Figure 6 It can be seen that in one or more embodiments of this specification, a distributed training device based on heterogeneous devices includes:

[0090] at least one processor; and,

[0091] a memory communicatively connected to the at least one processor; wherein,

[0092] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute any of the above-mentioned methods.

[0093] like Figure 7 As shown, the present specification provides a schematic diagram of the structure of a non-volatile storage medium. Figure 7 It can be seen that in one or more embodiments of this specification, a non-volatile storage medium stores computer executable instructions 701, and the computer executable instructions 701 can: execute any of the methods described above.

[0094] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0095] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0096] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A distributed training method based on heterogeneous devices, characterized in that: The method comprises: The hardware cluster of the current heterogeneous device is divided to obtain a plurality of logical computing units; wherein the logical computing unit comprises: an architecture computing unit; the architecture computing unit comprises a node computing unit; Determine an existing segmentation method corresponding to each operator in the current model training task, and determine multiple intra-operator parallel strategies and inter-operator parallel strategies of each operator according to the calculation graph corresponding to the current model training task and the existing segmentation method; Calculating the computation delay and the communication delay corresponding to each parallel strategy, so as to search and obtain the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy based on the computation delay and the communication delay; According to mapping the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy to corresponding node computing units, the node computing units of the heterogeneous devices perform distributed training on the current model training task based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy; The step of calculating the computation delay and the communication delay corresponding to each parallel strategy, and searching for an optimal intra-operator parallel strategy and an optimal inter-operator parallel strategy based on the computation delay and the communication delay, specifically includes: Performing mixed segmentation on the intra-operator parallel strategies to obtain intra-operator parallel strategies to be optimized, and determining the total delay of each intra-operator parallel strategy to be optimized according to the first calculation delay and the first communication delay of the intra-operator parallel strategies to be optimized; Performing pipeline parallel segmentation on the inter-operator parallel strategy to obtain the inter-operator parallel strategy to be optimized, and determining the total delay of each of the inter-operator parallel strategies to be optimized according to the second calculation delay and the second communication delay of the inter-operator parallel strategy to be optimized; Minimizing the total delay of the intra-operator parallel strategy to be optimized to search for the optimal intra-operator parallel strategy; Minimizing the total delay of the inter-operator parallel strategy to be optimized to search for the optimal inter-operator parallel strategy; The intra-operator parallel strategies are mixed and split to obtain the intra-operator parallel strategies to be optimized, and the total delay of each of the intra-operator parallel strategies to be optimized is determined according to the first calculation delay and the first communication delay of the intra-operator parallel strategies to be optimized, specifically including: By randomly searching the intra-operator parallel computing graph partition set, a mixed partition of the data parallel strategy and the model parallel strategy in the intra-operator parallel computing graph partition set is realized to obtain the intra-operator parallel strategy to be optimized; Determine a first communication volume of the parallel strategy within the operator to be optimized based on the matrix dimension data of the forward propagation and the backward propagation corresponding to the hybrid segmentation; Determining a first communication delay of the parallel strategy within the operator to be optimized according to the first communication volume and an actual hardware bandwidth of the hardware cluster; Determining a first computing delay of the parallel strategy within the operator to be optimized according to the computing cost of the parallel strategy within the operator to be optimized and the hardware computing power of the hardware cluster; The first computation delay and the first communication delay of each of the parallel strategies within the to-be-optimized operators are added together to determine a total delay of the parallel strategies within the to-be-optimized operators.

2. A distributed training method based on heterogeneous devices according to claim 1, characterized in that: Split the current hardware cluster of heterogeneous devices to obtain multiple logical computing units, including: Obtain the physical structure differences of each heterogeneous device in the current hardware cluster of heterogeneous devices; The current hardware cluster is divided according to the physical structure difference to obtain multiple logical computing units.

3. The distributed training method based on heterogeneous devices according to claim 1, characterized in that: Determine the existing segmentation method corresponding to each operator in the current model training task to obtain multiple intra-operator parallel strategies and inter-operator parallel strategies of each operator, specifically including: Determine the existing segmentation method corresponding to each operator in the current model training task; The computation graph is segmented according to an existing operator segmentation method to obtain an intra-operator parallel computation graph segmentation set and an inter-operator parallel computation graph segmentation set; wherein the intra-operator parallel computation graph segmentation set includes multiple intra-operator parallel strategies of each operator; and the inter-operator parallel computation graph segmentation set includes multiple inter-operator parallel strategies of each operator.

4. The distributed training method based on heterogeneous devices according to claim 1, characterized in that: The inter-operator parallel strategy is pipelined and parallelized to obtain the inter-operator parallel strategy to be optimized, and the total delay of each of the inter-operator parallel strategies to be optimized is determined according to the second calculation delay and the second communication delay of the inter-operator parallel strategy to be optimized, specifically including: By randomly searching the segmentation set of the inter-operator parallel computing graph, pipeline parallel segmentation is performed on the inter-operator parallel strategy to obtain the inter-operator parallel strategy to be optimized; Determining a second communication volume of the parallel strategy between operators to be optimized based on the matrix dimension data of the forward propagation and the backward propagation corresponding to the parallel segmentation of the pipeline; Determining a second communication delay of the inter-operator parallel strategy to be optimized based on the operation feedback data of the inter-operator parallel strategy to be optimized; The second computation delay and the second communication delay of each of the inter-operator parallel strategies to be optimized are added together to determine a total delay of the inter-operator parallel strategies to be optimized.

5. The distributed training method based on heterogeneous devices according to claim 1, characterized in that: Mapping the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy to corresponding node computing units, so that the node computing units of the heterogeneous devices perform distributed training on the current model training task based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy, specifically includes: Based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy, the model training task corresponding to each operator is divided to obtain multiple model training subtasks; Allocate each of the model training subtasks to a corresponding node computing unit based on the correspondence between the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy; Based on each of the node computing units, distributed training is performed on multiple model training subtasks corresponding to the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy.

6. A distributed training device based on heterogeneous devices, characterized in that: The device comprises: The unit splitting unit is used to split the hardware cluster of the current heterogeneous device to obtain multiple logical computing units; wherein the logical computing unit includes: an architecture computing unit; the architecture computing unit includes a node computing unit; A parallel strategy determination unit, used to determine an existing segmentation method corresponding to each operator in the current model training task, and determine multiple intra-operator parallel strategies and inter-operator parallel strategies of each operator according to the calculation graph corresponding to the current model training task and the existing segmentation method; A parallel strategy optimization unit, used to calculate the computation delay and communication delay corresponding to each parallel strategy, so as to search and obtain the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy based on the computation delay and communication delay; A training unit, configured to map the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy to corresponding node computing units, so that the node computing units of the heterogeneous devices perform distributed training on the current model training task based on the optimal intra-operator parallel strategy and the optimal inter-operator parallel strategy; The step of calculating the computation delay and the communication delay corresponding to each parallel strategy, and searching for an optimal intra-operator parallel strategy and an optimal inter-operator parallel strategy based on the computation delay and the communication delay, specifically includes: Performing mixed segmentation on the intra-operator parallel strategies to obtain intra-operator parallel strategies to be optimized, and determining the total delay of each intra-operator parallel strategy to be optimized according to the first calculation delay and the first communication delay of the intra-operator parallel strategies to be optimized; Performing pipeline parallel segmentation on the inter-operator parallel strategy to obtain the inter-operator parallel strategy to be optimized, and determining the total delay of each of the inter-operator parallel strategies to be optimized according to the second calculation delay and the second communication delay of the inter-operator parallel strategy to be optimized; Minimizing the total delay of the intra-operator parallel strategy to be optimized to search for the optimal intra-operator parallel strategy; Minimizing the total delay of the inter-operator parallel strategy to be optimized to search for the optimal inter-operator parallel strategy; The intra-operator parallel strategies are mixed and split to obtain the intra-operator parallel strategies to be optimized, and the total delay of each of the intra-operator parallel strategies to be optimized is determined according to the first calculation delay and the first communication delay of the intra-operator parallel strategies to be optimized, specifically including: By randomly searching the intra-operator parallel computing graph partition set, a mixed partition of the data parallel strategy and the model parallel strategy in the intra-operator parallel computing graph partition set is realized to obtain the intra-operator parallel strategy to be optimized; Determine a first communication volume of the parallel strategy within the operator to be optimized based on the matrix dimension data of the forward propagation and the backward propagation corresponding to the hybrid segmentation; Determining a first communication delay of the parallel strategy within the operator to be optimized according to the first communication volume and an actual hardware bandwidth of the hardware cluster; Determining a first computing delay of the parallel strategy within the operator to be optimized according to the computing cost of the parallel strategy within the operator to be optimized and the hardware computing power of the hardware cluster; The first computation delay and the first communication delay of each of the parallel strategies within the to-be-optimized operators are added together to determine a total delay of the parallel strategies within the to-be-optimized operators.

7. A distributed training device based on heterogeneous devices, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: execute any of the methods described in claims 1-5 above.

8. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions can: execute the method described in any one of claims 1 to 5.

Images