Cloud edge-end heterogeneous resource cooperative training strategy based on hierarchical hybrid parallelization

Abstract

The invention discloses a cloud edge end heterogeneous resource cooperative training strategy based on hierarchical hybrid parallel, and aims to research how a cross-domain distributed computing framework fully utilizes various cloud edge end heterogeneous computing resources, research an equipment graph and computing graph construction mechanism, construct a simplified and merged computing graph and equipment graph, and reduce the search space of the hierarchical hybrid parallel strategy. A search algorithm of the hierarchical hybrid parallel algorithm is researched, a possible hierarchical hybrid parallel algorithm is searched through a dynamic programming algorithm and a resource constraint mechanism, and efficient hierarchical hybrid parallel in a cloud edge-end collaborative cross-domain scene is realized; a cross-domain hierarchical hybrid parallel optimization strategy during operation is studied, the calculation speed between nodes is adjusted through calculation of data loads in clusters and secondary balanced distribution of the calculation loads, and slow nodes during operation are avoided. By means of the mechanism, efficient adaptation of the cloud edge heterogeneous computing resources and the hybrid parallel strategy is achieved, the utilization efficiency of the cloud edge heterogeneous computing resources is improved, and the cross-domain distributed training speed of the model is increased.

Description

Technical Field

[0001] The present invention relates to the field of computers, and particularly to a cloud-edge-end heterogeneous resource collaborative training strategy based on hierarchical hybrid parallelism. Background Art

[0002] During the training process of deep learning models, the models will be divided into different parts and run in different computing nodes with the same configuration and certain topology to achieve large-scale model training; in the cloud-edge computing environment, the diverse devices and heterogeneous environments have undoubtedly become obstacles to the collaborative training of cloud-edge models. Therefore, how to make full use of diverse cloud-edge heterogeneous computing resources has become an important issue in building a cross-domain distributed computing framework. Current research mainly focuses on optimizing the operation of resource-constrained device models and coordinating the joint operation of multiple heterogeneous nodes. The following will introduce the relevant work respectively.

[0003] 1. Optimizing model operation on resource-constrained devices

[0004] Optimizing model execution on resource-constrained devices means using sparsification, compilation, and conversion to enable model training on heterogeneous devices such as CPU, GPU, and DSP.

[0005] Zhu et al. proposed PockEngine, which can fine-tune large models in embedded devices. First, PockEngine supports sparse backpropagation, and updates the model sparsely while maintaining model quality by pruning the backpropagation graph and memory measurement. Second, PockEngine compiles the model before running to generate a complete training graph, reducing the computational overhead of edge heterogeneous devices. Finally, PockEngine integrates a rich training graph optimization to further reduce training costs. Jiang et al. proposed an MNN engine that can run models in heterogeneous devices. MNN optimizes the operators running in edge devices at the kernel level, provides the ability to run models in lightweight devices, even IoT devices, and provides support for multiple front-ends and back-ends. Patil et al. proposed the Poet system for fine-tuning models using external storage on tiny devices. Poet uses a comprehensive search optimization space of recomputation and paging mechanisms to reduce the memory consumption of backpropagation, and designs a MILP planner combined with energy-optimized training to achieve model fine-tuning in edge devices and minimize energy consumption. Wang et al. proposed AtRec, a technology for accelerating recommendation models using heterogeneous CPU hardware. AtRec identifies operators that take a long time to compute in the CPU and optimizes them so that they do not become bottlenecks in the computational graph. AtRec can also identify the overall inefficient parts of the running graph and optimize them by eliminating redundant computations and improving memory access efficiency. Zhang et al. proposed 2PGraph, a technology for accelerating graph neural network training in heterogeneous hardware. 2PGraph addresses the problems of long memory access latency and inefficient data transmission of vertex features from the CPU to the GPU, and proposes a GNN layer-aware feature caching method on available GPU resources to avoid redundant data transmission between the CPU and the GPU. Zhang et al. proposed AGL, a scalable heterogeneous training technology for graph neural networks. AGL designs a K-hop neighborhood data transmission algorithm for the underlying message passing mode of GNN computing to avoid message passing overhead across nodes and across heterogeneous hardware.

[0006] The existing optimization of single-device model operation mainly uses sparsification, compilation, conversion and other means to enable the model to run in heterogeneous devices such as CPU, GPU, DSP, etc. At present, it is already possible to train the model in most edge heterogeneous devices. In the future, sparsification, compilation and other means will continue to be used to optimize the running efficiency of the model in various heterogeneous devices.

[0007] 2. Related work on the collaborative multi-heterogeneous node joint operation model

[0008] The collaborative multi-heterogeneous node joint operation model refers to a technology that coordinates multiple heterogeneous resource nodes to collaboratively implement the model running in heterogeneous nodes at the cloud edge.

[0009] Ye et al. proposed the Asteroid system that uses pipeline parallelism to coordinate DNNs running on heterogeneous edges. Asteroid uses hybrid pipeline parallelism to coordinate distributed training in heterogeneous devices and can perform automatic parallelism planning to maximize throughput under certain resource constraints; in addition, Asteroid also developed fault-tolerant computing using pipeline replay technology. Mao et al. proposed the MoDNN system that coordinates multiple mobile computing devices for model computing. MoDNN can partition the trained DNN model onto multiple mobile devices, accelerating DNN computing by reducing device-level computing costs and memory usage; in addition, MoDNN also designed two model partitioning schemes to minimize non-parallel data transmission time. Zeng et al. proposed the CoEdge system that coordinates multiple edge devices through adaptive workloads. CoEdge utilizes the available computing and communication resources at the edge to dynamically divide DNN workloads according to the computing power and network conditions of the device. Zhao et al. proposed the DeepThings technology that runs in collaborative resource-constrained IoT devices and edge clusters. DeepThings uses Scalable Convolutional Fusion Partitioning (FTP) to achieve parallelism while minimizing memory usage; and implements a distributed work-stealing method to achieve dynamic workload distribution and balancing at runtime. Hao and Zhang et al. implemented an EDDL system for model training in resource-constrained devices. EDDL designed a dynamic training data distribution mechanism to distribute training data to worker nodes at runtime; in addition, EDDL proposed an adaptive inter-node communication mechanism and leader node rotation mechanism to accelerate model training at the edge. Yang et al. proposed the AutoSF technology for model training in dynamic edges. When the edge environment changes, AutoSF triggers an automated machine learning method to find the nearly optimal aggregation structure and frequency that adapts to time-varying edge resources; when the environment remains unchanged, AutoSF uses a heuristic method to adjust the aggregation structure and frequency to further suppress the heterogeneity caused by data distribution. Chen et al. proposed the FTPipeHD technology for pipeline parallelism in heterogeneous edge devices. FTPipeHD dynamically optimizes partition points based on real-time computing power, and also proposes a novel weight redistribution method to periodically copy weights to adjacent nodes and central nodes to solve the problem of failure during training.

[0010] The existing collaborative multi-heterogeneous node joint operation models mainly focus on heterogeneous node modeling and how to comprehensively utilize heterogeneous node computing resources through parallel strategies. The model collaborative reasoning technology is relatively mature, and the collaborative training technology is initially emerging. The computing efficiency and heterogeneous resource utilization are relatively low and far from mature. In the future, it can be foreseen that there will be more and more methods for collaborative multi-heterogeneous cloud-edge node model training.

[0011] In summary, for heterogeneous hardware training in cloud-edge computing environments, there are two main ideas: making full use of single-node resources and coordinating multiple heterogeneous nodes to run models. However, most of them are currently aimed at model heterogeneous reasoning scenarios, and related research on heterogeneous training has emerged in recent years. Specifically, the existing optimization of single-device model operation mainly uses sparsification, compilation, conversion and other means to enable the model to run in heterogeneous devices such as CPU, GPU, DSP, etc., and it can currently realize the operation of the model in edge heterogeneous devices, laying a good foundation; the existing collaborative multi-heterogeneous node joint operation model mainly focuses on heterogeneous node modeling and comprehensive utilization of heterogeneous computing resources through parallel strategies. In addition, the model collaborative reasoning technology is relatively mature, and the collaborative training technology has initially emerged. The computing efficiency and heterogeneous resource utilization are relatively low and far from mature. In general, the existing heterogeneous hardware collaborative training technology in the cloud-edge computing environment is far from mature, facing the problems of low computing efficiency and low utilization of heterogeneous resources, which limits the efficiency of comprehensive utilization of heterogeneous hardware for model training in the cloud-edge computing environment. Therefore, it is extremely important to study a heterogeneous hardware training strategy suitable for the cloud-edge computing environment. How to make full use of diverse cloud-edge heterogeneous computing resources has become an urgent problem to be solved. Summary of the invention

[0012] In response to the above problems, the present invention proposes a cloud-edge heterogeneous resource collaborative training strategy based on hierarchical hybrid parallelism, and studies and designs a heterogeneous resource collaborative training strategy for cloud-edge environment characteristics and applications. The overall idea of ​​the present invention is: first, the present invention obtains a hierarchical hybrid parallel strategy suitable for the current cloud-edge heterogeneous hardware configuration through device graph, calculation graph construction and fully automatic parallel plan search, and reasonably organizes cloud-edge heterogeneous hardware to jointly complete model training tasks, solving the challenge of collaborative cross-domain heterogeneous computing node operation; secondly, the present invention uses Micro Batch's dynamic load optimization during runtime to unload the computing load of slow nodes in real time, and combines multiple cloud-edge devices to complete the computing tasks of the same link, solving the challenge of insufficient resources of a single edge device to complete model training.

[0013] Specifically, the present invention consists of the following steps:

[0014] Step (1): Device graph and computation graph construction mechanism.

[0015] Step (2): Hierarchical hybrid parallel search algorithm.

[0016] Step (3): Model runtime optimization mechanism.

[0017] Each step is further broken down into a series of sub-steps.

[0018] Step (1.1) injects Profiler into each working node and collects node heterogeneous resource information through Profiler.

[0019] When collecting network information, the node matrix is ​​used To represent all nodes in the cloud-edge computing environment that start the current training task, these computing nodes are connected via Ethernet in the computing environment. Specifically, the present invention uses "psutil" to collect the resource information of the computing node p, and uses the node CPU core number C p 、Memory capacity M p , GPU computing power GPU memory capacity XPU computing power XPU storage capacity Network bandwidth between worker nodes Storage capacity p To express.

[0020] Step (1.2) constructs the above node resource information into a vector ρ and aggregates it to the master node to perform subsequent calculation processes.

[0021] Step (1.3) After the data of each node is uploaded to the master node, the present invention performs min-max normalization to construct different working node description vectors p.

[0022] Step (1.4) The present invention performs an improved Bi-kmeans algorithm on the working nodes to divide the device graph into classes, and uses the following calculation formula instead of the calculation of the sum of squared errors as the basis for clustering:

[0023]

[0024] Specifically, I and J represent the computing clusters after division, p i and p j Represent the description vectors of the working nodes in the computing cluster after division, N i and N jRepresents the network bandwidth between different nodes. The present invention minimizes the performance difference between different computing clusters and maximizes the network connection bandwidth between different computing clusters through the above formula, thereby achieving the aggregation of different device graphs to reduce the search space of the hierarchical hybrid parallel strategy.

[0025] Step (1.5) runs the device graph optimization algorithm and makes adjustments based on clustering to balance the computing power and communication capabilities between each computing cluster to avoid obvious slow nodes.

[0026] Specifically, after obtaining the optimal computation graph through affinity clustering, the present invention selects different computation clusters I and J in pairs, and traverses all nodes in computation clusters I and J, and obtains the minimum communication cost from any node in computation cluster I to any node in computation cluster J through the shortest path algorithm, using The communication cost between any two points in different computing clusters is represented by . Finally, the computing cluster optimization algorithm is executed to traverse different nodes in the computing cluster, color them according to their network connection relationship, computing power, and storage capacity, and pre-assign different roles to each node, such as heavy computing node and heavy communication node.

[0027] Step (1.6) performs model parsing through TorchFx to obtain the intermediate representation IR of the model, and passes the relevant IR information to the model code injection and calculation graph simplification module. The present invention makes full use of the model conversion capability provided by TorchFx. When performing TorchFx parsing, a graph object will be obtained, which describes the composition of the calculation graph, including objects such as placeholder, call_function, call_moudle, etc., and describes the composition of the calculation graph in detail.

[0028] Step (1.7) After the model code injection module receives the computational graph IR, it will run the communication code injection, optimizer conversion and model structure conversion procedures to replace the components in the model with the optimized distributed versions. The present invention defines an abstract Transform object that is responsible for the insertion of distributed communication code, optimizer conversion and specific structure conversion (including distributed optimization conversion of modules such as multi-head attention and CNN), and forms an optimized new Python code package for subsequent operation.

[0029] At the end of step (1.8), according to the computational graph IR, the computational graph simplification module calculates the activation value of the model, the size of the weight, and identifies the specific model structure to simplify the computational graph for subsequent model segmentation point search. First, the present invention searches for heavy computational modules, such as multi-head attention modules, GNN modules, etc., and merges their nodes so that they can be divided into the same computational cluster during hierarchical hybrid parallel search; then the present invention searches for heavy communication modules, such as RNN modules, etc., and labels and colors these modules so that they can work together with the device graph to better search the parallel space.

[0030] Step (2.1) analyzes the forward propagation and backward propagation time of each key calculation stage in the simplified calculation graph. The activation value and weight of the level l subgraph are respectively denoted as a l and w l . It is assumed that the time of forward propagation and back propagation is inversely proportional to the computing power and proportional to the batch size. The forward propagation and back propagation time of the l-th level subgraph with batch size β in the computing cluster d are respectively and Secondly, this method uses the forward propagation time and the backward propagation time to estimate the computational waiting time of the l-th level computational subgraph λ:

[0031]

[0032] This method uses the above formula to get the computation waiting time of the l-th level subgraph. The communication waiting time of the l-th level subgraph is given below. It is worth noting that we assume that AllReduce is used for communication:

[0033]

[0034] This method uses the above formula to estimate the AllReduce communication time of the model. The time to execute the calculation is given below, assuming that only forward propagation and directional propagation are generated:

[0035]

[0036] This method uses the above formula to get the computation time of each computation subgraph, so the computation time of each round can be defined as:

[0037]

[0038] Through the above, this method completes the modeling of the model calculation time in the cross-domain distributed environment, and models its calculation time as a unified calculation standard T, and gives a detailed calculation process.

[0039] Step (2.1) Based on the above modeling, the hierarchical hybrid parallel search algorithm is modeled as a dynamic programming algorithm. This method uses this algorithm to obtain a better hierarchical hybrid parallel strategy. This method uses Q(l,n,p) to represent the optimal delay when the last l-level subgraph is divided into p stages and placed in the last n devices. The goal of this method is to calculate and update:

[0040]

[0041] in It represents the minimum computation delay without adding the last level l subgraph and the last n devices. This method obtains multiple optimized hierarchical hybrid parallel solutions through the above optimization.

[0042] Step (3.1) When the model training task is started, the computational cluster and computational graph finally formed by the hierarchical hybrid parallel search algorithm will be distributed to each node. Each node will form a physical computational cluster and execute the corresponding model training task. When the task is started, each node in the computational cluster broadcasts a data packet to complete node election, and arranges the order and process of state information synchronization by combining the hierarchical hybrid parallel strategy.

[0043] Step (3.2) synchronizes the state information within the computing cluster, and completes the state information synchronization within the computing cluster by adding state information tensors during forward propagation and backward propagation.

[0044] Step (3.3) starts the computing status monitoring program on the master node, which counts the delays of different nodes in the computing cluster. The information reported by each node to the master node mainly includes the startup time of the computing node Forward propagation duration Backpropagation duration Activation value size a i , weight size w i , this method uses this information to estimate the end-to-end delay and slow node bottleneck. It is worth noting that if node i is the upstream node of the leader node, its weight-related data is zero; otherwise, the activation value-related data is zero. First, we use the following formula to get the end-to-end delay:

[0045]

[0046] Then the transmission delay is calculated for each upstream and downstream node. This method simply assumes that the transmission delay is proportional to the weight and activation value, and inversely proportional to the network bandwidth:

[0047]

[0048] Through the above simple calculation and modeling, the present invention detects fluctuations beyond the range through an outlier detection mechanism, namely:

[0049]

[0050] Where μ is a positive number. When the end-to-end delay between devices is less than the average value for more than a certain range, this method considers that the computing performance or communication capability is insufficient and the balancing mechanism needs to be triggered.

[0051] Step (3.4) Further, when the present invention detects insufficient performance, different problems are respectively dealt with through data balancing and tensor balancing. First, the critical path of the computing node where the performance bottleneck occurs is identified, as well as the steps where the performance bottleneck occurs. The present invention can mainly deal with the following problems: ① Insufficient computing performance; ② Poor communication with other pipeline nodes; ③ Poor communication with other pipeline nodes at the same stage, which can be identified by computing time, forward and reverse propagation time, and AllReduce time, respectively. Secondly, the present invention adopts different means to deal with the above problems. ① In view of the problem of insufficient computing performance, the present invention identifies the parallel stage in which it is located. If it adopts data parallelism, it reduces its data load according to the proportion of computing time through micro-batch scheduling. If it adopts tensor parallelism, it inserts AllReduce communication in the micro-batch, synchronizes weight information in advance, and re-divides the tensor partitioning method as soon as possible. ② In view of the problem of poor communication with other pipeline nodes, if it adopts data parallelism, all data loads are temporarily unloaded to the remaining nodes. If it adopts tensor parallelism, the preferred communication path forwards communication through other nodes. ③ To address the problem of poor communication with other pipeline nodes at the same stage, if data parallelism is used, the communication path is preferably forwarded through other nodes. If tensor parallelism is used, the tensor parallel nodes are temporarily excluded, and technologies such as ZeRO are used to use other nodes to complete the pipeline calculations at this stage.

[0052] The advantages of the present invention are:

[0053] By proposing a layered hybrid parallel cross-domain distributed training mechanism, cross-domain distributed training of heterogeneous hardware is realized to fully utilize diverse cloud-edge heterogeneous computing resources. This method generates a preliminary cross-domain parallel training plan by collecting information before running, and adjusts data and tensor partitioning at run time to further balance resources to achieve layered hybrid parallel cross-domain heterogeneous hardware training; compared with existing methods, it has the advantages of high environmental adaptability, high model adaptation flexibility, and low computing overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of the cloud-edge-end heterogeneous resource collaborative training strategy method based on layered hybrid parallelism proposed in the present invention.

[0055] Figure 2 This is a schematic diagram of the device graph construction mechanism based on computing cluster aggregation proposed in the present invention.

[0056] Figure 3 Schematic diagram of the computational graph construction mechanism based on model compilation proposed in the present invention.

[0057] Figure 4 This is a schematic diagram of the hierarchical hybrid parallel algorithm search process proposed in the present invention.

[0058] Figure 5 This is a schematic diagram of the model runtime state synchronization mechanism proposed by the present invention.

[0059] Figure 6 This is a schematic diagram of the data and tensor balancing mechanism based on Micro Batch when the model proposed in the present invention is running. DETAILED DESCRIPTION

[0060] Combined with the attached figure ( Figure 1-Figure 6 ) The present invention is further described in detail.

[0061] A cloud-edge-end heterogeneous resource collaborative training strategy based on hierarchical hybrid parallelism is studied to solve the problem of how the cross-domain distributed computing framework can fully utilize the diverse cloud-edge-end heterogeneous computing resources. Firstly, the device graph and computing graph construction mechanism are studied. Through the computing cluster aggregation algorithm and model compilation mechanism, the simplified and merged computing graph and device graph are constructed to narrow the search space of the hierarchical hybrid parallel strategy. Secondly, the search algorithm of the hierarchical hybrid parallel algorithm is studied. Through the dynamic programming algorithm and resource constraint mechanism, possible hierarchical hybrid parallel algorithms are searched to achieve efficient hierarchical hybrid parallelism in the cloud-edge-end collaborative cross-domain scenario. Finally, the cross-domain hierarchical hybrid parallel optimization strategy at runtime is studied. Through the secondary balanced distribution of data load and computing load within the computing cluster, the computing speed between each node is adjusted to avoid the appearance of slow nodes at runtime. Through the above mechanism, efficient adaptation of cloud-edge heterogeneous computing resources and hybrid parallel strategy is achieved, the utilization efficiency of cloud-edge heterogeneous computing resources is improved, and the cross-domain distributed training speed of the model is improved.

[0062] Figure 1The overall scheme of the cloud-edge-end heterogeneous resource collaborative training strategy of layered hybrid parallelism is presented, which includes steps (1), (2), and (3). First, the device graph and computation graph construction mechanism are studied. Through the computing cluster aggregation algorithm and model compilation mechanism, the simplified and merged computation graph and device graph are constructed to narrow the search space of the layered hybrid parallel strategy; secondly, the search algorithm of the layered hybrid parallel algorithm is studied. Through the dynamic programming algorithm and resource constraint mechanism, possible layered hybrid parallel algorithms are searched to achieve efficient layered hybrid parallelism in the cloud-edge-end collaborative cross-domain scenario; finally, the cross-domain layered hybrid parallel optimization strategy at runtime is studied. Through the secondary balanced distribution of data load and computing load within the computing cluster, the computing speed between each node is adjusted to avoid the emergence of slow nodes at runtime. Through the above mechanism, efficient adaptation of cloud-edge heterogeneous computing resources and hybrid parallel strategy is achieved, the utilization efficiency of cloud-edge heterogeneous computing resources is improved, and the cross-domain distributed training speed of the model is improved.

[0063] Figure 2 The workflow of the device graph construction method for computing cluster aggregation is shown, including steps (1.1) to (1.5). First, inject Profiler into each working node, collect node heterogeneous resource information through Profiler, and report the heterogeneous resource information to the master node to run the clustering algorithm and the device graph construction algorithm; secondly, run the affinity clustering algorithm on the master node, calculate the affinity between nodes through the computing, communication, and storage capabilities of the working nodes, and run the clustering algorithm to obtain a better match as the basis for target optimization; finally, run the device graph optimization algorithm, make adjustments based on clustering, balance the computing power and communication capabilities between each computing cluster, and avoid the appearance of obvious slow nodes.

[0064] Figure 3 The workflow of the computational graph construction method based on model compilation is shown, including steps (1.6) to (1.8). First, the model is parsed through TorchFx to obtain the intermediate representation IR of the model, and the relevant IR information is passed to the model code injection and computational graph simplification module; secondly, when the model code injection module receives the computational graph IR, it will run the communication code injection, optimizer conversion and model structure conversion program to replace the components in the model with the optimized distributed version; finally, according to the computational graph IR, the computational graph simplification module calculates the activation value and weight of the model, and identifies the specific model structure to simplify the computational graph for the subsequent search of model splitting points.

[0065] like Figure 4As shown, the search process of the hierarchical hybrid parallel algorithm is shown, including steps (2.1) to (2.2). First, the preferred device graph and computation graph generated in the first step are accepted, and the time of communication, computation, etc. is estimated through the relevant device capabilities and model information to model the delay information. Then, the dynamic programming algorithm with the running target Q is used to allocate different computing stages to different computing clusters, and tensors and data are simply proportionally divided in the computing cluster according to the computing power of a single device to obtain multiple hierarchical hybrid parallel plans.

[0066] Figure 5 The state synchronization mechanism of the model during cross-domain distributed runtime is demonstrated, including steps (3.1) and (3.2). It mainly consists of leader node election and inter-device state synchronization. First, when the model training task is started, the computing cluster and computing graph finally formed by the hierarchical hybrid parallel search algorithm will be distributed to each node. Each node will form a physical computing cluster on its own and execute the corresponding model training task. When the task is started, each node in the computing cluster broadcasts a data packet to complete the node election, and by combining the hierarchical hybrid parallel strategy, arranges the order and process of state information synchronization. Secondly, synchronize the state information within the computing cluster, such as Figure 5 As shown in the figure, an example of state synchronization of a computing cluster consisting of 4 computing nodes is shown. Among them, device 1 performs the first phase of calculation, devices 2 and 3 perform the second phase of calculation, and device 4 performs the third phase of calculation; device 3 is elected as the leader node to monitor the calculation status in the computing cluster. Six state synchronization opportunities are marked in the figure, in which the state information of devices 1, 4, 1, 4, 4, and 3 are synchronized respectively. Through the aggregation and synchronization of these state information, device 3 can obtain complete monitoring information in the computing cluster.

[0067] Figure 6The general workflow of the data and tensor balancing mechanism based on Micro Batch when the model is running is shown, including steps (3.3) and (3.4). First, the computing status monitoring program is started on the master node, which counts the delays of different nodes in the computing cluster. Furthermore, when the method detects insufficient performance, different problems are dealt with through data balancing and tensor balancing. First, the critical path of the computing node with performance bottleneck and the steps where the performance bottleneck occurs are identified. This method can mainly deal with the following problems: ① Insufficient computing performance; ② Poor communication with other pipeline nodes; ③ Poor communication with other pipeline nodes at the same stage, which can be identified by computing time, forward and backward propagation time, and AllReduce time respectively. Secondly, this method adopts different means to deal with the above problems. ① For the problem of insufficient computing performance, this method identifies the parallel stage in which it is located. If it adopts data parallelism, it reduces its data load according to the proportion of computing time through micro-batch scheduling. If it adopts tensor parallelism, it inserts AllReduce communication in the micro-batch, synchronizes weight information in advance, and re-divides the tensor partitioning method as soon as possible. ② For the problem of poor communication with other pipeline nodes, if data parallelism is used, all data loads are temporarily unloaded to the remaining nodes. If tensor parallelism is used, the communication path is preferably forwarded through other nodes. ③ For the problem of poor communication with other pipeline nodes in the same stage, if data parallelism is used, the communication path is preferably forwarded through other nodes. If tensor parallelism is used, the tensor parallel nodes are temporarily excluded, and other nodes are used to complete the pipeline calculation of this stage using technologies such as ZeRO.

[0068] The present invention proposes a cloud-edge heterogeneous resource collaborative training strategy based on hierarchical hybrid parallelism, and studies and designs a heterogeneous resource collaborative training strategy for cloud-edge environment characteristics and applications. The overall idea of ​​the present invention is: first, the present invention obtains a hierarchical hybrid parallel strategy suitable for the current cloud-edge heterogeneous hardware configuration through device graph, calculation graph construction and fully automatic parallel plan search, and reasonably organizes cloud-edge heterogeneous hardware to jointly complete model training tasks, solving the challenge of collaborative cross-domain heterogeneous computing node operation; secondly, the present invention uses Micro Batch's dynamic load optimization during runtime to unload the computing load of slow nodes in real time, and combines multiple cloud-edge devices to complete the computing tasks of the same link, solving the challenge of insufficient resources of a single edge device that cannot complete model training.

[0069] Specifically, the present invention consists of the following steps:

[0070] Step (1): Device graph and computation graph construction mechanism.

[0071] Step (2): Hierarchical hybrid parallel search algorithm.

[0072] Step (3): Model runtime optimization mechanism.

[0073] According to the method described in the above-mentioned cloud-edge heterogeneous resource collaborative training strategy, the specific process of step (1) is step (1.1) to step (1.8).

[0074] Step (1.1) injects Profiler into each working node and collects node heterogeneous resource information through Profiler.

[0075] When collecting network information, the node matrix is ​​used To represent all nodes in the cloud-edge computing environment that start the current training task, these computing nodes are connected via Ethernet in the computing environment. Specifically, the present invention uses "psutil" to collect the resource information of the computing node p, and uses the node CPU core number C p 、Memory capacity M p , GPU computing power GPU memory capacity XPU computing power XPU storage capacity Network bandwidth between worker nodes Storage capacity p To express.

[0076] Step (1.2) constructs the above node resource information into a vector ρ and aggregates it to the master node to perform subsequent calculation processes.

[0077] Step (1.3) After the data of each node is uploaded to the master node, the present invention performs min-max normalization to construct different working node description vectors p.

[0078] Step (1.4) The present invention performs an improved Bi-kmeans algorithm on the working nodes to divide the device graph into classes, and uses the following calculation formula instead of the calculation of the sum of squared errors as the basis for clustering:

[0079]

[0080] Specifically, I and J represent the computing clusters after division, p i and p j Represent the description vectors of the working nodes in the computing cluster after division, N i and N j Represents the network bandwidth between different nodes. The present invention minimizes the performance difference between different computing clusters and maximizes the network connection bandwidth between different computing clusters through the above formula, thereby achieving the aggregation of different device graphs to reduce the search space of the hierarchical hybrid parallel strategy.

[0081] Step (1.5) runs the device graph optimization algorithm and makes adjustments based on clustering to balance the computing power and communication capabilities between each computing cluster to avoid obvious slow nodes.

[0082] Specifically, after obtaining the optimal computation graph through affinity clustering, the present invention selects different computation clusters I and J in pairs, and traverses all nodes in computation clusters I and J, and obtains the minimum communication cost from any node in computation cluster I to any node in computation cluster J through the shortest path algorithm, using The communication cost between any two points in different computing clusters is represented by . Finally, the computing cluster optimization algorithm is executed to traverse different nodes in the computing cluster, color them according to their network connection relationship, computing power, and storage capacity, and pre-assign different roles to each node, such as heavy computing node and heavy communication node.

[0083] Step (1.6) performs model parsing through TorchFx to obtain the intermediate representation IR of the model, and passes the relevant IR information to the model code injection and calculation graph simplification module. The present invention makes full use of the model conversion capability provided by TorchFx. When performing TorchFx parsing, a graph object will be obtained, which describes the composition of the calculation graph, including objects such as placeholder, call_function, call_moudle, etc., and describes the composition of the calculation graph in detail.

[0084] Step (1.7) After the model code injection module receives the computational graph IR, it will run the communication code injection, optimizer conversion and model structure conversion procedures to replace the components in the model with the optimized distributed versions. The present invention defines an abstract Transform object that is responsible for the insertion of distributed communication code, optimizer conversion and specific structure conversion (including distributed optimization conversion of modules such as multi-head attention and CNN), and forms an optimized new Python code package for subsequent operation.

[0085] At the end of step (1.8), according to the computational graph IR, the computational graph simplification module calculates the activation value of the model, the size of the weight, and identifies the specific model structure to simplify the computational graph for subsequent model segmentation point search. First, the present invention searches for heavy computational modules, such as multi-head attention modules, GNN modules, etc., and merges their nodes so that they can be divided into the same computational cluster during hierarchical hybrid parallel search; then the present invention searches for heavy communication modules, such as RNN modules, etc., and labels and colors these modules so that they can work together with the device graph to better search the parallel space.

[0086] According to the method described in the above-mentioned cloud-edge heterogeneous resource collaborative training strategy, the specific process of step (2) is step (2.1) to step (2.2).

[0087] Step (2.1) analyzes the forward propagation and backward propagation time of each key calculation stage in the simplified calculation graph. The activation value and weight of the level l subgraph are respectively denoted as a l and w l . It is assumed that the time of forward propagation and back propagation is inversely proportional to the computing power and proportional to the batch size. The forward propagation and back propagation time of the l-th level subgraph with batch size β in the computing cluster d are respectively and Secondly, this method uses the forward propagation time and the backward propagation time to estimate the computational waiting time of the l-th level computational subgraph λ:

[0088]

[0089] This method uses the above formula to get the computation waiting time of the l-th level subgraph. The communication waiting time of the l-th level subgraph is given below. It is worth noting that we assume that AllReduce is used for communication:

[0090]

[0091] This method uses the above formula to estimate the AllReduce communication time of the model. The time to execute the calculation is given below, assuming that only forward propagation and directional propagation are generated:

[0092]

[0093] This method uses the above formula to get the computation time of each computation subgraph, so the computation time of each round can be defined as:

[0094]

[0095] Through the above, this method completes the modeling of the model calculation time in the cross-domain distributed environment, and models its calculation time as a unified calculation standard T, and gives a detailed calculation process.

[0096] Step (2.1) Based on the above modeling, the hierarchical hybrid parallel search algorithm is modeled as a dynamic programming algorithm. This method uses this algorithm to obtain a better hierarchical hybrid parallel strategy. This method uses Q(l,n,p) to represent the optimal delay when the last l-level subgraph is divided into p stages and placed in the last n devices. The goal of this method is to calculate and update:

[0097]

[0098] in It represents the minimum computation delay without adding the last level l subgraph and the last n devices. This method obtains multiple optimized hierarchical hybrid parallel solutions through the above optimization.

[0099] According to the method described in the above-mentioned cloud-edge heterogeneous resource collaborative training strategy, the specific process of step (3) is step (3.1) to step (3.4).

[0100] Step (3.1) When the model training task is started, the computational cluster and computational graph finally formed by the hierarchical hybrid parallel search algorithm will be distributed to each node. Each node will form a physical computational cluster and execute the corresponding model training task. When the task is started, each node in the computational cluster broadcasts a data packet to complete node election, and arranges the order and process of state information synchronization by combining the hierarchical hybrid parallel strategy.

[0101] Step (3.2) synchronizes the state information within the computing cluster, and completes the state information synchronization within the computing cluster by adding state information tensors during forward propagation and backward propagation.

[0102] Step (3.3) starts the computing status monitoring program on the master node, which counts the delays of different nodes in the computing cluster. The information reported by each node to the master node mainly includes the startup time of the computing node Forward propagation duration Backpropagation duration Activation value size a i , weight size w i , this method uses this information to estimate the end-to-end delay and slow node bottleneck. It is worth noting that if node i is the upstream node of the leader node, its weight-related data is zero; otherwise, the activation value-related data is zero. First, we use the following formula to get the end-to-end delay:

[0103]

[0104] Then the transmission delay is calculated for each upstream and downstream node. This method simply assumes that the transmission delay is proportional to the weight and activation value, and inversely proportional to the network bandwidth:

[0105]

[0106] Through the above simple calculation and modeling, the present invention detects fluctuations beyond the range through an outlier detection mechanism, namely:

[0107]

[0108] Where μ is a positive number. When the end-to-end delay between devices is less than the average value for more than a certain range, this method considers that the computing performance or communication capability is insufficient and the balancing mechanism needs to be triggered.

[0109] Step (3.4) Further, when the present invention detects insufficient performance, different problems are respectively dealt with through data balancing and tensor balancing. First, the critical path of the computing node where the performance bottleneck occurs is identified, as well as the steps where the performance bottleneck occurs. The present invention can mainly deal with the following problems: ① Insufficient computing performance; ② Poor communication with other pipeline nodes; ③ Poor communication with other pipeline nodes at the same stage, which can be identified by computing time, forward and reverse propagation time, and AllReduce time, respectively. Secondly, the present invention adopts different means to deal with the above problems. ① In view of the problem of insufficient computing performance, the present invention identifies the parallel stage in which it is located. If it adopts data parallelism, it reduces its data load according to the proportion of computing time through micro-batch scheduling. If it adopts tensor parallelism, it inserts AllReduce communication in the micro-batch, synchronizes weight information in advance, and re-divides the tensor partitioning method as soon as possible. ② In view of the problem of poor communication with other pipeline nodes, if it adopts data parallelism, all data loads are temporarily unloaded to the remaining nodes. If it adopts tensor parallelism, the preferred communication path forwards communication through other nodes. ③ To address the problem of poor communication with other pipeline nodes at the same stage, if data parallelism is used, the communication path is preferably forwarded through other nodes. If tensor parallelism is used, the tensor parallel nodes are temporarily excluded, and technologies such as ZeRO are used to use other nodes to complete the pipeline calculations at this stage.

Claims

1. A cloud-edge-device heterogeneous resource collaborative training strategy based on hierarchical hybrid parallelism, characterized in that: This includes building device graphs, computing graphs, and fully automatic parallel plan searches to obtain a hierarchical hybrid parallel strategy for cloud-edge heterogeneous hardware configurations, organizing cloud-edge heterogeneous hardware to jointly complete model training tasks, and achieving collaborative cross-domain heterogeneous computing node operations. During runtime, the computing load of slow nodes is unloaded in real time through Micro Batch's dynamic load optimization, and multiple cloud-edge devices are combined to complete computing tasks in the same link. The following steps are involved: Step (1): Establish a device graph construction mechanism based on computing cluster aggregation and a computing graph construction mechanism based on model compilation; Step (2): Establish a hierarchical hybrid parallel algorithm search process; Step (3): Establish a model runtime optimization mechanism, including a model runtime state synchronization mechanism, and a data balancing and tensor balancing mechanism based on Micro Batch during model runtime.

2. The cloud-edge-end heterogeneous resource collaborative training strategy according to claim 1 is characterized in that: Step (1) includes the following steps: Step (1.1) injects Profiler into each working node and collects node heterogeneous resource information through Profiler: When collecting network information, the node matrix is ​​used To represent all nodes in the cloud-edge computing environment that start the current training task, these computing nodes are connected through Ethernet in the computing environment; use "psutil" to collect the resource information of the computing node p, and use the node CPU core number C p 、Memory capacity M p , GPU computing power GPU memory capacity XPU computing power XPU storage capacity Network bandwidth between worker nodes Storage capacity p To express; Step (1.2) constructs the above node resource information into a vector ρ and aggregates it to the master node to perform subsequent calculation processes; Step (1.3) After the data of each node is uploaded to the master node, the present invention performs min-max normalization to construct different working node description vectors p; Step (1.4) performs the improved Bi-kmeans algorithm on the working nodes to divide the device graph into clusters, and uses the following calculation formula instead of the calculation of the error square sum S as the basis for clustering: Where I and J represent the computing clusters after division, respectively, i and p j Represent the description vectors of the working nodes in the computing cluster after division, N i and N j Represents the network bandwidth between different nodes. Through the above formula, the performance difference between different computing clusters is minimized, and the network connection bandwidth between different computing clusters is maximized, so as to achieve the aggregation of different device graphs and reduce the search space of the hierarchical hybrid parallel strategy. Step (1.5) runs the device graph optimization algorithm and makes adjustments based on clustering to balance the computing power and communication capabilities between the computing clusters and avoid obvious slow nodes: After obtaining the optimal computational graph through affinity clustering, different computational clusters I and J are selected in pairs, and all nodes in computational clusters I and J are traversed. The minimum communication cost from any node in computational cluster I to any node in computational cluster J is obtained through the shortest path algorithm. The communication cost between any two points in different computing clusters is represented by the computational cluster optimization algorithm, which traverses different nodes in the computing cluster, colors them according to their network connection relationship, computing power, and storage capacity, and pre-assigns different roles to each node, including heavy computing nodes and heavy communication nodes. Step (1.6) parse the model through TorchFx to obtain the intermediate representation IR of the model, and pass the relevant IR information to the model code injection and calculation graph simplification module; use the model conversion capability provided by TorchFx to perform TorchFx parsing to obtain a graph object that describes the composition of the calculation graph, including placeholder, call_function, and call_moudle objects; Step (1.7) After the model code injection module receives the computational graph IR, it will run the communication code injection, optimizer conversion, and model structure conversion programs to replace the components in the model with the optimized distributed versions; define the abstract Transform objects to be responsible for the insertion of distributed communication code, optimizer conversion, and specific structure conversion, including the distributed optimization conversion of multi-head attention and CNN modules, and form an optimized new Python code package for subsequent operation; Step (1.8) Based on the computational graph IR, the computational graph simplification module calculates the activation value and weight of the model, and identifies the specific model structure to simplify the computational graph for subsequent model splitting point search; finds heavy computational modules, including multi-head attention modules and GNN modules, and merges their nodes to divide them into the same computational cluster during hierarchical hybrid parallel search; finds heavy communication modules, including RNN modules, and annotates and colors them to work with the device graph to better search the parallel space.

3. The cloud-edge-end heterogeneous resource collaborative training strategy according to claim 1 is characterized in that: Step (2) includes the following steps: Step (2.1) analyzes the forward propagation and backward propagation time of each key calculation stage in the simplified calculation graph; the activation value and weight of the l-level subgraph are respectively recorded as a l and w l , and assume that the time of forward propagation and back propagation is inversely proportional to the computing power and proportional to the batch size; the forward propagation and back propagation time of the l-th level subgraph with a batch size of β in the computing cluster d are respectively recorded as and Use the forward propagation time and the backward propagation time to estimate the computational waiting time of the l-th level computational subgraph λ The above formula is used to get the computation waiting time of the l-th level subgraph; AllReduce is used for communication, and the communication waiting time of the l-th level subgraph is given by the following formula: Estimate the AllReduce communication time and give the time to execute the calculation using the following formula Only because of forward propagation and directional propagation: Using the above formula, we can get the computation time of each computation subgraph. The computation time T of each round is defined as: So far, the modeling of the model calculation time in the cross-domain distributed environment has been completed, and its calculation time is modeled as a unified calculation standard T; Step (2.1) models the hierarchical hybrid parallel search algorithm as a dynamic programming algorithm, and uses the algorithm to obtain a better hierarchical hybrid parallel strategy; Q(l,n,p) is used to represent the optimal delay when the last l-level subgraph is divided into p stages and placed in the last n devices; Q(l,n,p) is calculated and updated using the following formula: in Indicates the minimum computation delay without adding the last l-level subgraphs and the last n devices.

4. The cloud-edge-end heterogeneous resource collaborative training strategy according to claim 1 is characterized in that: Step (3) includes the following steps: Step (3.1) When the model training task is started, the computation cluster and computation graph finally formed by the hierarchical hybrid parallel search algorithm will be distributed to each node. Each node will form a physical computation cluster and execute the corresponding model training task. When the task is started, each node in the computation cluster broadcasts a data packet to complete node election and arranges the order and process of state information synchronization by combining the hierarchical hybrid parallel strategy. Step (3.2) synchronizes the state information within the computing cluster, and completes the state information synchronization within the computing cluster by adding state information tensors during forward propagation and backward propagation; Step (3.3) starts the computing status monitoring program on the master node, which counts the delays of different nodes in the computing cluster; the information reported by each node to the master node mainly includes the startup time of the computing node Forward propagation duration Backpropagation duration Activation value size a i , weight size w i , this method uses this information to estimate the end-to-end delay time and slow node bottleneck; if node i is the upstream node of the leader node, its weight-related data is zero; otherwise, the activation value-related data is zero; the end-to-end delay is obtained using the following formula: Then the transmission delay is calculated for the upstream and downstream nodes respectively, and the following expression is established with the transmission delay being proportional to the weight and activation value and inversely proportional to the network bandwidth: Through the outlier detection mechanism, fluctuations outside the range are detected, namely: Where μ is a positive number. When the end-to-end delay between devices is less than the average value for more than a certain range, the computing performance or communication capability is insufficient and the balancing mechanism needs to be triggered. When insufficient performance is detected in step (3.4), different problems are dealt with respectively through data balancing and tensor balancing: identifying the critical path of the computing node where the performance bottleneck occurs and the step where the performance bottleneck occurs; determining to deal with the following problems: ① insufficient computing performance; ② poor communication with other pipeline nodes; ③ poor communication with other pipeline nodes at the same stage, which can be identified by computing time, forward and reverse propagation time, and AllReduce time respectively; the coping measures are as follows: ① For the problem of insufficient computing performance, identify the parallel stage it is in. If it adopts data parallelism, reduce its data load according to the computing time ratio through micro-batch scheduling. If tensor parallelism is adopted, AllReduce communication is inserted in the micro-batch, weight information is synchronized in advance, and the tensor partitioning method is re-divided as soon as possible; ② For the problem of poor communication with other pipeline nodes, if data parallelism is adopted, all data loads are temporarily unloaded to the remaining nodes. If tensor parallelism is adopted, the communication path is preferably forwarded through other nodes; ③ For the problem of poor communication with other pipeline nodes in the same stage, if data parallelism is adopted, the communication path is preferably forwarded through other nodes. If tensor parallelism is adopted, the tensor parallel nodes are temporarily excluded, and ZeRO and other technologies are used to use other nodes to complete the pipeline calculation of this stage.

Images