A cloud-edge-end heterogeneous resource collaborative training strategy based on hierarchical hybrid parallelism

Abstract

A layered hybrid parallel cloud edge end heterogeneous resource collaborative training strategy, research on how to fully utilize the multi-layered hybrid parallel strategy of the search space of the search algorithm of the layered hybrid parallel algorithm, through the dynamic programming algorithm and the resource constraint mechanism to search the possible layered hybrid parallel algorithm, realize the efficient layered hybrid parallel of the cloud edge end collaborative cross-domain scene; research on the runtime cross-domain layered hybrid parallel optimization strategy, through the secondary balanced allocation of the data load and the calculation load in the calculation cluster, adjust the calculation speed between each node, avoid the emergence of slow nodes at runtime. Through the above mechanism, the efficient adaptation of cloud edge heterogeneous computing resources and hybrid parallel strategy is realized, the utilization efficiency of cloud edge heterogeneous computing resources is improved, and the model cross-domain distributed training speed is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer, and mainly relates to a cloud edge end heterogeneous resource collaborative training strategy based on hierarchical hybrid parallel. BACKGROUND

[0002] In the training process of a deep learning model, the model will be divided into different parts and run in different computing nodes with the same configuration and topology to realize the training of a large-scale model; in a cloud edge end computing environment, the diversity of devices and the heterogeneity of environment undoubtedly become an obstacle to the collaborative training of cloud edge end models. Therefore, how to fully utilize the diverse cloud edge end heterogeneous computing resources has become an important problem for building a cross-domain distributed computing framework. Current research mainly focuses on optimizing the model running of resource-constrained devices and the joint running of multiple heterogeneous nodes, which will be introduced respectively.

[0003] 1. Related work of optimizing the model running of resource-constrained devices

[0004] Optimizing the model running of resource-constrained devices refers to using sparsification, compilation and conversion to enable model training in heterogeneous devices such as CPU, GPU and DSP.

[0005] Zhu et al. proposed PockEngine, which enables fine-tuning of large models in embedded devices. First, PockEngine supports sparse backpropagation, updating the model sparsely while maintaining model quality through backpropagation graph pruning and memory measurements. Second, PockEngine compiles the model before execution, generating a complete training graph, reducing computational overhead on heterogeneous edge devices. Finally, PockEngine integrates rich training graph optimizations to further reduce training costs. Jiang et al. proposed the MNN engine, which enables running models on heterogeneous devices. MNN performs kernel-level optimizations for operators running on edge devices, providing the ability to run models on lightweight devices and even IoT devices, and offering support for various front-end and back-end implementations. Patil et al. proposed the Poet system, which uses external storage for model fine-tuning on micro-devices. Poet uses a combined search optimization space with recomputation and paging mechanisms to reduce backpropagation memory consumption and designs a MILP planner that incorporates energy-optimized training to achieve model fine-tuning on edge devices while minimizing energy consumption. Wang et al. proposed AtRec, a technique for accelerating recommendation models using heterogeneous CPU hardware. AtRec identifies operators with long computation times in the CPU and optimizes them to prevent them from becoming bottlenecks in the computation graph. AtRec can also identify inefficient parts of the overall computation graph and optimize them by eliminating redundant computations and improving memory access efficiency. Zhang et al. proposed 2PGraph, a technique for accelerating graph neural network training in heterogeneous hardware. Addressing the problems of long memory access latency and inefficient data transfer of vertex features from CPU to GPU, 2PGraph proposes a GNN layer-aware feature caching method on available GPU resources, avoiding redundant data transfer between CPU and GPU. Zhang et al. proposed AGL, a scalable heterogeneous graph neural network training technique. AGL designs a K-hop neighborhood data transfer algorithm for the message passing pattern at the bottom layer of GNN computation, avoiding message passing overhead across nodes and across heterogeneous hardware.

[0006] Existing methods for optimizing single-device model operation mainly use sparsification, compilation, and transformation to enable the model to run on heterogeneous devices such as CPUs, GPUs, and DSPs. Currently, it is possible to train the model on most edge heterogeneous devices. In the future, sparsification, compilation, and other methods will be used to further optimize the model's running efficiency on various heterogeneous devices.

[0007] 2. Related work on the collaborative multi-heterogeneous node joint operation model

[0008] The collaborative multi-heterogeneous node joint operation model refers to the technology of coordinating multiple heterogeneous resource nodes to collaboratively realize the operation of the model in heterogeneous nodes in the cloud, edge, and terminal.

[0009] Ye et al. proposed the Asteroid system, which uses pipelined parallelism to coordinate DNN operation on heterogeneous edge devices. Asteroid employs hybrid pipelined parallelism to coordinate distributed training across heterogeneous devices and can perform automatic parallelism planning to maximize throughput under certain resource constraints. Furthermore, Asteroid developed fault-tolerant computation using pipelined replay technology. Mao et al. proposed the MoDNN system, which coordinates model computation across multiple mobile computing devices. MoDNN can partition pre-trained DNN models across multiple mobile devices, accelerating DNN computation by reducing device-level computational costs and memory usage. Additionally, MoDNN designed two model partitioning schemes to minimize non-parallel data transmission time. Zeng et al. proposed the CoEdge system, which coordinates multiple edge devices through adaptive workload. CoEdge utilizes available computing and communication resources at the edge, dynamically partitioning the DNN workload based on device computing capabilities and network conditions. Zhao et al. proposed DeepThings technology, which coordinates resource-constrained IoT devices and runs in edge clusters. DeepThings employs Scalable Convolutional Fusion Partitioning (FTP) to minimize memory footprint while achieving parallelism; it also implements a distributed work-stealing method for dynamic workload allocation and balancing at runtime. Hao and Zhang et al. implemented an EDDL system for model training on resource-constrained devices. EDDL designs a dynamic training data distribution mechanism to distribute training data to worker nodes at runtime; furthermore, EDDL proposes an adaptive inter-node communication mechanism and a leader node rotation mechanism to accelerate model training at the edge. Yang et al. proposed AutoSF technology for model training at dynamic edges. When the edge environment changes, AutoSF triggers automated machine learning methods to find near-optimal aggregation structures and frequencies adapted to time-varying edge resources; when the environment remains unchanged, AutoSF uses heuristics to adjust the aggregation structure and frequency to further suppress heterogeneity caused by data distribution. Chen et al. proposed FTPipeHD technology for pipelined parallelism on heterogeneous edge devices. FTPipeHD dynamically optimizes partitioning points based on real-time computing capabilities and proposes a novel weight redistribution method that periodically replicates weights to neighboring nodes and the central node, thereby addressing fault issues during training.

[0010] The existing collaborative multi-heterogeneous node joint operation model mainly focuses on two aspects of heterogeneous node modeling and how to comprehensively utilize the computing resources of the heterogeneous nodes through parallel strategies, and the model collaborative reasoning technology is relatively mature, the collaborative training technology is initially emerging, and the computing efficiency and the utilization rate of the heterogeneous resources are relatively low, which is far from mature. It can be predicted that in the future, there will be more and more methods for model training of collaborative multi-heterogeneous cloud edge-end nodes.

[0011] In summary, for the heterogeneous hardware training in the cloud edge-end computing environment, there are mainly two ideas of fully utilizing single node resources and collaboratively training multiple heterogeneous nodes, but most of the current researches are mainly for model heterogeneous reasoning scenarios, and the related researches on heterogeneous training have initially emerged in recent years. Specifically, the existing optimization of single device model operation mainly uses means such as sparsification, compilation, conversion, etc. to enable the model to run on heterogeneous devices such as CPU, GPU, DSP, etc. At present, the running of the model in the edge-end heterogeneous device can be realized, which lays a good foundation. The existing collaborative multi-heterogeneous node joint operation model mainly focuses on two aspects of heterogeneous node modeling and how to comprehensively utilize the computing resources of the heterogeneous nodes through parallel strategies, and the model collaborative reasoning technology is relatively mature, the collaborative training technology is initially emerging, and the computing efficiency and the utilization rate of the heterogeneous resources are relatively low, which is far from mature. In summary, the existing heterogeneous hardware collaborative training technology in the cloud edge-end computing environment is far from mature, and it faces the problems of low computing efficiency and low utilization rate of heterogeneous resources, which limits the efficiency of the comprehensive utilization of heterogeneous hardware for model training in the cloud edge-end computing environment. Therefore, it is extremely important to study a heterogeneous hardware training strategy suitable for the cloud edge-end computing environment, and how to fully utilize the diverse cloud edge-end heterogeneous computing resources has become a problem to be solved. SUMMARY

[0012] In view of the above problems, the present application proposes a cloud edge-end heterogeneous resource collaborative training strategy based on hierarchical hybrid parallelism, and researches and designs a heterogeneous resource collaborative training strategy for the characteristics and applications of the cloud edge-end environment. The general idea of the present application is as follows: first, the present application obtains a hierarchical hybrid parallel strategy suitable for the current cloud edge-end heterogeneous hardware configuration through device graph, computation graph construction and full-automatic parallel plan search, reasonably organizes the cloud edge-end heterogeneous hardware to jointly complete the model training task, and solves the challenge of collaborative cross-domain heterogeneous computing node operation; second, the present application dynamically optimizes the Micro Batch load at runtime to real-time unload the computing load of the slow node, and jointly completes the computing task of the same link by multiple cloud edge-end devices, solving the challenge that a single edge-end device cannot complete the model training due to insufficient resources.

[0013] Specifically, the present application is composed of the following steps:

[0014] Step (1): Device graph and computation graph construction mechanism.

[0015] Step (2): Layered mixed and parallel search algorithm.

[0016] Step (3): Model runtime optimization mechanism.

[0017] Each step is further subdivided into a series of sub-steps.

[0018] Step (1.1) inject Profiler into each worker node, collect node heterogeneous resource information through Profiler.

[0019] When collecting network information, a node matrix is used to represent all nodes that start the current training task in the cloud edge computing environment, and these computing nodes are connected through Ethernet in the computing environment. Specifically, the present application uses "psutil" to collect resource information of computing node p, and uses node CPU core number C p , memory capacity M p , GPU computing power GPU memory capacity XPU computing power XPU storage capacity Network bandwidth between two worker nodes Storage capacity S p are represented.

[0020] Step (1.2) constructs the above node resource information into a vector p, and converges to the master node to perform subsequent calculation process.

[0021] Step (1.3) after the data of each node is uploaded to the master node, the present application performs min-max normalization to construct different worker node description vectors p.

[0022] Step (1.4) the present application performs improved Bi-kmeans algorithm on the worker nodes to divide the device graph into classes, and uses the following calculation formula instead of error square sum calculation as the basis for clustering division:

[0023]

[0024] Specifically, I and J represent the computing clusters after division, p i and p j represent the worker node description vectors in the computing clusters after division, N i and N jrepresent the network bandwidth between different nodes. The present application minimizes the performance difference between different computing clusters and maximizes the network connection bandwidth between different computing clusters through the above formula, thereby realizing the aggregation of different device graphs to reduce the search space of the hierarchical hybrid parallel strategy.

[0025] Step (1.5) runs the device graph optimization algorithm, adjusts on the basis of clustering, balances the computing power and communication power between different computing clusters, and avoids the presence of obvious slow nodes

[0026] Specifically, after obtaining the preferred computing graph through affinity clustering, the present application selects different computing clusters I and J in pairs, and traverses all nodes in computing clusters I and J to obtain the minimum communication cost from any node in computing cluster I to any node in computing cluster J through the shortest path algorithm, using to represent the communication cost between any two points in different computing clusters. Finally, the computing cluster optimization algorithm is executed to traverse different nodes in the computing cluster, and the nodes are colored according to the network connection relationship, computing power, and storage capacity, and different roles are pre-assigned to each node, such as heavy computing nodes and heavy communication nodes.

[0027] Step (1.6) performs model analysis through TorchFx to obtain the intermediate representation IR of the model, and transmits the relevant IR information to the model code injection and computing graph optimization module. The present application deeply utilizes the model conversion capability provided by TorchFx. When performing TorchFx analysis, a graph object will be obtained, which describes the composition of the computing graph and contains placeholder, call_function, call_moudle and other objects, which describe the composition of the computing graph in detail.

[0028] Step (1.7) When the model code injection module receives the computing graph IR, it will run the communication code injection, optimizer conversion and model structure conversion program to replace the components in the model with the optimized distributed version. The present application defines an abstract Transform object responsible for the insertion of distributed communication code, optimizer conversion and specific structure conversion (including distributed optimization conversion of multi-head attention, CNN and other modules), and forms an optimized new Python code package for subsequent running.

[0029] Step (1.8) Finally, according to the computation graph IR, the computation graph pruning module calculates the activation value, the size of the weight of the model, and identifies the specific model structure to prune the computation graph in order to find the subsequent model cut point. First, the present application finds the heavy computation module, such as the multi-head attention module, the GNN module, etc., and merges the nodes to divide them into the same computation cluster when searching in the hierarchical hybrid parallel manner; then the present application finds the heavy communication module, such as the RNN module, etc., and labels and colors these modules in order to work with the device graph to better search the parallel space.

[0030] Step (2.1) Analyze the forward propagation and backward propagation time of each key computation stage in the pruned computation graph. The activation value and weight of the l-level subgraph are denoted as a l and w l . And it is assumed that the forward propagation and backward propagation time is inversely proportional to the computing power and proportional to the Batch size. The forward propagation and backward propagation time of the l-level subgraph in the computation cluster d with the Batch size of β is denoted as and Secondly, the present method uses the forward propagation time and the backward propagation time to estimate the computation latency of the l-level computation subgraph λ:

[0031]

[0032] The present method uses the above formula to obtain the computation latency of the l-level subgraph. The communication latency of the l-level subgraph is given below, and it is worth noting that we assume that AllReduce is used for communication:

[0033]

[0034] The present method uses the above formula to estimate the AllReduce communication time of the model. The time of executing the computation is given below, and we assume that it is only because of the forward propagation and the backward propagation:

[0035]

[0036] The present method uses the above formula to obtain the computation time of each computation subgraph, so the computation time of each round can be defined as:

[0037]

[0038] Through the above, the present method completes the modeling of the model computation time in the cross-domain distributed environment, and models its computation time as a unified computation standard T, and gives the detailed computation process.

[0039] Step (2.1) Based on the above modeling, the hierarchical hybrid parallel search algorithm is modeled as a dynamic programming algorithm, and the method uses the algorithm to obtain an optimal hierarchical hybrid parallel strategy. The method uses Q(l, n, p) to represent the optimal delay when the last l-level subgraph is divided into p stages and placed in the last n devices. The goal of the method is to calculate and update:

[0040]

[0041] wherein represents the minimum calculation delay without joining the last l-level subgraph and the last n devices. The method obtains multiple optimized hierarchical hybrid parallel schemes through the above optimization.

[0042] Step (3.1) When the model training task is started, the computing cluster and the allocation of the computing graph finally formed by the hierarchical hybrid parallel search algorithm are issued to each node, and each node self-composes a physical computing cluster and executes the corresponding model training task; when the task is started, each node in the computing cluster broadcasts a data packet, completes node election, and arranges the order and process of state information synchronization by combining the hierarchical hybrid parallel strategy.

[0043] Step (3.2) Synchronize the state information in the computing cluster, and complete the synchronization of the state information in the computing cluster by adding a state information tensor during the forward propagation and the backward propagation.

[0044] Step (3.3) Start the computing state monitoring program in the master node, which counts the delays of different nodes in the computing cluster. The information reported by each node to the master node mainly includes the start time of the computing node the duration of forward propagation the duration of backward propagation the size of the activation value a i , the size of the weight w i , and the method uses these information to estimate the end-to-end delay time and the slow node bottleneck. It is worth noting that if node i is an upstream node of the leader node, the weight-related data is zero; otherwise, the activation value-related data is zero. First, we use the following formula to obtain the end-to-end delay:

[0045]

[0046] Then the transmission delay is calculated for the upstream and downstream nodes. The method simply assumes that the transmission delay is proportional to the weight and the activation value, and inversely proportional to the network bandwidth:

[0047]

[0048] Through the above simple calculation and modeling, the application detects fluctuations that exceed the range through an outlier detection mechanism, that is:

[0049]

[0050] μ is a positive number. When the end-to-end delay between devices is less than the average value and exceeds a certain range, this method considers the computing performance or communication capability to be insufficient and requires triggering a load balancing mechanism.

[0051] Step (3.4) Further, when the present invention detects insufficient performance, it addresses different problems through data balancing and tensor balancing respectively. First, it identifies the critical path of the computing node experiencing the performance bottleneck and the step where the bottleneck occurs. The present invention can mainly address the following problems: ① Insufficient computing performance; ② Poor communication with other pipeline nodes; ③ Poor communication with other pipeline nodes in the same stage, which can be identified by computing time, forward and backward propagation time, and AllReduce time, respectively. Second, the present invention adopts different methods to address the above problems. ① For the problem of insufficient computing performance, the present invention identifies the parallel stage in which it is located. If it uses data parallelism, it reduces its data load according to the computing time ratio through micro-batch scheduling. If it uses tensor parallelism, it inserts AllReduce communication in the micro-batch to synchronize weight information in advance and quickly re-divide the tensor partitioning method. ② For the problem of poor communication with other pipeline nodes, if it uses data parallelism, it temporarily offloads all data load to other nodes. If it uses tensor parallelism, it preferentially forwards communication through other nodes. ③ To address the issue of poor communication with other pipeline nodes in the same stage, if data parallelism is used, the preferred communication path is to forward communication through other nodes. If tensor parallelism is used, the tensor parallel node is temporarily excluded, and other nodes are used to complete the pipeline computation in this stage using technologies such as ZeRO.

[0052] The advantages of this invention are:

[0053] By proposing a hierarchical hybrid parallel cross-domain distributed training mechanism, this method achieves cross-domain distributed training on heterogeneous hardware, fully utilizing diverse cloud-edge-device heterogeneous computing resources. This approach generates an initial cross-domain parallel training plan by collecting information before execution, and further balances resources by adjusting data and tensor partitioning during runtime to achieve hierarchical hybrid parallel cross-domain heterogeneous hardware training. Compared with existing methods, it has advantages such as high environmental adaptability, high model adaptation flexibility, and low computational overhead. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the cloud-edge-device heterogeneous resource collaborative training strategy method based on hierarchical hybrid parallelism proposed in this invention.

[0055] Figure 2 This is a schematic diagram of the device graph construction mechanism based on computation cluster aggregation proposed in this invention.

[0056] Figure 3 A schematic diagram of a model compilation-based computation graph construction mechanism proposed by the present application.

[0057] Figure 4 A schematic diagram of a hierarchical hybrid parallel algorithm search process proposed by the present application.

[0058] Figure 5 A schematic diagram of a model runtime state synchronization mechanism proposed by the present application.

[0059] Figure 6 A schematic diagram of a model runtime Micro Batch-based data and tensor balancing mechanism proposed by the present application. DETAILED DESCRIPTION

[0060] The present application will be further described in detail in conjunction with the accompanying drawings. Figures 1-6 ).

[0061] A cloud-edge-end heterogeneous resource collaborative training strategy based on hierarchical hybrid parallelism, aiming at how to fully utilize the diverse cloud-edge-end heterogeneous computing resources of a cross-domain distributed computing framework, researches a cloud-edge-end heterogeneous resource collaborative training strategy based on hierarchical hybrid parallelism. First, the device graph and computation graph construction mechanism are researched, and through a computation cluster aggregation algorithm and a model compilation mechanism, a simplified and merged computation graph and device graph are constructed, so as to reduce the search space of the hierarchical hybrid parallelism strategy. Second, the search algorithm of the hierarchical hybrid parallel algorithm is researched, and through a dynamic programming algorithm and a resource constraint mechanism, possible hierarchical hybrid parallel algorithms are searched, so as to realize efficient hierarchical hybrid parallelism in the cloud-edge-end collaborative cross-domain scenario. Finally, the runtime cross-domain hierarchical hybrid parallel optimization strategy is researched, and through the secondary balancing distribution of the data load and the computation load in the computation cluster, the computation speed between nodes is adjusted, and the appearance of a slow node at runtime is avoided. Through the above mechanisms, efficient adaptation of the cloud-edge heterogeneous computing resources and the hybrid parallelism strategy is realized, the utilization efficiency of the cloud-edge heterogeneous computing resources is improved, and the model cross-domain distributed training speed is improved.

[0062] Figure 1The overall scheme of the layered mixed parallel cloud edge end heterogeneous resource collaborative training strategy is shown, including steps (1), (2), and (3). First, the device graph and computation graph construction mechanism is studied, and through the computation cluster aggregation algorithm and model compilation mechanism, a simplified and combined computation graph and device graph are constructed, reducing the search space of the layered mixed parallel strategy. Second, the search algorithm of the layered mixed parallel algorithm is studied, and through the dynamic programming algorithm and resource constraint mechanism, the possible layered mixed parallel algorithm is searched, realizing efficient layered mixed parallel in the cloud edge end collaborative cross-domain scenario. Finally, the runtime cross-domain layered mixed parallel optimization strategy is studied, and through the secondary balanced allocation of the data load and the computation load in the computation cluster, the computation speed between nodes is adjusted to avoid the appearance of slow nodes in runtime. Through the above mechanisms, efficient adaptation of cloud edge heterogeneous computing resources and mixed parallel strategies is realized, the utilization efficiency of cloud edge heterogeneous computing resources is improved, and the model cross-domain distributed training speed is improved.

[0063] Figure 2 The workflow of the computation cluster aggregation device graph construction method is shown, including steps (1.1) to (1.5). First, Profiler is injected into each work node, and the heterogeneous resource information is collected by Profiler and reported to the master node to run the clustering algorithm and device graph construction algorithm. Second, the affinity clustering algorithm is run on the master node, and the affinity between nodes is calculated based on the computing, communication, and storage capabilities of the work nodes, and the clustering algorithm is run to obtain the optimal match as the basis for target optimization. Finally, the device graph optimization algorithm is run to balance the computing and communication capabilities between the clusters and avoid the appearance of slow nodes.

[0064] Figure 3 The workflow of the computation graph construction method based on model compilation is shown, including steps (1.6) to (1.8). First, the model is parsed by TorchFx to obtain the intermediate representation IR, and the relevant IR information is passed to the model code injection and computation graph simplification module. Second, when the model code injection module receives the computation graph IR, it runs the communication code injection, optimizer conversion, and model structure conversion program to replace the components in the model with the optimized distributed version. Finally, based on the computation graph IR, the computation graph simplification module calculates the size of the activation value and weight of the model, and identifies specific model structures to simplify the computation graph for subsequent model cut point searching.

[0065] As Figure 4Figure 2 shows the search process of the hierarchical hybrid parallel algorithm, including steps (2.1) to (2.2). First, the preferred device graph and computation graph generated in the first step are accepted, and the delay information modeling is performed by estimating the time of communication, computation, etc. through the relevant device capabilities and model information. Then, the dynamic programming algorithm targeting Q is run to assign different computation stages to different computing clusters, and the tensor and data are simply scaled and divided among the computing clusters according to the computing capacity of a single device, resulting in multiple hierarchical hybrid parallel plans.

[0066] Figure 5 Figure 3 shows the state synchronization mechanism of the model cross-domain distributed runtime, including steps (3.1) and (3.2). It mainly consists of leader node election and inter-device state synchronization. First, when the model training task is started, the computing cluster and computation graph formed by the hierarchical hybrid parallel search algorithm are assigned to each node, and each node forms a physical computing cluster and executes the corresponding model training task. At the same time, each node in the computing cluster broadcasts a data packet to complete node election, and through the combination of the hierarchical hybrid parallel strategy, the order and process of state information synchronization are arranged. Second, the state information in the computing cluster is synchronized, such as Figure 5 Figure 4 shows an example of state synchronization of a computing cluster consisting of four computing nodes. Device 1 performs the first stage of computation, devices 2 and 3 perform the second stage of computation, and device 4 performs the third stage of computation; device 3 is elected as the leader node to monitor the computation in the computing cluster. There are six state synchronization opportunities in the figure, which synchronize the state information of devices 1, 4, 1, 4, 4, and 3, respectively. Through the aggregation and synchronization of these state information, device 3 can obtain complete monitoring information in the computing cluster.

[0067] Figure 6The model runtime Micro Batch-based data, tensor balancing mechanism is demonstrated, including steps (3.3) and (3.4). First, the master node starts the computing state monitoring program, which counts the delay of different nodes in the computing cluster. Further, when the performance is detected to be insufficient, the method deals with different problems through data balancing and tensor balancing respectively. First, the critical path where the computing node with performance bottleneck is located and the step where the performance bottleneck occurs are identified. The method can mainly deal with the following problems: ① insufficient computing performance; ② poor communication with other pipeline nodes; ③ poor communication with other pipeline nodes in the same stage, which can be identified by computing time, forward and backward propagation time, and AllReduce time respectively. Second, the method takes different means to deal with the above problems. ① For the problem of insufficient computing performance, the method identifies the parallel stage it is located in, such as using data parallelism, then reducing its data load according to the computing time ratio through micro batch scheduling, or inserting AllReduce communication in the micro batch to synchronize the weight information as soon as possible and redivide the tensor division mode as soon as possible. ② For the problem of poor communication with other pipeline nodes, such as using data parallelism, temporarily offloading all data loads to other nodes, or using tensor parallelism to forward the communication through other nodes. ③ For the problem of poor communication with other pipeline nodes in the same stage, such as using data parallelism, the communication path is optimized to forward the communication through other nodes, or using tensor parallelism to temporarily exclude the tensor parallel node and use other nodes to complete the current stage pipeline computation using ZeRO technology.

[0068] The application proposes a cloud edge end heterogeneous resource collaborative training strategy based on hierarchical hybrid parallelism, researches and designs a heterogeneous resource collaborative training strategy for the characteristics and applications of cloud edge end environment. The general idea of the application is: first, the application obtains a hierarchical hybrid parallel strategy suitable for the current cloud edge end heterogeneous hardware configuration through device graph, computing graph construction and full-automatic parallel plan search, reasonably organizes the cloud edge end heterogeneous hardware to complete the model training task together, and solves the challenge of running collaborative cross-domain heterogeneous computing nodes; second, the application performs real-time offloading of the computing load of slow nodes through dynamic load optimization of Micro Batch in runtime, and completes the computing task of the same link by combining multiple cloud edge end devices, thereby solving the challenge of insufficient resources of a single edge end device to complete model training.

[0069] Specifically, the application comprises the following steps:

[0070] Step (1): device graph and computing graph construction mechanism.

[0071] Step (2): hierarchical hybrid parallel search algorithm.

[0072] Step (3): Model runtime optimization mechanism.

[0073] According to the method described above, the specific process of step (1) is step (1.1) to step (1.8).

[0074] Step (1.1) injects Profiler into each working node to collect node heterogeneous resource information through Profiler.

[0075] When collecting network information, the node matrix is used to represent all nodes that start the current training task in the cloud edge end computing environment, and these computing nodes are connected through Ethernet in the computing environment. Specifically, the present application uses "psutil" to collect resource information of computing node p, and uses node CPU core number C p , memory capacity M p , GPU computing capability GPU memory capacity XPU computing capability XPU storage capability network bandwidth between working nodes storage capability S p to represent.

[0076] Step (1.2) constructs the above node resource information into a vector p, and converges to the master node to perform subsequent calculation process.

[0077] Step (1.3) after the data of each node is uploaded to the master node, the present application constructs different working node description vectors p after min-max normalization.

[0078] Step (1.4) the present application executes improved Bi-kmeans algorithm on working nodes to divide device graph into classes, and uses the following calculation formula instead of error square sum calculation as the basis for clustering division:

[0079]

[0080] Specifically, I and J represent the calculation clusters after division, p i and p j represent the working node description vectors in the calculation clusters after division, N i and N j represent the network bandwidth between different nodes. Through the above formula, the present application minimizes the performance difference between different calculation clusters, and maximizes the network connection bandwidth between different calculation clusters, so as to realize the aggregation of different device graphs, and reduce the search space of hierarchical hybrid parallel strategy.

[0081] Step (1.5) runs the device graph optimization algorithm, which adjusts the computing power and communication power between the clusters to avoid slow nodes.

[0082] Specifically, after obtaining the preferred computing graph through affinity clustering, the present application selects different computing clusters I and J, and traverses all nodes in computing clusters I and J to obtain the minimum communication cost between any node in computing cluster I and any node in computing cluster J through the shortest path algorithm, using to represent the communication cost between any two points in different computing clusters. Finally, the computing cluster optimization algorithm is executed to traverse different nodes in the computing cluster, and the network connection relationship, computing power, and storage capacity are used to color them to pre-allocate different roles to each node, such as heavy computing nodes and heavy communication nodes.

[0083] Step (1.6) analyzes the model through TorchFx to obtain the intermediate representation IR of the model, and passes the relevant IR information to the model code injection and computing graph optimization module. The present application deeply utilizes the model conversion capability provided by TorchFx. When TorchFx analysis is performed, a graph object will be obtained, which describes the composition of the computing graph, including placeholder, call_function, call_moudle, and other objects, which describe the composition of the computing graph in detail.

[0084] Step (1.7) When the model code injection module receives the computing graph IR, it will run the communication code injection, optimizer conversion, and model structure conversion program to replace the components in the model with the optimized distributed version. The present application defines an abstract Transform object responsible for the insertion of distributed communication code, optimizer conversion, and specific structure conversion (including distributed optimization conversion of multi-head attention, CNN, and other modules), and forms an optimized new Python code package for subsequent running.

[0085] Step (1.8) Finally, according to the computing graph IR, the computing graph optimization module calculates the size of the activation value and weight of the model, and identifies specific model structures to optimize the computing graph for subsequent model split point searching. First, the present application finds heavy computing modules such as multi-head attention modules and GNN modules, and combines them to divide them into the same computing cluster during hierarchical hybrid parallel search; then the present application finds heavy communication modules such as RNN modules, and labels and colors these modules to better search the parallel space in cooperation with the device graph.

[0086] According to the method described in the above cloud-edge-end heterogeneous resource collaborative training strategy, the specific process of step (2) is step (2.1) to step (2.2).

[0087] Step (2.1) Analyze the forward and backward propagation time of each critical computation stage in the pruned computation graph. Denote the activation value and weight of the l-th stage subgraph as a l and w l . Assume that the forward and backward propagation time is inversely proportional to the computation power and proportional to the batch size. Denote the forward and backward propagation time of the l-th stage subgraph in the computation cluster d with batch size β as and Second, the method uses the forward and backward propagation time to estimate the computation latency of the l-th stage computation subgraph λ:

[0088]

[0089] The method uses the above formula to obtain the computation latency of the l-th stage subgraph. The communication latency of the l-th stage subgraph is given below. It is worth noting that we assume that AllReduce is used for communication:

[0090]

[0091] The method uses the above formula to estimate the AllReduce communication time of the model. The time to perform computation is given below. We assume that it is only due to the forward and backward propagation:

[0092]

[0093] The method uses the above formula to obtain the computation time of each computation subgraph. Therefore, the computation time of each round can be defined as:

[0094]

[0095] Through the above, the method completes the modeling of the model computation time in the cross-domain distributed environment and models its computation time as a unified computation standard T, and gives the detailed computation process.

[0096] Step (2.1) Based on the above modeling, the hierarchical hybrid parallel search algorithm is modeled as a dynamic programming algorithm. The method uses this algorithm to obtain an optimal hierarchical hybrid parallel strategy. The method uses Q(l, n, p) to represent the optimal latency when the last l stage subgraph is divided into p stages and placed in the last n devices. The goal of the method is to compute and update:

[0097]

[0098] where The minimum computing delay is represented without adding the last l-level subgraph and the last n devices. The method obtains multiple optimized hierarchical hybrid parallel schemes through the optimization.

[0099] According to the method described in the cloud-edge-end heterogeneous resource collaborative training strategy, the specific process of step (3) is step (3.1) to step (3.4).

[0100] Step (3.1) When the model training task is started, the computing cluster and the computing graph formed by the hierarchical hybrid parallel search algorithm are distributed to each node, and each node forms a physical computing cluster and executes the corresponding model training task; when the task is started, each node in the computing cluster broadcasts a data packet, completes node election, and arranges the order and process of state information synchronization by combining the hierarchical hybrid parallel strategy.

[0101] Step (3.2) Synchronize state information in the computing cluster, and complete the synchronization of state information in the computing cluster by adding state information tensors during forward propagation and backward propagation.

[0102] Step (3.3) Start the computing state monitoring program on the master node, which counts the delays of different nodes in the computing cluster. The information reported by each node to the master node mainly includes the start time of the computing node Forward propagation duration Backward propagation duration Activation value a i , weight w i The method uses this information to estimate the end-to-end delay time and slow node bottleneck. It is worth noting that if node i is an upstream node of the leader node, the weight-related data is zero; otherwise, the activation value-related data is zero. First, we use the following formula to obtain the end-to-end delay:

[0103]

[0104] Then calculate the transmission delay of the upstream and downstream nodes. The method simply assumes that the transmission delay is proportional to the weight and activation value, and inversely proportional to the network bandwidth:

[0105]

[0106] Through the above simple calculation and modeling, the application detects fluctuations outside the range through an outlier detection mechanism, that is:

[0107]

[0108] Where μ is a positive number. When the end-to-end delay between devices is less than the average value by a certain range, the method considers that the computing performance or communication ability is insufficient, and needs to trigger the balancing mechanism.

[0109] Further, when the performance is detected to be insufficient, the application deals with different problems through data balancing and tensor balancing respectively. First, the critical path where the computing node with performance bottleneck is located and the step where the performance bottleneck occurs are identified. The application can mainly deal with the following problems: ① insufficient computing performance; ② poor communication with other pipeline nodes; ③ poor communication with other pipeline nodes in the same stage, which can be identified by computing time, forward and backward propagation time, and AllReduce time respectively. Second, the application adopts different means to deal with the above problems. ① For the problem of insufficient computing performance, the application identifies the parallel stage it is in, such as using data parallelism, reducing its data load according to the computing time ratio through micro-batch scheduling, or using tensor parallelism, inserting AllReduce communication in the micro-batch, synchronizing weight information in advance, and re-dividing the tensor division as soon as possible. ② For the problem of poor communication with other pipeline nodes, if it uses data parallelism, all data loads are temporarily offloaded to other nodes, and if it uses tensor parallelism, the communication path is optimized to forward the communication through other nodes. ③ For the problem of poor communication with other pipeline nodes in the same stage, if it uses data parallelism, the communication path is optimized to forward the communication through other nodes, and if it uses tensor parallelism, it is temporarily excluded from the tensor parallelism nodes, and other nodes are used to complete the pipeline computation in this stage using technologies such as ZeRO.

Claims

1. A cloud-edge-end heterogeneous resource collaborative training strategy based on hierarchical hybrid parallel, characterized in that, The layered mixed parallel strategy of cloud edge end heterogeneous hardware configuration is obtained through device graph construction, computation graph construction and full-automatic parallel plan search, and the cloud edge end heterogeneous hardware is organized to jointly complete the model training task to realize collaborative cross-domain heterogeneous computing node operation; the dynamic load optimization of Micro Batch is used in runtime to unload the computing load of slow nodes in real time, and multiple cloud edge end devices are combined to complete the computing task of the same link; The method comprises the following steps: Step (1): establishing a device graph construction mechanism based on computation cluster aggregation and a computation graph construction mechanism based on model compilation; Step (2): establishing a layered mixed parallel algorithm search process; Step (3): establishing a model runtime optimization mechanism, including a model runtime state synchronization mechanism and a model runtime data balancing and tensor balancing mechanism based on Micro Batch; Step (2) comprises the following steps: Step (2.1) analyzes the forward propagation and backward propagation time of each key computation stage in the simplified computation graph; the activation value and weight of the l-level subgraph are respectively denoted as a l and w l , and it is assumed that the time of forward propagation and backward propagation is inversely proportional to the computing power and proportional to the Batch size; the forward propagation and backward propagation time of the l-level subgraph in the computing cluster d with a Batch size of β is respectively denoted as and The computing waiting time of the l-level computation subgraph λ is estimated using the forward propagation time and the backward propagation time The calculation latency of the lth subgraph is obtained using the above formula; the communication latency of the lth subgraph is given by the following formula using AllReduce Estimate the AllReduce communication time, the time to perform the computation is given by the following formula Only because forward propagation and direction propagation produce: The computing time of each computation subgraph is obtained using the above formula, and the computing time T of each round is defined as: Thus, the model computing time in the cross-domain distributed environment is modeled, and the computing time is modeled as a unified computing standard T; Step (2.2) models the layered mixed parallel search algorithm as a dynamic programming algorithm, and uses the algorithm to obtain an optimal layered mixed parallel strategy; Q(l, n, p) is used to represent the optimal delay when the last l subgraphs are divided into p stages and placed in the last n devices; Q(l, n, p) is calculated and updated using the following formula: wherein denotes the minimum computation delay without the last l-stage subgraph and the last n devices.

2. The cloud-edge-heterogeneous resource collaborative training strategy according to claim 1, characterized in that, Step (1) comprises the following steps: Step (1.1) injects Profiler into each working node to collect node heterogeneous resource information through Profiler: When performing network information collection, a node matrix is adopted to represent all nodes that start the current training task in the cloud edge-end computing environment, and the computing nodes are connected in the computing environment through Ethernet; the resource information of the computing node p is collected using "psutil", and the number of CPU cores C p , memory capacity M p , GPU computing power GPU memory capacity XPU computing power XPU storage capacity network bandwidth between two working nodes storage capacity S p are represented; Step (1.2) constructs the above node resource information into a vector p and converges to the master node to perform subsequent calculation processes; Step (1.3) constructs different working node description vectors p after min-max normalization is performed on the data of each node after the data of each node is uploaded to the master node; Step (1.4) performs an improved Bi-kmeans algorithm on the working nodes to divide the device graph into classes, and uses the following calculation formula instead of the calculation of the sum of squared errors S as the basis for clustering division: where I and J represent the computing clusters after partition, respectively, p i and p j represent the worker node description vectors in the computing clusters after partition, respectively, N i and N j represent the network bandwidth between different nodes; by the above formula, the performance difference between different computing clusters is minimized, and the network connection bandwidth between different computing clusters is maximized, so as to realize the aggregation of different device graphs to reduce the search space of the hierarchical hybrid parallel strategy; Step (1.5) runs a device graph optimization algorithm to adjust the computing power and communication power between the clusters to avoid obvious slow nodes based on clustering: After the preferred computing graph is obtained by affinity clustering, different computing clusters I and J are selected pairwise, and all nodes in the computing clusters I and J are traversed to obtain the minimum communication cost from any node in the computing cluster I to any node in the computing cluster J by a shortest path algorithm, and the minimum communication cost is represented by the communication cost between any two points in different computing clusters; the computing cluster optimization algorithm is executed, different nodes in the computing cluster are traversed, and the nodes are colored according to network connection relationship, computing capacity and storage capacity, and different roles are pre-assigned to each node, including re-computing nodes and re-communication nodes; Step (1.6) analyzes the model through TorchFx to obtain the intermediate representation IR of the model, and transmits the related IR information to the model code injection and computation graph simplification module; the model conversion capability provided by TorchFx is used to perform TorchFx analysis to obtain a graph object describing the composition of the computation graph, including placeholder, call_function and call_moudle objects; Step (1.7) When the model code injection module receives the computation graph IR, it will run the communication code injection, optimizer conversion and model structure conversion program to replace the components in the model with the optimized distributed version; define an abstract Transform object to be responsible for the insertion of distributed communication code, optimizer conversion and specific structure conversion, including distributed optimization conversion of multi-head attention, CNN module, and form the optimized new Python code package for subsequent running; Step (1.8) According to the computation graph IR, the computation graph simplification module calculates the size of the activation value and weight of the model, and identifies the specific model structure to simplify the computation graph for subsequent model split point search; find the re-computation module, including the multi-head attention module and the GNN module, and combine the nodes to divide them into the same computing cluster in the hierarchical hybrid parallel search; find the re-communication module, including the RNN module, and mark and color them to work with the device graph to better search the parallel space. 3.The cloud-edge-end heterogeneous resource coordination training strategy according to claim 1, characterized in that, Step (3) includes the following steps: Step (3.1) When the model training task starts, the computing cluster and the allocation of the computation graph formed by the hierarchical hybrid parallel search algorithm will be distributed to each node, and each node will form a physical computing cluster and execute the corresponding model training task; when the task starts, each node in the computing cluster broadcasts a data packet to complete node election, and through the combination of hierarchical hybrid parallel strategy, the order and process of state information synchronization are arranged; Step (3.2) Synchronize state information in the computing cluster, and complete the synchronization of state information within the computing cluster by adding state information tensor during forward propagation and backward propagation; Step (3.3) The master node starts a computing state monitoring program, which counts the delay of different nodes in the computing cluster; the information reported by each node to the master node mainly includes the start time of the computing node Forward propagation duration Backward propagation duration Activation value size a i , weight size w i The method uses this information to estimate the end-to-end delay time and slow node bottleneck; if node i is an upstream node of the leader node, the weight-related data is zero; otherwise, the activation value-related data is zero; the end-to-end delay is obtained using the following formula: Then calculate the transmission delay of the upstream and downstream nodes respectively, and establish the following expression according to the transmission delay and the weight, activation value in direct proportion and the network bandwidth in inverse proportion: Through the outlier detection mechanism, detect the fluctuations beyond the range, that is: Where μ is a positive number, when the end-to-end delay between devices is less than the average value by a certain range, the computing performance or communication ability is insufficient, and the balancing mechanism needs to be triggered; When step (3.4) detects performance deficiency, it deals with different problems through data balancing and tensor balancing respectively: identifying the critical path where the computing node with performance bottleneck is located and the step where performance bottleneck occurs; determining the following problems: ① insufficient computing performance; ② poor communication with other pipeline nodes; ③ poor communication with other pipeline nodes in the same stage, which can be identified by computing time, forward and backward propagation time, and AllReduce time respectively; the countermeasures are as follows: ① for the problem of insufficient computing performance, identify the parallel stage it is in, if it uses data parallelism, reduce its data load according to the computing time proportion through micro-batch scheduling, if it uses tensor parallelism, insert AllReduce communication in the micro-batch, synchronize the weight information in advance, and redivide the tensor division mode as soon as possible; ② for the problem of poor communication with other pipeline nodes, if it uses data parallelism, temporarily unload all data loads to other nodes, if it uses tensor parallelism, optimize the communication path through other nodes to forward the communication; ③ for the problem of poor communication with other pipeline nodes in the same stage, if it uses data parallelism, optimize the communication path through other nodes to forward the communication, if it uses tensor parallelism, temporarily exclude it from the tensor parallel nodes, use other nodes to complete the pipeline computation in this stage using technologies such as ZeRO.

Images