Cloud edge end cross-domain training efficient communication technology based on partition sparse communication

Abstract

The invention discloses a cloud side end cross-domain training efficient communication technology based on partition sparse communication, and aims to research how a cross-domain distributed computing framework efficiently realizes quick communication of cloud side end cross-domain nodes, research a back propagation sparsification mechanism, determine the influence of sparsification of a model layer on the model accuracy and communication through a sparsity heuristic algorithm, and improve the communication efficiency. Therefore, the communication scale is greatly reduced; a partition and grading communication scheduling mechanism is researched, and mainly comprises a computing intra-cluster communication scheduling mechanism and a computing inter-cluster communication scheduling mechanism. Wherein the computing intra-cluster communication scheduling mechanism maximizes computing-communication overlapping of the computing cluster through an activation value and weight separation mechanism; and calculating an inter-cluster communication scheduling mechanism, decoupling forward propagation and back propagation, and obtaining a most suitable micro-batch scheduling target through a custom optimization target. Through the above mechanism, the communication scale is effectively reduced, the calculation-communication overlapping rate is improved, efficient and rapid cross-domain node communication is realized, and the efficiency of cross-domain distributed training is improved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and particularly to a cloud-edge-end cross-domain training efficient communication technology based on partitioned sparse communication. Background Art

[0002] During the training of deep learning models, synchronizing model parameters, gradients, and optimizer states between multiple GPUs or nodes generates huge communication overhead, which becomes a bottleneck for model training. In the cloud-edge computing environment, the scarce network resources undoubtedly magnify this problem. Therefore, how to efficiently achieve fast communication between cloud-edge nodes across domains has become an important issue in building a cross-domain distributed computing framework. Current research focuses on reducing communication volume, optimizing communication modes, and enhancing the overlap between computing and communication. The following will introduce the relevant work.

[0003] 1. Work related to reducing communication volume

[0004] Reducing the amount of communication means compressing gradients or model parameters before transmission to reduce the scale of data transmission and reduce the communication overhead of distributed model training. Common methods to reduce the amount of communication are quantization and sparsification.

[0005] Microsoft proposed the ZeRO optimization strategy to achieve ultra-large-scale model training. ZeRO can distribute model weights, gradients, and optimizer states among multiple GPUs to achieve parallel computing, thereby increasing the amount and frequency of communication, but introducing a huge amount of communication overhead; in order to minimize the communication overhead, Microsoft proposed the ZeRO++ communication optimization strategy based on ZeRO. It uses weight quantization to compress model parameters into smaller data types (INT8) in real time before communication, thereby reducing the required communication bandwidth and time; in addition, ZeRO++ maintains a complete model copy on each machine to enhance the communication bandwidth within the machine. Wang et al. proposed a hybrid compression technology CocktailSGD. This technology combines three different compression technologies, including random sparsification, Top-K sparsification, and parameter quantization, and cleverly overlaps communication with local gradient calculations, achieving excellent training performance in Ethernet with only 500Mbps. Song et al. creatively proposed a compression-based 3D parallel technology Optimus-CC. Based on data parallel process compression, this technology realizes a new inter-level back-propagation compression and embedded synchronous compression, and develops a new key communication path identification mechanism; in different training stages, it dynamically and selectively compresses according to communication needs, thereby minimizing unnecessary communication overhead and improving overall training efficiency while maintaining model accuracy as much as possible. Yu et al. proposed a dual quantization technology to accelerate the model training process. This technology first proposed a general scheme for quantizing model parameters and gradients at the same time, and then based on this general scheme, proposed three algorithms to improve communication efficiency, including asynchronous precision parallel algorithm, gradient sparse-dual quantization, and momentum quantization acceleration technology. Chen et al. proposed a scalable sparse gradient compression strategy ScaleCom. This technology uses the similarity of gradient distribution between working nodes to provide exchangeable compressors and keep the communication cost consistent with the number of working nodes, and uses a low-pass filter containing local gradient accumulation to reduce the impact of large-scale training. Hu et al. proposed a hybrid compression strategy HGC of hybrid gradient compression and gradient sparsification. This technology implements a dynamic selection strategy that can dynamically choose to use quantization or sparsification strategy during the operation of the working node, so as to combine the advantages of the two methods to achieve higher compression rate and better training effect.

[0006] Existing methods for reducing communication traffic generally use quantization and sparsification as the basic method, which has achieved amazing compression ratios and training acceleration ratios. In the future, it will develop in the direction of optimizing specific models and specific network conditions. However, existing compression methods for reducing communication traffic lack optimization for low-bandwidth and high-latency network scenarios.

[0007] 2. Optimizing communication mode

[0008] Optimizing the communication mode refers to a method of accelerating the communication speed between model training nodes by optimizing the communication library, changing the communication structure, optimizing the communication hardware, etc. during node communication.

[0009] Gan et al. proposed an MPI-style communication library called Bagua. Bagua provides a series of flexible and modular communication primitives to support the most advanced "system relaxations" techniques in data-parallel distributed model training, such as quantization, decentralized communication, and delayed communication. Through this design, Bagua achieves efficient support and expansion of various distributed training mechanisms. Duan et al. proposed a communication scheduling technology Mercury to accelerate model training at the transport layer. Mercury builds a simple transport layer scheduler, moves the priority scheduling of tensor transmission to the transport layer, and completes the scheduling at the granularity of data packets. Through the above means, Mercury eliminates the startup delay of tensor transmission scheduling and the quick adjustment of hyperparameters, accelerating model training. Chen et al. proposed a new communication scheduling framework Centauri for LLM training. This technology decomposes the scheduling tasks in hybrid parallel training into three levels: operator, model layer, and model, explores the optimization space composed of primitive replacement partitions, topology-aware group partitions, and workload partitions, and constructs a new communication scheduling framework suitable for heterogeneous environments. Fang et al. proposed a network aggregation routing GRID suitable for distributed training. GRID uses gradient routing and network aggregation in distributed training. On the control plane, an efficient gradient routing algorithm based on random rounding is used. On the data plane, intra-network aggregation is achieved by carefully designing working nodes and programmable switch logic. The above means alleviate the network bottleneck in the training cluster. Lao et al. proposed a network aggregation strategy ATP suitable for multi-tenant learning. ATP uses emerging programmable switches to support intra-network aggregation of multiple rack switches in a cluster to accelerate deep learning training jobs. ATP adopts a decentralized, dynamic, and best-effort aggregation strategy, which can efficiently and fairly share limited switch resources between simultaneously running deep learning training jobs and properly deal with the fierce competition for switch resources. Liu et al. proposed a network aggregation strategy InArt for routing aggregation to accelerate model training. InArt is the first work to introduce intra-network aggregation and routing selection in a multi-PS architecture. It uses a multi-PS architecture to split the model training task between multiple PSs and selects a suitable routing scheme to fully utilize the intra-network aggregation capabilities. Gebara et al. proposed a network architecture PANAMA for sharing cluster machine learning workloads. PANAMA consists of an efficient network acceleration hardware and a supporting congestion control protocol, which realizes the offloading of large-scale communication loads and fair sharing of network resources between different flows.

[0010] The existing methods for optimizing communication modes generally focus on optimizing the communication library, changing the communication structure, and optimizing the communication hardware, and have achieved good results in accelerating machine learning training. In the future, they will continue to develop in the direction of customized hardware and customized network structure optimization. In the cloud-edge computing environment, network communication hardware, network types, and network topologies are diverse, and the existing communication structure optimization and communication hardware optimization are difficult to apply to such scenarios and needs.

[0011] 3. Related work on enhancing the overlap between computing and communication

[0012] Enhancing the overlap of computing and communication refers to the means of making communication and computing fully overlap during deep learning training through communication splitting, communication scheduling, asynchronous transmission, etc.

[0013] Wang et al. proposed a new communication-computation decomposition technology to address the problem of over-scaling during intra-layer parallelism. This technology can identify the original communication set and related computing operations, and decompose them into a series of finer-grained operations to create more overlapping opportunities; and this technology can execute the newly created, finer-grained communication and computing operations in parallel, thereby effectively achieving the overlap of computing and data, hiding data transmission delays, and achieving better system utilization. Feng et al. proposed a pipeline strategy Mobius for heterogeneous memory. This strategy reduces model partitioning to an integer programming problem to achieve finer-grained model partitioning; this strategy performs fine-grained scheduling of model partitioning between GPU and DRAM, pre-fetches data from CPU to GPU memory for the next stage of computing, and fully achieves the overlap of communication and computing; in addition, it also adopts a cross-mapping strategy to reduce communication contention and further optimize overall performance. Oh et al. proposed an out-of-order backpropagation technology Out-Of-Order BackProp. This technology optimizes the order of calculating output gradients, weight gradients and parameter updates through three technologies: multi-stream scheduling, gradient reordering and key gradient identification for single GPU training, data parallelism and pipeline parallelism, and maximizes the overlap between communication and computing. Li et al. proposed a new communication pipeline called Fold3D using 3D parallelism. Fold3D proposed AIAO scheduling for 3D parallelism, which divides a deep learning model into multiple segments to uniformly schedule the segment computing tasks of the same deep learning model and start the synchronization of the communication tasks of the segment to overlap with the computing tasks of other segments. Lai et al. proposed an automatic 3D parallel optimization and communication technology Merak. This technology realizes automatic deployment of 3D parallelism through automatic model partitioning, and also integrates transfer critical path pipeline scheduling and idle memory-aware recomputation to achieve overlap of computing and communication. Zhang et al. proposed a memory-friendly MoE model pipeline parallel technology MPMoE. This technology can accelerate MoE training through adaptive and memory-efficient pipeline parallelism, divide the MoE training process into multiple independent sub-stages, and design a new computing-communication overlap to reduce communication latency.

[0014] The existing means of enhancing the overlap of computing and communication are mainly achieved through communication splitting, communication scheduling, asynchronous transmission, and pipeline parallelism, which have achieved good acceleration effects. In the future, it will develop in the direction of more fine-grained communication scheduling and 3D parallel optimization. In the cloud-edge computing environment, the network conditions are relatively poor and there is a lack of mature parallel strategies. There is still a lot of room for exploration and optimization on how to enhance the parallelism of computing and communication.

[0015] In summary, for communication optimization in cloud-edge computing environments, the existing solutions for reducing communication scale, optimizing communication modes, and enhancing the overlap of computing and communication are difficult to adapt to the complex network characteristics of cloud-edge computing environments. Specifically, the existing compression methods for reducing communication volume lack optimization for low-bandwidth, high-latency network scenarios; the existing communication structure optimization and communication hardware optimization are difficult to apply to scenarios and requirements with diverse network communication hardware, network types, and network topologies; the existing methods for enhancing computing and communication parallelism lack the combination of poor network conditions and cloud-edge parallel strategies. These problems make it difficult for existing communication optimization strategies to exert their optimization capabilities, limiting their performance benefits in cloud-edge computing environments. Therefore, it is extremely important to study a communication optimization strategy suitable for cloud-edge computing environments, and how to efficiently achieve fast communication between cloud-edge nodes across domains has become an urgent problem to be solved. Summary of the invention

[0016] In response to the above problems, the present invention proposes an efficient communication technology for cloud-edge cross-domain training based on partitioned sparse communication. First, through the runtime sparse backpropagation technology, under the premise of a certain impact on the model accuracy, the communication volume in the model training process is greatly reduced, and the model training process is accelerated to solve the problems of low network bandwidth and high latency in the cloud-edge computing environment; secondly, the present invention constructs a partitioned multi-level communication scheduling mechanism, and through designs such as separation of communication primitives, explores the communication optimization space in the cloud-edge computing environment, and achieves a good overlap of computing and communication in the cloud-edge environment to solve the overlapping mode challenges that arise in cross-domain distributed training modes.

[0017] Specifically, the present invention consists of the following steps:

[0018] Step (1): Sparse back-propagation mechanism.

[0019] Step (2): Establish a communication primitive separation model.

[0020] Step (3): Calculate the intra-cluster communication scheduling mechanism.

[0021] Step (4): Calculate the inter-cluster communication scheduling mechanism.

[0022] Each step is further broken down into a series of sub-steps.

[0023] Step (1.1) establishes a model weight sparsification model. In previous observations, when the accuracy of the model reaches a certain level, the weights of most models will not change, showing obvious sparsification characteristics. For example, for CNN, updating the weight of the first convolution in each CNN block is the most effective, while for the Transformer block, the weights of the first linear layer in the attention module and the feedforward network (FFN) are more important. At the same time, the sparse model back propagation brings about a significant reduction in video memory usage. For example, the Pointwise convolution in the CNN module and the input activation value of the second linear layer in the FFN module in the Transformer are large in size, and the present invention does not need to save these larger features, so the present invention can reduce video memory consumption while reducing the communication scale.

[0024] Step (1.2) The present invention establishes a correlation model between model quality and training cost. Specifically, the present method uses the improvement or decrease in training accuracy as the contribution of the i-th layer weight to the model accuracy, denoted as Similarly, this method also uses the improvement or decrease in training accuracy as the contribution of the k-th layer bias term to the model accuracy, denoted as This method randomly freezes the model layers at runtime, evaluates the impact of all layer weights and biases on model accuracy, and uses the following calculation formula to obtain the overall evaluation result:

[0025]

[0026] Among them, i is the layer index of the weight, k is the layer index of the bias, and r is the ratio of the learnable weights. At the same time, considering the memory shortage of cloud-edge devices, this method adds constraints to the above optimization objectives, namely:

[0027] M(k,i,r)≤M p

[0028] That is, the memory consumption of this method should be less than the memory capacity of the cloud-edge device. This method assumes that the accuracy contribution of each tensor, i.e., △acc, can be quantified, so the optimal sparse back-propagation configuration can be found through the above optimization objectives.

[0029] Step (1.3) After the model starts training, the user will set the threshold α for applying the sparse backpropagation mechanism during training based on accuracy heuristics. When the accuracy of the model reaches the threshold α, the sparse backpropagation mechanism during training based on accuracy heuristics will be activated.

[0030] Step (1.4) The present invention attempts to interrupt back propagation. The present invention analyzes typical back propagation models of various models and finds that the first half of most models has little impact on accuracy. Therefore, the present invention will first try to interrupt back propagation after a feature extraction layer or encoding layer, and inject a false back propagation process into the model by modifying the communication layer. Then, the impact of interrupting back propagation on model accuracy is evaluated. If it is within an acceptable range, the back propagation of the model is interrupted.

[0031] Step (1.5) After determining the interruption point of back propagation, the present invention will try to skip the back propagation of some model layers in the second half of some models, and reduce the communication overhead through sparse inter-layer back propagation. At the same time, the present invention will also evaluate the impact on the model accuracy to determine the model layer that is finally skipped.

[0032] Step (1.6) The present invention further performs intra-layer sparsification by freezing certain layers of multi-head attention, FNN, and CNN to obtain a higher communication compression ratio.

[0033] After completing the sparsity trial in step (1.7), the present invention will fix the sparsity back-propagation scheme and generate relevant configuration files and frozen computational graphs for subsequent reuse and further communication scheduling.

[0034] Step (2.1) In distributed training, collective communication can be composed of multiple basic communication primitives while maintaining the same communication effect, and can provide more possibilities for computation-communication overlap. For example, in data parallelism, AllReduce can be decomposed into Reduce and Broadcast, or ReduceScatter and AllGather, so that the communication during forward propagation and backward propagation can be separated and overlapped. Furthermore, it can be decomposed into more ReduceScatters with different Root Ranks, and AllGather can be considered equivalent to multiple Broadcast operations, providing more communication optimization space.

[0035] Step (2.2) The present invention groups the model running devices, establishes a preferred communication path through the communication capability of the network, aggregates different devices according to the communication capability, forms a device cluster, and searches for the best communication device and path.

[0036] In step (2.3), when only primitive partitioning and group partitioning are performed, the optimization space is limited, so the present invention further performs workload partitioning so that the communication can overlap with the computing chain. For example, the AllGather operation can overlap with the MatMul operation and activation functions such as GeLU and Dropout.

[0037] Step (3.1) In the scheduling within the computing cluster, the present invention mainly separates the activation value communication and the weight communication. First, we establish the time model of the activation value communication and the weight communication:

[0038]

[0039] This method uses T a The cost of the activation value, T a The first term describes the weight aggregation, the second and third terms describe the overlap of activation values ​​and communication, respectively, and the third term describes the overlap of weight aggregation and other operations. w The cost of the weight gradient is evaluated, T w It describes the activation value communication cost, overlapping calculation cost, data set communication cost and other calculation costs. a and T w The size of determines the scheduling scheme. Figure 3 As shown in the figure, these two scheduling situations are shown. The first scheduling window is a weight-priority communication scheduling, and the second scheduling window is an activation-priority communication scheduling. It can be found that this method improves the communication efficiency of communication and computing, and significantly saves the time of cross-domain distributed training.

[0040] Step (4.1) The present invention separates the AllGather communication operation and the ReduceScatter communication operation to provide maximum parallelism. The present invention receives the micro-batch state and the communication time as input, and monitors the computation-communication overlap between the computation clusters through the two states. If the overlap is low, the method regenerates the computation-communication overlap plan by optimizing the target T (the definition of T will be given later), and executes the current computation-communication overlap plan when the overlap is high.

[0041] Specifically, the optimization target T is defined as follows:

[0042] T=max(C AG -L1×C fw ,0)+max(C RS -L2×C bw ,0)

[0043] The above optimization target T is calculated by calculating the overlap between the forward propagation of multiple batches and AllGather, and the overlap between the back propagation of multiple batches and ReduceScatter. This method minimizes T by adjusting the size of L1 and L2 to find the appropriate value of L1 and L2. At the same time, this method sets the video memory restriction condition:

[0044]

[0045] And communication constraints:

[0046]

[0047] Through the above-mentioned optimization conditions and constraints, the present invention obtains a reasonable micro-batch number and a parallel strategy for communication operations, maximizing the degree of computation-communication overlap in cross-domain distributed training.

[0048] The advantages of the present invention are:

[0049] By proposing a partitioned communication mechanism for back-propagation sparsification, a cross-domain distributed training communication module is established to efficiently realize fast cross-domain node communication between cloud-edge and end. This method achieves efficient cross-domain node communication through sparse back-propagation with dynamic sparsity identification and two-layer communication scheduling with key path identification; compared with existing methods, it has the advantages of being adaptable to low-bandwidth and high-latency networks and having low computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This invention proposes a research plan for efficient communication technology for cloud-edge cross-domain training based on partitioned sparse communication.

[0051] Figure 2 The present invention proposes a variety of sparse back-propagation methods.

[0052] Figure 3 This is the communication scheduling mechanism within the computing cluster proposed by the present invention.

[0053] Figure 4 This is the computing cluster communication scheduling mechanism proposed by the present invention. DETAILED DESCRIPTION

[0054] Combined with the attached figure ( Figure 1-Figure 4 ) The present invention is described in further detail.

[0055] In order to solve the problem of how to efficiently realize fast cross-domain node communication between cloud-edge terminals in a cross-domain distributed computing framework, this paper studies efficient communication technology for cross-domain training between cloud-edge terminals based on partitioned sparse communication. First, the back-propagation sparsification mechanism is studied. The sparsity trial algorithm is used to determine the impact of sparsification of the model layer on model accuracy and communication, thereby greatly reducing the communication scale. Secondly, the partitioned hierarchical communication scheduling mechanism is studied, which mainly includes the intra-cluster communication scheduling mechanism and the inter-cluster communication scheduling mechanism. Among them, the intra-cluster communication scheduling mechanism maximizes the computing-communication overlap of the computing cluster through the mechanism of separating activation values ​​and weights. The inter-cluster communication scheduling mechanism decouples forward propagation and back propagation, and obtains the most appropriate micro-batch scheduling target through custom optimization targets. Through the above mechanism, the communication scale is effectively reduced, the computing-communication overlap rate is improved, efficient and fast cross-domain node communication is achieved, and the efficiency of cross-domain distributed training is improved.

[0056] Figure 1 The overall research scheme of efficient communication technology for cloud-edge cross-domain training based on partitioned sparse communication is presented, which includes steps (1), (2), (3) and (4). The efficient communication technology for cloud-edge cross-domain training based on partitioned sparse communication mainly consists of two parts: the first part is the runtime sparse backpropagation mechanism, and the second part is the partitioned multi-level communication scheduling mechanism. First, the sparse backpropagation mechanism for training runtime based on accuracy heuristic is studied. The sparsity heuristic algorithm is used to determine the impact of the sparsification of the model layer on the model accuracy and communication, thereby greatly reducing the communication scale; secondly, the partitioned hierarchical communication scheduling mechanism is studied, which mainly includes the intra-cluster communication scheduling mechanism and the inter-cluster communication scheduling mechanism. Among them, the intra-cluster communication scheduling mechanism maximizes the computing-communication overlap of the computing cluster through the mechanism of separating activation values ​​and weights; the inter-cluster communication scheduling mechanism decouples forward propagation and backpropagation, and obtains the most appropriate micro-batch scheduling target through a custom optimization target. Through the above mechanism, the communication scale is effectively reduced, the computing-communication overlap rate is improved, efficient and fast cross-domain node communication is achieved, and the efficiency of cross-domain distributed training is improved.

[0057] Figure 2 This method shows the process of trying to interrupt back propagation, including steps (1.1) to (1.7). This method analyzes typical back propagation models of various models and finds that the first half of most models has little impact on accuracy. Therefore, this method will first try to interrupt back propagation after a feature extraction layer or encoding layer, and inject a false back propagation process into the model by modifying the communication layer. Then, the impact of interrupting back propagation on model accuracy is evaluated. If it is within an acceptable range, the back propagation of the model is interrupted. Figure 2As shown in the upper part of , after determining the interruption point of back propagation, this method will try to skip the back propagation of some model layers in the second half of some models, and reduce the communication overhead through sparse inter-layer back propagation. At the same time, this method will also evaluate the impact on the model accuracy to determine the model layer to be skipped; as shown in Figure 2 As shown in the lower part of , this method will further perform intra-layer sparsification by freezing certain layers of multi-head attention, FNN, and CNN to obtain a higher communication compression ratio. After completing the sparsity trial, this method will fix this sparsity backpropagation scheme and generate relevant configuration files and frozen computational graphs for subsequent reuse and further communication scheduling and parallel plan regeneration.

[0058] Figure 3 The communication scheduling mechanism within the computing cluster is shown, including step (3). In the scheduling within the computing cluster, this method mainly separates the activation value communication and the weight communication. Figure 3 As shown in the figure, these two scheduling situations are shown. The first scheduling window is a weight-priority communication scheduling, and the second scheduling window is an activation-priority communication scheduling. It can be found that this method improves the communication efficiency of communication and computing, and significantly saves the time of cross-domain distributed training.

[0059] Figure 4 The inter-cluster communication scheduling mechanism is shown, including step (4). In the inter-cluster scheduling, this method separates the AllGather communication operation and the ReduceScatter communication operation to provide maximum parallelism. This method receives the micro-batch state and the communication time as input, and monitors the computation-communication overlap between the computation clusters through the two states. If the overlap is low, this method regenerates the computation-communication overlap plan by optimizing the target T (the definition of T will be given later). When the overlap is high, the current computation-communication overlap plan is executed.

[0060] The present invention proposes an efficient communication technology for cloud-edge cross-domain training based on partitioned sparse communication. First, through the runtime sparse backpropagation technology, under the premise of a certain impact on the model accuracy, the communication volume in the model training process is greatly reduced, and the model training process is accelerated to solve the problems of low network bandwidth and high latency in the cloud-edge computing environment; secondly, the present invention constructs a partitioned multi-level communication scheduling mechanism, and explores the communication optimization space in the cloud-edge computing environment through designs such as separation of communication primitives, and achieves a good overlap of computing and communication in the cloud-edge environment to solve the overlapping mode challenges that arise in cross-domain distributed training modes.

[0061] Specifically, the present invention consists of the following steps:

[0062] Step (1): Sparse back-propagation mechanism.

[0063] Step (2): Establish a communication primitive separation model.

[0064] Step (3): Calculate the intra-cluster communication scheduling mechanism.

[0065] Step (4): Calculate the inter-cluster communication scheduling mechanism.

[0066] According to the method described in the above-mentioned method of efficient communication technology for cloud-edge cross-domain training based on partitioned sparse communication, the specific process of step (1) is step (1.1) to step (1.7).

[0067] Step (1.1) establishes a model weight sparsification model. In previous observations, when the accuracy of the model reaches a certain level, the weights of most models will not change, showing obvious sparsification characteristics. For example, for CNN, updating the weight of the first convolution in each CNN block is the most effective, while for the Transformer block, the weights of the first linear layer in the attention module and the feedforward network (FFN) are more important. At the same time, the sparse model back propagation brings about a significant reduction in video memory usage. For example, the Pointwise convolution in the CNN module and the input activation value of the second linear layer in the FFN module in the Transformer are large in size, and the present invention does not need to save these larger features, so the present invention can reduce video memory consumption while reducing the communication scale.

[0068] Step (1.2) The present invention establishes a correlation model between model quality and training cost. Specifically, the present method uses the improvement or decrease in training accuracy as the contribution of the i-th layer weight to the model accuracy, denoted as Similarly, this method also uses the improvement or decrease in training accuracy as the contribution of the k-th layer bias term to the model accuracy, denoted as This method randomly freezes the model layers at runtime, evaluates the impact of all layer weights and biases on model accuracy, and uses the following calculation formula to obtain the overall evaluation result:

[0069]

[0070] Among them, i is the layer index of the weight, k is the layer index of the bias, and r is the ratio of the learnable weights. At the same time, considering the memory shortage of cloud-edge devices, this method adds constraints to the above optimization objectives, namely:

[0071] M(k,i,r)≤M p

[0072] That is, the memory consumption of this method should be less than the memory capacity of the cloud-edge device. This method assumes that the accuracy contribution of each tensor, i.e., △acc, can be quantified, so the optimal sparse back-propagation configuration can be found through the above optimization objectives.

[0073] Step (1.3) After the model starts training, the user will set the threshold α for applying the sparse backpropagation mechanism during training based on accuracy heuristics. When the accuracy of the model reaches the threshold α, the sparse backpropagation mechanism during training based on accuracy heuristics will be activated.

[0074] Step (1.4) The present invention attempts to interrupt back propagation. The present invention analyzes typical back propagation models of various models and finds that the first half of most models has little impact on accuracy. Therefore, the present invention will first try to interrupt back propagation after a feature extraction layer or encoding layer, and inject a false back propagation process into the model by modifying the communication layer. Then, the impact of interrupting back propagation on model accuracy is evaluated. If it is within an acceptable range, the back propagation of the model is interrupted.

[0075] Step (1.5) After determining the interruption point of back propagation, the present invention will try to skip the back propagation of some model layers in the second half of some models, and reduce the communication overhead through sparse inter-layer back propagation. At the same time, the present invention will also evaluate the impact on the model accuracy to determine the model layer that is finally skipped.

[0076] Step (1.6) The present invention further performs intra-layer sparsification by freezing certain layers of multi-head attention, FNN, and CNN to obtain a higher communication compression ratio.

[0077] After completing the sparsity trial in step (1.7), the present invention will fix the sparsity back-propagation scheme and generate relevant configuration files and frozen computational graphs for subsequent reuse and further communication scheduling.

[0078] According to the method described in the above-mentioned method of efficient communication technology for cloud-edge cross-domain training based on partitioned sparse communication, the specific process of step (3) is step (3.1).

[0079] Step (3.1) In the scheduling within the computing cluster, the present invention mainly separates the activation value communication and the weight communication. First, we establish the time model of the activation value communication and the weight communication:

[0080]

[0081] This method uses T a The cost of the activation value, T aThe first term describes the weight aggregation, the second and third terms describe the overlap of activation values ​​and communication, respectively, and the third term describes the overlap of weight aggregation and other operations. w The cost of the weight gradient is evaluated, T w It describes the activation value communication cost, overlapping calculation cost, data set communication cost and other calculation costs. a and T w The size of determines the scheduling scheme. Figure 3 As shown in the figure, these two scheduling situations are shown. The first scheduling window is a weight-priority communication scheduling, and the second scheduling window is an activation-priority communication scheduling. It can be found that this method improves the communication efficiency of communication and computing, and significantly saves the time of cross-domain distributed training.

[0082] According to the method described in the above-mentioned method of efficient communication technology for cloud-edge cross-domain training based on partitioned sparse communication, the specific process of step (4) is step (4.1).

[0083] Step (4.1) The present invention separates the AllGather communication operation and the ReduceScatter communication operation to provide maximum parallelism. The present invention receives the micro-batch state and the communication time as input, and monitors the computation-communication overlap between the computation clusters through the two states. If the overlap is low, the method regenerates the computation-communication overlap plan by optimizing the target T (the definition of T will be given later), and executes the current computation-communication overlap plan when the overlap is high.

[0084] Specifically, the optimization target T is defined as follows:

[0085] T=max(C AG -L1×C fw ,0)+max(C RS -L2×C bw ,0)

[0086] The above optimization target T is calculated by calculating the overlap between the forward propagation of multiple batches and AllGather, and the overlap between the back propagation of multiple batches and ReduceScatter. This method minimizes T by adjusting the size of L1 and L2 to find the appropriate value of L1 and L2. At the same time, this method sets the video memory restriction condition:

[0087]

[0088] And communication constraints:

[0089]

[0090] Through the above-mentioned optimization conditions and constraints, the present invention obtains a reasonable micro-batch number and a parallel strategy for communication operations, maximizing the degree of computation-communication overlap in cross-domain distributed training.

Claims

1. An efficient communication technology for cloud-edge-end cross-domain training based on partitioned sparse communication, characterized in that: This includes using runtime sparse backpropagation technology to reduce the amount of communication during model training and accelerate the model training process, while at the expense of certain impact on model accuracy, to address the problems of low network bandwidth and high latency in cloud-edge computing environments; building a partitioned multi-level communication scheduling mechanism, and optimizing the communication space in the cloud-edge computing environment through the separation of communication primitives, to achieve computing-communication overlap in the cloud-edge computing environment; The following steps are involved: Step (1): sparse back-propagation mechanism; Step (2): Establish a communication primitive separation model; Step (3): Calculate the intra-cluster communication scheduling mechanism; Step (4): Calculate the inter-cluster communication scheduling mechanism.

2. The cloud-edge-end cross-domain training efficient communication technology according to claim 1 is characterized in that: Step (1) includes the following steps: Step (1.1) establishes a model weight sparse model to reduce the communication scale and memory consumption; Step (1.2) establishes a correlation model between model quality and training cost, using the improvement or decrease in training accuracy as the contribution of the i-th layer weight to the model accuracy, denoted as The improvement or decrease in training accuracy is used as the contribution of the k-th layer bias term to the model accuracy, denoted as Randomly freeze the model layers at runtime, evaluate the impact of all layer weights and biases on model accuracy, and use the following calculation formula to get the overall evaluation result: Among them, i is the layer index of the weight, k is the layer index of the bias, and r is the ratio of the learnable weights. At the same time, based on the memory of the cloud-edge device, the following constraints are added: M(k,i,t)≤M p , That is, the video memory consumption does not exceed the video memory capacity of the cloud-edge device; by quantifying the accuracy contribution of each tensor, i.e., △acc, the optimal sparse backpropagation configuration is found for the optimization goal; Step (1.3) After the model starts training, the user sets the application threshold α of the sparse backpropagation mechanism during the training run based on the accuracy heuristic. When the accuracy of the model reaches the threshold α, the sparse backpropagation mechanism during the training run based on the accuracy heuristic will be started; Step (1.4) interrupts the back propagation. By modifying the communication layer, a false back propagation process is injected into the model. Then the impact of interrupting the back propagation on the model accuracy is evaluated. If it is within an acceptable range, the back propagation of the model is interrupted. Step (1.5) After determining the interruption point of back propagation, skip the back propagation of some model layers in the second half of some models, reduce the communication overhead through sparse inter-layer back propagation, evaluate the impact on model accuracy and determine the model layers to be skipped; Step (1.6) performs intra-layer sparsification by freezing certain layers of multi-head attention, FNN, and CNN to obtain a higher communication compression ratio; After completing the sparsity trial, step (1.7) fixes the sparsity back-propagation scheme and generates relevant configuration files and frozen computational graphs for subsequent reuse and further communication scheduling.

3. The cloud-edge-end cross-domain training efficient communication technology according to claim 1 is characterized in that: Step (3) includes the following steps: Step (3.1) In the scheduling within the computing cluster, separate the activation value communication and the weight communication, and establish the time model T for the activation value communication a The time model T for weight communication w as follows: By T a The cost of evaluating the activation value, T a The first term in describes weight aggregation, the second and third terms describe the overlap of activation values ​​and communications, respectively, and the third term describes the overlap of weight aggregation and other operations; w The cost of evaluating the weight gradient, T w It describes the activation value communication cost, overlapping calculation cost, data set communication cost and other calculation costs; by comparing T a and T w The size of determines the scheduling scheme, saving time for cross-domain distributed training.

4. The cloud-edge-end cross-domain training efficient communication technology according to claim 1 is characterized in that: Step (3) includes the following steps: Step (4.1) separates the AllGather communication operation and the ReduceScatter communication operation to provide maximum parallelism; receives the micro-batch state and the communication time as input, and monitors the computation-communication overlap between the computation clusters through the two states; if the overlap is low, regenerates the computation-communication overlap plan by optimizing the target T, and executes the current computation-communication overlap plan when the overlap is high; The optimization target T is defined as follows: T=max(C AG -L1×C fw ,0)+max(C RS -L2×C bw ,0), The optimization target T is calculated by calculating the overlap between the forward propagation of multiple batches and AllGather, and the overlap between the backpropagation of multiple batches and ReduceScatter, and minimizing T by adjusting the size of L1 and L2 to find the appropriate values ​​of L1 and L2; To set the video memory limit: To set communication restrictions: By optimizing conditions and constraints, we obtain a parallel strategy for the number of micro-batches and communication operations, maximizing the computation-communication overlap in cross-domain distributed training.

Images