Efficient distributed machine learning training system based on server-free architecture

Abstract

The invention discloses an efficient distributed machine learning training system based on a server-free architecture, and the system comprises a modeling module which is used for obtaining an input workload for modeling; the resource configuration module is configured with an optimized mathematical model and used for obtaining optimal parameter configuration based on the optimized mathematical model on the basis that the modeling module completes modeling; optimizing the relation between double-layer parameters and system-level parameters of mathematical model modeling server-free distributed training and training delay, cost and convergence efficiency; and the K-REDUCE training framework is used for triggering the server-free function, and the server-free function carries out distributed training according to the K-REDUCE framework. According to the method, a mathematical model is constructed to describe the relationship between parameters and performance, and optimal parameter configuration is searched by adopting a heuristic search algorithm based on pruning, so that the training performance is effectively improved, the economic cost is reduced, system resources are dynamically adjusted, and the optimal training performance is realized.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to an efficient distributed machine learning training system based on a serverless architecture. Background Art

[0002] Currently, there are three common parallelism mechanisms for distributed training of deep neural networks (DNNs): data parallelism, model parallelism, and pipeline parallelism. Parameter servers are a pioneering prototype of data parallelism, distributing training data across worker nodes to accelerate the training process. Model parallelism is helpful when the model size exceeds the memory capacity of a single machine. Pipeline parallelism combines the advantages of data and model parallelism to reduce the training time of large-scale models.

[0003] However, on serverless platforms, the communication overhead of these parallel mechanisms is more complex and demanding due to the lack of point-to-point communication between functions. To address this issue, researchers have proposed two common implementation strategies: server-based communication channels and storage-based communication channels. The server-based strategy starts a long-term server with a fixed communication address, allowing serverless functions to communicate with it through gRPC or HTTP. However, this implementation violates the core principles of serverless architecture, namely the absence of server management and automatic scaling, as it requires additional PaaS-based cloud services and lacks automatic scalability to adapt to increased communication traffic. On the other hand, the storage-based strategy uses object storage services as the communication channel. Compared with server-based solutions, object storage services provide greater scalability and are more cost-effective. However, the communication steps of this approach include upload, aggregation, and download, which may bring certain complexities.

[0004] Serverless computing has attracted the attention of deep learning practitioners. For example, LambdaML provides a comprehensive feature description for serverless deep model training. However, in practical applications, the following problems still exist: On the one hand, the stateless nature of serverless computing leads to the lack of point-to-point communication. This lack prevents the implementation of traditional communication patterns (such as Ring AllReduce and Decentralized Parallel) because these patterns rely on stable connections between worker nodes. Although LambdaML implements two communication patterns - AllReduce and ScatterReduce, and uses external storage as a communication channel, the communication overhead can still be as high as 6 times the computing latency; on the other hand, although serverless computing can reduce the operational burden, deep learning practitioners still need to configure system-level and application-level parameters to balance training latency, model accuracy and economic cost. However, errors in parameter configuration may lead to reduced training speed or increased costs.

[0005] LambdaML-based extensions of λDNN and Funcpipe also face similar challenges. While they optimize distributed training tasks for improved performance and predictability, communication overhead and parameter configuration still require improvement. For example, both technologies require extensive manual tuning and optimization to address communication overhead and parameter configuration, which undoubtedly increases the burden on users. In practical applications, those skilled in the art still need to overcome several key technical difficulties to better utilize serverless computing for deep learning training.

[0006] Therefore, technicians in this field are committed to developing an efficient distributed machine learning training system based on a serverless architecture. Summary of the Invention

[0007] In view of the above-mentioned shortcomings of the prior art, the present invention aims to solve the following technical problem: In serverless computing, due to its stateless nature, point-to-point communication is lacking. This lack hinders the implementation of traditional communication models that rely on stable connections between working nodes.

[0008] To achieve the above objectives, the present invention provides an efficient distributed machine learning training system based on a serverless architecture, the training system comprising:

[0009] K-REDUCE training framework, modeling module, and resource allocation module;

[0010] The modeling module is used to obtain an input workload for modeling; the workload includes a deep model, a training data set, a training cycle, and constraints;

[0011] The resource configuration module is configured with an optimization mathematical model, which is used to obtain the optimal parameter configuration based on the optimization mathematical model after the modeling module completes the modeling; the optimization mathematical model models the relationship between the two-layer parameters of serverless distributed training and the system-level parameters and training latency, cost and convergence efficiency, the two-layer parameters including memory quota, total number of working nodes and batch size, and the system-level parameters including the number of aggregators;

[0012] The K-REDUCE training framework is used to trigger a serverless function, and the serverless function performs distributed training according to the K-REDUCE framework.

[0013] Furthermore, the resource configuration module adaptively adjusts during training to obtain the optimal number of the aggregators, so that the aggregators are the optimal K working nodes (1≤K≤W), and the remaining WK ​​working nodes are non-aggregators.

[0014] Furthermore, the resource configuration module has a communication model, which obtains the relationship between communication overhead and the number of aggregators. The non-aggregators are allocated data set communication during idle time to accelerate training. The communication model is:

[0015] T=(T load +(T train iter +T comm )·I)·E

[0016] Among them, E is the total number of epochs, I is the number of iterations under each epoch, and T load is the model loading time, T train iter The training time for each iteration, T comm For each aggregate communication time, model it as follows:

[0017]

[0018] Among them, S m and S d are the model parameter size and training set size, respectively, which are determined by the training object. n and B a are the batch sizes of non-aggregate nodes and aggregate nodes respectively, M is the memory quota, K is the number of aggregate nodes, and W is the total number of nodes. The above five parameters are configuration parameters and are determined using a two-stage heuristic search algorithm. down and tp up is the communication bandwidth for downloading and uploading, and a, b and m are coefficients related to the model training speed.

[0019] Furthermore, the K-REDUCE training framework allocates additional data to the non-aggregator through an unequal dataset partitioning scheme, and the aggregator and the non-aggregator are assigned different synchronization protocols.

[0020] Furthermore, the unequal data set partitioning scheme includes two constraints: the total time of the non-aggregator does not exceed the time of the aggregator in each iteration, and the total number of iterations in one cycle remains the same for both the aggregator and the non-non-aggregator.

[0021] Furthermore, the hybrid asynchronous protocol adopts the BSP protocol for use between the aggregators and adopts the SSP protocol for communication between the aggregator and the non-aggregator.

[0022] Furthermore, the resource configuration module predicts the performance of all possible parameter configurations through the mathematical model constructed by the modeling module, and repeatedly prunes configurations that are expected to have poor performance until the optimal configuration is found.

[0023] Furthermore, the non-aggregator immediately transitions to the next iteration after obtaining the merging parameters of the previous iteration.

[0024] Furthermore, the pruning includes parameter range pruning and two-stage search. The parameter range pruning is to first determine the search range of the parameter and search for the optimal parameter value within this search range. The two-stage search is to divide the parameters into two groups and search in two stages respectively.

[0025] Furthermore, the communication channel is external storage.

[0026] The present invention provides an efficient distributed machine learning training system based on a serverless architecture. It constructs a mathematical model to describe the relationship between parameters and performance, adopts a pruning-based heuristic search algorithm to find the optimal parameter configuration and performs adaptive adjustments during the training process. It adopts an aggregator and non-aggregator adaptive adjustment configuration method to reduce excessive communication overhead, and adopts data partitioning and synchronization protocols to accelerate the training process, effectively improving training performance and reducing economic costs. In addition, it can also dynamically adjust system resources according to the characteristics of the training task to achieve optimal training performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of the working framework composition of the efficient distributed machine learning model training system based on a serverless architecture of the present invention. DETAILED DESCRIPTION

[0028] The following describes several preferred embodiments of the present invention with reference to the accompanying drawings to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0029] In the drawings, components with identical structures are denoted by the same reference numerals, and components with similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrary and are not limited by the present invention. For clarity, the thickness of components in some places in the drawings is appropriately exaggerated.

[0030] This paper proposes an efficient distributed machine learning training system (i.e., FasDL) based on a serverless architecture, aiming to reduce communication overhead and achieve automatic parameter configuration to be within time and convergence constraints and at minimal cost.

[0031] The core components of the system of the present invention mainly include: K-REDUCE training framework, modeling module, resource configuration module;

[0032] The modeling module is used to obtain input workloads for modeling; the workloads include deep models, training datasets, training cycles, and constraints;

[0033] The resource configuration module is configured with an optimization mathematical model to obtain the optimal parameter configuration based on the optimization mathematical model based on the modeling completed by the modeling module. The optimization mathematical model models the relationship between the two-layer parameters of serverless distributed training and system-level parameters and training latency, cost, and convergence efficiency. The two-layer parameters include memory quota, total number of worker nodes, and batch size, and the system-level parameters include the number of aggregators.

[0034] The K-REDUCE training framework is used to trigger serverless functions, which perform distributed training based on the K-REDUCE framework.

[0035] The components of the aforementioned system are organized as follows during the overall training process: Deep learning users submit their workloads, including deep models, training datasets, training cycles, and constraints, through the gateway. System modeling is performed first, including end-to-end training time, monetary cost, and convergence efficiency, all of which are workload-dependent. Workload parameter configurations include memory quota M, number of functions W, batch size B, and number of aggregators K. FasDL then uses an analyzer to determine workload-specific and platform-specific coefficients. The workload-specific coefficients describe the relationship between a given model's training speed and the two-layer parameter configuration, while the platform-specific coefficients describe the network capabilities provided by the platform. The modeling and analysis results are transferred to the optimizer to formulate the optimization problem, determine the parameter search range, and determine the parameter configuration based on a pruning-based heuristic algorithm. Finally, FasDL triggers serverless functions acting as aggregators and non-aggregators, allocating datasets of varying sizes. These functions perform distributed training according to the K-REDUCE framework, using external storage as a communication channel.

[0036] The training framework K-REDUCE, a core component of FasDL, reduces communication time by selecting the optimal number of aggregators, K, and accelerates training through unequal dataset partitioning and a hybrid asynchronous parallel protocol. This is a key innovation of FasDL, as traditional training frameworks typically cannot dynamically adjust parameter configurations to adapt to different training tasks and system environments.

[0037] First, communication is optimized through optimal K-aggregator selection. The K-REDUCE framework selects the optimal K worker nodes (1≤K≤W) as aggregators and the remaining WK ​​worker nodes as non-aggregators to minimize communication overhead. A model is established to describe the relationship between communication overhead and the number of aggregators, and a two-stage heuristic algorithm is designed to efficiently search for the optimal number of aggregators, K.

[0038] The present invention proposes to strengthen data partitioning and synchronization to accelerate training, and utilize the idle time of non-aggregators to accelerate the training process. To this end, an unequal dataset partitioning scheme and a hybrid asynchronous parallel protocol are proposed. The unequal dataset partitioning scheme accurately allocates additional data to non-aggregators, while the hybrid asynchronous parallel protocol flexibly implements different synchronization protocols for different roles of working nodes. The unequal dataset partitioning scheme accurately allocates additional training samples to non-aggregators. This partitioning needs to meet two constraints: the total time of the non-aggregator should not exceed the time of the aggregator in each iteration, otherwise it will cause idle waiting of the aggregator; at the same time, the total number of iterations in one cycle should be the same for the two roles (aggregator and non-aggregator) to ensure that they complete training in a similar time.

[0039] The present invention uses a hybrid asynchronous parallel (HAP) protocol for communication of training data. For communication between aggregators, the BSP protocol is used to maintain global model consistency; for communication between aggregators and non-aggregators, the SSP protocol is used, which allows worker nodes to asynchronously obtain outdated parameters and sets a staleness threshold to ensure convergence performance. To mitigate the impact of asynchronous updates on weights, the present invention fixes the staleness of non-aggregators to 1, meaning that non-aggregators obtain the merged parameters of the previous iteration and immediately transition to the next iteration without any additional waiting.

[0040] The present invention first constructs a mathematical model that describes the relationship between system-level and application-level parameters and training latency, model accuracy, and economic cost. The model's coefficients are then determined by collecting extensive experimental data. These coefficients help FasDL predict the performance of unseen parameter configurations and understand the impact of individual parameters on performance. Based on this modeling, a pruning-based heuristic search algorithm is used to find the optimal parameter configuration.

[0041] Specifically, the model describes the relationship between the two-layer parameters of serverless distributed training (including memory quota M, total number of worker nodes W, and batch size B) and the system-level parameters introduced by K-REDUCE (number of aggregators K) and training latency, economic cost, and convergence efficiency. The end-to-end training time is modeled as:

[0042] T=(T load +(T train iter +T comm )·I)·E

[0043] Among them, E is the total number of epochs, I is the number of iterations under each epoch, and T load is the model loading time, T train iter The training time for each iteration, T comm For each aggregate communication time, model it as follows:

[0044]

[0045] Among them, S m and S d are the model parameter size and training set size, respectively, which are determined by the training object. n and B a are the batch sizes of non-aggregate nodes and aggregate nodes respectively, M is the memory quota, K is the number of aggregate nodes, and W is the total number of nodes. The above five parameters are configuration parameters and are determined using a two-stage heuristic search algorithm. down and tp upis the download and upload communication bandwidth, and a, b, and m are coefficients related to model training speed. The above five parameters are determined through preliminary analysis of the serverless computing platform and training model.

[0046] Based on the predicted end-to-end training time and parameter configuration, combined with the billing rules for the serverless computing platform, the economic cost of training can be derived.

[0047] The above modeling describes three aspects of training performance: end-to-end training time, economic cost, and convergence efficiency. End-to-end training time is determined by factors such as communication overhead, single-iteration training time, and initialization overhead. Communication overhead includes three steps: upload, aggregation, and download. Each worker node splits the model parameters into K fragments and uploads them to the communication channel in the upload step. Each aggregator downloads the corresponding fragments from the other W-1 worker nodes and uploads the merged fragments back to the communication channel. Finally, each worker node downloads all merged fragments from the communication channel and reassembles them into model parameters. The time consumption of these three steps is thus described by a mathematical formula.

[0048] The training time for a single iteration depends primarily on the batch size B and the memory quota M. The batch size determines the computational effort, while the memory quota is proportional to the CPU capacity provided by the Lambda service. A mathematical formula describes the training time for a single iteration. Initialization overhead primarily includes the time to load the training model and partition the training data. Due to the maximum lifetime of the Lambda function, worker nodes must be stopped and retriggered in consistent iterations before the function times out. A mathematical formula describes the overhead of loading the model and training data.

[0049] Economic costs include Lambda function charges and network transmission charges. Lambda function charges are related to the memory quota M and the runtime T and can be calculated using a formula. Network transmission charges include charges for persistent storage (such as S3) and cache (such as ElasticCache).

[0050] Convergence efficiency is affected by the number of training rounds E and the global batch size Bg. As the global batch size increases, the model usually requires more rounds to reach the specified training loss value, that is, the convergence speed is slower.

[0051] Furthermore, after determining the mathematical model, the present invention provides a pruning-based heuristic search algorithm to find the optimal parameter configuration. This algorithm first uses the model to predict the performance of all possible parameter configurations, then prunes configurations predicted to have poor performance, narrowing the search space. This process is repeated until the optimal configuration is found. Pruning strategies include parameter range pruning and two-stage search. Parameter range pruning first determines the parameter search range and then searches for the optimal parameter value within this range. Two-stage search effectively narrows the search space by dividing the parameters into two groups and searching in two stages.

[0052] Specifically, we first prove that under the same parameter configuration M, W and B a Under this condition, the minimum cost of K-REDUCE does not exceed the cost of ScatterReduce. The proof is as follows: Since K-REDUCE further reduces the end-to-end training time T through the optimal K, the Lambda service billing of K-REDUCE does not exceed that of ScatterReduce under the same M and W. In addition, the number of requests in one communication of K-REDUCE is less than that of ScatterReduce, because K≤W, so the S3 service billing of K-REDUCE does not exceed that of ScatterReduce. In summary, under the same parameter configuration M, W, B a Under these conditions, the minimum cost of K-REDUCE does not exceed that of ScatterReduce.

[0053] Based on the above observations, we divide the five parameters into two groups: 1) M, W and B a , which are common parameters in ScatterReduce; 2) K and B n , which are additional parameters introduced in K-REDUCE. We then split the search for the original problem into two sequential phases, each targeting a set of parameters, effectively narrowing the search space. This two-phase search approach acts as a pruning strategy, reducing complexity by eliminating the need to consider all possible parameter combinations simultaneously.

[0054] In the first phase, we reformulate the optimization problem to minimize the training overhead of ScatterReduce as the objective function and determine the common parameters. Compared with ScatterReduce, the introduction of K-REDUCE in the second phase not only reduces the end-to-end training time, but also increases the global batch size because we convert multiple aggregators into non-aggregators and set a larger batch size. Taking this into account, we relax the constraints on the end-to-end training time and tighten the constraints on the global batch size by a constraint factor δ (0<δ≤1). We adjust the parameters W, M and B aPerform nested iterations. For a given W, we iterate over M in descending order. If no parameter configuration satisfies the time constraint under the current M, an early stopping mechanism is employed. This mechanism stops further iterations for smaller values ​​of M than the current W, thus avoiding unnecessary computations and improving the efficiency of the search process.

[0055] In the second stage, determine B a ,W and M, we determine the parameters K and B introduced by K-REDUCE n We iterate K from W to 1, and for the current K, we can determine the corresponding B based on the synchronization constraints between modeling and communication training. n We repeat the two-stage search with different constraint factors δ (i.e., 0.6, 0.7, 0.8, 0.9, and 1) and return the parameter configuration with the minimum cost. The synchronization constraint is represented as:

[0056] T train (B=B n )=T train (B=B a )+T agg

[0057] Among them, T agg is the aggregation phase time in the communication time, characterized by:

[0058]

[0059] After determining the optimal parameter configuration, the present invention adjusts system resources according to this configuration, including computing resources, storage resources, and network resources, which enables the present invention to achieve optimal training performance in a serverless computing environment.

[0060] Although LambdaML implements two communication modes (AllReduce and ScatterReduce) and uses external storage as a communication channel, the communication overhead can still be as high as 6 times the computing latency; serverless computing can ease operation and maintenance work, but deep learning practitioners still need to configure system-level and application-level parameters to balance training latency, model accuracy and economic costs. Incorrect configuration of system parameters may lead to slow training or high costs.

[0061] The proposed new training framework K-REDUCE can achieve a 16.8% performance improvement and a 28.3% cost reduction compared to the LambdaML architecture; verification on serverless computing platforms such as AWS Lambda shows that the prediction accuracy of the present invention exceeds 94%, and it can significantly improve the search speed by up to 11.4 times, giving it obvious advantages in training time and economic cost; the present invention has good scalability and compatibility, can be easily deployed on various serverless computing platforms, and supports the training of various deep learning models. The automated parameter configuration can reduce the workload of practitioners, making it more suitable for large-scale production environments.

[0062] This invention possesses excellence and flexibility, and has broad prospects for industrial application. First, it can be applied to various scenarios requiring deep learning training. Second, it can serve as a key component of serverless computing platforms, providing efficient deep learning training services. Third, its automated parameter configuration and resource adjustment capabilities can help enterprises reduce operation and maintenance costs and improve efficiency.

[0063] The preferred embodiments of the present invention have been described in detail above. It should be understood that numerous modifications and variations based on the concepts of the present invention are possible without inventive effort by those skilled in the art. Therefore, any technical solution that can be derived by one skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. An efficient distributed machine learning training system based on a serverless architecture, characterized in that: The training system comprises: K-REDUCE training framework, modeling module, and resource allocation module; The modeling module is used to obtain an input workload for modeling; the workload includes a deep model, a training data set, a training cycle and constraints; The resource configuration module is configured with an optimization mathematical model, which is used to obtain the optimal parameter configuration based on the optimization mathematical model on the basis of the modeling completed by the modeling module; the optimization mathematical model models the relationship between the two-layer parameters of serverless distributed training and the system-level parameters and the training delay, cost and convergence efficiency, the two-layer parameters include memory quota, total number of working nodes and batch size, and the system-level parameters include the number of aggregators; The K-REDUCE training framework is used to trigger a serverless function, and the serverless function performs distributed training according to the K-REDUCE framework.

2. The efficient distributed machine learning training system based on serverless architecture according to claim 1, characterized in that: The resource configuration module adaptively adjusts during training to obtain the optimal number of the aggregators, so that the aggregators are the optimal K working nodes (1≤K≤W), and the remaining WK ​​working nodes are non-aggregators.

3. The efficient distributed machine learning training system based on serverless architecture as claimed in claim 2, characterized in that: The resource configuration module has a communication model, which obtains the relationship between the communication overhead and the number of aggregators. The non-aggregators are assigned data set communication during idle time to accelerate training; the communication model is: T=(T load +(T trainiter +T comm )·I)·E Among them, E is the total number of epochs, I is the number of iterations under each epoch, and T load is the model loading time, T trainiter The training time for each iteration round, T comm For each aggregate communication time, the model is as follows: Among them, S m and S d are the model parameter size and the training set size, respectively, which are determined by the training object, and B n and B a are the batch sizes of non-aggregate nodes and aggregate nodes, M is the memory quota, K is the number of aggregate nodes, and W is the total number of nodes. The above five parameters are configuration parameters, which are determined by a two-stage heuristic search algorithm. down and tp up is the communication bandwidth for downloading and uploading, and a, b and m are coefficients related to the model training speed.

4. The efficient distributed machine learning training system based on serverless architecture as claimed in claim 3, characterized in that: The K-REDUCE training framework allocates additional data to the non-aggregator through an unequal dataset partitioning scheme, and the aggregator and the non-aggregator are assigned different synchronization protocols.

5. The efficient distributed machine learning training system based on serverless architecture as claimed in claim 4, characterized in that: The unequal data set partitioning scheme includes two constraints: the total time of the non-aggregator does not exceed the time of the aggregator in each iteration, and the total number of iterations in a cycle remains the same for both the aggregator and the non-non-aggregator.

6. The efficient distributed machine learning training system based on serverless architecture as claimed in claim 5, characterized in that: The hybrid asynchronous protocol adopts the BSP protocol for use between the aggregators and adopts the SSP protocol for communication between the aggregator and the non-aggregator.

7. The efficient distributed machine learning training system based on serverless architecture according to claim 1, characterized in that: The resource configuration module predicts the performance of all possible parameter configurations through the mathematical model constructed by the modeling module, and repeatedly prunes configurations that are expected to have poor performance until the optimal configuration is found.

8. The efficient distributed machine learning training system based on serverless architecture as claimed in claim 3, characterized in that: The non-aggregator immediately transitions to the next iteration after obtaining the merging parameters of the previous iteration.

9. The efficient distributed machine learning training system based on serverless architecture as claimed in claim 8, characterized in that: The pruning includes parameter range pruning and two-stage search. The parameter range pruning is to first determine the search range of the parameter and search for the optimal parameter value within the search range. The two-stage search is to divide the parameters into two groups and search in two stages respectively.

10. The efficient distributed machine learning training system based on serverless architecture according to claim 8, characterized in that: The communication channel is external storage.

Images