A framework system construction method for deep learning model training
By providing programming interfaces and logical timestamp synchronization methods in the big data computing framework, the memory and throughput issues in deep learning model training are solved, achieving resource optimization and throughput improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2023-04-03
- Publication Date
- 2026-07-03
AI Technical Summary
Existing deep learning model training suffers from memory limitations and slow training speeds in big data computing frameworks, especially in IoT streaming data scenarios, where direct-method frameworks suffer from insufficient memory and synchronous update limitations that restrict system throughput.
It provides a programming interface to transform the model into an operator execution graph of a big data computing framework, and achieves asynchronous updates and resource optimization of parameter server operator instances through fast parameter partitioning and logical timestamp synchronization methods.
It alleviates the memory pressure on big data computing framework clusters, improves hardware resource utilization and system throughput, and reduces development costs and data exchange overhead.
Smart Images
Figure CN116402127B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a framework system construction method for deep learning model training, and more particularly to a framework system construction method for deep learning model training in IoT streaming data scenarios, belonging to the fields of big data processing and deep learning training environment optimization technology. Background Technology
[0002] The vision of the Internet of Things (IoT) is to make traditional embedded devices and other objects more intelligent by widely utilizing advanced technologies such as Internet communication protocols and data analytics. In recent years, much work has brought machine learning techniques such as deep learning and neural networks to IoT tasks, which has had a positive impact on data analysis, intelligent decision-making, and other tasks in IoT scenarios.
[0003] IoT tasks involve processing with big data frameworks, such as Spark and Flink, which are Big Data Analytics (BDA) system frameworks. Big data frameworks express data flow relationships by providing operators; the output data stream of one operator can serve as the input data stream of another. Furthermore, each operator can have its own set number of instances; instances of the same operator share the operator's state, while different operators are isolated from each other, thus achieving secure stateful computation.
[0004] Deep learning is a technique that uses deep neural networks for training and inference, including common structures such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Graph Neural Networks (GNNs). These models typically use the training set to continuously perform forward propagation, backpropagation, gradient calculation, and parameter updates, thereby allowing the model to converge and achieve better performance.
[0005] Currently, the application of deep learning in BDA systems falls into two main directions: the connector approach and the direct approach. The connector approach involves using additional components to connect different big data systems with deep learning systems, integrating them into a complete workflow. Frameworks like TFX and TensorflowOnSpark belong to the connector approach. However, this approach introduces additional overhead in practice (such as inter-process communication, data serialization, and persistence). The direct approach, on the other hand, aims to address these additional overheads, as exemplified by the BigDL framework. The direct approach involves training the model directly within the BDA system. Current work uses a single operator to train a single model replica, performs data parallelization training on multiple model replicas, and finally establishes a parameter synchronization mechanism with the parameter server operator.
[0006] While the direct approach is superior to the connector approach—lacking additional communication overhead and facilitating direct computation and inference of deep learning models through big data frameworks (such as Spark and Flink), reducing development costs and significantly minimizing data exchange overhead between system components, resulting in better performance in throughput and latency—current direct approach frameworks suffer from memory and training speed issues. Memory-wise, the assumption that each node has sufficient memory to accommodate a model presents a significant limitation as models become increasingly complex. Furthermore, the synchronous updates between model operators and parameter server operators restrict system throughput. Summary of the Invention
[0007] Objective: To address the shortcomings and limitations of existing BDA frameworks for deep learning model training, this invention designs a framework system construction method for deep learning training in streaming data scenarios. By providing a programming interface, the model is transformed into an operator execution graph of a big data computing framework. Then, rapid online parameter partitioning is performed, with each parameter server operator instance responsible for the aggregation calculation of a portion of the parameters. Finally, a parameter synchronization method based on logical timestamps is used to synchronize parameters between the various model trainer operators and the parameter server operators during model training. This method effectively alleviates the memory pressure on each node of the big data computing framework cluster and improves hardware resource utilization and system throughput.
[0008] Technical solution: A framework system construction method for deep learning model training in IoT streaming data scenarios, including the following:
[0009] 1) When running at the underlying level of a big data computing framework (a big data computing framework refers to a framework that supports operator chain operations, such as Flink, Spark, etc.), provide programming interface tools to parse and construct the user's task into the operator execution graph of the underlying big data computing framework, and then use the big data computing framework for model training.
[0010] 2) By using a fast model parameter splitting method, the model parameters are distributed across multiple instances of the parameter server operator, with each instance responsible for a portion of the parameter aggregation calculation;
[0011] 3) The model parameter synchronization method using logical timestamps enables parameter synchronization between the model trainer operator and the parameter server operator.
[0012] (1) A runtime tool based on a big data computing framework is provided:
[0013] 11) Based on the big data computing framework, it provides rich and flexible programming interface tools. The model operators can express computational semantics with multiple inputs and single outputs, enabling users to define models through programming interface tools.
[0014] 12) The programming interface tool parses the user-defined model computation graph into an operator execution graph of the big data computing framework, including a data distributor operator, a model trainer operator, and a parameter server operator, and then submits it to the big data computing framework for scheduling and execution; the operator execution graph includes one data distributor operator, N*M model trainer operators, and one parameter server operator. N refers to the data parallelism configured by the user, that is, N model replicas; M refers to the number of operators owned by a model. The number of instances of the data distributor operator is 1, the number of instances of the model trainer operator is 1, and the number of instances of the parameter server operator is K, where K is a configuration item of the user.
[0015] (2) Fast model parameter splitting method, which realizes the fast partitioning of the parameter list of the model when the parameter server operator is started, obtains a parameter allocation scheme for a model, and each instance is responsible for the aggregation calculation of the allocated parameters, specifically including:
[0016] Use the online allocation method, that is, the parameters are allocated when the operator is executed. After the operator is scheduled in the entire cluster, each instance of the operator performs independent parameter splitting to obtain a splitting scheme. The process is as follows:
[0017] Step 1: Each instance of the parameter server operator attempts to obtain a unique lock and register the model on the node it belongs to; if the lock cannot be obtained, it waits;
[0018] Step 2: The instance that obtains the lock is responsible for executing the fast model parameter splitting method to obtain a splitting scheme;
[0019] Step 3: The instance in the waiting state then obtains the scheme at this time;
[0020] Step 4: Each instance initializes its own responsible parameter list.
[0021] The process in Step 2 includes: by analyzing the parameters of the model, obtaining a parameter information list, and determining the cost of each parameter according to the size of each parameter, and then using a fast splitting algorithm with a greedy strategy for the cost list. The steps are as follows:
[0022] Step 2-1: Obtain the parameter list P of the model = {w i ∣0 ≤ i < l}, where l is the number of model parameters, and w i is the parameter of the model.
[0023] Step 2-2: Obtain the cost list C = {c i ∣0 ≤ i < l} according to the parameter list, and c i refers to the estimated computational cost of the parameter w i .
[0024] Steps 2-3: Set the total cost value sum, where sum is the sum of the elements in cost list C.
[0025] Steps 2-4: Set the current number of groups split to 0, the maximum allowed cost per group max to 0, the left boundary left to max, the right boundary right to sum, and the mid value mid to 0;
[0026] Step 2-5: Determine if left is less than right. If yes, proceed to step 2-6; otherwise, proceed to step 2-10.
[0027] Steps 2-6: Set mid to the average of left and right and round down, split to 1, and the maximum interval cost sum curIntervalSum of the current partitioning strategy to 0.
[0028] Step 2-7: Traverse list C. Assuming the current element is j, perform the following two steps 2-7-1) and 2-7-2) for each element until list C is completely traversed:
[0029] 2-7-1) If the sum of the current element and curIntervalSum is greater than mid, then set curIntervalSum to 0 and increment split by 1.
[0030] 2-7-2) Let curIntervalSum=curIntervalSum+1.
[0031] Step 2-8: Determine the relationship between split and the number of server instances configured by the user. If split is greater, set left = mid + 1; if split is less, set right = mid - 1; if split is equal, set right = mid.
[0032] Step 2-9: Jump to step 2-5.
[0033] Step 2-10: Set curIntervalSum = 0 and pre = 0, then start iterating from the first element of P. Assuming the current element is j, perform the following two steps until the iteration ends:
[0034] 2-10-1) If the sum of the current element and curIntervalSum is greater than right, then set curIntervalSum to 0 and increment index by 1. In this case, the parameters in the range [pre,j) are a partitioned subset.
[0035] 2-10-2) Let curIntervalSum=curIntervalSum+1.
[0036] (3) A model parameter synchronization method based on logical timestamps, which realizes the model training and parameter synchronization process between components such as data distributor, model trainer, and parameter server.
[0037] The data distributor is responsible for distributing global data and causes the model trainer to take different actions by defining message types. The model trainer operators with N*M model replicas take different actions after receiving data from the data distributor, including model calculation and parameter synchronization. Multiple instances of the parameter server operator are responsible for parameter aggregation.
[0038] At the start of the task, the data distributor maintains a global timestamp, initially set to 0, and then processes it according to the following steps:
[0039] Step 3-1: Obtain a data sample from an IoT input stream, perform user-defined preprocessing operations on the data sample, and generate an iterator.
[0040] Step 3-2: Determine if the iterator has ended. If it has ended, jump to step 3-1; otherwise, proceed to the following steps:
[0041] 3-2-1) Determine whether the Delay round has been sent. Delay is a parameter configured by the user. If so, increment the global timestamp by 1 and broadcast a token data with the global timestamp to N model replicas; otherwise, retrieve the next small batch of training samples from the iterator and send the samples sequentially to one of the N model replicas.
[0042] 3-2-2) Jump to step 3-2.
[0043] Each model trainer operator maintains its own parameter version information and local logical timestamp upon startup. The specific steps are as follows:
[0044] Step 31: Obtain all parameters of itself, mark the version of all parameters as 0; and set the local logical timestamp.
[0045] Step 32: Wait for the parameter server operator to send back the initial model parameters and use them for initialization.
[0046] Step 33: Begin processing the input stream, performing different processing based on the type of data in the stream:
[0047] 33-1. If it is training data, after aligning the data, determine whether the local parameter version is valid. If it is, calculate the model output and output the result; otherwise, wait for the next data.
[0048] 33-2. If it is token data, update the local timestamp according to the token and send the parameters to the corresponding instance of the parameter server, along with the local timestamp.
[0049] 33-3 If it is parameter aggregation data, then update the local parameters and their logical versions.
[0050] In step 33 above, step 33-1 determines the validity period as follows: Traverse the parameter version information table. If the difference between the smallest logical version number and the operator's local logical timestamp is greater than S, where S is a user configuration item, then the operator parameter is considered not to have an expiration period; otherwise, it is valid.
[0051] Each instance of the parameter server receives parameter data from the model trainer operators, performs aggregation calculations on N copies of each parameter, and sends the aggregation results to the model trainer. The specific steps are as follows:
[0052] Step (1): Call the above quick splitting method to obtain the parameter list you assigned.
[0053] Step (2): Initialize each parameter and send the parameter to the corresponding model trainer operator.
[0054] Step (3): Create N queues for each parameter, corresponding to N copies of the model.
[0055] Step (4): Place the received parameter data into the corresponding queue. If N queues are not empty, take one data from each queue and combine them to perform parameter aggregation. Then return the result to the model training operator where the parameter is located, and add version information; otherwise, wait until new data arrives, and then proceed to step (3).
[0056] Framework systems for training deep learning models include:
[0057] The API module is responsible for exposing the system's interfaces to the outside world, mainly including defining training tasks and submitting tasks to the big data computing framework cluster.
[0058] The data distributor is responsible for controlling the training of the entire task. It processes the IoT input data stream and sends training data or token data to the model trainer.
[0059] The model trainer is responsible for data processing, including training the model, sending parameters to the parameter server, receiving and updating the parameter aggregation results.
[0060] The parameter server is responsible for executing a greedy strategy for fast model splitting, aggregating and calculating the parameters of the model trainer, and returning the results to the model trainer.
[0061] The data distributor, model trainer, and parameter server coordinate their work through a model parameter synchronization method based on logical timestamps.
[0062] Beneficial effects: Compared with existing technical solutions, the present invention has the following advantages:
[0063] 1) It provides a flexible programming interface that can represent complex model operator computation graphs.
[0064] 2) A greedy strategy for online fast parameter splitting is adopted, which places the costly operations in the big data computing framework cluster for execution, and the computing overhead of each parameter server operator instance is relatively balanced.
[0065] 3) A logical timestamp-based parameter synchronization method is adopted, with each operator performing asynchronous parameter updates, which improves hardware utilization and system throughput. Attached Figure Description
[0066] Figure 1 An architecture diagram of the framework system constructed by the method in the embodiments of the present invention;
[0067] Figure 2 This is a user-defined AlexNet model computation graph in an embodiment of the present invention;
[0068] Figure 3 This is an operator execution diagram of the AlexNet network in a big data computing framework in an embodiment of the present invention. Detailed Implementation
[0069] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.
[0070] like Figure 1 As shown, the framework system architecture constructed using a method for building a deep learning model training framework system for IoT streaming data scenarios includes:
[0071] 1) The API module is responsible for exposing the system's interfaces to the outside world, mainly including defining training tasks and submitting tasks to the big data computing framework cluster.
[0072] 2) The data distributor is responsible for controlling the training of the entire task. It processes the IoT input data stream and sends training data or token data to the model trainer.
[0073] 3) The model trainer is responsible for data processing, including training the model, sending parameters to the parameter server, receiving parameter aggregation results and updating them.
[0074] 4) The parameter server is responsible for executing the greedy strategy for fast model splitting, and for aggregating and calculating the parameters of the model trainer, and then returning the results to the model trainer.
[0075] The data distributor, model trainer, and parameter server coordinate their work through a model parameter synchronization method based on logical timestamps.
[0076] When using this system, the user's workflow consists of two parts: defining the training task and task execution (which involves parsing the task into the operator execution graph of the underlying big data framework and the coordinated work of the three components: data distributor, model trainer, and parameter server).
[0077] First, the training task is defined as follows: Users use the model operator definition interface provided by the API module to represent computational semantics through the programming interface, construct a directed acyclic computation graph, and define various configurations related to training.
[0078] The next phase is task execution: This system parses the user-defined model computation graph into an operator execution graph of the big data computing framework, including a data distributor operator, a model trainer operator, and a parameter server operator. This is then handed over to the big data computing framework for scheduling and execution. The operator execution graph includes one data distributor operator, N*M model trainer operators, and one parameter server operator. N refers to the user-configured data parallelism, i.e., N model replicas; M refers to the number of operators a model has. The data distributor operator has one instance, the model trainer operator has one instance, and the parameter server operator has K instances, where K is a user-configured value.
[0079] The parameter server operator quickly partitions the model's parameter list upon startup, resulting in a parameter allocation scheme for the model. Each instance is responsible for the aggregation calculation of the allocated parameters, specifically including:
[0080] 1. Once operators are scheduled across the entire cluster, an online allocation method is executed. Each operator instance undergoes independent parameter partitioning to obtain a partitioning scheme. The process is as follows:
[0081] Step 1: Each instance of the parameter server operator attempts to acquire a unique lock and register the model on its respective node; if the lock cannot be acquired, it waits.
[0082] Step 2: The instance that obtains the lock is responsible for executing the fast model parameter splitting method to obtain the splitting scheme;
[0083] Step 3: Instances in the waiting state then obtain a solution;
[0084] Step 4: Each instance initializes its own parameter list.
[0085] 2. The process in Step 2 above includes: obtaining a parameter information list by analyzing the parameters of the model, determining the cost of each parameter according to its size, and then using a fast splitting algorithm with a greedy strategy for the cost list. The steps are as follows:
[0086] Step 1: Obtain the parameter list P = {w i ∣0 ≤ i < l}, where l is the number of model parameters and w i is the parameter of the model.
[0087] Step 2: Obtain the cost list C = {c i ∣0 ≤ i < l} according to the parameter list, where c i refers to the estimated computational cost of the parameter w i .
[0088] Step 3: Set the total cost value sum, and the sum value is the sum of the elements in the cost list C.
[0089] Step 4: Set the currently split number of groups split to 0, the maximum allowed cost number per group max to 0, the left boundary left to max, the right boundary right to sum, and the middle value mid to 0;
[0090] Step 5: Judge whether left is less than right. If so, jump to Step 6; otherwise, jump to Step 10.
[0091] Step 6: Set mid to the average of left and right and round down, split to 1, and curIntervalSum to 0. <Step 10: Set curIntervalSum = 0 and pre = 0, then start iterating from the first element of P. Assuming the current element is j, perform the following two steps until the iteration ends:
[0098] a) If the sum of the current element and curIntervalSum is greater than right, then set curIntervalSum to 0 and increment the index by 1. In this case, the parameters in the range [pre,j) are a partitioned subset.
[0099] b) Let curIntervalSum=curIntervalSum+1.
[0100] The data distributor, model trainer, and parameter server coordinate their work: the data distributor is responsible for distributing global data, and the model trainer takes different actions by defining message types; the model trainer operators, consisting of N*M model replicas, take different actions after receiving data from the data distributor, including model computation and parameter synchronization; multiple instances of the parameter server operator are responsible for parameter aggregation. Details are as follows:
[0101] At the start of the task, the data distributor maintains a global timestamp, initially set to 0, and then processes it according to the following steps:
[0102] Step 1: Obtain a data sample from an IoT input stream, perform user-defined preprocessing operations on the data sample, and generate an iterator.
[0103] Step 2: Determine if the iterator has ended. If it has ended, jump to Step 1; otherwise, proceed to the following steps:
[0104] 1. Determine whether the Delay round has been sent. Delay is a user-configured parameter. If so, increment the global timestamp by 1 and broadcast a token data with the global timestamp to N replicas. Otherwise, retrieve a small batch of training samples from the iterator and send the samples sequentially to one of the N model replicas.
[0105] 2. Jump to step 2.
[0106] Each model trainer operator maintains its own parameter version information and local logical timestamp upon startup. The specific steps are as follows:
[0107] Step 1: Obtain all parameters of itself, mark the version of all parameters as 0; at the same time, set the local logical timestamp.
[0108] Step 2: Wait for the parameter server operator to send back the initial model parameters and use them for initialization.
[0109] Step 3: Begin processing the input stream, performing different processing based on the type of data in the stream:
[0110] 1. If it is training data, after aligning the data, check if the local parameter version is valid. If it is, calculate the model output and output the result; otherwise, wait for the next data.
[0111] 2. If it is token data, update the local timestamp according to the token, and send the parameters to the corresponding instance of the parameter server, along with the local timestamp;
[0112] 3. If it is parameter aggregation data, then update the local parameters and their logical versions.
[0113] In step 3a) above, the validity period is determined as follows: Traverse the parameter version information table. If the difference between the smallest logical version number and the operator's local logical timestamp is greater than S, where S is a user configuration item, then the operator parameter is considered not to have an expiration period; otherwise, it is valid.
[0114] Each instance of the parameter server receives parameter data from the model trainer operators, performs aggregation calculations on N copies of each parameter, and sends the aggregation results to the model trainer. The specific steps are as follows:
[0115] Step 1: Call the above quick splitting method to obtain the list of parameters you have assigned.
[0116] Step 2: Initialize each parameter and send the parameters to the corresponding model trainer operator.
[0117] Step 3: Create N queues for each parameter, corresponding to N copies of the model.
[0118] Step 4: Place the received parameter data into the corresponding queue. If all N queues are not empty, take one data point from each queue and combine them to perform parameter aggregation. Then, return the result to the model training operator containing the parameters, along with version information. Otherwise, wait until new data arrives, and then proceed to Step 3.
[0119] like Figure 2 As shown in the figure, the user defines a model computation graph containing four model operators through the API interface. They have linear data dependencies, and each model operator has a different number of model parameters, including convolutional layers, pooling layers, and fully connected layers.
[0120] Figure 3 This is a schematic diagram of the operator execution graph of the underlying big data framework. Figure 2The model computation graph is transformed into an operator graph of the underlying computation framework, which includes a data distributor operator, multiple model trainer operators, and a parameter server operator. The parameter server operator has multiple instances, each responsible for a different list of parameters; each model trainer operator contains parameters corresponding to those defined by the user through an API interface.
Claims
1. A method for constructing a framework system for training deep learning models, characterized in that, Includes the following: 1) When the big data computing framework is running at the bottom layer, a programming interface tool is provided to parse and construct the user's task into the operator execution graph of the bottom big data computing framework, and then use the big data computing framework to train the model. 2) By using a fast model parameter splitting method, the model parameters are distributed across multiple instances of the parameter server operator, with each instance responsible for a portion of the parameter aggregation calculation; 3) The model parameter synchronization method using logical timestamps enables parameter synchronization between the model trainer operator and the parameter server operator; The fast model parameter partitioning method implements the partitioning of the model's parameter list during startup of the parameter server operator, resulting in a parameter allocation scheme for the model. Each instance is responsible for the aggregation calculation of the allocated parameters, specifically including: The online allocation method is used, meaning that parameters are allocated during operator execution. After the operator is scheduled across the entire cluster, each operator instance undergoes independent parameter splitting to obtain a splitting scheme. The implementation process is as follows: Step 1: Each instance of the parameter server operator attempts to acquire a unique lock and register the model on its respective node; if the lock cannot be acquired, it waits. Step 2: The instance that obtains the lock is responsible for executing the fast model parameter splitting method to obtain the splitting scheme; Step 3: Instances in the waiting state then obtain a solution; Step 4: Each instance initializes its own parameter list; The process in step 2 includes: performing parameter analysis on the model to obtain a list of parameter information; determining the cost of each parameter based on its size; and then applying a greedy strategy-based fast segmentation algorithm to the cost list, as follows: Step 2-1: Obtain the model's parameter list , It is the number of model parameters. These are the parameters of the model; Step 2-2: Obtain the cost list based on the parameter list. , Refer to parameters Estimating the computational cost; Steps 2-3: Set the total cost value to sum, where sum is the sum of the elements in cost list C; Steps 2-4: Set the current number of groups split to 0, the maximum allowed cost per group max to 0, the left boundary left to max, the right boundary right to sum, and the mid value mid to 0; Step 2-5: Determine if left is less than right. If yes, proceed to step 2-6; otherwise, proceed to step 2-10. Steps 2-6: Set mid to the average of left and right and round down, split to 1, and the maximum interval cost sum curIntervalSum of the current partitioning strategy to 0; Step 2-7: Traverse list C. Assuming the current element is j, perform the following two steps (2-7-1) and (2-7-2) for each element until list C is completely traversed: 2-7-1) If the sum of the current element and curIntervalSum is greater than mid, then set curIntervalSum to 0 and increment split by 1; 2-7-2) Let curIntervalSum=curIntervalSum+1; Steps 2-8: Determine the relationship between split and the number of server instances configured by the user. If split is greater, set left = mid + 1; if split is less, set right = mid - 1; if split is equal, set right = mid. Step 2-9: Jump to step 2-5; Step 2-10: Set curIntervalSum=0, pre=0, and then start iterating from the first element of P. Assuming the current element is j, perform the following two steps until the iteration ends: 2-10-1) If the sum of the current element and curIntervalSum is greater than right, then set curIntervalSum to 0 and increment the index by 1. At this time, the parameters in the range [pre,j) are a partitioned subset. 2-10-2) Let curIntervalSum=curIntervalSum+1.
2. The method for constructing a framework system for deep learning model training according to claim 1, characterized in that, Provide a runtime tool based on a big data computing framework: 11) Based on the big data computing framework, a programming interface tool is provided, and the model operator can express the computational semantics of multiple inputs and single output, enabling users to define models through the programming interface tool; 12) The programming interface tool parses the user-defined model computation graph into an operator execution graph of the big data computing framework, including a data distributor operator, a model trainer operator, and a parameter server operator, and then hands it over to the big data computing framework for scheduling and execution; the operator execution graph includes one data distributor operator, N*M model trainer operators, and one parameter server operator; N refers to the user-configured data parallelism, i.e., N model replicas; M refers to the number of operators a model has; the number of data distributor operator instances is 1, the number of model trainer operator instances is 1, and the number of parameter server operator instances is K, where K is a user configuration item.
3. The method for constructing a framework system for deep learning model training according to claim 1, characterized in that, The data distributor is responsible for distributing global data and causes the model trainer to take different actions by defining message types. The model trainer operators with N*M model replicas take different actions after receiving data from the data distributor, including model calculation and parameter synchronization. Multiple instances of the parameter server operator are responsible for parameter aggregation.
4. The method for constructing a framework system for deep learning model training according to claim 3, characterized in that, At the start of the task, the data distributor maintains a global timestamp, initially set to 0, and then processes it according to the following steps: Step 3-1: Obtain a data sample from an IoT input stream, perform user-defined preprocessing operations on the data sample, and generate an iterator; Step 3-2: Determine if the iterator has ended. If it has ended, jump to step 3-1; otherwise, proceed to the following steps: 3-2-1) Determine whether the Delay round has been sent. Delay is a parameter configured by the user. If so, increment the global timestamp by 1 and broadcast a token data with the global timestamp to N model replicas. Otherwise, retrieve the next small batch of training samples from the iterator and send the samples sequentially to one of the N model replicas; 3-2-2) Jump to step 3-2; Each model trainer operator maintains its own parameter version information and local logical timestamp upon startup. The specific steps are as follows: Step 31: Obtain all parameters of itself, mark the version of all parameters as 0; and set the local logical timestamp. Step 32: Wait for the parameter server operator to send back the initial model parameters and initialize with them; Step 33: Begin processing the input stream, performing different processing based on the type of data in the stream; Each instance of the parameter server receives parameter data from the model trainer operator, performs aggregation calculations on N copies of each parameter, and sends the aggregation results to the model trainer.
5. The method for constructing a framework system for training deep learning models according to claim 4, characterized in that, Step 33: Begin processing the input stream, performing different processing based on the type of data in the stream: 33-1. If it is training data, after aligning the data, check if the local parameter version is valid. If it is, calculate the model output and output the result; otherwise, wait for the next data. 33-2. If it is token data, update the local timestamp according to the token and send the parameters to the corresponding instance of the parameter server, along with the local timestamp. 33-3 If it is parameter aggregation data, then update the local parameters and their logical versions.
6. The method for constructing a framework system for training deep learning models according to claim 4, characterized in that, Each instance of the parameter server receives parameter data from the model trainer operators, performs aggregation calculations on N copies of each parameter, and sends the aggregation results to the model trainer; the specific steps are as follows: Step (1): Call the above fast splitting method to obtain the list of parameters you have assigned; Step (2): Initialize each parameter and send the parameter to the corresponding model trainer operator; Step (3): Create N queues for each parameter, corresponding to N replicas of the model; Step (4): Place the received parameter data into the corresponding queue; if N queues are not empty, take one data from each queue and combine them to perform parameter aggregation, then return the result to the model training operator where the parameter is located, and add version information; otherwise, wait until new data arrives, and then proceed to step (3).
7. The method for constructing a framework system for deep learning model training according to claim 4, characterized in that, In step 33 above, step 33-1 determines the validity period as follows: Traverse the parameter version information table. If the difference between the smallest logical version number and the operator's local logical timestamp is greater than S, where S is a user configuration item, then the operator parameter is considered not to have an expiration period; otherwise, it is valid.
8. A framework system for training deep learning models constructed by the method of claim 1, characterized in that, include: The API module is responsible for exposing the system's interfaces to the outside world, mainly including defining training tasks and submitting tasks to the big data computing framework cluster; The data distributor is responsible for controlling the training of the entire task. It processes the IoT input data stream and sends training data or token data to the model trainer. The model trainer is responsible for data processing, including training the model, sending parameters to the parameter server, receiving and updating parameter aggregation results; The parameter server is responsible for executing the greedy strategy for fast model splitting, aggregating and calculating the parameters of the model trainer, and returning the results to the model trainer. The data distributor, model trainer, and parameter server coordinate their work through a model parameter synchronization method based on logical timestamps.