Distributed machine learning parameter updating method and device, and medium

Abstract

The present disclosure provides a distributed machine learning parameter updating method, device and medium, relates to the technical field of machine learning, and aims to solve the problem of how to improve the performance of distributed training of a machine learning model. The method comprises the following steps: in a first stage of model training, after each working node completes local model training once, each working node cooperates with a service node to update first model parameters until a first stage of model training end condition is reached, and a synchronization period of each working node and a self-difference updating parameter of each working node are obtained; in a second stage of model training, after each working node completes local model training several times according to the self-difference updating parameter in each synchronization period, each working node cooperates with the service node to update second model parameters until a second stage of model training end condition is reached, and a final training model is obtained. The present disclosure improves the performance of distributed training of a machine learning model by dividing the model training process of machine learning into a first stage and a second stage.

Description

Technical Field

[0001] This disclosure relates at least to the field of machine learning technology, and in particular to a distributed machine learning parameter update method, a service node, a worker node, a distributed machine learning system, and a computer-readable storage medium. Background Technology

[0002] Current distributed training of model parameters commonly employs two communication modes during the transmission of gradient or parameter information: BSP (Bulk Synchronous Parallel) and ASP (Asynchronous Parallel). In BSP mode, the parameter server updates its parameters only after all worker nodes have sent gradient or parameter information to the parameter server. In ASP mode, the parameter server updates its parameters immediately when a worker node sends gradient or parameter information back to the parameter server.

[0003] In BSP mode, performance differences among worker nodes lead to prolonged synchronization wait times, and the parameter server extends training time by waiting for slower nodes to upload parameters, impacting cluster resource utilization. In ASP mode, if some nodes are significantly slower than others, the iteration difference between fast and slow nodes becomes too large, affecting model convergence and degrading learning performance. Furthermore, during distributed training, the parameter server and worker nodes frequently communicate about gradients and parameters; as cluster and data sizes increase, this communication overhead easily becomes a performance bottleneck. Therefore, improving the performance of distributed training of machine learning models is a pressing issue that needs to be addressed. Summary of the Invention

[0004] The technical problem to be solved by this disclosure is to address the above-mentioned shortcomings of the prior art by providing a distributed machine learning parameter update method, service node, worker node, distributed machine learning system, and computer-readable storage medium to solve the problem of how to improve the performance of distributed training of machine learning models.

[0005] In a first aspect, this disclosure provides a distributed machine learning parameter update method, which is applied to a service node, worker nodes, or a distributed machine learning system including a service node and multiple worker nodes, and includes:

[0006] In the first stage of model training, after each local model training is completed, each working node collaborates with the service node to update the first model parameters until the end condition of the first stage of model training is met, and obtains the synchronization cycle and the different update parameters of each working node.

[0007] In the second stage of model training, each worker node updates its parameters according to its own differences and completes several local model trainings in each synchronization cycle. Then, each worker node and the service node collaborate to update the second model parameters until the end conditions of the second stage of model training are met, and the final trained model is obtained.

[0008] Secondly, this disclosure provides a service node, including:

[0009] The first module of global training is used to update the first model parameters in collaboration with each working node after each local model training is completed in the first stage of model training, until the end condition of the first stage of model training is met, and to obtain the synchronization period of each working node and its own differential update parameters.

[0010] The second global training module, connected to the first global training module, is used in the second stage of model training. After each working node updates its parameters according to its own differences and completes several local model trainings in each synchronization cycle, it collaborates with each working node to update the second model parameters until the end condition of the second stage of model training is met, and the final trained model is obtained.

[0011] Thirdly, this disclosure provides a working node, including:

[0012] The first module of local training is used to update the parameters of the first model in collaboration with the other working nodes and service nodes after each local model training is completed in the first stage of model training, until the end condition of the first stage of model training is met, and obtain the synchronization cycle with the other working nodes and the parameter update with its own differences.

[0013] The second local training module, connected to the first local training module, is used in the second stage of model training. After updating the parameters according to its own differences and completing several local model trainings in each synchronization cycle, it collaborates with other working nodes and service nodes to update the second model parameters until the end conditions of the second stage of model training are met, and the final trained model is obtained.

[0014] Fourthly, this disclosure provides a distributed machine learning system, including a service node and multiple worker nodes, and includes:

[0015] The first module of model training is used to update the first model parameters of each working node and the service node after each local model training is completed in the first stage of model training, until the end condition of the first stage of model training is met, and to obtain the synchronization period and the different update parameters of each working node.

[0016] The second module of model training is connected to the first module of model training. In the second stage of model training, each working node updates its parameters according to its own differences and completes several local model trainings in each synchronization cycle. Then, each working node and the service node update the second model parameters until the end condition of the second stage of model training is met, and the final trained model is obtained.

[0017] Fifthly, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the distributed machine learning parameter update method described above.

[0018] This disclosure provides a distributed machine learning parameter update method, service nodes, worker nodes, a distributed machine learning system, and a computer-readable storage medium. The method divides the machine learning model training process into a first stage and a second stage. In the first stage, based on the actual performance of each worker node after completing its local model training, it obtains its own differential update parameters for local model training in the second stage and a synchronization period for controlling the synchronization of model parameters between worker nodes in the second stage. By understanding the performance differences of each worker node in the first stage, a synchronization control mechanism for the second stage is formulated. In the second stage, the synchronization period is used to simultaneously synchronize the parameter updates of each worker node after several local model training iterations, reducing the waiting time of the service node, reducing the frequency of communication between worker nodes and the service node, improving resource utilization, and reducing communication overhead. The differential update parameters control the number of local model training iterations of each worker node within the same synchronization period, thereby reducing the iteration difference between worker nodes and improving model convergence performance. Attached Figure Description

[0019] Figure 1 This is a flowchart of a distributed machine learning parameter update method according to an embodiment of this disclosure;

[0020] Figure 2 This is a flowchart of another distributed machine learning parameter update method according to an embodiment of this disclosure;

[0021] Figure 3 This is a flowchart of the first stage of a distributed machine learning parameter update method according to an embodiment of the present disclosure;

[0022] Figure 4 This is a flowchart of the second stage of a distributed machine learning parameter update method according to an embodiment of the present disclosure;

[0023] Figure 5 This is an interaction diagram of a distributed machine learning parameter update method according to an embodiment of this disclosure;

[0024] Figure 6This is a schematic diagram of the structure of a service node according to an embodiment of this disclosure;

[0025] Figure 7 This is a schematic diagram of the structure of a working node according to an embodiment of this disclosure;

[0026] Figure 8 This is a schematic diagram of the structure of a distributed machine learning system according to an embodiment of the present disclosure. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions of this disclosure, the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0028] It is understood that the specific embodiments and accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0029] It is understood that, without conflict, the various embodiments and features in the embodiments of this disclosure can be combined with each other.

[0030] It is understood that, for ease of description, only the parts relevant to this disclosure are shown in the accompanying drawings, while parts unrelated to this disclosure are not shown in the drawings.

[0031] It is understood that each unit or module involved in the embodiments of this disclosure may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0032] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this disclosure may occur in a different order than that marked in the accompanying drawings.

[0033] It is understood that the flowcharts and block diagrams of this disclosure illustrate the architecture, functions, and operations of possible implementations of systems, apparatuses, devices, and methods according to various embodiments of this disclosure. Each block in a flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagrams and flowcharts may be implemented using a hardware-based system to implement the specified function, or using a combination of hardware and computer instructions.

[0034] It is understood that the units and modules involved in the embodiments of this disclosure can be implemented by software or by hardware, for example, the units and modules can be located in a processor.

[0035] Example 1:

[0036] like Figure 1As shown, this disclosure provides a distributed machine learning parameter update method, which is applied to a service node, worker nodes, or a distributed machine learning system including a service node and multiple worker nodes, and includes:

[0037] S1. In the first stage of model training, after each working node completes its local model training, each working node and the service node collaborate to update the first model parameters until the end condition of the first stage of model training is met, and the synchronization cycle and the self-differential update parameters of each working node are obtained.

[0038] S2. In the second stage of model training, each working node updates its parameters according to its own differences and completes several local model trainings in each synchronization cycle. Then, each working node and the service node collaborate to update the second model parameters until the end condition of the second stage of model training is met, and the final trained model is obtained.

[0039] Specifically, with the rapid growth in the size of model parameters, the training of large models becomes inseparable from the support of AI (Artificial Intelligence) computing power. The computing power of a single training GPU is far from sufficient to support the training resource requirements of large models, leading to a stronger demand for large-scale distributed parallel training. The Parameter Server (PS) architecture is a commonly used distributed training architecture in deep learning. Its basic idea is to store model parameters on one or more central servers. Each worker node can obtain the current model parameters from the parameter server (i.e., the service node) and return the calculation results to the parameter server for updates. Based on the above distributed training foundation, this embodiment proposes... Figure 1 The method described here is a parameter update approach suitable for distributed machine learning. It mainly includes two stages of model training. The first stage primarily obtains the synchronization period and differential update parameters. In the second stage, each worker node performs distributed training based on the synchronization period and differential update parameters obtained in the first stage. Within each synchronization period in the second stage, each worker node trains its local model based on the existing model and data, combined with the differential update parameters. A global model update is performed when the synchronization period ends. This method reduces the synchronization waiting time between worker nodes, avoids excessive iteration differences between nodes, and alleviates the communication pressure between the parameter server and worker nodes. This achieves improved performance of distributed deep learning model training while simultaneously increasing the utilization of computing resources.

[0040] A more specific implementation example is as follows Figure 2 As shown, a parameter update method suitable for distributed machine learning is presented. The distributed training architecture adopts a PS architecture, which mainly includes two stages of model training. The main steps include:

[0041] S01. The service node distributes the initial global model to each worker node. Specifically, after starting a distributed training task, the service node distributes the initial global model to each worker node.

[0042] S02. Each working node obtains training data. Specifically, the service node distributes the data to each working node, or each working node uses its local data as model training data.

[0043] S03. Distributed training using a single-training synchronous update mechanism: Specifically, during the first stage of model training, each worker node uploads the parameters obtained from local model training to the service node once after completing local training. Each time the service node receives all the parameters from each worker node, it updates the parameters of the global model to each worker node.

[0044] S04. Determine whether the first stage end condition has been met. Specifically, after N distributed trainings using the single training synchronization update mechanism, the service node will send the updated global model and the calculated synchronization period and difference update parameters to each working node.

[0045] S05. Distributed training using a differential training and synchronous update mechanism: Specifically, during the second stage of model training, each working node performs local model training within the synchronization period. When the synchronization period time is reached, each working node uploads its local model parameters to the service node, and the service node distributes the aggregated and updated global model to each working node.

[0046] S06. Determine whether the end condition of the second stage has been met. Specifically, determine whether the model training has reached the number of iterations. If the model training has not reached the number of iterations, continue to the second stage of model training in step S05.

[0047] S07. Output the final trained model. Specifically, if the model training has reached the required number of iterations, output the global model obtained from distributed training.

[0048] It is understandable that some technical features in the above specific examples can be replaced. For example, the initial global model can have another source, the training data can be obtained through different means, and the termination conditions of the first and second stages can be set to other criteria besides the number of training sessions mentioned above.

[0049] In one implementation, after each worker node completes its local model training, each worker node collaborates with the service node to update the first model parameters until the end condition of the first stage of model training is met, thereby obtaining the synchronization period and its own differential update parameters for each worker node, specifically including:

[0050] Each worker node completes a local model training based on the updated global model parameters from the service node. After obtaining its own local model parameters and local training time parameters, it sends its own local model parameters and local training time parameters to the service node. The service node receives the local model parameters and local training time parameters from each worker node.

[0051] After the service node updates the global model parameters based on the local model parameters of each worker node each time, it sends the updated global model parameters to each worker node. Each worker node then receives the updated global model parameters from the service node.

[0052] Until the global model parameters are updated N times, the service node calculates the synchronization period and its own differential update parameters for each worker node based on the N local training time parameters of each worker node, and then sends the synchronization period and its own differential update parameters to each worker node. Each worker node receives the synchronization period and its own differential update parameters from the service node.

[0053] Specifically, in this embodiment, the parameter update method for distributed machine learning, the first stage of model training is mainly used to obtain the synchronization period and difference update parameters to provide for the subsequent second stage of model training. It can be regarded as the preprocessing stage of the entire distributed training process.

[0054] Another more specific example of parameter updates in distributed machine learning is as follows: Figure 3 As shown, in a specific distributed machine learning system, there is one service node and three worker nodes. The service node distributes initial data, including the initial global model, to each worker node. During the first stage of model training, a single-train synchronous update mechanism is used for distributed training twice. The three worker nodes train the model locally, and the training process includes forward propagation and backward propagation. The local parameters are uploaded to the service node. The local parameters include local model parameters, update count, forward propagation time, and backward propagation time. After all three worker nodes have completed uploading their local parameters, the service node updates the global model only once, and calculates the second-stage parameters a second time, including updating the global model, calculating the synchronization period and differential update parameters, and then distributes the second-stage parameters to the three worker nodes.

[0055] In one implementation, each worker node performs local model training based on the updated global model parameters from the service node, obtaining its own local model parameters and local training time parameters, specifically including:

[0056] Each worker node determines its own current local model training iterations t. iAfter <N>, based on its own local model training data and the updated global model parameters w from the service node this time k , complete the current local model training of the t-th time based on the mini-batch stochastic gradient descent algorithm i to obtain the local model parameters of its own this time in the first stage as and the local training time parameters of its own this time, including the forward propagation time and the backward propagation time where r represents the learning rate, is to calculate the gradient by backpropagation, and record t i = t i + 1.

[0057] In one embodiment, the service node updates the global model parameters once according to the local model parameters of each worker node each time, specifically including:

[0058] The service node aggregates according to the current from all M worker nodes to obtain the updated global model parameters of this time as and record the global model update count k = k + 1.

[0059] In one embodiment, until the global model parameters are updated N times, the service node calculates the synchronization period and respective differential update parameters of each worker node according to the N local training time parameters of each worker node, specifically including:

[0060] Until the service node determines that k = N, the service node calculates the average calculation time of each worker node's N training times to obtain the common synchronization period of each worker node where T is a preset synchronization period threshold, calculate the average backward propagation time of each worker node's N training times to obtain the respective differential update parameters of each worker node where α is an integer constant greater than 1.

[0061] Specifically, in this embodiment, in the first-stage model training in the overall process of parameter update in distributed machine learning, its detailed process is as Figure 4 shown, and the main steps include:

[0062] S11. Enter the first-stage model training link. At this time, the global model update count k that has completed distributed training is recorded as zero;

[0063] S12. Each worker node completes one local model training iteration and uploads the local parameters obtained from this training to the service node. Specifically, there are M worker nodes, each labeled i, and the current training iteration is denoted as t, where i∈[1,M], t∈[1,N]. Each worker node is based on the existing model w... t The model is trained locally on the data, and the parameters of the local model are updated using the mini-batch stochastic gradient descent algorithm. The gradient is calculated through backpropagation. And complete the local model parameter update. Where r represents the learning rate, and also records the forward propagation time during training. Backpropagation time After the current training is completed, each worker node will transfer the local model parameters w i Number of updates (t) i Forward propagation time and backpropagation time Uploaded to the service node, in the first phase t i =t=k;

[0064] S13. The service node completes a global model update and distributes the updated global model parameters to each worker node. Specifically, the service node waits for all worker nodes to upload their local model parameters, update count, forward propagation time, and backpropagation time. The service node then aggregates the local model parameters from each worker node to obtain the updated global model parameters. At the same time, k = k + 1;

[0065] S14. Determine whether N rounds of distributed training have been completed. Specifically, the service node determines whether the global model has been updated N times in the distributed training, i.e., whether k equals N. If N rounds of distributed training have not been completed, repeat steps S12-S14. After each round of operation, t = t + 1.

[0066] S15. The service node calculates the synchronization period and the differential update parameters for each worker node, and distributes them to each worker node. Specifically, if distributed training has been completed N times, the service node calculates the global model update synchronization period T based on the forward propagation time and backpropagation time of each worker node. syn The corresponding difference update parameter α is calculated based on the backpropagation time of each working node. i Among them, the average computation time for each training session at each working node. Synchronization period T is a preset synchronization period threshold, and the average backpropagation time for each worker node during each training iteration is... Difference update parameters for each working node α is an integer constant greater than 1; the service node will update the global model parameter w.t And the calculated synchronization period T syn And the corresponding difference update parameter α for each working node i Distribute to each work node.

[0067] In one implementation, each worker node updates its parameters according to its own differences and completes several local model training iterations within each synchronization cycle. Then, each worker node collaborates with the service node to update the second model parameters until the end condition of the second stage of model training is met, thus obtaining the final trained model. This specifically includes:

[0068] Each worker node uses the updated global model parameters from the service node for each synchronization period as the initial model for each synchronization period. Based on its own differences, it updates the parameters and performs several local model trainings on the initial model for each synchronization period. After obtaining its own local model parameters for this synchronization period, it sends its own local model parameters for this synchronization period to the service node. The service node receives the local model parameters from each worker node for this synchronization period.

[0069] After the service node updates the global model parameters once based on the local model parameters of each worker node in each synchronization period, it sends the updated global model parameters for the current synchronization period to each worker node. Each worker node receives the updated global model parameters for the current synchronization period from the service node.

[0070] Once the service node determines that the conditions for ending the second stage of model training have been met, the service node generates and sends a model training termination instruction to each worker node. Each worker node and the service node then obtain the final trained model based on the model training termination instruction.

[0071] Specifically, in this embodiment, the second-stage model training phase is the main execution phase of distributed training in the parameter update method of distributed machine learning. Each working node performs distributed training based on the synchronization period and differential update parameters obtained in the first-stage model training phase. Within the synchronization period, each working node performs local model training based on the existing model and data. When the synchronization period is reached, a global model update is performed.

[0072] In such Figure 3In another more specific example of distributed machine learning parameter updates, during the second-stage model training, each worker node begins its second-stage model training based on the second-stage parameters it receives. Within each synchronization cycle, worker node 1 performs local model updates 6 times, worker node 2 performs local model updates 4 times, and worker node 3 performs local model updates 3 times. The server node performs a global model update and distributes it to the three worker nodes, then proceeds to the next synchronization cycle. Using this distributed machine learning parameter update method, the first-stage model training process primarily obtains the synchronization cycle and differential update parameters for subsequent second-stage model training. During the second-stage model training, each worker node performs distributed training based on the synchronization cycle and differential update parameters obtained in the first-stage model training. Within each synchronization cycle, each worker node performs local model training based on the existing model and data, combined with the differential update parameters, and performs a global model update upon reaching the synchronization cycle. By adopting this parameter update method, the synchronization period reduces the idle waiting time of each worker node and alleviates the communication pressure between worker nodes and service nodes. At the same time, it avoids excessive differences in the local model update iterations of each worker node. Combined with the differential update parameter, the local model update rhythm of each worker node is as balanced as possible within the synchronization period, reducing the impact of iteration differences on the global model update. Thus, it can improve the performance of distributed training of deep learning models while improving the utilization of computing resources.

[0073] In one implementation, each worker node uses the updated global model parameters from the service node for each synchronization cycle as its initial model for that cycle. Based on its own differences, it updates the parameters and performs several local model training iterations on the initial model for each synchronization cycle to obtain its own local model parameters for that synchronization cycle. Specifically, this includes:

[0074] Each worker node responds to the updated global model parameter w received from the server node during the current synchronization cycle. k Based on the current number of training iterations t of the local model i w k As its current local model Or, in response to completing its current t-th iteration... i Secondary local model training, recording t i =t i +1, obtain the current local model.

[0075] Each working node responds if its current time has not reached the synchronization cycle, based on its own local model training data. Update parameter α based on its own differences i Based on the mini-batch stochastic gradient descent algorithm, complete its current t-th step. iThe second stage of local model training is used to obtain the parameters of the local model itself in this second stage. Where r represents the learning rate. It is backpropagation to calculate the gradient, where k is the number of times the global model has been updated, and the constant β∈(0,1);

[0076] Each working node responds at the current time to reach its current synchronization period, and sends its last response within the current synchronization period. These are local model parameters for this synchronization cycle itself.

[0077] In one implementation, the service node updates the global model parameters once based on the local model parameters of each worker node during each synchronization cycle, specifically including:

[0078] The service node synchronizes with all M worker nodes for the current synchronization period. The updated global model parameters obtained after aggregation in this synchronization cycle are: And record the number of global model updates, k = k + 1.

[0079] In one implementation, until the service node determines that the conditions for ending the second stage of model training have been met, the service node generates and sends a model training termination instruction to each worker node. Each worker node and the service node obtain the final trained model based on the model training termination instruction, specifically including:

[0080] Until the service node determines that k = K, the service node generates and sends a model training end instruction to each worker node. Each worker node and the service node then use the global model parameters updated after the Kth time as the final training model parameters according to the model training end instruction to obtain the final training model.

[0081] Specifically, in this embodiment, the detailed process of the second stage of model training in the overall parameter update process of distributed machine learning is as follows: Figure 5 As shown, the main steps include:

[0082] S21. Enter the second stage of model training. At this point, the training is based on the first stage.

[0083] S22. Each working node updates its parameters based on its own differences to complete a local model training. Specifically, each working node updates its parameters based on the existing model. The model is trained locally on the data, and the parameters of the local model are updated using the mini-batch stochastic gradient descent algorithm. The gradient is calculated through backpropagation. And complete the local model parameter update. Where t ik∈[N+1,∞) represents the number of local model updates performed by worker node i, k∈[N+1,∞) represents the number of global model updates performed by service node i, and the constant β∈(0,1);

[0084] S23. Determine if the synchronization period has been reached. Specifically, each working node determines whether the time since completing the training of the current local model has reached the synchronization period T. syn If the synchronization period T is not reached syn Then each working node repeats steps S22-S23. After each round of operations, the number of node updates is t. i =t i +1;

[0085] S24. Each working node uploads the final local parameters obtained during the training of this synchronization period to the service node. Specifically, if the synchronization period T has been reached... syn Then each working node will use the local model parameters trained iteratively within this period. Update count t i Upload to the service node;

[0086] S25. After completing a global model update, the service node distributes the updated global model parameters to each worker node. Specifically, after obtaining the local model parameters and update count uploaded by all worker nodes, the service node aggregates the local model parameters from each worker node to obtain the updated global model parameters. At the same time, k = k + 1, the service node will update the global model w. k And the number of times to synchronize the update, k, is distributed to each working node;

[0087] S26. Determine whether the model training has been iterated K times. Specifically, the service node determines whether the global model update of distributed training has been completed K times, i.e., whether k is equal to K. If distributed training has not been completed K times, repeat steps S22-S26.

[0088] S27. The service node and each worker node end the model training and obtain the final trained model. Specifically, if the global model has been updated K times, the service node ends the training and obtains the final trained model, and can also instruct each worker node to end the training and obtain the final trained model.

[0089] This embodiment proposes a parameter update method for distributed machine learning, mainly comprising two stages of model training. The first stage primarily obtains the synchronization period and differential update parameters. In the second stage, each worker node performs distributed training based on the synchronization period and differential update parameters obtained in the first stage. Detailed flowcharts of the two-stage model training process and the calculation methods for the synchronization period and differential update parameters are provided. Within the synchronization period, each worker node performs local model training based on the existing model and data, combined with the differential update parameters. A global model update is performed when the synchronization period is reached. This method can reduce the synchronization waiting time between worker nodes, avoid excessively large iteration differences between nodes, and alleviate the communication pressure between the parameter server and worker nodes, thereby improving the performance of distributed training of deep learning models while increasing the utilization of computing resources.

[0090] Example 2:

[0091] like Figure 6 As shown, this disclosure provides a service node, including:

[0092] The first module of global training 11 is used to update the first model parameters in collaboration with each working node after each local model training is completed in the first stage of model training, until the end condition of the first stage of model training is met, and to obtain the synchronization period and the self-differential update parameters of each working node.

[0093] The second global training module 12 is connected to the first global training module 11. It is used in the second stage of model training. After each working node updates its parameters according to its own differences and completes several local model trainings in each synchronization cycle, it collaborates with each working node to update the second model parameters until the end condition of the second stage of model training is met, and the final trained model is obtained.

[0094] In one embodiment, the global training first module 11 specifically includes:

[0095] The first local model parameter receiving unit is used for each working node to complete a local model training once based on the updated global model parameters from the service node, obtain its own local model parameters and local training time parameters, send its own local model parameters and local training time parameters to the service node, and then receive the local model parameters and local training time parameters from each working node.

[0096] The first global model parameter sending unit, connected to the first local model parameter receiving unit, is used to update the global model parameter once according to the local model parameters of each worker node each time, and then send the updated global model parameter of this time to each worker node, so that each worker node can receive the updated global model parameter of this time from the service node respectively;

[0097] The synchronization control parameter sending unit, connected to the first global model parameter sending unit, is used to calculate the synchronization period and their respective differential update parameters of each worker node according to the N local training time parameters of each worker node until the global model parameter is updated N times, and then send the synchronization period and their respective differential update parameters to each worker node, so that each worker node can receive the synchronization period from the service node and its own differential update parameter respectively.

[0098] In an embodiment, the first local model parameter receiving unit is specifically used to receive the following parameters:

[0099] Each worker node respectively determines its own current local model training times t i After <N, based on its own local model training data and the updated global model parameter w of this time from the service node k , complete its own current local model training of the t i th time based on the mini-batch stochastic gradient descent algorithm, and obtain the local model parameter of its own in the first stage of this time And its own local training time parameter of this time includes the forward propagation time And the backward propagation time Where r represents the learning rate, Is the gradient calculated by the backward propagation, and / or the recorded t i =t i +1.

[0100] In an embodiment, the first global model parameter sending unit specifically includes:

[0101] The first global model parameter update subunit is used to aggregate the current From all M worker nodes to obtain the updated global model parameter of this time as And record the global model update times k = k + 1.

[0102] In an embodiment, the synchronization control parameter sending unit specifically includes:

[0103] The synchronization control parameter generation subunit is used to calculate the average calculation time of each worker node's N training times until it is judged that k = N To obtain the common synchronization period of each worker node Where T is a preset synchronization period threshold, and the average backpropagation time of each working node for N training iterations is calculated. Obtain the difference update parameters for each working node. Where α is an integer constant greater than 1.

[0104] In one embodiment, the global training second module 12 specifically includes:

[0105] The second local model parameter receiving unit is used for each working node to take the updated global model parameters from the service node in each synchronization period as the initial model in each synchronization period, update the parameters according to its own differences, complete the local model training for the initial model in each synchronization period several times, obtain its own local model parameters in this synchronization period, send its own local model parameters in this synchronization period to the service node, and then receive the local model parameters from each working node in this synchronization period.

[0106] The second global model parameter sending unit is connected to the second local model parameter receiving unit. It is used to update the global model parameters once according to the local model parameters of each working node in each synchronization period, and then send the updated global model parameters for the current synchronization period to each working node so that each working node can receive the updated global model parameters for the current synchronization period from the service node.

[0107] The model training end instruction sending unit is connected to the second global model parameter sending unit. It is used to generate and send model training end instructions to each working node until the current condition for the end of the second stage of model training is met, and each working node obtains the final trained model according to the model training end instructions.

[0108] In one embodiment, the second local model parameter receiving unit is specifically used to receive the following parameters:

[0109] The global model parameter w is determined by each worker node in its response to receiving the update from the service node during the current synchronization cycle. k Based on the current number of training iterations t of the local model i w k As its current local model Or, in response to completing its current t-th iteration... i Secondary local model training, recording t i =t i +1, obtain the current local model.

[0110] And, if each responds before the current time reaches its synchronization cycle, it will do so based on its own local model training data. Update parameter α based on its own differences iBased on the mini-batch stochastic gradient descent algorithm, complete its current t-th step. i The second stage of local model training is used to obtain the parameters of the local model itself in this second stage. Where r represents the learning rate. It is backpropagation to calculate the gradient, where k is the number of times the global model update has been completed, and the constant β∈(0,1);

[0111] Each response reaches its current synchronization period at the current time, and sends its last response within the current synchronization period. The parameters used are the local model parameters for this synchronization cycle itself.

[0112] In one embodiment, the second global model parameter sending unit specifically includes:

[0113] The second global model parameter update subunit is used to update parameters based on the current synchronization cycle from all M working nodes. The updated global model parameters obtained after aggregation in this synchronization cycle are: And record the number of global model updates, k = k + 1.

[0114] In one embodiment, the model training completion instruction sending unit specifically includes:

[0115] The model training end command sending sub-unit is used to generate and send the model training end command to each worker node until k = K is determined.

[0116] The first final training model obtains a sub-unit, which is connected to the sub-unit that sends the model training end instruction. This sub-unit is used to obtain the final training model by having each working node use the global model parameters updated after the Kth time as the final training model parameters according to the model training end instruction.

[0117] Example 3:

[0118] like Figure 7 As shown, this disclosure provides a working node, including:

[0119] The first local training module 21 is used to update the first model parameters in collaboration with other working nodes and service nodes after each local model training is completed in the first stage of model training, until the end condition of the first stage of model training is met, and obtain the synchronization cycle with other working nodes and the parameter update with its own differences.

[0120] The second local training module 22 is connected to the first local training module 21. It is used to update the parameters according to its own differences and complete several local model trainings in each synchronization cycle during the second stage of model training. Then, it collaborates with other working nodes and service nodes to update the second model parameters until the end condition of the second stage of model training is met, and the final trained model is obtained.

[0121] In one embodiment, the first module 21 is locally trained, specifically including:

[0122] The first local model parameter sending unit is configured to complete one local model training according to the updated global model parameters from the service node each time. After obtaining the local model parameters and local training time parameters of this time for itself, it sends the local model parameters and local training time parameters of this time for itself to the service node, so that the service node receives the local model parameters and local training time parameters of each of the working nodes this time;

[0123] The first global model parameter receiving unit, connected to the first local model parameter sending unit, is configured to, after the service node updates the global model parameters once according to the local model parameters of each working node each time and sends the updated global model parameters of this time to each working node, receive the updated global model parameters of this time from the service node;

[0124] The synchronization control parameter receiving unit, connected to the first global model parameter receiving unit, is configured to until the global model parameters are updated N times, the service node calculates the synchronization period and respective differential update parameters of each working node according to the N local training time parameters of each working node, and after sending the synchronization period and respective differential update parameters to each working node, receive the synchronization period and its own differential update parameters from the service node.

[0125] In one embodiment, the first local model parameter sending unit specifically includes:

[0126] The first local model parameter obtaining subunit is configured to determine its own current local model training times t i <After N, based on its own local model training data and the updated global model parameters w of this time from the service node k , complete its own current t i th local model training based on the mini-batch stochastic gradient descent algorithm, so as to obtain the local model parameters of this time in the first stage as and the local training time parameters of this time for itself include the forward propagation time and the backward propagation time where r represents the learning rate, is the gradient calculated by the backward propagation, and record t i =t i +1.

[0127] In one embodiment, the first global model parameter receiving unit is specifically configured to receive the following parameters:

[0128] From the service node according to the current The aggregated global model parameters obtained after this update And / or the number of global model updates recorded, k = k + 1.

[0129] In one embodiment, the synchronization control parameter receiving unit is specifically used to receive the following parameters:

[0130] Until the service node determines k=N, the service node calculates the average computation time of each worker node for each of the N training iterations. The common synchronization period of all working nodes. Where T is a preset synchronization period threshold, and the average backpropagation time of each working node for N training iterations is calculated. The difference update parameters obtained for each working node Where α is an integer constant greater than 1.

[0131] In one embodiment, the local training second module 22 specifically includes:

[0132] The second local model parameter sending unit is used to take the updated global model parameters from the service node in each synchronization period as the initial model of each synchronization period, update the parameters according to its own differences, complete the local model training of the initial model of each synchronization period several times, obtain the local model parameters of the current synchronization period, and then send the local model parameters of the current synchronization period to the service node so that the service node can receive the local model parameters of each working node in the current synchronization period.

[0133] The second global model parameter receiving unit is connected to the second local model parameter receiving unit. It is used for the service node to update the global model parameters once according to the local model parameters of each working node in each synchronization period, send the updated global model parameters of the current synchronization period to each working node, and receive the updated global model parameters of the current synchronization period from the service node.

[0134] The model training end instruction receiving unit is connected to the second global model parameter sending unit. It is used until the service node determines that the end condition of the second stage of model training has been met. After the service node generates and sends the model training end instruction to each worker node, it obtains the final trained model with the other worker nodes and service nodes according to the model training end instruction.

[0135] In one embodiment, the second local model parameter sending unit specifically includes a second local model parameter obtaining subunit, used for:

[0136] In response to receiving the updated global model parameter w from the service node during the current synchronization cycle k Based on the current number of training iterations t of the local model i w kAs its current local model Or, in response to completing its current t-th iteration... i Secondary local model training, recording t i =t i +1, obtain the current local model.

[0137] In response to the current time not reaching the synchronization cycle, based on its own local model training data, Update parameter α based on its own differences i Based on the mini-batch stochastic gradient descent algorithm, complete its current t-th step. i The second stage of local model training is used to obtain the parameters of the local model itself in this second stage. Where r represents the learning rate. It is backpropagation to calculate the gradient, where k is the number of times the global model has been updated, and the constant β∈(0,1);

[0138] In response to the current time reaching the current synchronization period, the last time its own time within the current synchronization period will be recorded. These are local model parameters for this synchronization cycle itself.

[0139] In one embodiment, the second global model parameter receiving unit is specifically used to receive the following parameters:

[0140] The service node determines the synchronization period based on data from all M worker nodes. The updated global model parameters obtained from aggregation during this synchronization cycle are as follows: And / or the number of global model updates recorded, k = k + 1.

[0141] In one embodiment, the model training completion instruction receiving unit specifically includes:

[0142] The model training end instruction receiving subunit is used to receive the model training end instruction generated by the service node and sent to each worker node until the service node determines k=K.

[0143] The second final training model acquisition subunit is connected to the model training end instruction receiving subunit. It is used to obtain the final training model by having the other working nodes and service nodes use the global model parameters after the Kth update as the final training model parameters according to the model training end instruction.

[0144] Example 4:

[0145] like Figure 8 As shown, this disclosure provides a distributed machine learning system, including a service node and multiple worker nodes, and includes:

[0146] The first model training module 1 is used to update the first model parameters between each working node and the service node after each local model training is completed by each working node during the first stage of model training, until the end condition of the first stage of model training is reached, to obtain the synchronization period of each working node and its own differential update parameters;

[0147] The second model training module 2, connected to the first model training module 1, is used to update the second model parameters between each working node and the service node after each working node completes a number of local model trainings according to its own differential update parameters within each synchronization period during the second stage of model training, until the end condition of the second stage of model training is reached, to obtain the final trained model.

[0148] In one embodiment, the first model training module 1 specifically includes:

[0149] The first local model parameter interaction unit is used for each working node to complete a local model training according to the updated global model parameters from the service node each time, and after obtaining the local model parameters and local training time parameters of its own this time, send the local model parameters and local training time parameters of its own this time to the service node, and the service node receives the local model parameters and local training time parameters of each working node this time;

[0150] The first global model parameter interaction unit, connected to the first local model parameter interaction unit, is used for the service node to update the global model parameters once according to the local model parameters of each working node each time, and then send the updated global model parameters of this time to each working node, and each working node receives the updated global model parameters of this time from the service node;

[0151] The synchronization control parameter interaction unit, connected to the first global model parameter interaction unit, is used until the global model parameters are updated N times, the service node calculates the synchronization period of each working node and its own differential update parameters according to the N local training time parameters of each working node, and then sends the synchronization period and its own differential update parameters to each working node, and each working node receives the synchronization period and its own differential update parameters from the service node.

[0152] In one embodiment, the first local model parameter interaction unit specifically includes:

[0153] The first local model parameter obtaining subunit is used for each working node to judge its own current local model training times t i <After N, based on its own local model training data and the updated global model parameters w of this time from the service node k , complete its own current t iThe local model is trained again to obtain the parameters of the local model in the first stage. And its own local training time parameters, including forward propagation time. and backpropagation time Where r represents the learning rate. The gradient is calculated via backpropagation, and t is recorded. i =t i +1.

[0154] In one embodiment, the first global model parameter interaction unit specifically includes:

[0155] The first global model parameter acquisition sub-unit is used by the service node to obtain the parameters from all M working nodes for this current iteration. The aggregated results show the updated global model parameters as follows: And record the number of global model updates, k = k + 1.

[0156] In one embodiment, the synchronous control parameter interaction unit specifically includes:

[0157] The synchronization control parameter acquisition sub-unit is used until the service node determines k=N, and the service node calculates the average computation time of each worker node for N training iterations. Obtain the common synchronization period of all working nodes. Where T is a preset synchronization period threshold, and the average backpropagation time of each working node for N training iterations is calculated. Obtain the difference update parameters for each working node. Where α is an integer constant greater than 1.

[0158] In one implementation, the model training second module 2 specifically includes:

[0159] The second local model parameter interaction unit is used by each working node to take the updated global model parameters from the service node in each synchronization period as the initial model for each synchronization period, update the parameters according to its own differences, complete the local model training for the initial model of each synchronization period several times, obtain its own local model parameters for the current synchronization period, and then send its own local model parameters for the current synchronization period to the service node. The service node receives the local model parameters from each working node for the current synchronization period.

[0160] The second global model parameter interaction unit is connected to the second local model parameter interaction unit. It is used to send the updated global model parameters for the current synchronization period to each working node after the service node updates the global model parameters once according to the local model parameters of each working node in each synchronization period. Each working node receives the updated global model parameters for the current synchronization period from the service node.

[0161] The model training termination unit is connected to the second global model parameter interaction unit. It is used until the service node determines that the conditions for the termination of the second stage of model training have been met. The service node generates and sends the model training termination instruction to each worker node. Each worker node and the service node obtain the final trained model according to the model training termination instruction.

[0162] In one embodiment, the second local model parameter interaction unit specifically includes:

[0163] The second local model parameter acquisition subunit is used by each working node in response to receiving the updated global model parameters w from the service node during the current synchronization cycle. k Based on the current number of training iterations t of the local model i w k As its current local model Or, in response to completing its current t-th iteration... i Secondary local model training, recording t i =t i +1, obtain the current local model.

[0164] Each working node responds if its current time has not reached the synchronization cycle, based on its own local model training data. Update parameter α based on its own differences i Based on the mini-batch stochastic gradient descent algorithm, complete its current t-th step. i The second stage of local model training is used to obtain the parameters of the local model itself in this second stage. Where r represents the learning rate. It is backpropagation to calculate the gradient, where k is the number of times the global model has been updated, and the constant β∈(0,1);

[0165] Each working node responds at the current time to reach its current synchronization period, and sends its last response within the current synchronization period. These are local model parameters for this synchronization cycle itself.

[0166] In one embodiment, the second global model parameter interaction unit specifically includes:

[0167] The second global model parameter acquisition sub-unit is used by the service node to obtain the current synchronization cycle based on the parameters from all M working nodes. The updated global model parameters obtained after aggregation in this synchronization cycle are: And record the number of global model updates, k = k + 1.

[0168] In one embodiment, the model training termination unit specifically includes:

[0169] The model training end instruction subunit is used until the service node determines k=K. The service node then generates and sends the model training end instruction to each worker node.

[0170] The final trained model obtains a sub-unit, which is connected to the model training end instruction sub-unit. This sub-unit is used by each worker node and service node to use the global model parameters after the Kth update as the final trained model parameters according to the model training end instruction, so as to obtain the final trained model.

[0171] Example 5:

[0172] Embodiment 5 of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the distributed machine learning parameter update method as described in Embodiment 1, or implements the apparatus as described in any one of Embodiments 2, 3, and 4.

[0173] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules, or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.

[0174] In addition, this disclosure may also provide a computer device including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the distributed machine learning parameter update method as described in Embodiment 1, and the computer device may be the device as described in any of Embodiments 2, 3, and 4.

[0175] The memory is connected to the processor. The memory can be flash memory, read-only memory or other types of memory. The processor can be a central processing unit or a microcontroller.

[0176] Embodiments 1-5 of this disclosure provide a distributed machine learning parameter update method, a service node, worker nodes, a distributed machine learning system, and a computer-readable storage medium. The machine learning model training process is divided into a first stage and a second stage. In the first stage, based on the actual performance of each worker node after completing its local model training, self-difference update parameters for each worker node's local model training in the second stage and a synchronization period for controlling the synchronization of model parameters between worker nodes in the second stage are obtained. By understanding the performance differences of each worker node in the first stage, a synchronization control mechanism for the second stage is formulated. In the second stage, the synchronization period is used to simultaneously synchronize the parameter updates of each worker node after several local model training iterations, reducing the waiting time of the service node, reducing the frequency of communication between worker nodes and the service node, improving resource utilization, and reducing communication overhead. The difference update parameters are used to control the number of local model training iterations of each worker node within the same synchronization period, thereby reducing the iteration difference between worker nodes and improving model convergence performance.

[0177] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.

Claims

1. A distributed machine learning parameter update method, characterized in that, The method is applied to service nodes, worker nodes, or a distributed machine learning system including service nodes and multiple worker nodes, and includes: In the first phase of model training, after each local model training iteration, each worker node collaborates with the service node to update the first model parameters until the end condition of the first phase of model training is met. This process yields the synchronization period and individual difference update parameters for each worker node. The individual difference update parameters for each worker node i are as follows: α is an integer constant greater than 1. is the average backpropagation time of each worker node i during the first stage of N training iterations, and M is the total number of worker nodes. In the second phase of model training, each worker node updates its parameters based on its own differences and completes several local model training iterations within each synchronization cycle. Then, each worker node collaborates with the service node to update the second model parameters until the end condition of the second phase of model training is met, resulting in the final trained model. Each of these local model training iterations involves updating the local model parameters. t i Let r be the number of times the local model parameters of worker node i are updated, and r be the learning rate. It calculates the gradient through backpropagation, where k is the number of times the global model has been updated, and the constant β∈(0,1).

2. The method according to claim 1, characterized in that, After each local model training iteration, each worker node collaborates with the service node to update the first model parameters until the end condition of the first phase of model training is met. This process yields the synchronization cycle and individual update parameters for each worker node, specifically including: Each worker node completes a local model training based on the updated global model parameters from the service node. After obtaining its own local model parameters and local training time parameters, it sends its own local model parameters and local training time parameters to the service node. The service node receives the local model parameters and local training time parameters from each worker node. After the service node updates the global model parameters based on the local model parameters of each worker node each time, it sends the updated global model parameters to each worker node. Each worker node then receives the updated global model parameters from the service node. Until the global model parameters are updated N times, the service node calculates the synchronization period and its own differential update parameters for each worker node based on the N local training time parameters of each worker node, and then sends the synchronization period and its own differential update parameters to each worker node. Each worker node receives the synchronization period and its own differential update parameters from the service node.

3. The method according to claim 2, characterized in that, Each worker node performs local model training based on the updated global model parameters from the service node, obtaining its own local model parameters and local training time parameters, specifically including: Each working node independently determines its current local model training count \(t\). i After \(N\), based on its own local model training data and the updated global model parameters \(w\) from the service node in this round k , it completes its current \(t\)-th local model training based on the mini-batch stochastic gradient descent algorithm i to obtain the local model parameters of its own in this stage as and its own local training time parameters for this round, including the forward propagation time and the backward propagation time where \(r\) represents the learning rate, is to calculate the gradient during backpropagation, and record \(t\) i \(=t\) i \(+ 1\).

4. The method according to claim 3, characterized in that, The service node updates the global model parameters once based on the local model parameters of each worker node, specifically including: The service node is based on the current data from all M worker nodes. The aggregated results show the updated global model parameters as follows: And record the number of global model updates, k = k + 1.

5. The method according to claim 4, characterized in that, Until the global model parameters are updated N times, the service node calculates the synchronization period and its respective differential update parameters for each worker node based on the N local training time parameters of each worker node, specifically including: Until the service node determines k=N, the service node calculates the average computation time of each worker node for N training iterations. Obtain the common synchronization period of all working nodes. Where T is a preset synchronization period threshold, and the average backpropagation time of each working node for N training iterations is calculated. Obtain the difference update parameter α for each working node. i .

6. The method according to any one of claims 1-5, characterized in that, Each worker node updates its parameters based on its own differences and completes several local model training iterations within each synchronization cycle. Then, each worker node collaborates with the service node to update the second model parameters until the conditions for ending the second phase of model training are met, resulting in the final trained model. This process includes: Each worker node uses the updated global model parameters from the service node for each synchronization period as the initial model for each synchronization period. Based on its own differences, it updates the parameters and performs several local model trainings on the initial model for each synchronization period. After obtaining its own local model parameters for this synchronization period, it sends its own local model parameters for this synchronization period to the service node. The service node receives the local model parameters from each worker node for this synchronization period. After the service node updates the global model parameters once based on the local model parameters of each worker node in each synchronization period, it sends the updated global model parameters for the current synchronization period to each worker node. Each worker node receives the updated global model parameters for the current synchronization period from the service node. Once the service node determines that the conditions for ending the second stage of model training have been met, the service node generates and sends a model training termination instruction to each worker node. Each worker node and the service node then obtain the final trained model based on the model training termination instruction.

7. The method according to claim 6, characterized in that, Each worker node uses the updated global model parameters from the service node for each synchronization cycle as its initial model for that cycle. Based on its own differences, it updates the parameters and performs several local model training iterations on the initial model for each synchronization cycle to obtain its own local model parameters for that synchronization cycle. Specifically, these parameters include: Each worker node responds to the updated global model parameter w received from the server node during the current synchronization cycle. k Based on the current number of training iterations t of the local model i w k As its current local model Or, in response to completing its current t-th iteration... i Secondary local model training, recording t i =t i +1, obtain the current local model. Each working node responds if its current time has not reached the synchronization cycle, based on its own local model training data. Update parameter α based on its own differences i Based on the mini-batch stochastic gradient descent algorithm, complete its current t-th step. i The second stage of local model training is used to obtain the parameters of the local model itself in this second stage. Each working node responds at the current time to reach its current synchronization period, and sends its last response within the current synchronization period. These are local model parameters for this synchronization cycle itself.

8. The method according to claim 7, characterized in that, The service node updates the global model parameters once based on the local model parameters of each worker node in each synchronization cycle, specifically including: The service node synchronizes with all M worker nodes for the current synchronization period. The updated global model parameters obtained after aggregation in this synchronization cycle are: And record the number of global model updates, k = k + 1.

9. The method according to claim 8, characterized in that, Until the service node determines that the conditions for ending the second stage of model training have been met, the service node generates and sends a model training termination command to each worker node. Each worker node and the service node then obtain the final trained model based on the model training termination command, specifically including: Until the service node determines that k = K, the service node generates and sends a model training end instruction to each worker node. Each worker node and the service node then use the global model parameters updated after the Kth time as the final training model parameters according to the model training end instruction to obtain the final training model.

10. A service node, characterized in that, include: The first module of global training is used in the first stage of model training. After each local model training iteration on each worker node, it collaborates with each worker node to update the first model parameters until the end condition of the first stage of model training is met. It obtains the synchronization period and the individual difference update parameters of each worker node, where the individual difference update parameters of worker node i are: α is an integer constant greater than 1. is the average backpropagation time of each worker node i during the first stage of N training iterations, and M is the total number of worker nodes. The second global training module, connected to the first global training module, is used in the second phase of model training. After each worker node completes several local model training iterations based on its own differences within each synchronization cycle, the second module collaborates with each worker node to update the second model parameters until the end condition of the second phase of model training is met, resulting in the final trained model. Each of these local model training iterations involves updating the local model parameters. t i Let r be the number of times the local model parameters of worker node i are updated, and r be the learning rate. It calculates the gradient through backpropagation, where k is the number of times the global model has been updated, and the constant β∈(0,1).

11. A working node, characterized in that, include: The first module of local training is used to update the first model parameters in collaboration with other worker nodes and service nodes after each local model training iteration, until the end condition of the first phase of model training is met. It obtains a shared synchronization period with the other worker nodes and its own differential update parameters. The differential update parameters for each worker node i are as follows: α is an integer constant greater than 1. is the average backpropagation time of each worker node i during the first stage of N training iterations, and M is the total number of worker nodes. The second local training module, connected to the first local training module, is used in the second phase of model training. After completing several local model training iterations within each synchronization cycle by updating parameters based on its own differences, it collaborates with other worker and service nodes to update the second model parameters until the end condition of the second phase of model training is met, resulting in the final trained model. Each of these local model training iterations involves updating the local model parameters. t i Let r be the number of times the local model parameters of worker node i are updated, and r be the learning rate. It calculates the gradient through backpropagation, where k is the number of times the global model has been updated, and the constant β∈(0,1).

12. A distributed machine learning system, characterized in that, It includes service nodes and multiple worker nodes, and includes: The first module of model training is used in the first stage of model training. After each local model training cycle is completed, each worker node updates the first model parameters with the server node until the end condition of the first stage of model training is met. This module obtains the synchronization period and the individual difference update parameters of each worker node. The individual difference update parameters of each worker node i are as follows: α is an integer constant greater than 1. is the average backpropagation time of each worker node i during the first stage of N training iterations, and M is the total number of worker nodes. The second model training module, connected to the first module, is used in the second phase of model training. After each worker node completes several local model training iterations based on its own differences within each synchronization cycle, the worker nodes and the server node update the second model parameters until the end condition of the second phase of model training is met, resulting in the final trained model. Each of these local model training iterations involves updating the local model parameters. t i Let r be the number of times the local model parameters of worker node i are updated, and r be the learning rate. It calculates the gradient through backpropagation, where k is the number of times the global model has been updated, and the constant β∈(0,1).

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the distributed machine learning parameter update method as described in any one of claims 1-9.

Images