Distributed deep learning method based on communication frequency optimization and error compensation mechanism

Abstract

The present invention discloses a distributed deep learning method based on communication frequency optimization and error compensation mechanism, comprising: initializing global model parameters and local model parameters, and synchronizing the local model parameters between computing nodes; the computing nodes obtain a training data set to train the local model; based on the current batch gradient obtained from the training, the local model parameters are updated using a communication frequency optimization algorithm and an error compensation mechanism; based on the updated local model parameters, the current number of iterations of the computing node and the preset number of iterations, the global model is updated. In this way, the present invention can reduce the amount of data transmission between nodes by lowering the synchronous communication frequency to solve the communication bottleneck problem existing in the distributed deep learning optimization algorithm; introduce an error compensation mechanism of synchronous model averaging to improve the generalization accuracy of the model to solve the generalization error problem caused by the distributed deep learning communication optimization algorithm.

Description

Technical Field

[0001] The present invention relates to, but is not limited to, the field of distributed computers for deep learning, and in particular to a distributed deep learning method based on communication frequency optimization and error compensation mechanism. Background Art

[0002] With the development of internet technology, distributed deep learning has been widely applied in fields such as computer vision, speech recognition, and natural language processing. However, as the number of nodes increases, the communication overhead between distributed computing nodes offsets the performance gains brought by the addition of new nodes, resulting in a non-linear increase in training speed and affecting the efficiency of distributed computing.

[0003] Communication frequency optimization techniques are usually used, such as the iterative algorithm Synchronous Stochastic Gradient Descent (SSGD), to reduce data transmission volume, improve computing efficiency, and optimize the training of distributed machine learning models. Among them, local gradient descent is a common strategy to reduce communication frequency. Local gradient descent performs multiple local iterations in each computing node before synchronization, which can reduce the number of communications to The proposed method reduces the communication overhead of the quadratic form by a factor of , where T is the number of iterations. However, its performance lower bound is worse than that of traditional SSGD, and it often suffers from large convergence errors, resulting in reduced model test accuracy. Therefore, addressing the communication bottleneck in distributed deep learning and improving the generalization accuracy of the model while optimizing the synchronous communication frequency has become an urgent problem to be solved. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a distributed deep learning method, device, terminal and storage medium based on communication frequency optimization and error compensation.

[0005] The technical solution of the present invention is achieved as follows:

[0006] A distributed deep learning method based on communication frequency optimization and error compensation, the method comprising:

[0007] S101: Build a global model of distributed deep learning and multiple computing nodes, and deploy copies of the global model as local models on the computing nodes;

[0008] S102: Initializing global model parameters, and synchronizing copies of the global model parameters as local model parameters to each of the computing nodes;

[0009] S103: The computing node obtains a training data set as a sample vector, trains the corresponding local model, and obtains the current batch gradient of each computing node;

[0010] S104: updating local model parameters using a communication frequency optimization algorithm and an error compensation mechanism according to the global model parameters and the current batch gradient;

[0011] S105: updating the global model according to the updated local model parameters, the current iteration number of the computing node and the preset iteration number;

[0012] S106: Perform performance evaluation on the updated global model and save the trained global model.

[0013] In some embodiments, the method further includes: using the color image dataset CIFAR-10 as an original dataset, and dividing the original dataset into a validation dataset and the training dataset.

[0014] In some embodiments, the method further comprises:

[0015] Allocating memory for the global model parameters and initializing the global model parameters using a preset initialization method;

[0016] The copies of the global model parameters are copied to each of the computing nodes using a Broadcast operation via the MPI communication protocol as local model parameters of the computing node.

[0017] In some embodiments, the method further comprises:

[0018] Splitting the training data set into sub-data sets of the same size and the same number as the computing nodes;

[0019] The computing node obtains a corresponding sub-dataset from the training data set and preprocesses the sub-dataset;

[0020] Input each of the preprocessed sub-datasets as a sample vector into the corresponding local model for forward propagation to obtain the prediction vector of the current batch;

[0021] Calculate a first difference between the prediction vector and the sample vector using a loss function as a local model error of the computing node;

[0022] Back propagation is performed using the local model error to obtain the gradient of the current batch of samples relative to the local model parameters of the computing node, that is, the current batch gradient.

[0023] In some embodiments, the method further comprises:

[0024] Determining a preset update frequency of local model parameters using the communication frequency optimization algorithm;

[0025] Calculating a second difference between the global model parameter and the local model parameter;

[0026] multiplying the second difference by a preset compensation coefficient to obtain an error compensation term between the global model and the local model;

[0027] Determining an error compensation gradient between the global model parameters and the local model parameters according to the error compensation term, the current batch gradient, and the local model parameters;

[0028] According to the preset update frequency, the error compensation gradient is updated to the local model parameter.

[0029] In some embodiments, the method further comprises:

[0030] Obtaining the current iteration number of the computing node;

[0031] Determine whether synchronous communication is required in the current round based on the current number of iterations and the preset number of iterations; if so, synchronous communication is required, and use a global communication algorithm to aggregate the updated local model parameters of each computing node to update the global model parameters;

[0032] The global model is updated using the updated global model parameters, and a copy of the updated global model parameters is synchronized to each of the computing nodes as a local model parameter.

[0033] In some embodiments, the method further comprises:

[0034] If not, there is no need for synchronous communication, the global model parameters inherit the global model parameters of the previous batch iteration, and the local model is continuously trained using the updated local model parameters.

[0035] An embodiment of the present invention further provides a distributed deep learning device based on communication frequency optimization and error compensation, comprising:

[0036] An initialization module, configured to construct a global model of distributed deep learning and multiple computing nodes, and deploy copies of the global model as local models on the computing nodes;

[0037] The initialization module is also used to initialize global model parameters and synchronize copies of the global model parameters as local model parameters to each computing node;

[0038] A training processing module is used for the computing node to obtain a training data set as a sample vector, train the corresponding local model, and obtain the current batch gradient of each computing node;

[0039] The training processing module is also used to update local model parameters based on the global model parameters and the current batch gradient using a communication frequency optimization algorithm and an error compensation mechanism;

[0040] The training processing module is further configured to update the global model according to the updated local model parameters, the current iteration number of the computing node, and the preset iteration number;

[0041] The evaluation module is used to perform performance evaluation on the updated global model and save the trained global model.

[0042] An embodiment of the present invention also provides a terminal, comprising a processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, it implements the distributed deep learning method described in any embodiment of the present invention.

[0043] An embodiment of the present invention further provides a storage medium, wherein the storage medium contains computer-executable instructions, characterized in that the computer-executable instructions are executed by a processor to implement the distributed deep learning method described in any embodiment of the present invention.

[0044] The embodiment of the present invention provides a distributed deep learning method based on communication frequency optimization and error compensation, which constructs a global model and multiple computing nodes for distributed deep learning, and deploys a copy of the global model as a local model on the computing node; initializes the global model parameters, and synchronizes the copies of the global model parameters as local model parameters to each computing node; the computing node obtains a training data set as a sample vector, trains the corresponding local model, and obtains the current batch gradient of each computing node; based on the global model parameters and the current batch gradient, the local model parameters are updated using a communication frequency optimization algorithm and an error compensation mechanism; based on the updated local model parameters, the current number of iterations of the computing node and the preset number of iterations, the global model is updated; the performance of the updated global model is evaluated, and the trained global model is saved. In this way, the present invention can reduce the amount of data transmission between nodes by reducing the synchronous communication frequency to solve the communication bottleneck problem existing in the distributed deep learning optimization algorithm; and introduces a synchronous model average error compensation mechanism to improve the generalization accuracy of the model to solve the generalization error problem caused by the distributed deep learning communication optimization algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0046] Figure 1This is a flow chart of a distributed deep learning method based on communication frequency optimization and error compensation mechanism provided by an embodiment of the present invention;

[0047] Figure 2 Schematic diagram of a node communication architecture of a distributed deep learning method based on communication frequency optimization and error compensation mechanism provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of broadcast propagation between nodes of a distributed deep learning method based on communication frequency optimization and error compensation mechanism provided by an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of Gather propagation between distributed nodes of a distributed deep learning method based on communication frequency optimization and error compensation mechanism provided by an embodiment of the present invention;

[0050] Figure 5 Schematic diagram of a distributed deep learning device based on communication frequency optimization and error compensation mechanism provided by an embodiment of the present invention;

[0051] Figure 6 Comparison of the Cifar10 test set accuracy results of a distributed deep learning method based on communication frequency optimization and error compensation mechanism provided by an embodiment of the present invention;

[0052] Figure 7 Comparative experimental results of Cifar10 training set loss for a distributed deep learning method based on communication frequency optimization and error compensation mechanism provided by an embodiment of the present invention;

[0053] Figure 8 This is a schematic diagram of the terminal hardware structure of a distributed deep learning method based on communication frequency optimization and error compensation mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0055] In the subsequent description, suffixes such as "module," "component," or "unit" used to represent elements are only used to facilitate the description of the present invention and have no specific meaning. Therefore, "module," "component," or "unit" can be used interchangeably.

[0056] like Figure 1As shown, the embodiment of the present invention provides a distributed deep learning method based on communication frequency optimization and error compensation mechanism, which specifically includes the following steps:

[0057] S101: Build a global model of distributed deep learning and multiple computing nodes, and deploy copies of the global model as local models on the computing nodes;

[0058] S102: Initializing global model parameters, and synchronizing copies of the global model parameters as local model parameters to each of the computing nodes;

[0059] S103: The computing node obtains a training data set as a sample vector, trains the corresponding local model, and obtains the current batch gradient of each computing node;

[0060] S104: updating local model parameters using a communication frequency optimization algorithm and an error compensation mechanism according to the global model parameters and the current batch gradient;

[0061] S105: Determine whether to perform synchronous communication based on the current iteration number of the computing node and the preset iteration number; if so, update the global model based on the updated local model parameters;

[0062] S106: Perform performance evaluation on the updated global model and save the trained global model for subsequent deep learning.

[0063] The method described in the embodiment of the present invention is executed by a terminal. The terminal can be any type of terminal; for example, the terminal can be, but is not limited to, at least one of the following: a server, a computer, a tablet computer, or other electronic devices.

[0064] Here, the computing nodes can be heterogeneous physical devices distributed locally, at the edge, and in the cloud in a distributed deep learning network; each terminal device performs its own local computation, but their outputs are aggregated between local exit points.

[0065] In some embodiments, the method further includes: using the color image dataset CIFAR-10 as an original dataset, and dividing the original dataset into a validation dataset and the training dataset.

[0066] For example, CIFAR-10, or CIFAR-100, is a labeled subset of the 80 million small image dataset collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The original CIFAR-10 dataset can be split into five training datasets and one test dataset, each with 10,000 images. The test dataset contains 1,000 images randomly selected from each class. The training dataset contains the remaining images in a random order, but some training datasets may contain more images from one class than another.

[0067] Here, the color image dataset can be replaced by other datasets that can be used for deep learning model training.

[0068] In some embodiments, the step S101 includes: constructing a distributed deep learning model including an input layer, a hidden layer, and an output layer and multiple computing nodes; the distributed deep learning model is the global model; and one computing node corresponds to one local model.

[0069] It should be noted that the hidden layer of the deep learning model can include multiple levels. In the embodiments of the present invention, a three-layer neural network, namely an input layer, a hidden layer, and an output layer, is used as an example for specific description. Moreover, each network layer can include multiple neurons. The number of neurons included in the input layer of the three-layer neural network is defined as a, the number of neurons included in the hidden layer is defined as b, and the number of neurons included in the output layer is defined as c. According to common knowledge of deep learning models, the number of prediction vectors is the same as the number of output neurons, and the sample vectors are distributed to each neuron in the input layer according to the number of input neurons.

[0070] It should be noted that each of the neurons consists of two parts. The first part is the sum e of the input value and the weight product, and the second part is the output y of an activation function f(e), that is, y=f(e); the activation function is a nonlinear function, such as a sigmoid function.

[0071] In some embodiments, step S102 includes:

[0072] Allocating memory for the global model parameters and initializing the global model parameters using a preset initialization method;

[0073] The copies of the global model parameters are copied to each of the computing nodes using a Broadcast operation via the MPI communication protocol as local model parameters of the computing node.

[0074] Exemplarily, the global model parameters are A copy of the global model parameters The local model parameters are synchronized to computing node 1, computing node 2, ..., computing node n; and t is the number of iterations of the current training of the local model.

[0075] Here, the global model parameters Including all weights between each network layer in the global model, it can be understood that there is a weight between all neurons between the input layer and the hidden layer, there is a weight between all neurons between the hidden layer and the output layer, and the weights between all neurons of the global waveform are all global model parameters.

[0076] Here, the parameter initialization method may be Xavier initialization or He initialization.

[0077] For example, during forward propagation with Xavier initialization and He initialization, the variance of the state value remains unchanged; during backward propagation, the variance of the gradient with respect to the activation value remains unchanged.

[0078] Here, the global model parameters are parameters created in the global model.

[0079] It should be understood that global model parameters do not have a specific category and can be assigned to multiple families. Global model parameters contain simple values ​​that can be used to establish parametric relationships in the global model. Because global model parameters can be assigned to multiple families and types, they can be used to control multiple elements at once.

[0080] In some embodiments, the step S102 further includes: constructing a communication architecture between the computing nodes based on the MPI communication protocol; and based on the communication architecture, using a Broadcast operation to copy a copy of the global model parameters to each of the computing nodes.

[0081] Here, the communication architecture between the computing nodes is a decentralized distributed communication architecture.

[0082] For example, Figure 2In the decentralized architecture shown, each computing node plays the same role in model training, and each point-to-point computing node has a complete copy of the global model, namely the local model. Each computing node has a left computing node and a right computing node, and each computing node can only receive from the left computing node and send to the right computing node. Specifically, computing node 1 receives data from computing node 2 and sends data to computing node 3; computing node 3 receives data from computing node 1 and sends data to computing node 5; computing node 5 receives data from computing node 3 and sends data to computing node 4; computing node 4 receives data from computing node 5 and sends data to computing node 2; computing node 2 receives data from computing node 4 and sends data to computing node 1.

[0083] For example, Figure 3 The broadcast propagation diagram between distributed computing nodes shown in Figure 2 The five computing nodes shown deploy a global model and four local models, including local model 1, local model 2, local model 3, and local model 4. The global model parameters, i.e., the initialization parameters, are synchronized to local model 1, local model 2, local model 3, and local model 4 at the same time, and the local model parameters of the corresponding four computing nodes are updated to the initialization parameters.

[0084] In some embodiments, step S103 includes:

[0085] Splitting the training data set into sub-data sets of the same size and the same number as the computing nodes;

[0086] The computing node obtains a corresponding sub-dataset from the training data set and preprocesses the sub-dataset;

[0087] Input each of the preprocessed sub-datasets as a sample vector of the current batch into the corresponding local model for forward propagation to obtain the prediction vector of the current batch;

[0088] Calculate a first difference between the prediction vector and the sample vector using a loss function as a local model error of the computing node;

[0089] Back propagation is performed using the local model error to obtain the gradient of the current batch of samples relative to the local model parameters, that is, the current batch gradient.

[0090] It's important to note that each node maintains a complete local copy of the model. Each node reads and processes only a unique subset of the dataset and updates the local model during training. These local model parameters are then synchronized with other nodes to compute the global model parameters. These global model parameters are then distributed to each node via the network for the next iteration.

[0091] It can be understood that the current batch gradient of the local model is determined by the gradients corresponding to all neurons included in each network layer, that is, all neuron gradients are determined first, and then the current batch gradient of the local model is determined based on the neuron gradients.

[0092] Here, the neuron gradient calculation formula is expressed as:

[0093]

[0094] in, is the local model parameter of computing node i, x ij is the sample vector of neuron j of computing node i, δ j is the error corresponding to neuron j, y j is the output of neuron j.

[0095] Here, the gradient of the local model parameters is expressed as:

[0096]

[0097] in, is the current batch gradient of computing node i, x ij is the sample vector of neuron j of computing node i, is the local model parameter of computing node i, is the objective function gradient of the computation node i.

[0098] In some embodiments, dividing the training data set into sub-data sets of equal size and equal in number to the number of computing nodes; obtaining, by the computing nodes, corresponding sub-data sets from the training data set and preprocessing the sub-data sets includes:

[0099] Obtaining the number of computing nodes, dividing the training data set into sub-data sets of equal size according to the number of computing nodes, and labeling the sub-data sets;

[0100] Allocating the sub-datasets to the computing nodes according to the labels;

[0101] Each computing node reads the corresponding sub-dataset and stores it in the memory, performs data enhancement and normalization preprocessing on the sub-dataset, and converts the sub-dataset into the input format required by the local model.

[0102] For example, the number of computing nodes is n, the training data set is divided into n sub-data sets b of the same size, and the sub-data sets are labeled to obtain sub-data sets b (i) , and i=1,…,m; sub-dataset b (i) Assigned to computing nodes 1, ..., sub-dataset b (n) Assigned to compute node n; compute node 1 reads subdataset b (1) Save in memory, perform data enhancement and trannsform normalization on the sub-dataset, and convert the sub-dataset into the sample vector x required by the local model i .

[0103] In one embodiment, the training dataset is partitioned using PyTorch's dataset and dataloader classes.

[0104] For example, if there are 8 computing nodes, the subscript of the dataset is evenly divided into 8 sequences, namely the sub-datasets, and each computing node is assigned a sub-dataset. The computing node uses the dataloader to read the current batch data, namely the sub-dataset, and saves it to the memory to ensure that the 8 GPUs do not read duplicate data.

[0105] Here, the forward propagation is to use the output of the previous layer as the input of the next layer, and calculate the output of the next layer, and continue to calculate until the output layer.

[0106] It can be understood that, according to the sample vector x i And the weights between each network layer, that is, the local model parameters, are weighted and summed up nonlinearly to obtain the prediction vector y1.

[0107] Here, the first difference may be the mean square error between the sample vector and the prediction vector, and the loss function is an error function, as follows:

[0108]

[0109] Among them, x i is the sample vector of the local model input to the computing node i, and j = 0, 1, ..., n; n is a given value, which is the number of neurons in the output layer of the local model; is the weight of neuron j, y jis the prediction vector output by neuron j, and δ is the local model error.

[0110] Here, the back propagation is performed in the opposite order of the forward propagation, and the corresponding errors of the neurons in each network layer are calculated and updated in sequence according to the local model error; then, in the same order as the forward propagation, the gradients corresponding to the neurons in each network layer are calculated using the activation function according to the corresponding errors of the neurons in each network layer, and the calculation is continued until the output layer.

[0111] It can be understood that first, in the order opposite to the forward propagation, according to the local model error δ and the weights between the network layers, that is, the local model parameters, the error corresponding to the neurons of each network layer is weighted; then, in the same order as the forward propagation, according to the error corresponding to the neurons of each network layer obtained by back propagation, the weight and the activation function, the derivative of each neuron (weight) is calculated, that is, the gradient corresponding to the neuron.

[0112] In some embodiments, step S104 includes:

[0113] Determining a preset update frequency of local model parameters using the communication frequency optimization algorithm;

[0114] Calculating a second difference between the global model parameter and the local model parameter;

[0115] multiplying the second difference by a preset compensation coefficient to obtain an error compensation term between the global model and the local model;

[0116] Determining an error compensation gradient between the global model parameters and the local model parameters according to the error compensation term, the current batch gradient, and the local model parameters;

[0117] According to the preset update frequency, the error compensation gradient is updated to the local model parameter.

[0118] Here, the communication frequency optimization algorithm selects a local gradient descent algorithm, and sets a hyperparameter according to the local gradient descent algorithm, namely, the preset update frequency τ.

[0119] Here, the error compensation term is calculated as follows:

[0120]

[0121] in, is the local model parameter of computing node i, are the global model parameters, is the second difference, α is the preset compensation coefficient, is the error compensation term.

[0122] Here, the error compensation gradient The calculation formula is as follows:

[0123]

[0124] Where η is the step size, which is a given value; are the local model parameters of iteration round t, is the current batch gradient at iteration t, is the error compensation term of iteration round t.

[0125] In this way, the embodiment of the present invention can adjust the local model parameters based on the determined local model error, introduce an error compensation mechanism while synchronously training the local model of each computing node, and synchronously update the local model parameters of each computing node.

[0126] In some embodiments, step S105 includes:

[0127] Obtaining the current iteration number of the computing node;

[0128] Determine whether synchronous communication is required in the current round based on the current number of iterations and the preset number of iterations; if so, synchronous communication is required, and use a global communication algorithm to aggregate the updated local model parameters of each computing node to update the global model parameters;

[0129] The global model is updated using the updated global model parameters, and a copy of the updated global model parameters is synchronized to each of the computing nodes as a local model parameter.

[0130] Here, the preset number of iterations T of the synchronous communication may be calculated by the communication frequency optimization algorithm.

[0131] Preferably, the preset number of iterations can be selected as 105, that is, 105 rounds of training are performed on the local model to achieve an accuracy level of 80%.

[0132] In one embodiment, the determination of whether synchronous communication is required in the current round based on the current number of iterations and the preset number of iterations includes: performing a remainder operation on the current number of iterations and the preset number of iterations, and determining whether the remainder obtained is equal to zero; if so, it is considered that the final prediction accuracy of the local model can reach the preset accuracy threshold, and the local model parameters of the current batch can be synchronized to the global model, and synchronous communication is required.

[0133] Exemplarily, a remainder operation is performed on the current iteration number t and the preset iteration number T to determine whether the obtained remainder is equal to zero; if tmodT=0, synchronous communication is required.

[0134] In this way, the embodiment of the present invention can regularly upload the local model parameters to the global model, reduce the frequency of synchronous communication, and thereby reduce the amount of data transmission between computing nodes, thereby optimizing the training process of distributed deep learning.

[0135] In one embodiment, the utilizing a global communication algorithm to aggregate the updated local model parameters of each computing node to update the global model parameters includes:

[0136] Using the MPI communication protocol, using a Gather operation to collect local model parameters on each of the computing nodes;

[0137] According to the amount of data of each computing node, the global communication algorithm is used to perform weighted averaging on the collected local model parameters to update the global model parameters.

[0138] Here, the amount of data of each computing node is the same, and the weighted average of the collected local model parameters using the global communication algorithm is calculated using the global communication algorithm to obtain the average value of the collected local model parameters.

[0139] For example, Figure 4 As shown in the schematic diagram of Gather propagation between distributed nodes, the local model parameters after the local model 1 is updated are training parameters A, the local model parameters after the local model 2 is updated are training parameters B, the local model parameters after the local model 3 is updated are training parameters C, and the local model parameters after the local model 4 is updated are training parameters D. The various training parameters are collected through the Gather operation and weighted averaged using the global communication algorithm to update the global model.

[0140] Here, the global model parameter update formula can be:

[0141]

[0142] in, are the updated local model parameters, is the updated global model parameter, and n is the number of computing nodes.

[0143] In one embodiment, updating the global model using the updated global model parameters and synchronizing a copy of the updated global model parameters as local model parameters to each computing node includes:

[0144] Using the updated global model parameters Update the global model and use the Broadcast operation to transmit the updated global model parameters to the A copy of The local model parameters are synchronized to each computing node.

[0145] It is understood that, in an embodiment of the present invention, after the local model parameters of each computing node are updated, the local model parameters of all the computing nodes can be averaged to update the global model parameters and the global model, and the global model can be further updated as the local model of each computing node. In this way, by incorporating the error compensation method of local gradient descent into synchronous model averaging, the embodiment of the present invention can effectively reduce the training divergence problem of multiple node local models during distributed training and improve the generalization accuracy of the model.

[0146] In some embodiments, step S7 includes:

[0147] If so, there is no need for synchronous communication, and the local model training is continued using the updated local model parameters.

[0148] In one embodiment, continuing to train the local model using the updated local model parameters includes: updating the local model using the updated local model parameters, and continuing to train the local model using the sub-dataset corresponding to the local model.

[0149] In some embodiments, step S106 includes: using the verification data set to evaluate the updated global model, and calculating the accuracy of the global model on the verification data set; if the accuracy is not less than a preset accuracy threshold, the final global model is considered to have converged, and the global model is saved for subsequent deep learning.

[0150] In some other embodiments, the step S106 further includes: if the accuracy is less than the preset accuracy threshold, it is considered that the updated global model has not converged, and the training of the local model is continued according to the updated global model parameters.

[0151] In some embodiments, the step S106 further includes: selecting a suitable file format to save the updated global model for subsequent use.

[0152] Here, the suitable file format may be, but is not limited to, HDF5, SavedModel, or ONNX.

[0153] In an embodiment of the present invention, a global model and multiple computing nodes for distributed deep learning are constructed, and copies of the global model are deployed on the computing nodes as local models; global model parameters are initialized, and copies of the global model parameters are synchronized to each computing node as local model parameters; the computing nodes obtain training data sets as sample vectors, train corresponding local models, and obtain current batch gradients of each computing node; local model parameters are updated according to the global model parameters and the current batch gradients using a communication frequency optimization algorithm and an error compensation mechanism; the global model is updated according to the updated local model parameters, the current number of iterations of the computing node, and the preset number of iterations; performance evaluation is performed on the updated global model, and the trained global model is saved. In this way, the present invention can update the local model parameters of each computing node based on synchronous training, and regularly upload the local model to the global model, reduce the frequency of synchronous communication, and further reduce the amount of data transmission between computing nodes, thereby optimizing the distributed deep learning training process; by averaging all the updated local model parameters to update the global model parameters, an error compensation mechanism of synchronous model averaging is implemented, which effectively reduces the training divergence problem of local models of multiple computing nodes in the distributed training process, and improves the generalization accuracy of the distributed deep learning model.

[0154] like Figure 5 As shown, the embodiment of the present invention also provides a distributed deep learning device based on communication frequency optimization and error compensation, the device comprising: an initialization module 201, a training processing module 202, and an evaluation module 203; wherein,

[0155] The initialization module 201 is used to build a global model of distributed deep learning and multiple computing nodes, and deploy copies of the global model as local models on the computing nodes;

[0156] The initialization module 201 is also used to initialize global model parameters and synchronize copies of the global model parameters as local model parameters to each computing node;

[0157] The training processing module 202 is used for the computing node to obtain a training data set as a sample vector, train the corresponding local model, and obtain the current batch gradient of each computing node;

[0158] The training processing module 202 is also used to update local model parameters based on the global model parameters and the current batch gradient using a communication frequency optimization algorithm and an error compensation mechanism;

[0159] The training processing module 202 is further configured to update the global model according to the updated local model parameters, the current iteration number of the computing node, and the preset iteration number;

[0160] The evaluation module 203 is used to perform performance evaluation on the updated global model and save the trained global model.

[0161] In some embodiments, the method further comprises:

[0162] The initialization model is used to use the color image dataset CIFAR-10 as an original dataset, and divide the original dataset into a verification dataset and the training dataset.

[0163] In some embodiments, the method further comprises:

[0164] The training processing module is used to allocate memory for the global model parameters and initialize the global model parameters using a preset initialization method;

[0165] The training processing module is used to copy a copy of the global model parameters to each computing node using a Broadcast operation through the MPI communication protocol as the local model parameters of the computing node.

[0166] In some embodiments, the method further comprises:

[0167] The training processing module is used to divide the training data set into sub-data sets of the same size and the same number as the computing nodes;

[0168] The training processing module is used for the computing node to obtain the corresponding sub-dataset from the training data set and pre-process the sub-dataset;

[0169] The training processing module is used to input each of the preprocessed sub-datasets as a sample vector into the corresponding local model for forward propagation to obtain the prediction vector of the current batch;

[0170] The training processing module is configured to calculate a first difference between the prediction vector and the sample vector using a loss function as a local model error of the computing node;

[0171] The training processing module is used to use the local model error to perform back propagation to obtain the gradient of the current batch of samples relative to the local model parameters of the computing node, that is, the current batch gradient.

[0172] In some embodiments, the method further comprises:

[0173] The training processing module is used to determine the preset update frequency of local model parameters using the communication frequency optimization algorithm;

[0174] The training processing module is used to calculate the second difference between the global model parameter and the local model parameter;

[0175] The training processing module is configured to multiply the second difference by a preset compensation coefficient to obtain an error compensation term between the global model and the local model;

[0176] The training processing module is configured to determine an error compensation gradient between the global model parameters and the local model parameters based on the error compensation term, the current batch gradient, and the local model parameters;

[0177] The training processing module is used to update the error compensation gradient to the local model parameter according to the preset update frequency.

[0178] In some embodiments, the method further comprises:

[0179] The training processing module is used to obtain the current iteration number of the computing node;

[0180] The training processing module is configured to determine whether synchronous communication is required for the current round based on the current number of iterations and the preset number of iterations; if so, synchronous communication is required, and the global model parameters inherit the global model parameters of the previous batch iteration, and use a global communication algorithm to aggregate the updated local model parameters of each computing node to update the global model parameters;

[0181] The training processing module is used to update the global model using the updated global model parameters, and synchronize a copy of the updated global model parameters as local model parameters to each of the computing nodes.

[0182] In some embodiments, the method further comprises:

[0183] The training processing module is used to, if not, not need to perform synchronous communication, and continue to train the local model using the updated local model parameters.

[0184] As an embodiment, the present invention compares the proposed method with other methods. Specifically:

[0185] The LSGDMA method of the present invention is compared with the Local SGD and SSGD algorithms. In the experiment, both Local SGD and SSGD algorithms are configured with the same number of local iterations. The communication parameter comparison table 1 of Cifar10 under different distributed communication algorithms is obtained as follows:

[0186] Table 1 Comparison of Cifar10 communication parameters under different distributed communication algorithms

[0187]

[0188] As well as Figure 6 The comparative experimental results of different distributed communication algorithms on the Cifar10 test set are shown in Figure 2. Figure 7 The accuracy comparison experimental results of different distributed communication algorithms on the Cifar10 training set are shown.

[0189] By analyzing the experimental results, we found that in the experiment, both Local SGD and LSGDMA algorithms were configured with 32 local iterations, which means that when local gradient descent is used to update local model parameters, global model synchronization is performed only once every 32 updates.

[0190] Based on CIFAR10 experimental results, LSGDMA converges faster than Local SGD with the same number of iterations. This indicates that Local SGDMA can approach the optimal solution more quickly. Furthermore, we observed that LSGDMA significantly reduces the loss function value in the early stages of the iterations, while Local SGD requires more iterations to achieve similar results.

[0191] With the same training time, LSGDMA achieved higher final accuracy than Local SGD. This means that LSGDMA can better optimize model parameters, thereby improving prediction performance. We ran LSGDMA and Local SGD multiple times and observed that LSGDMA exhibited greater stability than Local SGD. This indicates that under the same initialization conditions and random influences, LSGDMA can more reliably converge to an optimal solution.

[0192] Figure 7 The term "epoch" in this article refers to the number of rounds required to reach 80% accuracy on CIFAR10. Our proposed algorithm achieves significant improvements over SSGD by reducing the frequency of communication, or traffic, during upload and download processes. Specifically, on the CIFAR10 dataset, our algorithm reduces traffic by 32x compared to SSGD, achieving the specified accuracy level in epoch 105, and reduces communication by 47 rounds compared to Local SGD. These results further validate our algorithm's effectiveness in reducing communication costs, making the training process more efficient and feasible.

[0193] See Figure 8An embodiment of the present invention further provides a terminal, comprising a processor 301 and a memory 302; wherein the processor 301 is configured to implement the distributed deep learning method described in any embodiment of the present invention when running a computer program, and the memory 302 stores instructions and data.

[0194] In some embodiments, the memory 302 in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 302 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0195] Processor 301 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 301. The above processor 301 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 302. Processor 301 reads information from memory 302 and, in conjunction with its hardware, completes the steps of the above method.

[0196] In some embodiments, the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, or other electronic units or combinations thereof for performing the functions described herein.

[0197] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0198] Another embodiment of the present invention provides a computer storage medium, which stores an executable program. When the executable program is executed by the processor 301, the steps of the information processing method applied to the terminal can be implemented. For example, Figure 1 - one or more of the methods shown in Figure 4.

[0199] In some embodiments, the computer storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program codes.

[0200] It should be noted that the technical solutions described in the embodiments of the present invention can be arbitrarily combined without conflict.

[0201] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A distributed deep learning method based on communication frequency optimization and error compensation, characterized in that: The method comprises the following steps: S101: Build a global model of distributed deep learning and multiple computing nodes, and deploy copies of the global model as local models on the computing nodes; S102: Initializing global model parameters, and synchronizing copies of the global model parameters as local model parameters to each of the computing nodes; S103: The computing node obtains a training data set as a sample vector, trains the corresponding local model, and obtains the current batch gradient of each computing node; S104: updating local model parameters using a communication frequency optimization algorithm and an error compensation mechanism according to the global model parameters and the current batch gradient; S105: updating the global model according to the updated local model parameters, the current iteration number of the computing node and the preset iteration number; S106: Performing performance evaluation on the updated global model and saving the trained global model; Step S104 includes: Determining a preset update frequency of local model parameters using the communication frequency optimization algorithm; Calculating a second difference between the global model parameter and the local model parameter; multiplying the second difference by a preset compensation coefficient to obtain an error compensation term between the global model and the local model; Determining an error compensation gradient between the global model parameters and the local model parameters according to the error compensation term, the current batch gradient, and the local model parameters; According to the preset update frequency, the error compensation gradient is updated to the local model parameter.

2. The method according to claim 1, wherein: The color image dataset CIFAR-10 is used as an original dataset, and the original dataset is divided into a validation dataset and the training dataset.

3. The method according to claim 1, characterized in that The step S102 includes: Allocate memory for the global model parameters and initialize the global model parameters using a preset initialization method; and use a Broadcast operation via the MPI communication protocol to copy a copy of the global model parameters to each computing node as the local model parameters of the computing node.

4. The method according to claim 2, characterized in that The step S103 includes: The training data set is divided into sub-data sets of the same size and the same number as the computing nodes; the computing nodes obtain corresponding sub-data sets from the training data set and pre-process the sub-data sets; The preprocessed sub-datasets are used as sample vectors and input into the corresponding local model for forward propagation to obtain the prediction vector of the current batch; Calculate a first difference between the prediction vector and the sample vector using a loss function as a local model error of the computing node; Back propagation is performed using the local model error to obtain the gradient of the current batch of samples relative to the local model parameters of the computing node, that is, the current batch gradient.

5. The method according to claim 1, wherein The step S105 includes: Obtaining the current iteration number of the computing node; Determine whether synchronous communication is required in the current round based on the current number of iterations and the preset number of iterations; if so, synchronous communication is required, and use a global communication algorithm to aggregate the updated local model parameters of each computing node to update the global model parameters; The global model is updated using the updated global model parameters, and a copy of the updated global model parameters is synchronized to each of the computing nodes as a local model parameter.

6. The method according to claim 5, characterized in that The method further comprises: If not, there is no need for synchronous communication, the global model parameters inherit the global model parameters of the previous batch iteration, and the local model is continuously trained using the updated local model parameters.

7. A distributed deep learning device based on communication frequency optimization and error compensation, characterized in that: The device comprises: An initialization module, configured to construct a global model of distributed deep learning and multiple computing nodes, and deploy copies of the global model as local models on the computing nodes; The initialization module is also used to initialize global model parameters and synchronize copies of the global model parameters as local model parameters to each computing node; A training processing module is used for the computing node to obtain a training data set as a sample vector, train the corresponding local model, and obtain the current batch gradient of each computing node; The training processing module is also used to update local model parameters based on the global model parameters and the current batch gradient using a communication frequency optimization algorithm and an error compensation mechanism; The training processing module is further configured to update the global model according to the updated local model parameters, the current iteration number of the computing node, and the preset iteration number; An evaluation module, configured to perform performance evaluation on the updated global model and save the trained global model; The training processing module is specifically configured as follows: Determining a preset update frequency of local model parameters using the communication frequency optimization algorithm; Calculating a second difference between the global model parameter and the local model parameter; multiplying the second difference by a preset compensation coefficient to obtain an error compensation term between the global model and the local model; Determining an error compensation gradient between the global model parameters and the local model parameters according to the error compensation term, the current batch gradient, and the local model parameters; According to the preset update frequency, the error compensation gradient is updated to the local model parameter.

8. A terminal, characterized in that: The terminal includes a processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, it implements the distributed deep learning method according to any one of claims 1 to 6.

9. A computer-readable storage medium containing computer-executable instructions, characterized in that: The computer-executable instructions are executed by a processor to implement the distributed deep learning method according to any one of claims 1 to 6.

Images