Decentralized parallel gradient descent training method and system for deep learning
By introducing a pre-training and pipeline communication mechanism based on historical weights into deep learning, combined with a parallel training method using buffered weights, the problem of high communication overhead in decentralized algorithms is solved, and fast and robust training of deep convolutional neural networks is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHWEST A & F UNIV
- Filing Date
- 2023-07-27
- Publication Date
- 2026-05-29
AI Technical Summary
In existing technologies, the decentralized parallel gradient descent algorithm for deep learning suffers from excessive communication overhead, resulting in slow training speeds for deep convolutional neural networks.
We employ a pre-training method based on historical weights, pipeline communication, and parallel training based on buffered weights. By combining topological matrices and hybrid matrices, we achieve asynchronous training, reduce communication tasks, and improve information exchange efficiency.
It accelerates the training speed of deep convolutional neural network models, improves the stability of the training process and the utilization of hardware resources, and realizes more efficient distributed deep learning training.
Smart Images
Figure CN116894461B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of deep learning parallelization, specifically relating to a decentralized parallel gradient descent training method and system for deep learning. Background Technology
[0002] With the advent of the big data era, training datasets have become incredibly abundant. For example, the ImageNet dataset contains over 14 million images, the Google Books Ngrams dataset contains over 5.2 trillion word frequencies, and the memory capacity reaches 2.2TB. The massive datasets and large-scale deep neural networks have led to a dramatic increase in the number of parameters involved in training. The ResNet-152 model has 60 million parameters (He et al. 2016), the VGGNet19 model has 140 million parameters (Simonyan et al. 2014), and the BERT model has a staggering 300 million parameters (Devlin et al. 2018). Model training is extremely time-consuming, often taking months, which hinders the development and application of deep learning.
[0003] Large-scale datasets and deep neural networks place stringent demands on the hardware environment for model training. A single server can no longer handle such massive computations, making distributed server-based deep neural network training a hot research topic in the field of deep learning parallelization. Distributed training reduces training time by distributing deep neural network models across multiple worker nodes for parallel computation. It primarily includes centralized algorithms (based on parameter servers) and decentralized algorithms. Centralized algorithms manage and distribute model parameters through a central node (parameter server). However, a failure of the central node can cause the entire training process to crash. Furthermore, since the central node is responsible for merging gradients from multiple nodes, data inconsistencies can occur. The communication pressure between the central node and worker nodes also limits the training speed, resulting in low robustness. In decentralized algorithms, each worker node maintains a complete copy of the model, and communication between nodes coordinates model gradients. Compared to fully connected structures, partially connected distributed networks have gained widespread attention due to their lower cost, as they only establish connections between adjacent nodes. However, the overhead of global communication is enormous in decentralized training methods. Despite extensive research on decentralized distributed gradient descent algorithms, they still face the problem of high communication costs, which slow down the training of deep convolutional neural networks. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a decentralized parallel gradient descent training method and system for deep learning, so as to solve the problem that the training speed of deep convolutional neural networks is slow due to excessive communication overhead in the prior art.
[0005] To achieve the above objectives, the present invention employs the following technical solution:
[0006] A decentralized parallel gradient descent training method for deep learning includes:
[0007] Calculate the topology matrix and the hybrid matrix, and assign the same neural network to each working node in the distributed network topology matrix. Set the initial weight parameters for the neural network on each working node, and assign data samples to each working node.
[0008] A pipeline communication mechanism is created. Based on the weight parameters of the previous round, the neural network weight parameters on each worker node are updated through the pipeline communication mechanism. Based on the data samples allocated to each node, the neural network on each worker node is trained. The trained neural network is obtained after updating and training a set number of times. Among them, the neural network weight parameters on neighboring worker nodes are exchanged in the form of key-value pairs.
[0009] In each worker node, the main thread calculates the gradient of the neural network parameters and obtains the buffer weights; a communication thread is created, which exchanges and updates the neural network weight parameters on the worker node based on the buffer weights. The main thread and the communication thread train asynchronously and iteratively until the neural network obtains the optimal solution, and the neural network training ends.
[0010] A further improvement of the present invention is that:
[0011] Preferably, the initial weight parameters of the neural network on the worker node are set using a Gaussian function.
[0012] Preferably, before exchanging neural network weight parameters on neighboring worker nodes through the pipeline communication mechanism, the neural networks on each worker node in the distributed network topology matrix are pre-trained.
[0013] The pre-training process involves exchanging neural network weight parameters of neighboring working nodes through a pipeline communication mechanism, weighting the neural network parameter matrix using a topology matrix, and updating the weight parameters of each neural network based on the weighting result.
[0014] Preferably, the process of updating the neural network weight parameters on each working node through the mixing matrix is as follows:
[0015]
[0016] Among them, X kThe weights are the network weights from the previous iteration, and α represents the learning rate.
[0017] Preferably, when the model of the neural network weight parameters obtains the optimal solution, the gradient of the neural network satisfies...
[0018] Preferably, the buffer weights are used by the worker nodes to store the neural network weight parameters for each round.
[0019] Preferably, the main thread performs gradient calculation using the buffered weights from the previous round, and at the same time, the main thread passes the neural network weight parameters of the current round to the communication thread, which then exchanges and updates the parameters based on the neural network weight parameters.
[0020] Preferably, the neural network weight parameters on the updated working node are updated according to the following formula:
[0021]
[0022] Among them, X k+2 X represents the network weights after the (K+2)th iteration. COPY X represents the buffer weight. k This represents the historical weights during the training process, and α represents the fixed learning rate. This represents the gradient information calculated in the (k+1)th iteration. This represents the difference between the gradient in the previous iteration and the historical gradient.
[0023] A decentralized parallel gradient descent training system for deep learning includes:
[0024] The initialization unit calculates and obtains the topology matrix and the hybrid matrix, and uses them to allocate the same neural network to each working node in the distributed network topology matrix. It sets the initial weight parameters for the neural network on each working node and allocates data samples to each working node.
[0025] The synchronous training unit is used to create a pipeline communication mechanism. Based on the weight parameters of the previous round, the neural network weight parameters on each working node are updated through the pipeline communication mechanism. Based on the data samples allocated to each node, the neural network on each working node is trained. The trained neural network is obtained after updating and training a set number of times. Among them, the neural network weight parameters on neighboring working nodes are exchanged in the form of key-value pairs.
[0026] An asynchronous training unit is used in each working node for the main thread to calculate the gradient of the neural network parameters and obtain buffered weights; a communication thread is created, which exchanges and updates the neural network weight parameters on the working node based on the buffered weights. The main thread and the communication thread train asynchronously and iteratively until the neural network obtains the optimal solution, and the neural network training ends.
[0027] Preferably, the pipeline communication mechanism includes:
[0028] The communication unit is used to handle communication and information exchange between worker nodes;
[0029] Cache units are used to store key-value pairs in the neural network;
[0030] The computational unit is used to perform weighted operations on the weights of the neural network;
[0031] The replica unit is used to store the key-value pairs processed by the computation unit and output them to the buffer weight.
[0032] Compared with the prior art, the present invention has the following beneficial effects:
[0033] This invention proposes a decentralized parallel gradient descent training method for deep learning. Addressing the problem of excessive communication overhead in distributed deep learning training tasks—namely, the high communication cost of central nodes in existing centralized algorithms and the frequent synchronization requirements of decentralized algorithms—this method combines pre-training based on historical weights, pipeline communication, and parallel training based on buffered weights. This accelerates the training process of deep neural networks in distributed systems, speeding up the training of deep convolutional neural network models. This method aims to provide a more efficient training approach for distributed deep learning tasks, enabling rapid training of deep neural network models. The pre-training method based on historical weights ensures the stability of model convergence, allowing the network model to converge more robustly to the global optimum during training. The pipeline communication mechanism decomposes the communication task, increases communication parallelism, and improves the efficiency of information exchange between worker nodes. The parallel training method based on buffered weights decouples communication and computation operations, maximizing the utilization of hardware computing resources.
[0034] This invention proposes a decentralized parallel gradient descent training system for deep learning. First, a pre-training phase is introduced to avoid accuracy fluctuations during model training and ensure algorithm stability. Second, buffered weights and a mixing matrix are introduced into the calculation of each iteration, allowing the algorithm to be trained with a fixed learning rate. Then, each worker node allocates a physical space to store the current model weights, which are used to calculate gradients in the next training round. Finally, the computation and communication operations of the worker nodes are executed asynchronously. Experimental results show that the AD-PEXTR algorithm achieves speedups of 1.5x and 1.3x compared to All-Reduce SGD on the CIFAR 10 and ImageNet 100 datasets, respectively. It achieves competitive performance in distributed deep learning training tasks and has broad application prospects in practical production. Overlapping worker node communication and computation overhead improves the utilization of hardware computing resources. In addition, this invention has the following beneficial effects:
[0035] Furthermore, this invention designs and optimizes existing gradient descent training methods, establishing a training mechanism based on buffered weights. The Laplacian weight matrix is used as the topology matrix W, representing the link information between distributed network nodes, ensuring that a fixed learning rate can be used for model training.
[0036] Furthermore, this invention constructs a pipeline communication mechanism based on KV key-value pairs. By designing four working units, the parallel granularity of communication operations is increased, thereby accelerating the communication speed between working nodes.
[0037] Furthermore, this invention constructs an asynchronous training mechanism that, based on buffered weights, creates new threads for asynchronous communication and updates to the weight information of the network model, and modifies the update formula based on a hybrid matrix, thus overlapping computational and communication overhead and maximizing the utilization of hardware computing resources. Attached Figure Description
[0038] Figure 1 This is a flowchart of the network model initialization process of the present invention;
[0039] Figure 2 The flowchart for constructing the algorithm of this invention is shown below;
[0040] Figure 3 This is a flowchart of the distributed training process of the present invention;
[0041] Figure 4 The test results of this invention are shown in the figure: (A) Comparison of network model training accuracy and test accuracy; (B) Comparison of network model training time; (C) Comparison of scalability under different communication delays and the number of working nodes. Detailed Implementation
[0042] The present invention will now be described in further detail with reference to the accompanying drawings:
[0043] This invention discloses a decentralized parallel gradient descent training method for deep learning. The algorithm proposes a solution for distributed deep learning training tasks, which can maximize the overlap of computation and communication overhead and increase the utilization of hardware computing resources. By using buffered weights and a hybrid matrix, the algorithm can effectively accelerate the training process of deep convolutional neural network models by asynchronously executing gradient calculation and weight update tasks.
[0044] This invention is implemented based on the MXNet and Horovod frameworks. The entire method consists of several sub-methods, including initialization units, synchronous training, asynchronous mode switching, overlapping computation, and communication overhead. First, the initialization of the network model ensures that each worker node has different weights at the start of training, avoiding getting trapped in the same local optima during training. Then, the distributed system performs pre-training synchronously. Next, a copy of the model weights is created based on the network weights in the num-th round, and a new thread is created to initiate asynchronous training. Finally, a delayed update protocol is used, where worker nodes use buffered weights to calculate gradients and delay updating the buffered weights, achieving overlap of computation and communication overhead.
[0045] The decentralized parallel gradient descent training method for deep learning mainly includes five steps: network model initialization, synchronous training, pipeline communication, asynchronous mode switching, overlapping computation and communication overhead. (See [link to documentation]). Figure 3 After completing these four steps, you will obtain the final trained network model. See [link / reference]. Figure 2 and Figure 3 The method specifically includes the following steps:
[0046] Step 1, as follows Figure 1 As shown, the training parameters are initialized and the data is prepared.
[0047] The distributed network topology matrix is a crucial parameter in distributed deep learning, and it needs to be initialized before model training. Simultaneously, reassigning the neural network weights to each worker node before training the neural network helps obtain the optimal network model. Then, data samples are allocated to each worker node. This process includes dataset partitioning and data allocation. See also... Figure 2 The specific steps are as follows:
[0048] Step 1.1 Calculate the topology matrix and the hybrid matrix. Calculate the link information between the working nodes in the distributed system and initialize the topology matrix W with formula (3). When there is a network connection between node i and node j, W[i,j] has a definite value; otherwise, W[i,j] = W[j,i] = 0. At the same time, W[i,i] = 0.
[0049]
[0050] Where I is the identity matrix, L represents the Laplacian matrix of the distributed network, and τ is a constant greater than 0.5. Next, the mixture matrix is calculated:
[0051] Step 1.2 Initialize network weights. Distribute the designed deep neural network model to each working node, and then use the Gaussian function (2) to initialize the weight parameters of the deep neural network model in the working nodes, thereby optimizing the initial weights of the deep neural network model in the working nodes.
[0052]
[0053] Step 1.3 Dataset Allocation. During the experiment, the dataset was randomly divided into training and testing sets in an 8:2 ratio and stored as documents, with each line recording an image name. Simultaneously, the samples in the dataset were further partitioned, and data samples were assigned to each worker node, ensuring that each worker node trained with different data samples.
[0054] Step 2, synchronous training and communication.
[0055] First, a pipeline communication mechanism is established to exchange network model weights with neighboring working nodes. Simultaneously, the aggregated network weights are weighted using the topology matrix W. Then, new network weights are calculated using the learning rate and gradient information. Finally, the network weights X from the previous iteration are introduced. K-1 A network model can be trained using a fixed learning rate.
[0056] Step 2.1 involves dividing the trained neural network into key-value pairs by layer. In this step, the deep neural network model is subdivided into multiple key-value pairs according to the network layers, and then the network weight information is exchanged in the form of key-value pairs to reduce the communication granularity. The specific implementation process is as follows:
[0057] First, the deep neural network model is divided into layers. The weights of each layer are assigned to independent key-value pairs, where the key represents the layer and the value contains the weights of that layer. During training, the current node aggregates the network weights of its neighboring worker nodes through an All-Gather communication mechanism. This ensures that each node receives global network weight information, maintaining consistency across all nodes. By dividing the network into layers, the algorithm efficiently utilizes the communication bandwidth between nodes, reduces communication overhead, and fully leverages the parallel computing advantages of distributed systems.
[0058] Step 2.2 Create four work units
[0059] This invention designs multiple processing units, including a communication unit (COM), a cache unit, a computation unit (COP), and a copy unit, each undertaking different tasks to increase parallel granularity.
[0060] Communication Unit: Responsible for handling communication and information exchange between nodes. During training, the communication unit uses the All-Gather communication mechanism to aggregate the network weight information of neighboring worker nodes. In this way, each node can obtain global network weight information, maintaining consistency of weights among all nodes.
[0061] Caching Unit: Responsible for storing key-value (KV) pairs of the network model. During training, each key-value pair contains weight information for a network layer. The caching unit temporarily stores these key-value pairs, allowing for quick access and weighting when needed. The introduction of caching units reduces the communication load on the distributed system, improving training efficiency and performance.
[0062] Computational unit: Responsible for weighting operations on the network model, assigning W... ij X ij The person in charge of operation, disassembled into Where X ij For the network weights of other work units obtained through communication, This represents the network weight of the Kth layer, i.e., the value of the Kth key-value pair. By decomposing the computation process, the granularity of parallelism is increased.
[0063] The replica unit is responsible for storing the key-value pairs processed by the computation unit and outputting them to the buffer weights. Through the processing of the replica unit, the data size of each key-value pair is relatively small, reducing the burden on network transmission. At the same time, different nodes can process weights at different levels in parallel, accelerating the training process.
[0064] Step 2.3, Training and Communication
[0065] The topology matrix W carries the network link information of the distributed system. After steps 2.1 and 2.2, the topology matrix W is used for synchronous pre-training. See [link to relevant documentation]. Figure 3 The specific steps are as follows:
[0066] Step 2.3.1: The deep convolutional neural network initialized with a Gaussian function in Step 1 is used as the training target. Each image in the training set is processed by this network model. For each image sample, convolution, pooling, and other operations are performed. Based on the feedback from the forward training, the gradient of the weights of each layer of the neural network is obtained using the chain rule. Then, the nodes communicate with each other to exchange their respective network weight parameters, and use the topological matrix for weighted summation to perform synchronous updates. The training samples of the worker nodes are uniformly distributed, and the training images of each worker node are different.
[0067] Before training begins, pre-training is performed to ensure that the model training results maintain high accuracy. The network weights of neighboring working nodes are aggregated through a pipeline communication mechanism and weighted using a topology matrix W. Based on the weighting result, the network model is updated using formula (3). This global weight aggregation and weighted update mechanism helps each node obtain more comprehensive network model information and perform accurate parameter updates locally. This operation only needs to be performed in the first round of training and will not be performed again in subsequent training processes.
[0068]
[0069] Step 2.3.2: Based on the network weights and gradient information obtained in Step 2.1, the network weights from the previous iteration are introduced, and the synchronous pre-training round number num is input based on experience. Training is then performed using a fixed learning rate based on formula (4), where X k The weights are the network weights from the previous iteration, and α represents the learning rate.
[0070]
[0071] The network weights from the previous iteration contain "outdated" information, so the topology matrix W cannot be directly used in the calculation. Therefore, a hybrid matrix is used instead. The weights are then applied and the network model weights are updated using the calculation process described in formula (4).
[0072] This invention introduces a weighted fusion of historical weights, i.e., a hybrid matrix. This effectively offsets potential drastic fluctuations in network weight updates, enabling the network model to be trained stably with a fixed learning rate. This allows the invention to better control learning rate changes in a distributed environment, avoiding excessively large or small learning rates in a single weight update, thus resulting in more robust convergence to the global optimum during training.
[0073] Step 3: Asynchronous mode performs overlapping computations and communication overhead.
[0074] Following the synchronization phase, a buffer weight X is created based on the weights of the deep convolutional neural network model in the num-th iteration. copy Before each weight update, the current network weights are copied to a buffer weight to save the previous model state. Then, a new thread is created to control communication between nodes and the updating of the buffer weights, and updates the network weights based on the buffer weights and the current gradient information.
[0075] The process creates buffered weights and a communication thread, enabling asynchronous training and achieving efficient training in a distributed system. The communication thread acts as a coordinator in the distributed training system, responsible for coordinating information exchange and synchronization between different nodes, while the main thread handles gradient calculation. Specific details are as follows:
[0076] Step 3.1, Create buffer weights
[0077] When the model training reaches the num-th round, the worker node pauses the current training task and allocates a new block of physical memory as a buffer for the weights X. copy Store the network weights for the num-th round. By introducing buffered weights, worker nodes can retain the model state for this round during training, for use in subsequent training sessions.
[0078] In deep learning training, model optimization often requires multiple iterations. Assuming the model parameters are X and the loss function is f(X), the training objective is to find the optimal parameter values X. * This minimizes the loss function, i.e., X. * =argminL(X), when the model reaches the num-th training round, a buffer weight X is introduced. copy , is used to store the network weight state of the num-th training round. The update of the buffer weight can be expressed as formula (5), where X(num) represents the model parameters obtained after the num-th training round.
[0079] X copy =X(num) (5)
[0080] Step 3.2 Create a communication thread
[0081] Before the next training round begins, a new thread is created to control communication between worker nodes and update buffer weights. This communication thread acts as a coordinator in the distributed training system, responsible for coordinating information exchange and synchronization between different nodes to ensure the smooth progress of the training process.
[0082] In distributed deep learning training, assuming there are n worker nodes participating in the training, each node i has its own local model parameters X. i The communication thread is responsible for aggregating the network weight information of neighboring worker nodes to achieve information transmission and synchronization between nodes, thereby maintaining the consistency of the global model. Each worker node calculates its local network weights and gradients during training, and the communication thread is responsible for collecting this information so that each node can understand the training status of the distributed system.
[0083] To effectively integrate network weight information, the communication thread uses a topology matrix W to weight the collected network weight information. The topology matrix W carries network link information for the distributed system, with parameters representing the network bandwidth and connectivity between different nodes. This weighting provides crucial information for subsequent network model updates.
[0084] After each training round, node i sets its local model parameters X. i The data is aggregated to buffer weight X via a communication thread. copy The communication thread updates the global model parameters using a weighted average method, as shown in formula (6).
[0085]
[0086] Among them, X copy w represents the buffer weight on the i-th working node. ij x represents the network link between worker node i and worker node j. ij This represents the network weights aggregated from worker node j. Due to the definition of the mixing matrix W, worker nodes only need to communicate with their neighboring nodes, and global communication is not required.
[0087] Step 3.3 Start asynchronous training
[0088] In step 3.2, based on the previously created communication thread, asynchronous training mode is initiated, decoupling the communication thread of the worker node from the main thread. The main thread is responsible for calculating the local gradient in each training round, while the communication thread runs independently, periodically aggregating and updating parameters. This asynchronous mode allows worker nodes to continue training for the next round while aggregating parameters, without waiting for other nodes to complete, thus significantly improving training efficiency.
[0089] Specifically, the main thread uses the buffered weights obtained in the previous round to calculate the gradient. Simultaneously, the main thread passes the network weights for the current round to the communication thread for exchanging and updating network weight information. Then, after the communication thread completes the weighting process, the main thread applies the updated buffered weights to the next iteration of gradient calculation to obtain new gradient information.
[0090] Local gradient calculation is performed in the main thread of each worker node. Assuming the loss function is f(X), where X represents the model parameters, the local gradient is... The communication thread asynchronously aggregates the network model parameters X into the local node's buffer weights X. copy Above. In the communication thread, the buffer weight X copy This will be used as the current global model parameter and participate in the weight update of the next iteration. At the end of each training round, X copy This saves the current state of the global model.
[0091] The network weights are updated according to the description in formula (7). The buffer weights are based on the delayed update protocol, which ensures that the model retains the previous model state during training, avoiding information loss and instability. The introduction of formula (7) makes full use of the role of the buffer weights, enabling the algorithm to optimize the network model more robustly and efficiently during training. The new thread controls the asynchronous communication between nodes and updates the network buffer weights. At the same time, the worker nodes use the backed-up buffer weights to calculate the gradient of the network model parameters in the new round, and update the network weights asynchronously according to formula (7).
[0092]
[0093] Among them, X k+2 X represents the network weights after the (K+2)th iteration. COPY Represents the buffer weight, X k The historical weights during the training process are represented by α, which represents a fixed learning rate. This represents the gradient information calculated in the (k+1)th iteration. This term represents the difference between the gradient in the previous iteration and the historical gradient. Introducing this term allows the current node to fully consider the model state from the previous training iteration during parameter updates, thereby better adjusting network weights. This design helps avoid overly rapid weight updates, ensuring a stable training process and preventing model overfitting.
[0094] The cumulative update formula yields formula (8). When the model obtains the optimal solution, the distributed system nodes satisfy... At this point, the gradient satisfies make Because null{U} = span{1}, therefore This can be offset by accumulating historical weights. Asynchronous training ends when the model obtains the optimal solution, or when the training accuracy exceeds the threshold or set value.
[0095]
[0096] During parameter aggregation and buffer weight updates, the main thread of the worker node can continue with the next round of gradient calculation and training. In this way, each node can autonomously complete gradient updates and parameter aggregation, achieving an efficient asynchronous training mode.
[0097] In asynchronous training mode, computational and communication overheads overlap. Operations between the communication thread and the main thread are performed in parallel without waiting for the computation results from other nodes. While the main thread is calculating gradients, the communication thread can aggregate and process gradient information from other nodes, and then pass the weighted global gradient information to the main thread for gradient calculation and updating the network model.
[0098] One embodiment of the present invention discloses a decentralized parallel gradient descent training system for deep learning, comprising:
[0099] The initialization unit is used to allocate the same neural network to each working node in the distributed network topology matrix, set the initial weight parameters of the neural network on each working node, and allocate data samples to each working node.
[0100] The synchronous training unit is used to create a pipeline communication mechanism. Based on the weight parameters of the previous round, a mixing matrix is obtained through the pipeline communication mechanism. The neural network weight parameters on each working node are updated through the mixing matrix. When the model with updated neural network weight parameters obtains the optimal solution, the trained neural network is obtained. Among them, the neural network weight parameters on neighboring working nodes are exchanged in the form of key-value pairs.
[0101] An asynchronous training unit is used in each neural network for the main thread to calculate the local gradient and obtain buffered weights; a communication thread is created, which periodically exchanges and updates the neural network weight parameters on the worker nodes based on the buffered weights. The main thread and the communication thread perform asynchronous iterative training until the set number of iterations is completed, at which point the neural network training ends.
[0102] Example
[0103] The decentralized parallel gradient descent algorithm requires four steps to train a deep neural network model: parameters, synchronous training and communication, asynchronous mode switching, and overlapping computation and communication overhead. This invention is implemented based on the MXNet and Horovod frameworks. (1) Initialization of training parameters: First, the distributed network topology matrix W is initialized, and the mixing matrix is calculated. Then, the designed deep neural network model is distributed to each working node, and the weight parameters in the network model are initialized using a Gaussian function. Finally, the original dataset is divided into training and test sets in an 8:2 ratio and stored in document form, with each line recording an image name. At the same time, data samples are allocated to each working node according to certain rules to ensure that each node has different training samples. (2) Synchronous training and communication: First, the trained neural network is divided into KV key-value pairs according to the layers. Each key-value pair represents a network layer and its corresponding weight. Then, four processing units are designed, namely, communication unit, cache unit, computing unit and copy unit. Then, the deep convolutional neural network initialized by the Gaussian function is used as the training target. Each image sample is processed by convolution, pooling and other operations and gradient information is calculated using the chain rule. The network weights of neighboring working nodes are summarized using the pipeline communication mechanism, and synchronous training is performed using the topology matrix W. The topology matrix W carries the connection information and network bandwidth of the working nodes in the distributed system. Finally, before each update of the network weights, the network weights of the previous iteration are introduced to participate in the update of the network weights, so that the network model maintains a certain historical trend during the update process and prevents drastic fluctuations. Finally, a convolutional deep neural network model is trained using a fixed learning rate. (3) Asynchronous mode switching: First, buffer weights are created to save the network model weight state of the num-th iteration for subsequent training. Then, a new communication thread is created to act as the coordinator in the distributed training system. The main task of the communication thread is to coordinate the information transmission and synchronization between nodes to maintain the consistency of the global model. Finally, due to the decoupling between the communication thread and the main thread, the worker nodes can achieve asynchronous training mode. (4) Overlapping computation and communication overhead: First, the communication thread summarizes the network weight information of other worker nodes and uses the topology matrix W to weight these information. Then, the main thread uses the buffer weights to perform gradient calculation and passes the network weights of the current round to the communication thread. Next, the main thread updates the network weights according to formula (9), which introduces the terms of buffer weights and historical gradients. Finally, in the asynchronous training mode, the overlap of computation and communication overhead is achieved, and the optimal training effect is obtained, such as Figure 4 As shown.
[0104] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A decentralized parallel gradient descent training method for deep learning, characterized in that, include: Calculate the topology matrix and the hybrid matrix, and assign the same neural network to each working node in the distributed network topology matrix. Set the initial weight parameters for the neural network on each working node, and assign data samples to each working node. A pipeline communication mechanism is created. Based on the weight parameters of the previous round, the neural network weight parameters on each working node are updated through the pipeline communication mechanism. Based on the data samples allocated to each node, the neural network on each working node is trained. The trained neural network is obtained by updating and training a set number of times; wherein, the neural network weight parameters on neighboring worker nodes are exchanged in the form of key-value pairs. In each worker node, the main thread calculates the gradient of the neural network parameters and obtains the buffer weights; a communication thread is created, which exchanges and updates the neural network weight parameters on the worker node based on the buffer weights. The main thread and the communication thread train asynchronously and iteratively until the neural network obtains the optimal solution, and the neural network training ends. The pipeline communication mechanism, based on the weight parameters of the previous round, updates the neural network weight parameters on each working node through the pipeline communication mechanism. The specific process of training the neural network on each working node based on the data samples allocated to each node is as follows: Step 2.1: Divide the trained neural network into KV key-value pairs by layer, further subdivide the neural network into multiple key-value pairs by layer, and exchange weights to reduce the communication granularity. Specifically, each layer weight corresponds to an independent key-value pair, and the weights of neighboring nodes are aggregated through the All-Gather communication mechanism. Step 2.2: Create a communication unit, a cache unit, a computation unit, and a replica unit. The communication unit is responsible for communication between nodes and the aggregation of weight information. The cache unit stores the model's key-value pairs to reduce communication load and improve access efficiency. The computation unit decomposes the weighting operation to increase parallel granularity. The replica unit stores the processed key-value pairs and outputs them to the buffer weights to reduce transmission burden and speed up training. Step 2.3 Conduct training and communication, using the topology matrix W for synchronous pre-training, specifically including: Step 2.3.1: Using the neural network as the training target, the weight gradient of each layer is calculated by the chain rule through forward training feedback on the image samples. The nodes communicate to exchange weight parameters and update synchronously by weighting through the topology matrix W. Pre-training and global weight aggregation weighting update are performed before the first round of training, and are not performed in subsequent training. Step 2.3.2 Introduce the weights from the previous iteration based on the existing network weights and gradients. Input the number of pre-training rounds num, train with a fixed learning rate, and update the model weights by weighting the weights from the previous iterations using a mixing matrix.
2. The decentralized parallel gradient descent training method for deep learning according to claim 1, characterized in that, The initial weight parameters of the neural network on the worker node are set using a Gaussian function.
3. The decentralized parallel gradient descent training method for deep learning according to claim 1, characterized in that, Before exchanging neural network weight parameters on neighboring worker nodes through the pipeline communication mechanism, the neural networks on each worker node in the distributed network topology matrix are pre-trained. The pre-training process involves exchanging neural network weight parameters of neighboring working nodes through a pipeline communication mechanism, weighting the neural network parameter matrix using a topology matrix, and updating the weight parameters of each neural network based on the weighting result.
4. The decentralized parallel gradient descent training method for deep learning according to claim 1, characterized in that, The process of updating the neural network weight parameters on each working node using the mixing matrix is as follows: （4） in, These are the network weights from the previous iteration. This represents the learning rate.
5. The decentralized parallel gradient descent training method for deep learning according to claim 1, characterized in that, When the model of the neural network weight parameters obtains the optimal solution, the gradient of the neural network satisfies .
6. The decentralized parallel gradient descent training method for deep learning according to claim 1, characterized in that, The buffer weights are used by the worker nodes to store the neural network weight parameters for each round.
7. The decentralized parallel gradient descent training method for deep learning according to claim 1, characterized in that, The main thread calculates gradients using the buffered weights from the previous round, and simultaneously passes the neural network weight parameters for the current round to the communication thread, which then exchanges and updates the parameters based on the neural network weights.
8. The decentralized parallel gradient descent training method for deep learning according to claim 1, characterized in that, The neural network weight parameters on the updated working node are updated according to the following formula: （8） in, Indicates the first Network weights after rounds of iteration Indicates buffer weight, Represents the historical weights during the training process. This indicates a fixed learning rate. Indicates the first The gradient information calculated in each iteration This represents the difference between the gradient in the previous iteration and the historical gradient.
9. A decentralized parallel gradient descent training system for deep learning, used to implement the decentralized parallel gradient descent training method of claim 1, characterized in that, include: The initialization unit calculates and obtains the topology matrix and the hybrid matrix, and uses them to allocate the same neural network to each working node in the distributed network topology matrix. It sets the initial weight parameters for the neural network on each working node and allocates data samples to each working node. The synchronous training unit is used to create a pipeline communication mechanism. Based on the weight parameters of the previous round, the neural network weight parameters on each working node are updated through the pipeline communication mechanism. Based on the data samples allocated to each node, the neural network on each working node is trained. The trained neural network is obtained by updating and training a set number of times; wherein, the neural network weight parameters on neighboring worker nodes are exchanged in the form of key-value pairs. An asynchronous training unit is used in each working node for the main thread to calculate the gradient of the neural network parameters and obtain buffered weights; a communication thread is created, which exchanges and updates the neural network weight parameters on the working node based on the buffered weights. The main thread and the communication thread train asynchronously and iteratively until the neural network obtains the optimal solution, and the neural network training ends.
10. A decentralized parallel gradient descent training system for deep learning according to claim 9, characterized in that, The pipeline communication mechanism includes: The communication unit is used to handle communication and information exchange between worker nodes; Cache units are used to store key-value pairs in the neural network; The computational unit is used to perform weighted operations on the weights of the neural network; The replica unit is used to store the key-value pairs processed by the computation unit and output them to the buffer weight.