Low-bandwidth distributed deep learning methods

Abstract

The present invention relates to a low-bandwidth distributed deep learning method, comprising: constructing a global deep learning model of a decentralized network according to a local data set; constructing a distributed deep learning model; based on a local local data set, using a randomized decentralized parallel stochastic gradient descent algorithm to train the parameters of the distributed deep learning model of the decentralized network topology working nodes; performing weighted averaging on the model parameters of all working nodes of the decentralized network topology to obtain global model parameters. The method of the present invention only needs to randomly select some parameters of the deep learning model for information transmission, and does not need to perform time-consuming operations such as sorting the model parameters. While meeting the low bandwidth requirements, it effectively reduces the computational complexity of model parameter screening.

Description

Technical Field

[0001] The present invention relates to the field of computer deep learning technology, and in particular to a low-bandwidth distributed deep learning method. Background Art

[0002] With the breakthrough development of artificial intelligence and the rapid improvement of on-board computing and storage capabilities, it is an inevitable development trend for remote sensing satellite systems to directly generate information required by users through on-orbit intelligent processing. Since the scenes and targets that users are concerned about are usually highly dynamic, the traditional working mode of "offline training on the ground-on-orbit prediction on the satellite" can no longer meet user needs. Therefore, it is urgent to learn and update the intelligent model on-orbit to adapt to the dynamic changes of the observation scene or meet the perception needs of unfamiliar scenes.

[0003] At the same time, affected by factors such as satellite orbits, payloads, target physical characteristics, and imaging methods, more and more intelligent tasks require multiple satellites of different types to work together to complete. For example, emergency observations in disaster areas and searches for missing Malaysia Airlines flights require the coordination of multiple satellites or spacecraft. If we only rely on observation data from a single satellite, it will be difficult to achieve accurate learning of the global intelligent model.

[0004] With the development of satellite technology and the reduction of costs, the number of satellites in orbit has increased sharply, and inter-satellite networks have been gradually established. Remote sensing satellite systems have the foundation for multi-satellite collaborative work. Remote sensing satellite systems work together in a self-organizing manner and use distributed deep learning technology to integrate the real-time perception data and computing power of multiple satellites. At the same time, by using communication and interaction between satellites, the collaborative operation of the system can be achieved without global information or human intervention to solve more and more complex problems and broaden applications. However, due to the particularity of the operating environment of artificial satellites or spacecraft, the network topology of remote sensing satellite systems has the characteristics of low bandwidth. Therefore, the research on distributed deep learning in low-bandwidth network environments is of great significance and practical urgency.

[0005] Existing solutions for distributed deep learning in bandwidth-constrained scenarios are mainly divided into two categories. The first category is to use a decentralized network structure to avoid network congestion of the central server. In the traditional centralized network structure, all nodes need to transmit their trained parameters or gradients to the central server, and then perform parameter or gradient fusion on the central server, and then the central server distributes the fused parameters or gradients to each node. The problem is that the network transmission pressure of the central server is very large. By adopting a decentralized network structure, by removing the central server and allowing all nodes to perform parameter or gradient fusion with adjacent nodes, the network transmission pressure can be distributed to each node, avoiding congestion while improving the timeliness of distributed deep learning. Typical methods of this type include the Decentralized Parallel Stochastic Gradient Descent (D-PSGD) algorithm and the Asynchronous Decentralized Parallel Stochastic Gradient Descent (AD-PSGD) algorithm. Each node calculates the stochastic gradient locally, averages the model parameters with neighboring nodes, and finally updates the gradient to the averaged model parameters.

[0006] The second category is to reduce the amount of data transmission and save bandwidth usage. There are mainly three methods: delayed synchronization, quantization and sparsification. Among them, delayed synchronization is no longer required to start parameter or gradient fusion after each node trains a batch, but to perform parameter or gradient fusion with other nodes or servers after training multiple batches locally. Delayed synchronization is represented by Local SGD and Federated Averaging. Quantization mainly replaces model parameters with low-bit and low-precision representation methods, such as quantizing model parameters from 64-bit double-precision floating point numbers to 32-bit integers, thereby greatly reducing network transmission. Representative methods include QSGD, TernGrad, DoReFa-Net, etc. The sparsification method refers to synchronizing only part of the model parameters instead of transmitting all model parameters. Representative methods include Top-K Sparsification, DeepGradient Compression, Sparse Binary Compression, etc., all of which require certain rules to filter model parameters. For example, the Top-K Sparsification method selects the largest K model parameters for transmission. However, the existing sparsification methods have high parameter screening complexity and require operations such as sorting, which results in high computational complexity. Summary of the invention

[0007] In order to solve the defects of high complexity in model parameter screening of existing low-bandwidth distributed deep learning solutions based on sparsification, and difficulty in giving full play to the advantages of distributed deep learning in quickly completing collaborative tasks, while meeting the intelligent learning needs of remote sensing satellite systems in low-bandwidth network environments, the present invention provides a low-bandwidth distributed deep learning method.

[0008] In order to achieve the above-mentioned object of the invention, the technical solution of the present invention is:

[0009] The present invention provides a low-bandwidth distributed deep learning method, comprising: constructing a global deep learning model of a decentralized network based on a local data set; constructing a distributed deep learning model; based on a local local data set, using a randomized decentralized parallel stochastic gradient descent algorithm to train the parameters of the distributed deep learning model of the working nodes of the decentralized network topology; performing weighted averaging on the model parameters of all working nodes of the decentralized network topology to obtain global model parameters.

[0010] Preferably, the process of constructing a global deep learning model of a decentralized network based on a local data set includes: defining a global deep learning model, and the following stochastic optimization problem:

[0011]

[0012] Where D is the local dataset, ξ is a data sample in the local dataset, and x∈R N represents the serialized parameter vector of the global deep learning model, N is the total number of model parameters, and F(x;ξ) represents the loss function of the predefined global deep learning model.

[0013] Preferably, the process of constructing a distributed deep learning model includes: distributing data of a local data set for the decentralized network topology, defining that the local local data set on the working node of the decentralized network topology is co-distributed with the local data set, and the following random optimization problem:

[0014]

[0015] Where i represents the serial number of the working node, and 1≤i≤n, D i is the local data set, F i (x;ξ) is the local loss function, and F i (·;·)=F(·;·).

[0016] Preferably, the process of training the parameters of the distributed deep learning model using a randomized decentralized parallel stochastic gradient descent algorithm on a working node of the decentralized network topology based on a local local data set includes:

[0017] Input initialization model parameters x 0，i = x0, learning rate γ, weight matrix W and maximum number of iterations K, where W∈R n ×n is the network topology of all working nodes, satisfying the relationship: (i) Wij∈[0,1], (ii)∑ j W ji =1, j represents the working node number, and 1≤j≤n;

[0018] Initialize the number of iterations k = 0;

[0019] Compare the current number of iterations k with the maximum number of iterations K. When the current number of iterations k is less than the maximum number of iterations K, continue to perform the following steps. When the current number of iterations k is not less than the maximum number of iterations K, output the model parameter x k，i ;

[0020] Construct vector a on the same working node of the decentralized network topology k , and set the vector a k The probability that each element belongs to 1 is α, and then the vector a k Transmit to other working nodes;

[0021] Randomly collect data samples ξ from the local local data set k，i ;

[0022] Calculate the gradient of the current model parameters

[0023] Get the vector a from the adjacent working node k Specify the model parameters and calculate the neighborhood weighted average of the model parameters:

[0024]

[0025] Where Γ(a k ) represents vector a k The set of positions of non-zero elements in ;

[0026] Let x k+1 / 2，i (s) = x k，j (s), s∈[N]\Γ(a k ), where [N]:={1, 2, ..., N};

[0027] renew

[0028] The number of iterations k is updated to k+1, and the step of comparing the updated number of iterations with the maximum number of iterations K is repeated.

[0029] Preferably, the vector a is obtained from an adjacent working node. k The specified model parameters are calculated and the neighborhood weighted average of the model parameters and the set x k+1 / 2，i (s) = x k，j (s), s∈[N]\Γ(a k ) is equivalent to:

[0030]

[0031] Among them, X k =[x k，1 , x k，2 , …, x k，n ]∈R N×n represents the concatenation of the model parameter vectors of all working nodes at the kth iteration, P α ∈R N×N Represents the matrix Diag(a k ), a k ∈R N is a vector of independent Bernoulli random variables, The probability that each element of the vector belongs to 1 is α∈(0,1].

[0032] Beneficial effects of the present invention:

[0033] The low-bandwidth distributed deep learning method of the present invention provides an improved randomized sparsification method based on the decentralized network topology, forming a randomized decentralized parallel stochastic gradient descent (RD-PSGD) algorithm. Compared with the traditional distributed deep learning solution of the decentralized parallel stochastic gradient descent algorithm, the present invention can meet the requirements of intelligent learning tasks in lower bandwidth network scenarios.

[0034] Compared with the existing sparsification methods, in the process of real-time data perception and calculation synchronization of the distributed deep learning model of adjacent working nodes in the decentralized network topology structure, the present invention only needs to randomly select some parameters of the deep learning model for information transmission and sharing, and does not require time-consuming operations such as sorting of the model parameters. While meeting the low bandwidth requirements, it effectively reduces the computational complexity of model parameter screening and greatly improves the working efficiency of the distributed deep learning model.

[0035] In addition, the present invention can also be well applied to remote sensing satellite systems. When performing complex intelligent learning tasks such as perception or recognition, remote sensing satellites form a decentralized network topology structure, and use the RD-PSGD algorithm of the present invention to train the models of each satellite. Through intelligent learning, satellites can interact and communicate on orbit in a lower bandwidth operating environment, integrate the real-time perception and computing information of multiple satellites, and complete the collaborative tasks on orbit. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart schematically showing a low-bandwidth distributed deep learning method according to an embodiment of the present invention;

[0037] Figure 2 A flowchart schematically showing a low-bandwidth distributed deep learning method for training distributed deep learning model parameters of network topology nodes according to an embodiment of the present invention;

[0038] Figure 3 A diagram schematically showing a comparison of the network topology node information transmission process of a randomized decentralized parallel stochastic gradient descent algorithm and a decentralized parallel stochastic gradient descent algorithm proposed in a low-bandwidth distributed deep learning method according to an embodiment of the present invention;

[0039] Figure 4 A diagram schematically showing a comparison of training loss and accuracy between a randomized decentralized parallel stochastic gradient descent algorithm and a decentralized parallel stochastic gradient descent algorithm proposed in a low-bandwidth distributed deep learning method according to an embodiment of the present invention;

[0040] Figure 5 Schematically represents the bandwidth consumption of the randomized decentralized parallel stochastic gradient descent algorithm at different sparsity rates proposed in the low-bandwidth distributed deep learning method of one embodiment of the present invention;

[0041] Figure 6 A diagram schematically shows a comparison of the net time per epoch synchronization between a randomized decentralized parallel stochastic gradient descent algorithm proposed in a low-bandwidth distributed deep learning method according to an embodiment of the present invention and a decentralized parallel stochastic gradient descent algorithm in a low-bandwidth scenario. DETAILED DESCRIPTION

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments cannot be described one by one here, but the embodiments of the present invention are not therefore limited to the following embodiments.

[0044] Figure 1 A flowchart schematically illustrates a low-bandwidth distributed deep learning method according to an embodiment of the present invention, revealing the technical concept of the present invention. Figure 2 A flowchart schematically shows a low-bandwidth distributed deep learning method for training distributed deep learning model parameters of network topology nodes according to an embodiment of the present invention. Figure 1 As shown, after the implementation of this embodiment starts, a local data set D of a perception or cognitive task is first given, and ξ is a data sample therein. A global deep learning model of a decentralized network topology is constructed based on the local data set. The specific process includes: first defining a global deep learning model, whose serialized parameter vector is x∈R N , then the global deep learning model learning is equivalent to solving the following stochastic optimization problem,

[0045]

[0046] Where x∈R N represents the serialized parameter vector of the global deep learning model, N is the total number of model parameters, and F(x;ξ) represents the loss function of the predefined global deep learning model. Model learning here refers to the process of model construction.

[0047] Secondly, a distributed deep learning model is constructed. The decentralized network topology contains n working nodes. The process includes: distributing data samples of local data sets for the decentralized network topology, defining the local local data sets D on the working nodes of the decentralized network topology, and i The local loss function corresponding to the distributed deep learning model is F i (·;·)=F(·;·), then the process of building a distributed deep learning model or learning a distributed deep learning model is equivalent to solving the following stochastic optimization problem,

[0048]

[0049] Wherein, i represents the serial number of the working node, and 1≤i≤n.

[0050] Then, based on the above local data set D i , the randomized decentralized parallel stochastic gradient descent (RD-PSGD) algorithm is used to train the parameters of the distributed deep learning model of the decentralized network topology working nodes. Figure 2 As shown, this process specifically includes the following steps:

[0051] First enter the initialized model parameters x 0，i =x0, learning rate γ, weight matrix W and maximum number of iterations K. Where W∈R n×n is the network topology of all working nodes, satisfying the relationship: (i) Wij∈[0,1], (ii)∑ j W ji=1, j represents the working node number, and 1≤j≤n.

[0052] At the same time, the number of iterations k is initialized to 0.

[0053] Next, a judgment instruction is executed to compare the current iteration number k with the maximum iteration number K. When the current iteration number k is less than the maximum iteration number K, the following steps are continued. When the current iteration number k is not less than (including equal to and greater than) the maximum iteration number K, the model parameter x is output. k，i That is, when the current iteration number k is equal to or greater than the maximum iteration number K, the output model parameter x k，i .

[0054] When the current iteration number k is less than the maximum iteration number K, construct vector a on the same working node in the decentralized network topology k , and set the vector a k The probability that each element in belongs to 1 is α, and then the vector a k Transmitted to other working nodes. Here, vector a k Refers to the vector composed of the parameters of the distributed deep learning model of the worker node. If a model parameter in the vector is 1, it is transmitted to the remaining worker nodes. If a model parameter is 0, it is not transmitted to the remaining worker nodes. In each iteration, vector a is constructed on the same worker node. k , and then the worker node passes vector a k The parameters of the corresponding position are transmitted to the remaining working nodes, that is, the information received by all working nodes is kept consistent through broadcasting, thereby ensuring that all working nodes in the entire network topology remain synchronized.

[0055] Next, randomly collect data samples ξ from the local dataset k，i . And calculate the gradient of the current model parameters Then obtain the above vector a from the adjacent working node k The specified model parameters are calculated as the neighborhood weighted average of the model parameters:

[0056]

[0057] Where Γ(a k ) represents vector a k In this way, the working nodes only need to obtain some parameters of the distributed deep learning model to achieve interaction and synchronization, without the need for time-consuming sorting and screening of model parameters, thus reducing computational complexity and improving work efficiency.

[0058] Let x k+1 / 2，i(s) = x k，j (s), s∈[N]\Γ(a k ), where [N]: = {1, 2, ..., N}, and update At the same time, the number of iterations is updated k=k+1, and then the above judgment instruction and subsequent steps are repeatedly executed. The parameters of the distributed deep learning model are trained through continuous updating and iteration.

[0059] Finally, the parameters of the distributed deep learning model that have been trained on all working nodes of the decentralized network topology are weighted averaged to obtain the final model parameters. The model parameters at this time are the parameters of the global deep learning model. By broadcasting part of the parameter information of the distributed deep learning model of a certain working node to each working node of the decentralized network topology, the parameters of the distributed deep learning model of the working node are continuously trained to obtain the model parameters of the global network, and finally realize the synchronization and coordinated operation of each working node, that is, Figure 1 The process shown ends.

[0060] It should be noted here that, from a global perspective, in the RD-PSGD algorithm training distributed deep learning model parameters, vector a is obtained from adjacent working nodes. k The specified model parameters are used to calculate the neighborhood weighted average of the model parameters, and x k+1 / 2，i (s) = x k，j (s), s∈[N]\Γ(a k ) These two steps are equivalent to:

[0061]

[0062] Among them, X k =[x k，1 , x k，2 , …, x k，n ]∈R N×n represents the concatenation of the model parameter vectors of all working nodes in the kth iteration, α represents the sparsity rate, and P α ∈R N×N Represents the matrix Diag(a k ), a k ∈R N is a vector of independent Bernoulli random variables, Vector a k The probability that each element in belongs to 1 is α∈(0, 1]. It can be proved that the communication complexity of the RD-PSGD algorithm is O(αN·Deg(network)), where Deg(network) represents the network topology, that is, the degree of the weight matrix.

[0063] Figure 3A diagram schematically shows a comparison of the network topology node information transmission process of the randomized decentralized parallel stochastic gradient descent algorithm and the decentralized parallel stochastic gradient descent algorithm proposed in the low-bandwidth distributed deep learning method of an embodiment of the present invention. Figure 3 As shown in Figure 2, compared with the D-PSGD algorithm, each working node in the RD-PSGD algorithm no longer needs to transmit all model parameters X = [x(s)], s∈[N], but only needs to transmit P α The corresponding information specified, that is, X′=[x(s)], s∈Γ(a k ), the overall optimization and global convergence of the distributed deep learning model can be achieved. When α<1, the transmission complexity of the RD-PSGD algorithm will be lower than the transmission complexity of the D-PSGD algorithm O(N·Deg(network)), and will decrease accordingly with the decrease of the sparsity rate α. For remote sensing satellite systems, Figure 3 Each working node in the remote sensing satellite system is a satellite in the system, which together form a decentralized network topology. When training a distributed deep learning model, each satellite only needs to interact with its neighboring satellites to achieve overall optimization and global convergence of the model.

[0064] Preferably, the local data set D of this embodiment is the image classification data set ImageNet-1k. The network architecture of the global deep learning model is ResNet-50, and its loss function F(x; ξ) is the cross entropy loss. The model parameter x0 is initialized using Kaiming initialization settings, and the initial learning rate γ is initialized to 0.1. Every 30 epochs, that is, every time the distributed deep learning model is trained as a whole, the learning rate γ is divided by 10. The decentralized network topology, that is, the weight matrix W is a ring structure of 8 working nodes, in which each working node samples data ξ for each iteration. k，i =64, the maximum number of iterations K is 225000, that is, the number of epochs for training is 90. Every 100 iterations, the adjacent working nodes perform model synchronization, that is, when the model parameters are merged, the training loss and accuracy of the D-PSGD algorithm (sparse rate α is 1) and the RD-PSGD algorithm (sparse rate α is 0.1) are as follows: Figure 4 As shown in (a) and (b). Figure 4 The displayed results show that the RD-PSGD algorithm proposed in this embodiment can reach convergence even when only 10% of the model parameters are transmitted, and the accuracy at the time of convergence can remain basically unchanged.

[0065] Figure 5Schematically represents the bandwidth consumption of a single working node single weight transmission at different sparsity rates of the randomized decentralized parallel stochastic gradient descent algorithm proposed in the low-bandwidth distributed deep learning method of one embodiment of the present invention. Figure 5 As shown in the figure, when the sparsity rate is close to 1, the network traffic consumption will increase due to the need for additional transmission of weight positions, but as the sparsity increases, the network bandwidth occupancy decreases approximately linearly. Furthermore, we use trickle software to limit the connection speed of the local wireless network (200kb / s) to verify the acceleration performance of the RD-PSGD algorithm in low-bandwidth scenarios. Figure 6 As shown in the figure, when the sparsity rate is 0.1, compared with the D-PSGD algorithm, the net synchronization time per epoch of the RD-PSGD algorithm (i.e., the total time required per epoch minus the GPU training time) is reduced from 285.6s to 58.3s, which is nearly 5 times faster. Since the RD-PSGD algorithm can effectively reduce the amount of parameter transmission, it can effectively reduce the transmission time in low-bandwidth scenarios.

[0066] The RD-PSGD algorithm of this embodiment only needs to randomly select some parameters for model synchronization. Compared with the existing low-bandwidth distributed deep learning algorithm based on sparsification, the RG-PSGD algorithm does not need to perform time-consuming operations such as sorting model parameters, which can greatly reduce the computational complexity of parameter screening. Table 1 shows the time consumption comparison of model parameter selection at different sparsity rates between the randomized decentralized parallel stochastic gradient descent algorithm proposed in the low-bandwidth distributed deep learning method of this embodiment and the low-bandwidth distributed deep learning method based on Top-K sparsification. As shown in Table 1, at different sparsity rates, compared with the low-bandwidth distributed deep learning method based on Top-K sparsification, the RD-PSGD algorithm of this embodiment takes less time to screen model parameters and has a more obvious acceleration effect.

[0067] α＝0.1 α＝0.5 Top-K 0.659 1.619 RD-PSGD 0.099 0.419

[0068] Table 1

[0069] The above is only one embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A low-bandwidth distributed deep learning method, comprising: Build a global deep learning model for decentralized networks based on local datasets; Build distributed deep learning models; Based on the local local data set, a randomized decentralized parallel stochastic gradient descent algorithm is used to train the parameters of the distributed deep learning model of the decentralized network topology working nodes; Performing weighted averaging on the model parameters of all working nodes of the decentralized network topology to obtain global model parameters; The process of constructing a global deep learning model of a decentralized network based on a local data set includes: defining a global deep learning model, and the following random optimization problem, Where D is the local dataset, ξ is a data sample in the local dataset, and x∈R N represents the serialized parameter vector of the global deep learning model, N is the total number of model parameters, and F(x;ξ) represents the loss function of the predefined global deep learning model; The process of training the parameters of the distributed deep learning model using a randomized decentralized parallel stochastic gradient descent algorithm based on a local local data set on a working node of the decentralized network topology includes: Input initialization model parameters x 0,i = x0, learning rate γ, weight matrix W and maximum number of iterations K, where W∈R n×n is the network topology of all working nodes, satisfying the relationship: (i) Wij∈[0,1], (ii)∑ j W ji =1, j represents the working node number, and 1≤j≤n; Initialize the number of iterations k = 0; Compare the current number of iterations k with the maximum number of iterations K. When the current number of iterations k is less than the maximum number of iterations K, continue to perform the following steps. When the current number of iterations k is not less than the maximum number of iterations K, output the model parameter x k,i ; Construct vector a on the same working node of the decentralized network topology k , and set the vector a k The probability that each element belongs to 1 is α, and then the vector a k Transmit to other working nodes; Randomly collect data samples ξ from the local local data set k,i ; Calculate the gradient of the current model parameters Get the vector a from the adjacent working node k Specify the model parameters and calculate the neighborhood weighted average of the model parameters: Where Γ(a k ) represents vector a k The set of positions of non-zero elements in ; Let x k+1 / 2,i (s) = x k,j (s), s ∈ [N]\Γ(a k ), where [N] := {1, 2, …, N}; renew The number of iterations k is updated to k+1, and the step of comparing the updated number of iterations with the maximum number of iterations K is repeated.

2. The low-bandwidth distributed deep learning method according to claim 1, characterized in that: The process of constructing a distributed deep learning model includes: distributing data of a local data set for the decentralized network topology, defining that the local local data set on the working node of the decentralized network topology is co-distributed with the local data set, and the following random optimization problem: Where i represents the serial number of the working node, and 1≤i≤n, D i is the local data set, F i (x;ξ) is the local loss function, and F i (·;·)=F(·;·).

3. The low-bandwidth distributed deep learning method according to claim 1, characterized in that: The vector a is obtained from the adjacent working node k The specified model parameters are calculated and the neighborhood weighted average of the model parameters and the set x k+1 / 2,i (s) = x k,j (s),s∈[N]\Γ(a k ) is equivalent to: Among them, X k =[x k,1 ,x k,2 ,…,x k,n ]∈R N×n represents the concatenation of the model parameter vectors of all working nodes in the kth iteration, α represents the sparsity rate, and P α ∈R N×N Represents the matrix Diag(a k ), a k ∈R N is a vector of independent Bernoulli random variables, The probability that each element of the vector belongs to 1 is α∈(0,1].

Images