Centerless distributed network optimization training method and device, equipment and medium

Abstract

The present disclosure relates to the technical field of communication, in particular to a centerless distributed network optimization training method, device and equipment and medium. The method comprises: based on the connection relationship between N distributed nodes in a centerless distributed network, a random mixing matrix is obtained, a preset initialization parameter is obtained, and an initial model of an i-th node in the N nodes is obtained; a target compression algorithm is determined from at least one compression algorithm that satisfies a preset communication compression condition; based on the row random mixing matrix, the initialization parameter and the target compression algorithm, the initial model of the i-th node is iteratively trained, and when a training stop condition is satisfied, a trained model of the i-th node is obtained. The scheme is suitable for generalized communication compression assumptions containing relative and absolute compression errors, so that a more suitable compression method can be selected to improve communication efficiency while reducing communication cost.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of communication, and in particular to a centerless distributed network optimization training method, device, equipment and medium. BACKGROUND

[0002] The explosive growth of distributed big data brings opportunities to data-driven machine learning. In order to train a machine learning model based on massive distributed data, the data is usually uploaded from distributed devices to a data center, and the parameter optimization is performed in a center-distributed manner in the data center. However, this center-distributed optimization algorithm lacks scalability and privacy protection capability. Therefore, in recent years, centerless distributed optimization algorithms based on centerless network topology have gradually attracted research attention. The centerless distributed architecture does not rely on a central master node for coordination, but each node makes autonomous decisions based on limited information exchange with its neighbor nodes. It has the advantages of low communication cost, strong data security, and suitability for real-time applications.

[0003] However, due to the influence of high-dimensional model parameters and limited network communication bandwidth, the centerless distributed network optimization method in the related art still faces challenges in communication efficiency. In the related art, methods such as quantization and sparsification are used to reduce the number of information bits transmitted between nodes each time. However, this method is usually only applicable to certain specific error conditions introduced by compression, and due to the limitations of the compression method, the communication efficiency achieved is poor.

[0004] Therefore, there is a need for a centerless distributed network optimization method that can improve communication efficiency. SUMMARY

[0005] To solve the problems in the related art, the embodiments of the present disclosure provide a centerless distributed network optimization training method, device, equipment and medium.

[0006] In a first aspect, the embodiments of the present disclosure provide a centerless distributed network optimization training method, comprising:

[0007] Based on the connection relationship between N distributed nodes in the centerless distributed network, a random mixing matrix is obtained, the random mixing matrix includes a row random mixing matrix and a column random mixing matrix, and N is an integer greater than 1;

[0008] An initial parameter is obtained, and an initial model of an i-th node in the N nodes is obtained, i is greater than 0 and less than N;

[0009] A target compression algorithm is determined from at least one compression algorithm that satisfies a preset communication compression condition, the preset communication compression condition includes at least one of the following: a relative error compression condition, an absolute error compression condition;

[0010] perform iterative training on the initial model of the i-th node based on the random mixing matrix, the initialization parameter and the target compression algorithm;

[0011] obtain the model of the i-th node after training when a training stop condition is met.

[0012] In a second aspect, the embodiments of the present disclosure provide a centerless distributed network optimization training apparatus, comprising:

[0013] A first obtaining module is configured to obtain a random mixing matrix based on a connection relationship between N distributed nodes in the centerless distributed network, the random mixing matrix comprising a row random mixing matrix and a column random mixing matrix, N being an integer greater than 1;

[0014] A second obtaining module is configured to obtain a preset initialization parameter and obtain an initial model of an i-th node in the N nodes, i being greater than 0 and less than N;

[0015] A first determining module is configured to determine a target compression algorithm from at least one compression algorithm meeting a preset communication compression condition, the preset communication compression condition comprising at least one of the following: a relative error compression condition and an absolute error compression condition;

[0016] An iterative module is configured to perform iterative training on the initial model of the i-th node based on the random mixing matrix, the initialization parameter and the target compression algorithm;

[0017] A second determining module is configured to obtain the model of the i-th node after training when a training stop condition is met.

[0018] In a third aspect, the embodiments of the present disclosure provide an electronic device, comprising a memory and a processor, wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method of the first aspect.

[0019] In a fourth aspect, the embodiments of the present disclosure provide a computer readable storage medium having computer instructions stored thereon, the computer instructions being executed by a processor to implement the method of the first aspect.

[0020] According to the technical scheme provided by the embodiment of the present disclosure, the connection relationship between N distributed nodes in a centerless distributed network can be used to obtain a random mixing matrix, the random mixing matrix includes a row random mixing matrix, a preset initialization parameter is obtained, and an initial model of an i th node in the N nodes is obtained. A target compression algorithm is determined from at least one compression algorithm that meets a preset communication compression condition, the preset communication compression condition includes at least one of the following: a relative error compression condition, an absolute error compression condition; the initial model of the i th node is iteratively trained based on the row random mixing matrix, the initialization parameter and the target compression algorithm, and when a training stop condition is met, a model of the i th node after training is obtained. Through the scheme, under the condition of considering the relative error and the absolute error existing when the communication compression, a more suitable compression algorithm can be selected, so that the communication efficiency is improved under the premise of reducing the communication cost, and the application of the scheme is more extensive.

[0021] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0022] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of the non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

[0023] Figure 1 A flowchart of a centerless distributed network optimization training method according to an embodiment of the present disclosure is shown.

[0024] Figure 2 A flowchart of a centerless distributed network optimization training device according to an embodiment of the present disclosure is shown.

[0025] Figure 3 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0026] Figure 4 A structural schematic diagram of a computer system suitable for implementing the method according to the embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily implement them. In addition, parts irrelevant to the description of the exemplary embodiments are omitted in the drawings for the sake of clarity.

[0028] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate that there exist the features, numbers, steps, actions, components, parts or combinations thereof disclosed in the specification, and do not exclude the possibility that one or more other features, numbers, steps, actions, components, parts or combinations thereof exist or are added.

[0029] It should also be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] In the present disclosure, if the operation of acquiring user information or user data or the operation of showing user information or user data to others is involved, the operation is an operation authorized, confirmed by the user, or actively selected by the user.

[0031] It is mentioned above that the explosive growth of distributed big data brings opportunities to data-driven machine learning. In order to train a machine learning model based on massive distributed data, the data is usually uploaded from distributed devices to a data center, and the parameter optimization is performed in a central distributed manner in the data center. However, this central distributed optimization algorithm lacks scalability and privacy protection capability. Therefore, in recent years, the non-central distributed optimization algorithm based on non-central network topology has gradually attracted research attention. The non-central distributed architecture does not rely on a central master node for coordination, but each node makes autonomous decisions based on limited information exchange with its neighbor nodes, has the advantages of low communication cost, strong data security, and suitability for real-time applications, etc.

[0032] However, affected by high-dimensional model parameters and limited network communication bandwidth, the non-central distributed network optimization method in the related art still faces challenges in communication efficiency. In the related art, methods such as quantization and sparsification are adopted to reduce the information bits transmitted between nodes each time, but this method is usually only applicable to certain specific error conditions introduced by compression, so due to the limitations of the compression method, the communication efficiency achieved is poor.

[0033] To solve the above technical problems, in the technical scheme provided by the embodiments of the present disclosure, a random mixing matrix can be obtained based on the connection relationship between N distributed nodes in a centerless distributed network, the random mixing matrix including a row random mixing matrix and a column random mixing matrix, a preset initialization parameter is obtained, and an initial model of an i-th node of the N nodes is obtained. A target compression algorithm is determined from at least one compression algorithm that satisfies a preset communication compression condition, the preset communication compression condition including at least one of a relative error compression condition and an absolute error compression condition. The initial model of the i-th node is iteratively trained based on the random mixing matrix, the initialization parameter, and the target compression algorithm, and when a training stop condition is met, a model of the i-th node after training is obtained. Through this scheme, under the condition of considering the relative error and the absolute error existing when communication compression, a more suitable compression algorithm can be selected, so that the model after training improves the communication efficiency on the premise of reducing the communication cost, and also makes the application of the scheme more widely.

[0034] Figure 1 A flowchart of a centerless distributed network optimization training method according to an embodiment of the present disclosure is shown. As shown in Figure 1 The optimization training method includes the following steps S101-S105:

[0035] In step S101, a random mixing matrix is obtained based on the connection relationship between N distributed nodes in the centerless distributed network, the random mixing matrix including a row random mixing matrix and a column random mixing matrix, N being an integer greater than 1;

[0036] In step S102, a preset initialization parameter is obtained, and an initial model of an i-th node of the N nodes is obtained, i being greater than 0 and less than N;

[0037] In step S103, a target compression algorithm is determined from at least one compression algorithm that satisfies a preset communication compression condition, the preset communication compression condition including at least one of a relative error compression condition and an absolute error compression condition;

[0038] In step S104, the initial model of the i-th node is iteratively trained based on the random mixing matrix, the initialization parameter, and the target compression algorithm;

[0039] In step S105, when a training stop condition is met, a model of the i-th node after training is obtained.

[0040] In an embodiment of the present disclosure, the random mixing matrix is obtained based on the connection relationship between the N distributed nodes in the decentralized distributed network. It can be understood that in the decentralized distributed network environment, the N distributed nodes in the network topology are connected by point-to-point communication to form a connected graph, and the row random mixing matrix and the column random mixing matrix can be obtained based on the connected graph. The transpose of the column random mixing matrix and the row random mixing matrix need to have the same root node.

[0041] Exemplarily, the connected graph is Exemplarily, the connected graph is is a set of nodes, and is a set of edges in the connected graph. For a directed network The row random mixing matrix can be obtained It should be understood that R can correspond to a subnetwork of the connected graph.

[0042] Exemplarily, the connected graph is Exemplarily, the connected graph is is a set of nodes, and is a set of edges in the connected graph. For a directed network The column random mixing matrix can be obtained It should be understood that c and R can correspond to two different subnetworks of

[0043] In an embodiment of the present disclosure, the initialization parameters can be understood to include an input learning step, a consistency step, an error control parameter, and other initial values.

[0044] In an embodiment of the present disclosure, the initial model of the i-th node can be understood as a machine learning model, which can be used for image recognition. The machine learning model can be a linear regression model, a neural network model, etc., and the present disclosure does not limit the same.

[0045] In an embodiment of the present disclosure, the target compression algorithm satisfies the relative error compression condition and the absolute error compression condition at the same time, thereby avoiding the performance of the decentralized distributed optimization algorithm being affected, and making the model of the i-th node obtained by training be a converged optimal solution.

[0046] In an embodiment of the present disclosure, satisfying the training stop condition can be understood as the number of iterations of the training reaching a preset number, or the model of the i-th node reaching a converged state.

[0047] ​It should be noted that the embodiments of the present disclosure are exemplified by optimizing the model of the i th node in the N distributed nodes in the centerless distributed network. In actual application, the same compression algorithm can be used to optimize the model of each node in the N distributed nodes, and the specific manner can refer to the above steps S101-S105. It should be understood that after the model training of all nodes is completed, the same model can be obtained, that is, the N distributed nodes share the same model.

[0048] In an embodiment of the present disclosure, the initialization parameters include an input learning step length of the i th node, an absolute communication compression error control parameter, a consistency step length corresponding to the model of the i th node, initial decision compression auxiliary variables and initial decision compression mixed auxiliary variables corresponding to each node in the N nodes, and an initial random gradient of a loss function corresponding to the model of each node in the N nodes.

[0049] The iterative training of the initial model of the i th node based on the random mixing matrix, the initialization parameters and the target compression algorithm includes:

[0050] Based on the initial model of the i th node and the corresponding input learning step length and initial random gradient, a decision variable of the i th node is obtained, and the decision variable is used to indicate the local model of the i th node.

[0051] Based on the target compression algorithm, the absolute communication compression error control parameter and the initial decision compression auxiliary variable corresponding to the i th node, the decision variable is compressed to obtain a compressed decision variable corresponding to the i th node.

[0052] Based on the absolute communication compression error control parameter and the initial decision compression auxiliary variable corresponding to the i th node, the compressed decision variable is compressed to obtain a recovered decision variable.

[0053] Based on the absolute communication compression error control parameter, the row random mixing matrix and the initial decision compression mixed auxiliary variable corresponding to the i th node, the compressed decision variable is communicated and recovered between nodes to obtain mixed decision variable information.

[0054] Based on the decision variable, the recovered decision variable, the mixed decision variable information and the consistency step length corresponding to the model of the i th node, the initial model of the i th node is updated once to obtain an updated model of the i th node.

[0055] In an embodiment of the present disclosure, the initial random gradient of the i-th node can be understood as an initial value of the random gradient calculated by collecting independent random samples in each step of iteration of the i-th node. For example, taking independent random samples as For example, the random gradient can be represented as g i (x i ,v i ), and the initial random gradient can be represented as y i (0) = g i (x i (0), ξ i (0)).

[0056] In an embodiment of the present disclosure, based on the initial model of the i-th node, the corresponding input learning step and the initial random gradient, the decision variable of the i-th node can be calculated by the following formula:

[0057]

[0058] wherein, x i (k) represents the local decision variable of the i-th node after the k-th iteration training, x (k) is a model trained based on the initial model of the i-th node for the k-th iteration, λ i is an input learning step of the i-th node, y i (k) is a gradient tracking variable trained based on the initial random gradient of the i-th node for the k-th iteration, and k can be an integer greater than or equal to 0.

[0059] In an embodiment of the present disclosure, the initial decision compression auxiliary variable corresponding to the i-th node can be understood as the decision compression auxiliary variable corresponding to the i-th node at the zeroth iteration training, and the initial decision compression auxiliary variable can be a zero vector.

[0060] In an embodiment of the present disclosure, based on the target compression algorithm, the absolute communication compression error control parameter and the initial decision compression auxiliary variable corresponding to the i-th node, the decision variable is compressed for communication to obtain the compressed decision variable corresponding to the i-th node, which can be calculated by the following formula:

[0061]

[0062] wherein, c x,i (k) represents the compressed decision variable corresponding to the i-th node after the k-th iteration training, represents the target compression algorithm, x x,i (k) represents the decision variable of the i-th node after the k-th iteration training, h kdenotes the absolute communication compression error control parameter after the kth iteration training.

[0063] In an embodiment of the present disclosure, the compressed decision variable is compressed information recovered based on the absolute communication compression error control parameter and the initial decision compression auxiliary variable corresponding to the ith node, to obtain the recovered decision variable, which can be calculated by the following formula:

[0064]

[0065] wherein q x,i (k) = s k c x,i (k). denotes the recovered decision variable corresponding to the ith node after the kth iteration training, h x,i (k) denotes the decision compression auxiliary variable after the kth iteration training, c x,i (k) denotes the compressed decision variable corresponding to the ith node after the kth iteration training, s k denotes the absolute communication compression error control parameter after the kth iteration training.

[0066] In an embodiment of the present disclosure, the initial decision compression mixed auxiliary variable corresponding to the ith node can be understood as the decision compression mixed auxiliary variable corresponding to the ith node when the zeroth iteration training is performed, and the initial decision compression mixed auxiliary variable can be a zero vector.

[0067] In an embodiment of the present disclosure, the compressed decision variable is communicated and recovered between nodes based on the absolute communication compression error control parameter, the row random mixing matrix and the initial decision compression mixed auxiliary variable corresponding to the ith node, to obtain the mixed decision variable information, which can be calculated by the following formula:

[0068]

[0069] wherein, denotes the mixed decision variable information corresponding to the ith node after the kth iteration training, h R,i (k) denotes the decision compression mixed auxiliary variable corresponding to the ith node after the kth iteration training, denotes the weighted sum of the decision variables of all neighbor nodes adjacent to the ith node after communication compression and recovery; r ij denotes any one of the neighbor nodes adjacent to the ith node in the row random mixing matrix, q x,i (k) = s k c x,i (k), c x,i (k) denotes the compressed decision variable corresponding to the ith node after the kth iteration training, s kdenotes the absolute communication compression error control parameter after the kth iteration training.

[0070] It should be noted that the mixed decision variable information is used to aggregate the communication information of all neighbor nodes of the ith node, so there is no need to store the decision compression mixed auxiliary variables of all neighbor nodes.

[0071] In an embodiment of the present disclosure, the initial model of the ith node is updated once based on the decision variable, the recovered decision variable, the mixed decision variable information and the consistency step length corresponding to the model of the ith node, to obtain an updated model of the ith node, which can be calculated by the following formula:

[0072]

[0073] wherein x i (k+1) denotes the model of the ith node after k iterations of training updates, denotes the local decision variable of the ith node after the kth iteration training, denotes the recovered decision variable corresponding to the ith node after the kth iteration training, denotes the mixed decision variable information corresponding to the ith node after the kth iteration training, and γ x denotes the consistency step length corresponding to the model of the ith node.

[0074] In an embodiment of the present disclosure, the initialization parameter further includes a first relative communication compression error control parameter related to the model of the ith node; and the iterative training of the initial model of the ith node based on the random mixing matrix, the initialization parameter and the target compression algorithm comprises:

[0075] updating the initial decision compression auxiliary variable corresponding to the ith node based on the first relative communication compression error control parameter and the recovered decision variable;

[0076] updating the initial decision compression mixed auxiliary variable corresponding to the ith node based on the first relative communication compression error control parameter and the mixed decision variable information;

[0077] wherein the updated initial decision compression auxiliary variable and the updated initial decision compression mixed auxiliary variable are used for the next update of the updated model of the ith node.

[0078] In an embodiment of the present disclosure, the initial decision compression auxiliary variable corresponding to the ith node is updated based on the first relative communication compression error control parameter and the recovered decision variable, which can be calculated by the following formula:

[0079]

[0080] wherein, h x,i (k+1) represents the decision compression auxiliary variable corresponding to the i-th node after k times of iterative training, and α x represents the first relative communication compression error control parameter, represents the recovered decision variable corresponding to the i-th node after k times of iterative training, and h x,i (k) represents the decision compression auxiliary variable corresponding to the i-th node after k times of iterative training, and when k = 0, h x,i (0) is the initial decision compression auxiliary variable corresponding to the i-th node.

[0081] In an embodiment of the present disclosure, the initial decision compression hybrid auxiliary variable corresponding to the i-th node is updated based on the first relative communication compression error control parameter and the hybrid decision variable information, which can be calculated by the following formula:

[0082]

[0083] wherein, h x,i (k+1) represents the decision compression hybrid auxiliary variable corresponding to the i-th node after k times of iterative training, and α x represents the first relative communication compression error control parameter, represents the hybrid decision variable information corresponding to the i-th node after k times of iterative training, and h R,i (k) represents the decision compression hybrid auxiliary variable corresponding to the i-th node after k times of iterative training, and when k = 0, h R,i (0) is the initial decision compression hybrid auxiliary variable corresponding to the i-th node.

[0084] In an embodiment of the present disclosure, the initialization parameters further include a consistent step size of random gradient corresponding to the N nodes, and an initial gradient tracking compression auxiliary variable and an initial gradient tracking compression hybrid auxiliary variable corresponding to each node of the N nodes; the iterative training of the initial model of the i-th node based on the random mixing matrix, the initialization parameters and the target compression algorithm comprises:

[0085] The local gradient tracking variable is obtained based on the random gradient difference and the initial random gradient corresponding to the i-th node, and the random gradient difference is obtained according to the random gradient corresponding to the model of the i-th node after updating and the random gradient corresponding to the initial model of the i-th node;

[0086] Based on the target compression algorithm, the absolute communication compression error control parameters, and the initial gradient tracking compression auxiliary variable corresponding to the i-th node, the gradient tracking variable is compressed through communication to obtain the compressed gradient tracking variable corresponding to the i-th node;

[0087] Based on the absolute communication compression error control parameters and the initial gradient tracking compression auxiliary variable corresponding to the i-th node, the compressed gradient tracking variable is restored to obtain the restored gradient tracking variable.

[0088] Based on the absolute communication compression error control parameters, the column random mixing matrix, and the initial gradient tracking compression mixing auxiliary variable corresponding to the i-th node, the compressed gradient tracking variable is transmitted and recovered between nodes to obtain the mixed gradient tracking variable information.

[0089] Based on the gradient tracking variables, the recovered gradient tracking variables, the mixed gradient tracking variable information, and the consistency step size of the stochastic gradient corresponding to the i-th node, the initial stochastic gradient corresponding to the i-th node is updated once, and the updated initial stochastic gradient corresponding to the i-th node is used to update the model of the i-th node in the next update.

[0090] In one embodiment of this disclosure, in a decentralized distributed network environment, for N distributed nodes in the network topology, each node forms a connected graph through point-to-point communication, and a column random mixing matrix can be obtained based on this connected graph.

[0091] In one embodiment of this disclosure, the gradient tracking variable is obtained based on the stochastic gradient difference and the initial stochastic gradient corresponding to the i-th node, and can be calculated using the following formula:

[0092]

[0093] in, Let y represent the local gradient tracking variable of the i-th node after the k-th iteration of training. i (k) represents the stochastic gradient after the k-th iteration training based on the initial stochastic gradient of the i-th node, g i (x i (k+1), ξ i (k+1)) represents the stochastic gradient corresponding to the i-th node after k iterations of training and updating the model, g i (x i (k), ξ i (k) represents the stochastic gradient corresponding to the initial model of the i-th node.

[0094] In an embodiment of the present disclosure, the initial gradient tracking compression auxiliary variable corresponding to the i-th node can be understood as the gradient tracking compression auxiliary variable corresponding to the i-th node at the zeroth iteration training, and the initial gradient tracking compression auxiliary variable can be a zero vector.

[0095] In an embodiment of the present disclosure, the gradient tracking variable is compressed based on the target compression algorithm, the absolute communication compression error control parameter and the initial gradient tracking compression auxiliary variable corresponding to the i-th node, to obtain the compressed gradient tracking variable corresponding to the i-th node, which can be calculated by the following formula:

[0096]

[0097] wherein c y,i (k) represents the compressed gradient tracking variable corresponding to the i-th node after the k-th iteration training, represents the target compression algorithm, represents the local gradient tracking variable of the i-th node after the k-th iteration training, h y,i (k) represents the gradient tracking compression auxiliary variable after the k-th iteration training, s k represents the absolute communication compression error control parameter after the k-th iteration training.

[0098] In an embodiment of the present disclosure, the compressed gradient tracking variable is recovered based on the absolute communication compression error control parameter and the initial gradient tracking compression auxiliary variable corresponding to the i-th node, to obtain the recovered gradient tracking variable, which can be calculated by the following formula:

[0099]

[0100] wherein q y,i (k) = s k c y,i (k). represents the recovered gradient tracking variable corresponding to the i-th node after the k-th iteration training, h y,i (k) represents the gradient tracking compression auxiliary variable after the k-th iteration training, c y,i (k) represents the compressed gradient tracking variable corresponding to the i-th node after the k-th iteration training, s k represents the absolute communication compression error control parameter after the k-th iteration training.

[0101] In an embodiment of the present disclosure, the compressed gradient tracking variable is communicated and recovered between nodes based on the absolute communication compression error control parameter, the column random mixing matrix and the initial gradient tracking compression mixing auxiliary variable corresponding to the i-th node, to obtain the mixed gradient tracking variable information, which can be calculated by the following formula:

[0102]

[0103] wherein, denotes the mixed gradient tracking variable information corresponding to the ith node after the kth iteration of training, h C,i (k) denotes the gradient tracking compressed mixed auxiliary variable corresponding to the ith node after the kth iteration of training, denotes the weighted sum of the gradient tracking variable communication compression recovery of all neighbor nodes adjacent to the ith node; c ij denotes any neighbor node adjacent to the ith node in the column random mixed matrix, q y,i (k) = s k c y,i (k), c y,i (k) denotes the compressed gradient tracking variable corresponding to the ith node after the kth iteration of training, s k denotes the absolute communication compression error control parameter after the kth iteration of training.

[0104] It should be noted that the mixed gradient tracking variable information is used to aggregate the communication information of all neighbor nodes of the ith node, so there is no need to store the gradient tracking compressed mixed auxiliary variables of all neighbor nodes.

[0105] In an embodiment of the present disclosure, the initial random gradient corresponding to the ith node is updated based on the gradient tracking variable, the recovered gradient tracking variable, the mixed gradient tracking variable information, and the consistent step size of the random gradient corresponding to the ith node, which can be calculated by the following formula:

[0106]

[0107] wherein, y i (k+1) denotes the updated gradient tracking variable corresponding to the ith node after k iterations of training, denotes the local gradient tracking variable of the ith node after the kth iteration of training, denotes the recovered gradient tracking variable corresponding to the ith node after the kth iteration of training, denotes the mixed gradient tracking variable information corresponding to the ith node after the kth iteration of training, γ y denotes the consistent step size of the random gradient corresponding to the ith node.

[0108] It should be understood that the gradient tracking variable after multiple iterations of training is used to track the global average gradient, and the gradient tracking variable gradually converges, which can effectively process heterogeneous data in the training process.

[0109] In an embodiment of the present disclosure, the relative communication compression error control parameter comprises a second relative communication compression error control parameter related to a random gradient corresponding to the i-th node; and the iterative training of the initial model of the i-th node based on the random mixing matrix, the initialization parameter and the target compression algorithm comprises:

[0110] updating the initial gradient tracking compression auxiliary variable corresponding to the i-th node based on the second relative communication compression error control parameter and the recovered gradient tracking variable;

[0111] updating the initial gradient tracking compression mixed auxiliary variable corresponding to the i-th node based on the second relative communication compression error control parameter and the mixed gradient tracking variable information;

[0112] wherein the updated initial gradient tracking compression auxiliary variable and the updated initial gradient tracking compression mixed auxiliary variable are used for next updating of the updated gradient tracking variable corresponding to the i-th node.

[0113] In an embodiment of the present disclosure, the updating of the initial gradient tracking compression auxiliary variable corresponding to the i-th node based on the second relative communication compression error control parameter and the recovered gradient tracking variable can be calculated by the following formula:

[0114]

[0115] wherein h y,i (k+1) represents the gradient tracking compression auxiliary variable corresponding to the i-th node after k times of iterative training, a y represents the second relative communication compression error control parameter, represents the recovered gradient tracking variable corresponding to the i-th node after k times of iterative training, h y,i (k) represents the gradient tracking compression auxiliary variable corresponding to the i-th node after k times of iterative training, and when k=0, h y,i (0) is the gradient tracking compression auxiliary variable corresponding to the i-th node.

[0116] In an embodiment of the present disclosure, the updating of the initial gradient tracking compression mixed auxiliary variable corresponding to the i-th node based on the second relative communication compression error control parameter and the mixed gradient tracking variable information can be calculated by the following formula:

[0117]

[0118] wherein h C,i (k+1) represents the gradient tracking compression mixed auxiliary variable corresponding to the i-th node after k times of iterative training, a ya second relative communication compression error control parameter, h represents the mixed gradient tracking variable information corresponding to the i-th node after the k-th iteration training, C,i (k) represents the gradient tracking compression mixed auxiliary variable corresponding to the i-th node after the k-th iteration training, and when k=0, h C,i (0) is the initial gradient tracking compression mixed auxiliary variable corresponding to the i-th node.

[0119] It can be understood that in the above embodiment, by performing differential compression on the recovered decision variable and the initial decision compression auxiliary variable corresponding to the i-th node, the relative communication compression error can be reduced, and the decision variable and the gradient tracking variable of the i-th node are transmitted to its neighbor nodes after performing communication compression, and then recovered at the neighbor nodes. This process uses a dynamic scaling compression operator, thereby avoiding the problem of affecting the final convergence result of the model caused by the continuous accumulation of absolute error.

[0120] According to the technical scheme provided by the embodiment of the present disclosure, the connection relationship between N distributed nodes in a centerless distributed network can be used to obtain a random mixing matrix, the random mixing matrix includes a row random mixing matrix, a preset initialization parameter is obtained, and an initial model of an i-th node in the N nodes is obtained. From at least one compression algorithm that satisfies a preset communication compression condition, a target compression algorithm is determined, the preset communication compression condition includes at least one of the following: a relative error compression condition, an absolute error compression condition; and the initial model of the i-th node is iteratively trained based on the row random mixing matrix, the initialization parameter and the target compression algorithm. When a training stop condition is met, a model of the i-th node after training is obtained. Through this scheme, under the condition of considering the relative error and the absolute error existing when communication compression is performed, a more suitable compression algorithm can be selected, so that the communication efficiency is improved under the premise of reducing the communication cost, and the application of the compression algorithm is more widely.

[0121] In an embodiment of the present disclosure, the target compression algorithm is determined from at least one compression algorithm that satisfies a preset compression condition, including:

[0122] Based on a set of convergence parameters corresponding to each compression algorithm in the at least one compression algorithm, a convergence rate corresponding to each compression algorithm is obtained;

[0123] Based on the convergence rate of each compression algorithm, a compression algorithm corresponding to the maximum convergence rate is determined from the at least one compression algorithm;

[0124] When the compression algorithm corresponding to the maximum convergence rate satisfies the network bandwidth and system computing resources, the compression algorithm corresponding to the maximum convergence rate is determined as the target compression algorithm.

[0125] In an embodiment of the present disclosure, a set of convergence parameters can include a constant C and a constant σ. Wherein, C≥0 and σ≥0. The constant C satisfies and there is a constant δ, σ r ≥0 and r>0 such that its r-scale operator satisfies It should be understood that different compression algorithms correspond to different sets of convergence parameters.

[0126] According to the technical scheme provided by the embodiments of the present disclosure, the convergence rate corresponding to each compression algorithm can be obtained based on a set of convergence parameters corresponding to each compression algorithm in at least one compression algorithm, the compression algorithm corresponding to the maximum convergence rate can be determined from the at least one compression algorithm based on the convergence rate of each compression algorithm, and when the compression algorithm corresponding to the maximum convergence rate meets the network bandwidth and system computing resources, the compression algorithm corresponding to the maximum convergence rate is determined as the target compression algorithm, so that the compression algorithm with the fastest convergence rate can be selected to perform communication compression on the model to facilitate subsequent inter-node communication, thereby improving the communication efficiency and improving the efficiency of the centerless distributed optimization training.

[0127] Figure 2 A structural block diagram of a centerless distributed network optimization training apparatus according to an embodiment of the present disclosure is shown. Wherein, the apparatus can be realized as part or all of an electronic device through software, hardware or a combination of both.

[0128] As Figure 2 shown, the apparatus 200 includes:

[0129] A first obtaining module 201 configured to obtain a random mixing matrix based on a connection relationship between N distributed nodes in the centerless distributed network, the random mixing matrix including a row random mixing matrix and a column random mixing matrix, N being an integer greater than 1;

[0130] A second obtaining module 202 configured to obtain a preset initialization parameter and obtain an initial model of an i-th node in the N nodes, i being greater than 0 and less than N;

[0131] A first determining module 203 configured to determine a target compression algorithm from at least one compression algorithm that meets a preset communication compression condition, the preset communication compression condition including at least one of the following: a relative error compression condition, an absolute error compression condition;

[0132] An iteration module 204 configured to perform iterative training on the initial model of the i-th node based on the random mixing matrix, the initialization parameter and the target compression algorithm;

[0133] The second determination module 205 is configured to obtain the model of the i-th node after training when the training stop condition is met.

[0134] In an implementation of the present disclosure,

[0135] The initialization parameters include an input learning step length of the i-th node, an absolute communication compression error control parameter, a consistency step length corresponding to the model of the i-th node, initial decision compression auxiliary variables and initial decision compression hybrid auxiliary variables corresponding to each node of the N nodes, and an initial random gradient of a loss function corresponding to the model of each node of the N nodes; the iteration module is configured to:

[0136] obtain a decision variable of the i-th node based on the initial model of the i-th node and the corresponding input learning step length and the initial random gradient, the decision variable being used to indicate a local model of the i-th node;

[0137] perform communication compression on the decision variable based on the target compression algorithm, the absolute communication compression error control parameter, and the initial decision compression auxiliary variable corresponding to the i-th node, to obtain a compressed decision variable corresponding to the i-th node;

[0138] perform compression information recovery on the compressed decision variable based on the absolute communication compression error control parameter and the initial decision compression auxiliary variable corresponding to the i-th node, to obtain a recovered decision variable;

[0139] perform communication transmission and recovery between nodes on the compressed decision variable based on the absolute communication compression error control parameter, the row random hybrid matrix, and the initial decision compression hybrid auxiliary variable corresponding to the i-th node, to obtain hybrid decision variable information;

[0140] update the initial model of the i-th node based on the decision variable, the recovered decision variable, the hybrid decision variable information, and the consistency step length corresponding to the model of the i-th node, to obtain an updated model of the i-th node.

[0141] In an implementation of the present disclosure, the initialization parameters further include a first relative communication compression error control parameter related to the model of the i-th node; the iteration module is configured to:

[0142] update the initial decision compression auxiliary variable corresponding to the i-th node based on the first relative communication compression error control parameter and the recovered decision variable;

[0143] updating the initial decision compressed auxiliary variable corresponding to the i-th node based on the first relative communication compression error control parameter and the mixed decision variable information;

[0144] wherein the updated initial decision compressed auxiliary variable and the updated initial decision compressed mixed auxiliary variable are used for next updating of the updated model of the i-th node.

[0145] In an implementation manner of the present disclosure, the initialization parameter further comprises a consistent step length of the random gradient corresponding to the N nodes and initial gradient tracking compressed auxiliary variables and initial gradient tracking compressed mixed auxiliary variables corresponding to each of the N nodes; the iteration module is configured to:

[0146] obtaining a local gradient tracking variable based on the random gradient difference and the initial random gradient corresponding to the i-th node, wherein the random gradient difference is obtained according to the random gradient corresponding to the updated model of the i-th node and the random gradient corresponding to the initial model of the i-th node;

[0147] performing communication compression on the gradient tracking variable based on the target compression algorithm, the absolute communication compression error control parameter and the initial gradient tracking compressed auxiliary variable corresponding to the i-th node, to obtain a compressed gradient tracking variable corresponding to the i-th node;

[0148] performing compressed information recovery on the compressed gradient tracking variable based on the absolute communication compression error control parameter and the initial gradient tracking compressed auxiliary variable corresponding to the i-th node, to obtain a recovered gradient tracking variable;

[0149] performing communication transmission and recovery of the compressed gradient tracking variable between nodes based on the absolute communication compression error control parameter, the column random mixing matrix and the initial gradient tracking compressed mixed auxiliary variable corresponding to the i-th node, to obtain mixed gradient tracking variable information;

[0150] updating the initial random gradient corresponding to the i-th node once based on the gradient tracking variable, the recovered gradient tracking variable, the mixed gradient tracking variable information and the consistent step length of the random gradient corresponding to the i-th node, wherein the updated gradient tracking variable corresponding to the i-th node is used for next updating of the updated model of the i-th node.

[0151] In an implementation manner of the present disclosure, the relative communication compression error control parameter comprises a second relative communication compression error control parameter related to the random gradient corresponding to the i-th node; the iteration module is configured to:

[0152] update the initial gradient tracking compressed auxiliary variable corresponding to the i-th node based on the second relative communication compressed error control parameter and the recovered gradient tracking variable;

[0153] update the initial gradient tracking compressed mixed auxiliary variable corresponding to the i-th node based on the second relative communication compressed error control parameter and the mixed gradient tracking variable information;

[0154] wherein the updated initial gradient tracking compressed auxiliary variable and the updated initial gradient tracking compressed mixed auxiliary variable are used for next update of the updated gradient tracking variable corresponding to the i-th node.

[0155] In an implementation of the present disclosure, the first determining module is configured to:

[0156] obtain a convergence rate corresponding to each compression algorithm based on a set of convergence parameters corresponding to each compression algorithm in the at least one compression algorithm;

[0157] determine a compression algorithm corresponding to a maximum convergence rate from the at least one compression algorithm based on the convergence rate of each compression algorithm;

[0158] determine the compression algorithm corresponding to the maximum convergence rate as the target compression algorithm when the compression algorithm corresponding to the maximum convergence rate meets the network bandwidth and the system computing resources.

[0159] According to the technical scheme provided by the embodiments of the present disclosure, the connection relationship between N distributed nodes in a centerless distributed network can be used to obtain a random mixed matrix, the random mixed matrix includes a row random mixed matrix, and a preset initialization parameter and an initial model of an i-th node in the N nodes are obtained. A target compression algorithm is determined from at least one compression algorithm that meets a preset communication compression condition, the preset communication compression condition includes at least one of the following: a relative error compression condition and an absolute error compression condition. The initial model of the i-th node is iteratively trained based on the row random mixed matrix, the initialization parameter and the target compression algorithm. When a training stop condition is met, a trained model of the i-th node is obtained. Through the scheme, a more suitable compression algorithm can be selected under the condition of considering the relative error and the absolute error existing when communication compression, so as to improve the communication efficiency on the premise of reducing the communication cost, and also make the application of the scheme more widely.

[0160] The present disclosure also discloses an electronic device, Figure 3 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0161] As Figure 3As shown, the electronic device includes a memory and a processor, wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to the embodiments of the present disclosure.

[0162] Figure 4 A structural diagram of a computer system suitable for implementing the method according to the embodiments of the present disclosure is shown.

[0163] As Figure 4 shown, the computer system includes a processing unit which can execute various methods in the above embodiments according to programs stored in a read-only memory (ROM) or programs loaded from a storage section into a random access memory (RAM). In the RAM, various programs and data required for the operation of the computer system are also stored. The processing unit, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0164] The following components are connected to the I / O interface: an input section including a keyboard, a mouse, etc.; an output section including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section including a hard disk, etc.; and a communication section including a network interface card such as a LAN card, a modem, etc. The communication section performs communication processes via a network such as the Internet. A drive is also connected to the I / O interface as necessary. A removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive as necessary, so that a computer program read out from the removable medium is installed into the storage section as necessary. The processing unit can be implemented as a CPU, a GPU, a TPU, a FPGA, a NPU, etc.

[0165] In particular, according to the embodiments of the present disclosure, the above-described method can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for executing the above-described method. In such embodiments, the computer program can be downloaded and installed from a network via the communication section, and / or installed from a removable medium.

[0166] The flow and block diagrams in the drawings represent possible architectural, functional, and operational architectures of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block can represent a module, a segment, or a portion of code that comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0167] The units or modules described in the embodiments of the present disclosure can be implemented by software, or by programmable hardware. The described units or modules can also be arranged in a processor, and the names of the units or modules do not constitute a limitation on the units or modules themselves in some cases.

[0168] As another aspect, the present disclosure also provides a computer readable storage medium, which can be the computer readable storage medium included in the electronic device or the computer system in the above embodiments, or can exist separately from the device and not be assembled into the device. The computer readable storage medium stores one or more programs for execution by one or more processors to perform the methods described in the present disclosure.

[0169] The above description is merely preferred embodiments of the present disclosure and a description of principles of applied technologies. It should be understood by those skilled in the art that the scope of the application involved in the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and also covers other technical solutions formed by the arbitrary combinations of the above technical features or equivalent features without departing from the inventive concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features disclosed in the present disclosure (but not limited to) having similar functions.

Claims

1. A decentralized distributed network optimization training method, characterized in that, The method comprises: obtaining a random mixing matrix based on the connection relationship between N distributed nodes in the ad-hoc distributed network, the random mixing matrix comprising a row random mixing matrix and a column random mixing matrix, N being an integer greater than 1; obtaining a preset initialization parameter and an initial model of an i-th node in the N distributed nodes, i being greater than 0 and less than N; determining a target compression algorithm from at least one compression algorithm satisfying a preset communication compression condition, the preset communication compression condition comprising at least one of a relative error compression condition and an absolute error compression condition; wherein the determination of the target compression algorithm from the at least one compression algorithm satisfying the preset communication compression condition comprises: obtaining a convergence rate corresponding to each compression algorithm in the at least one compression algorithm based on a set of convergence parameters corresponding to each compression algorithm; determining a compression algorithm corresponding to a maximum convergence rate from the at least one compression algorithm based on the convergence rate of each compression algorithm; and determining the compression algorithm corresponding to the maximum convergence rate as the target compression algorithm when the compression algorithm corresponding to the maximum convergence rate satisfies a network bandwidth and a system computing resource; iteratively training the initial model of the i-th node based on the random mixing matrix, the initialization parameter and the target compression algorithm; obtaining a trained model of the i-th node when a training stop condition is satisfied.

2. The method of claim 1, wherein, The initialization parameter comprises an input learning step of the i-th node, an absolute communication compression error control parameter, a consistency step corresponding to the model of the i-th node, an initial decision compression auxiliary variable and an initial decision compression mixing auxiliary variable corresponding to each node in the N distributed nodes, and an initial random gradient of a loss function corresponding to the model of each node in the N distributed nodes; The iterative training of the initial model of the i-th node based on the random mixing matrix, the initialization parameter and the target compression algorithm comprises: obtaining a decision variable of the i-th node based on the initial model of the i-th node and the corresponding input learning step and initial random gradient, the decision variable being used to indicate a local model of the i-th node; performing communication compression on the decision variable based on the target compression algorithm, the absolute communication compression error control parameter and the initial decision compression auxiliary variable corresponding to the i-th node to obtain a compressed decision variable corresponding to the i-th node; performing compression information recovery on the compressed decision variable based on the absolute communication compression error control parameter and the initial decision compression auxiliary variable corresponding to the i-th node to obtain a recovered decision variable; performing communication transmission and recovery of the compressed decision variable between nodes based on the absolute communication compression error control parameter, the row random mixing matrix and the initial decision compression mixing auxiliary variable corresponding to the i-th node to obtain mixed decision variable information; updating the initial model of the ith node based on the decision variable, the recovered decision variable, the mixed decision variable information and a consistency step corresponding to the model of the ith node, to obtain an updated model of the ith node.

3. The method of claim 2, wherein, The initialization parameters further include a first relative communication compression error control parameter related to the model of the ith node. The iterative training of the initial model of the ith node based on the random mixing matrix, the initialization parameters and the target compression algorithm includes: updating the initial decision compression auxiliary variable corresponding to the ith node based on the first relative communication compression error control parameter and the recovered decision variable; updating the initial decision compression mixed auxiliary variable corresponding to the ith node based on the first relative communication compression error control parameter and the mixed decision variable information; The updated initial decision compression auxiliary variable and the updated initial decision compression mixed auxiliary variable are used for the next update of the updated model of the ith node.

4. The method of claim 2, wherein, The initialization parameters further include a consistency step of the random gradient corresponding to the N distributed nodes, and an initial gradient tracking compression auxiliary variable and an initial gradient tracking compression mixed auxiliary variable corresponding to each node of the N distributed nodes. The iterative training of the initial model of the ith node based on the random mixing matrix, the initialization parameters and the target compression algorithm includes: obtaining a local gradient tracking variable based on a random gradient difference and the initial random gradient corresponding to the ith node, wherein the random gradient difference is obtained according to a random gradient corresponding to the updated model of the ith node and a random gradient corresponding to the initial model of the ith node; performing communication compression on the gradient tracking variable based on the target compression algorithm, the absolute communication compression error control parameter and the initial gradient tracking compression auxiliary variable corresponding to the ith node, to obtain a compressed gradient tracking variable corresponding to the ith node; performing compression information recovery on the compressed gradient tracking variable based on the absolute communication compression error control parameter and the initial gradient tracking compression auxiliary variable corresponding to the ith node, to obtain a recovered gradient tracking variable; performing communication transmission and recovery between nodes on the compressed gradient tracking variable based on the absolute communication compression error control parameter, the column random mixing matrix and the initial gradient tracking compression mixed auxiliary variable corresponding to the ith node, to obtain mixed gradient tracking variable information; updating the initial random gradient corresponding to the ith node based on the gradient tracking variable, the recovered gradient tracking variable, the mixed gradient tracking variable information and a consistency step of the random gradient corresponding to the ith node, wherein the updated gradient tracking variable corresponding to the ith node is used for the next update of the updated model of the ith node.

5. The method of claim 4, wherein, The initialization parameters further include a second relative communication compression error control parameter related to the random gradient corresponding to the ith node. The initial model of the ith node is iteratively trained based on the random mixing matrix, the initialization parameter and the target compression algorithm, including: updating the initial gradient tracking compression auxiliary variable corresponding to the ith node based on the second relative communication compression error control parameter and the recovered gradient tracking variable; updating the initial gradient tracking compression mixed auxiliary variable corresponding to the ith node based on the second relative communication compression error control parameter and the mixed gradient tracking variable information; wherein the updated initial gradient tracking compression auxiliary variable and the updated initial gradient tracking compression mixed auxiliary variable are used to update the updated gradient tracking variable corresponding to the ith node next time.

6. A device for training optimization of a central-less distributed network, characterized in that, The device comprises: The first acquisition module is configured to acquire a random mixing matrix based on the connection relationship between N distributed nodes in the non-central distributed network, the random mixing matrix comprising a row random mixing matrix and a column random mixing matrix, N being an integer greater than 1; The second acquisition module is configured to acquire a preset initialization parameter and acquire an initial model of an ith node in the N distributed nodes, i being greater than 0 and less than N; The first determination module is configured to determine a target compression algorithm from at least one compression algorithm satisfying a preset communication compression condition, the preset communication compression condition comprising at least one of the following: a relative error compression condition, an absolute error compression condition; wherein the determination of the target compression algorithm from the at least one compression algorithm satisfying the preset communication compression condition comprises: obtaining a convergence rate corresponding to each compression algorithm in the at least one compression algorithm based on a set of convergence parameters corresponding to each compression algorithm; determining a compression algorithm corresponding to a maximum convergence rate from the at least one compression algorithm based on the convergence rate of each compression algorithm; when the compression algorithm corresponding to the maximum convergence rate satisfies the network bandwidth and the system computing resources, determining the compression algorithm corresponding to the maximum convergence rate as the target compression algorithm; The iteration module is configured to iteratively train the initial model of the ith node based on the random mixing matrix, the initialization parameter and the target compression algorithm; The second determination module is configured to obtain a trained model of the ith node when a training stop condition is satisfied.

7. An electronic device, comprising: The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps of any one of claims 1 to 5.

8. A computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are executed by the processor to implement the method steps of any one of claims 1 to 5. The computer instructions are executed by the processor to implement the method steps of any one of claims 1 to 5.

Images