A distributed machine learning method for a heterogeneous scene

Abstract

The present application belongs to the field of information technology, and specifically provides a distributed machine learning method for heterogeneous scenarios, aiming to solve the problems of the existing federated learning efficiency improvement methods, such as the non-independent and identically distributed data widely existing in the distributed scenarios, the large calculation burden, the dependence on experience and the like. The present application realizes the dynamic selection of the participant nodes by dynamically updating the related matrix in the training process, effectively incorporates the non-independent and identically distributed data existing in the distributed environment into the node selection consideration, and also considers the training effect and historical training information of the nodes, so that the model can learn unknown knowledge or knowledge with poor learning effect more quickly, thereby effectively improving the model training efficiency of the federated learning in the actual application scenarios.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of information technology, mainly aiming at the distributed multi-party joint modeling scene in the heterogeneous scene, and in this scene, data sharing is not allowed due to privacy security problems, so it is a typical federated learning scene. For the above-mentioned scene, the present application proposes an efficient federated learning method based on heterogeneity. BACKGROUND

[0002] Information technology is a general term for various technologies used to manage and process information, and distributed machine learning is one of the efficient and secure technologies. Distributed machine learning refers to a system that uses multiple computing nodes to collaboratively train a global machine learning model, aiming to improve performance and be scalable to larger training data and larger models. As a method under the distributed machine learning framework that focuses on privacy protection, federated learning integrates multi-party data information through differential privacy, homomorphic encryption and other technologies to collaboratively build a global model. Although federated learning can effectively deal with common data security problems in distributed scenarios, it also faces some challenges. In federated learning, the local data owned by each node is not leaked to other nodes or the central server, and the construction of the global model depends on the local model trained by each participant node itself, and its efficiency is affected by various factors. For example, the communication bandwidth limits the exchange between participants, the down participant node affects the model training, and the non-identically distributed data makes it difficult for the global model to converge.

[0003] The existing methods for improving the efficiency of federated learning can be mainly divided into two categories: one is to reduce the overhead of federated learning, and the other is to customize the selection strategy of participant nodes; although both methods can improve the efficiency of federated learning to some extent, due to some defects of the existing methods, such as lack of consideration of non-identically distributed data existing in actual application scenarios, the need to solve complex multi-objective optimization problems, etc., the actual application effect is not ideal, and it is difficult to meet the actual use demand.

[0004] Methods for reducing the overhead of federated learning can be divided into two aspects: reducing communication consumption and reducing resource consumption. In terms of reducing communication consumption, existing research has applied model compression techniques to reduce the size of model parameters transmitted during federated learning, thereby reducing communication consumption. In terms of reducing resource consumption, multi-objective optimization methods are applied to consider factors such as network conditions, computing capacity, and resource consumption of distributed nodes, thereby minimizing resource consumption for model training in federated learning. However, the above methods do not work well in actual scenarios, mainly due to two reasons. First, non-identically distributed data widely exists in actual scenarios, making the model training process of federated learning unstable. Existing methods are mostly based on the assumption of identical distribution, so unexpected negative effects may occur when they are applied in actual scenarios. Second, model compression and multi-objective optimization increase the computational burden of nodes and centers, making it difficult for them to effectively function in resource-constrained scenarios.

[0005] Methods for customizing the selection strategy of participant nodes. Existing research mainly improves the efficiency of federated learning by selecting participant nodes through a selection strategy, which enables the federated learning model to learn global data knowledge more quickly, thereby accelerating the convergence of the federated model and improving the efficiency of federated learning. In addition, some research has introduced reinforcement learning and added prior knowledge to enable the reinforcement learning model to intelligently select participants. However, this method also has some problems, such as high additional computational burden and the need to introduce experience, making it difficult to be widely applied in actual scenarios. SUMMARY

[0006] The present invention aims to address the shortcomings of existing technologies by providing a distributed machine learning method for heterogeneous scenarios, which aims to solve the problems of existing federated learning efficiency improvement methods, such as non-identically distributed data widely existing in distributed scenarios, high computational burden, and dependence on experience, thereby effectively improving the efficiency of federated learning applications in actual scenarios.

[0007] To achieve the above-mentioned purpose, the technical solutions adopted by the present invention are as follows:

[0008] A distributed machine learning method for heterogeneous scenarios, comprising the following steps:

[0009] Step S1, based on the scenario of distributed machine learning, each distributed computing node with system heterogeneity and data heterogeneity, the center initializes a global model, a probability matrix, a cumulative model intensity matrix, and a count matrix;

[0010] Step S2, the center randomly selects nodes with a preset selection fixed C as participant nodes to participate in training based on the probability matrix;

[0011] Step S3: Extract key node information of participating nodes. Key node information includes: data non-independent and identically distributed degree index and training loss.

[0012] Step S4: Based on the key information of the nodes, calculate the cumulative model strength value for each node and update the cumulative model strength matrix;

[0013] Step S5: Update the counting matrix based on the node selection information in step S2;

[0014] Step S6: Update the probability matrix based on the node key information and the counting matrix.

[0015] Furthermore, in step S1, the probability matrix is ​​represented as Q = {q1, q2, ..., q}. n}, q i Let be the probability that node i is selected to participate in the joint model training, i = 1, 2, ..., n, where n is the number of nodes; initialized so that each value in the matrix is ​​equal.

[0016] The cumulative model strength matrix is ​​represented as H = {h} (1) ,h (2) ,...,h (n)}, h (i) This is an indicator representing the degree of data heterogeneity and training performance of node i; it is initialized so that each value in the matrix is ​​0.

[0017] The counting matrix is ​​represented as cnt = {c1, c2, ..., c n}, c i This represents the number of times node i has been selected consecutively, initialized so that each value in the matrix is ​​0.

[0018] Furthermore, in step S3, the index emd of the degree of non-independent and identically distributed data at node i is... i Represented as:

[0019]

[0020] Where P represents the global data distribution, p i Denotes the distribution of local training data at node i, ∏(P,p) i ) represents P and p i The set of joint distributions, where γ is a subset of Π(P,p). i A joint distribution of γ, where x and y represent samples in the joint distribution γ;

[0021] The loss during the training process of node i is recorded and represented as the training loss. i .

[0022] Furthermore, in step S4, the specific steps are as follows:

[0023] Step 401, for the participant node i, the node calculates the current model strength value m i is:

[0024]

[0025] wherein emd i represents the data non-independent identically distributed degree index of the node i, loss i represents the training loss value of the node i in the current round of training;

[0026] Step 402, calculate the cumulative model strength value is:

[0027]

[0028] wherein, represents the cumulative model strength value of the node i after the current round of training, represents the cumulative model strength value of the node i after the last round of training, and p is the decay coefficient;

[0029] After each participant node calculates the cumulative model strength value, it is uploaded to the center, and the center updates the cumulative model strength matrix.

[0030] Further, in step S5, in each round of training, the corresponding count value of the participant node selected continuously in the count matrix is increased by 1, and the corresponding count value of the remaining nodes is reduced by 1 (until 0); if the calculation value of the node is greater than the preset threshold value, it is determined as a poor node, otherwise, it is a good node.

[0031] Further, in step S6, specifically:

[0032] Step S601, according to the node selection information in step S2, update the probability value of each node;

[0033] For the participant node i, the probability value is updated as:

[0034]

[0035] wherein, represents the probability value of the node i after the current round of training, represents the probability value of the node i after the last round of training, and e is a preset non-negative value;

[0036] For the remaining nodes j, the probability value is updated as:

[0037]

[0038] Step S602, according to the count matrix, the corresponding probability value of the poor node in the probability matrix is set to zero;

[0039] Step S603, normalize the probability matrix.

[0040] Based on the above technical solutions, the present application has the following beneficial effects:

[0041] Based on the data non-independent and identically distributed degree measurement method and training loss, the present application provides an efficient federated learning method for dynamically selecting participant nodes, aiming at the phenomenon that the "poor" participant nodes affect the model performance in the distributed machine scene. By dynamically updating the related matrix during the training process, the dynamic selection of participant nodes is realized, and the non-independent and identically distributed data existing in the distributed environment is effectively included in the node selection consideration. At the same time, the training effect and historical training information of the node are also considered, so that the model can learn unknown knowledge or knowledge with poor learning effect more quickly, thereby effectively improving the model training efficiency of federated learning in actual application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The figure is a flowchart of the distributed machine learning method in the present application for heterogeneous scenarios.

[0043] Figure 2 The figure is a flowchart of the cumulative model strength matrix update process in the present application.

[0044] Figure 3 The figure is a flowchart of the count matrix update process in the present application.

[0045] Figure 4 The figure is a flowchart of the probability matrix update process in the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to the embodiments and drawings.

[0047] The present embodiment provides an efficient federated learning method based on data heterogeneity and training loss applied to distributed machine learning scenarios, i.e. a distributed machine learning method for heterogeneous scenarios, as shown in Figure 1 The specific steps include the following steps:

[0048] Step S1, initialization; based on the scene of distributed machine learning, each distributed computing node with system heterogeneity and data heterogeneity, the center initializes the global model, the probability matrix, the cumulative model strength matrix and the count matrix, while each node initializes its own cumulative model strength value;

[0049] This invention mainly revolves around the above three matrices to realize the dynamic selection of participants in the federated learning training process. The matrices are dynamically updated after each round of training, so that the model can learn unknown and poorly learned data knowledge as quickly as possible, thereby accelerating the convergence process of the federated learning model and improving the efficiency of federated learning.

[0050] For a federated learning training task scenario, the number of all distributed nodes that can participate in the training task is denoted as n. The above three key matrices are initialized: the cumulative model strength matrix, the probability matrix, and the counting matrix, with a matrix size of 1×n.

[0051] The probability matrix is ​​represented as Q = {q1, q2, ..., q}. n} represents the probability that each node is selected to participate in the joint model training. After each round of model fusion in federated learning, it is updated based on the cumulative model strength matrix. The initial matrix has equal values, representing random selection.

[0052] The cumulative model strength matrix is ​​represented as H = {h} (1) ,h (2) ,...,h (n)} represents the degree of heterogeneity of data and training effect of each node. If the degree of non-independent and identically distributed data is stronger or the training effect is worse, the cumulative model strength value is smaller, and vice versa. This matrix will be updated after each round of training. Each value in the initial matrix is ​​0.

[0053] The counting matrix is ​​represented as cnt = {c1, c2, ..., c n}, c i This matrix represents the number of times node i has been selected consecutively. It is mainly used to remove nodes with poor data quality, i.e., to exclude them from subsequent training. Each value in the matrix is ​​initialized to 0.

[0054] Step S2: Joint Model Training; The center randomly selects a fixed proportion C (0 < C ≤ 1) of nodes as participating nodes based on the probability matrix to participate in the training; that is, the global model is distributed to the participating nodes, and the participating nodes use local data to update the model.

[0055] Step S3: Extract key node information of participating nodes; key node information includes: data non-independent and identically distributed degree index and training loss;

[0056] Specifically, for the index of the degree of non-independent and identically distributed data, the earthwork haulage distance between the local data distribution of the participating node and the global data distribution P is calculated based on the local data distribution of the participating node, which is used to represent the degree of non-independent and identically distributed local training data at the node; for participating node i, it is assumed that the distribution of its local training data is p. iThe data non-independent and identically distributed degree index on the node is represented as:

[0057]

[0058] Wherein, P represents a global data distribution, p i represents a distribution of local training data of the node i, Π(P, p i ) represents a set of joint distributions of P and p i , γ is a joint distribution belonging to ∏(P, p i ), x and y represent samples in the joint distribution γ, and E (x,y)～γ [||x-y||2] represents an expected value of the distance between the samples x and y in the joint distribution γ.

[0059] For the training loss, the loss in the current training process of the node is recorded, and for the participant node i, it is represented as Loss i .

[0060] Step S4, updating the cumulative model strength matrix; based on the node key information, each node updates the cumulative model strength value, and the corresponding value in the cumulative model strength matrix is updated;

[0061] The cumulative model strength value is determined by the historical cumulative model strength value, the data non-independent and identically distributed degree on the node, and the current training effect, that is, the cumulative model strength of the node is positively correlated with the training effect, and is negatively correlated with the data non-independent and identically distributed degree, the better the training effect of the node, the lower the data non-independent and identically distributed degree, and the higher the cumulative model strength, the cumulative model strength value reflects the comprehensive effect of the model on the node; as shown in the following formula, the update of the cumulative model strength matrix is divided into the following steps: Figure 2

[0062] Step 401, calculating the current model strength value; the node calculates the current model strength value, and for the node i, the current model strength value m i is:

[0063]

[0064] Wherein, emd i represents the data non-independent and identically distributed degree index of the node i, and loss i represents the training loss value of the node i in the current training.

[0065] Step 402, updating the cumulative model strength matrix; the cumulative model strength value of the node is updated by taking the current model strength value of the node and the historical cumulative model strength value as input data, and the calculation process applies the idea of exponential average, which is widely used in the training optimization method of deep learning; for the node i, the cumulative model strength value is:​

[0066]

[0067] Where t represents the time when the current training round is completed, and t-1 represents the time when the previous training round is completed; ρ is the decay coefficient, 0≤ρ<1, which represents the degree of retention of historical cumulative model strength information;

[0068] After each participating node updates its own cumulative model strength value, it uploads it to the center. The center then updates the corresponding values ​​of each node in the cumulative model strength matrix to obtain the latest cumulative model strength matrix H. t That is, if node i is a participating node, then the cumulative model strength matrix H is calculated after training. t middle The value will then be updated;

[0069] Step S5: Update the counting matrix; the counting matrix is ​​positively correlated with the number of times a node is selected. If a node is continuously selected for training, it indicates that the model performs poorly at that node, and the data is highly non-independent and identically distributed. Furthermore, if it exceeds a certain threshold, it indicates that the data quality of that node is likely poor and it should not be selected for subsequent training. Figure 3 As shown, the update of the counting matrix involves the following steps:

[0070] Based on the node selection information in step S2, in each round of training, the corresponding count value of the participating node that is continuously selected in the counting matrix is ​​incremented by 1, and the corresponding count value of the other nodes is decremented by 1 (until 0); if the calculated value of a node is greater than the preset threshold, it is determined to be a poor node, otherwise, it is a good node.

[0071] Step S6: Update the probability matrix; Update the probability matrix based on the cumulative model strength matrix. The probability of a node being selected is determined by its cumulative model strength, meaning the selection probability is negatively correlated with the cumulative model strength. A higher cumulative model strength value indicates a better performance of the model at that node, meaning it may not participate in the current training, thus reducing its selection probability. Conversely, a lower cumulative model strength value indicates a poor performance of the model at that node, or a high degree of non-independent and identically distributed data, thus increasing the probability of it being selected for training. Figure 3 As shown, the update of the probability matrix consists of the following steps:

[0072] Step S601: Match the node selection information in step S2 and update the probability value of each node;

[0073] For participating nodes (those that participated in the previous training round), such as node i, the probability value is updated as follows:

[0074]

[0075] wherein, Pi,j represents the probability value of node i after the current round of training, Pi-1,j represents the probability value of node i after the last round of training, and ε is a preset non-negative value to avoid the denominator being 0;

[0076] The probability value of the remaining nodes (not participating in the last round of training), such as node j, is updated as:

[0077]

[0078] Step S602: deleting the poor nodes, i.e. setting the corresponding probability values of the poor nodes in the probability matrix to zero;

[0079] Step S603: normalizing the updated probability matrix to ensure that the sum of all probability values in the probability matrix is 1.

[0080] The process of steps S2 to S6 is repeated until a preset ending condition is met, such as model convergence or a specified number of training rounds is reached, i.e. the training is completed.

[0081] The above is only a specific implementation of the present application, and any feature disclosed in the specification can be replaced by other equivalent or similar purpose features unless specifically described; all features disclosed or all steps in the method or process can be combined in any way, except for mutually exclusive features and / or steps.

Claims

1. A distributed machine learning method for heterogeneous scenarios, applied to a distributed machine learning system, the distributed machine learning system comprising a central server and n distributed computing nodes, wherein the computing nodes interact with the central server, comprising the following steps, Step S1: In a distributed machine learning scenario, for each distributed computing node with system heterogeneity and data heterogeneity, the central system initializes the global model, probability matrix, cumulative model strength matrix, and counting matrix. The probability matrix is ​​represented as Q = {q1, q2, ..., q} n }, q i Let be the probability that node i is selected to participate in the joint model training, i = 1, 2, ..., n, where n is the number of nodes; initialized so that each value in the matrix is ​​equal. The cumulative model strength matrix is ​​represented as H = {h} (1) ,h (2) ,...,h (n) }, h (i) This is an indicator representing the degree of data heterogeneity and training performance of node i; it is initialized so that each value in the matrix is ​​0. The counting matrix is ​​represented as cnt = {c1, c2, ..., c n }, c i This represents the number of times node i has been selected consecutively, initialized to 0 for each value in the matrix. Step S2: The center randomly selects a fixed proportion C of nodes based on the probability matrix to participate in the training. Step S3: Extract key node information from participating nodes. Key node information includes: Indices of the degree of non-independent and identically distributed data and training loss; The index of the degree of non-independence and identical distribution of data at node i: emd i Represented as: Where P represents the global data distribution, p i Π(P,p) represents the distribution of local training data at node i. i ) represents P and p i The set of joint distributions, where γ is a subset of Π(P,p). i A joint distribution of γ, where x and y represent samples in the joint distribution γ; The loss during the training process of node i is recorded and represented as the training loss. i ; Step S4: Based on the key information of each node, calculate the cumulative model strength value for each node and update the cumulative model strength matrix; specifically: Step 401: For participant node i, the node calculates the current model strength value m. i for: Among them, emd i The loss function represents the degree to which the data at node i is not independently and identically distributed. i This represents the training loss value of node i in this round of training; Step 402: Calculate the cumulative model strength value for: in, This represents the cumulative model strength value of node i after this round of training. This represents the cumulative model strength value after the previous training round for node i, where ρ is the decay coefficient; Each participating node calculates the cumulative model strength value and uploads it to the center, which then updates the cumulative model strength matrix. Step S5: Update the counting matrix based on the node selection information in step S2; In each round of training, the corresponding count value of the participating node that is selected consecutively in the counting matrix is ​​incremented by 1, and the corresponding count value of the other nodes is decremented by 1 until 0; if the calculated value of a node is greater than a preset threshold, it is judged as a poor node, otherwise, it is a good node. Step S6: Update the probability matrix based on the node key information and the counting matrix; specifically: Step S601: Update the probability value of each node according to the node selection information in step S2; For participant node i, the probability value is updated as follows: in, This represents the probability value of node i after this round of training. ε represents the probability value of node i after the previous training round, and ε is a preset non-negative value; For the remaining node j, the probability value is updated as follows: Step S602: Based on the counting matrix, set the probability values ​​of the inferior nodes in the probability matrix to zero; Step S603: Normalize the probability matrix.

Images