Privacy protection algorithm and device based on efficient sparse communication

Abstract

The application provides a privacy protection algorithm and device based on efficient sparse communication, the algorithm comprising a distributed ADMM algorithm based on an LBSA communication model, which comprises the following steps: S1, each node reads a respective data set, counts associated features, and adds the associated features to a set Ψ i ; S2, all nodes perform an allgather operation, store associated features of all nodes into a two-dimensional vector ARI, count a weight W of each feature value by using a value of the ARI, and divide the feature values according to the weight; S3, after the associated features are recorded, each node sequentially updates local variables x i , dual variables y i , local model parameters χ i , and aggregates local model parameters χ i of all nodes; and S4, after the local model parameters χ of all nodes are aggregated by using a load-balanced sparse allreduce algorithm, a global variable z is updated. In the algorithm, sparse vectors are used for calculation and communication, which can greatly reduce a single-node calculation cost and a communication load.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a privacy protection algorithm and device based on efficient sparse communication. Background Technology

[0002] In the era of big data, many machine learning applications need to solve optimization problems involving billions of variables on massive training data. Therefore, using multi-node parallelism to solve large-scale machine learning applications has become a natural choice. Among the many parallelism methods, data parallelism is the most commonly used in machine learning. When training machine learning models based on data parallelism, each computing node stores the model parameters and performs local computations using local data. After completing the local computations, the allreduce operation is usually used to aggregate the parameters from each computing node. When exchanging parameters, the number of parameters transferred between nodes is usually the same as the number of features in the dataset and model. As the number of model features increases, the communication overhead between nodes becomes one of the main factors limiting the scalability of distributed machine learning algorithms. At the same time, the exchange of parameters between nodes also introduces the risk of data leakage.

[0003] I. Distributed ADMM for Large-Scale Machine Learning

[0004] When using distributed nodes to solve large-scale machine learning problems, the large-scale problem can usually be transformed into a consistency optimization problem as shown in equation (1).

[0005]

[0006] in Represents a dataset, Let m be the model parameters, p be the number of datasets, and p be the number of dataset features. Global consistency optimization introduces a local variable to replace the model parameters for each node, and maintains consistency among the computation nodes by constructing constraint terms for the global and local variables, thus transforming equation (1) into the global consistency problem shown in equation (2).

[0007]

[0008] The augmented Lagrangian function of equation (2) can be expressed as:

[0009]

[0010] Where x i z is a local variable, y is a global variable. i For x i The dual variable is then the iterative formula for solving (2) based on the ADMM algorithm is:

[0011]

[0012] From equations (4)-(6), it can be seen that all x i It can be solved in parallel, and all y i It can also be solved in parallel. Before updating the global variable z, all x values ​​need to be aggregated. i -y i / ρ。x i Solving x is usually also an optimization problem, called solving subproblems. i The computational complexity is related to the number of features (p) in the dataset. For example, when using exact algorithms to solve problems such as Logistic Regression and Lasso, the computational complexity is O(p). 3 When solving for global variables, the `allreduce` operation aggregates the parameters. In this case, the communication load between nodes is typically linearly related to `p`. Therefore, as `p` increases, reducing the computational and communication load on each node is crucial for improving algorithm performance. Utilizing sparse vectors for computation and communication is one of the most direct and effective methods, but the scalability of sparse `Allreuce` is not ideal.

[0013] II. Scalability issues of sparse allreduce

[0014] In some application scenarios (such as text classification), the dataset has the following characteristics: (1) the frequency of occurrence of each feature in the dataset varies greatly, with only a few features appearing in all datasets and other features appearing only in a few datasets; (2) the dataset processed by each node contains only a portion of the features of the entire dataset. For example Figure 1 The frequency distribution of features in two public datasets is shown.

[0015] Sparse allreduce typically consists of two stages: split-reduce and allgather. Figure 2 The figure shows a typical implementation of sparse allreduce. When transmitting sparse data, it is necessary to transmit the characteristics and the index values ​​corresponding to the characteristics. Due to the special nature of sparse data, the sparse allreduce algorithm may have two problems: (1) the communication load of each node is unbalanced in the split-reduce stage, such as Figure 2 As shown, the worker node Worker0 receives the most parameters in each round of communication; (2) Although the data sent by each node is sparse in the split-reduce stage, after the split-reduce stage is completed, the data that each node is responsible for aggregating is transformed into dense data. Figure 2 As shown, in the allgather phase, dense data needs to be transferred between nodes. Due to the two problems mentioned above, sparse allreduce has poor scalability. Summary of the Invention

[0016] In view of the above, the present invention provides a privacy protection algorithm and apparatus based on efficient sparse communication to reduce the computational cost and communication load of a single node.

[0017] The technical solution of the present invention:

[0018] In a first aspect, the present invention provides a privacy protection algorithm based on efficient sparse communication, including a distributed ADMM algorithm based on the LBSA communication model, comprising:

[0019] Step S1: Each node reads its own dataset, calculates its correlation features, and adds them to set Ψ. i middle;

[0020] Step S2: Perform the allgather operation on all nodes, store the associated features of all nodes in the two-dimensional vector ARI, use the values ​​of ARI to calculate the weight W of each feature value, and divide the feature values ​​according to the weights;

[0021] Step S3: After recording the association features, each node updates its local variable x sequentially using local data. i dual variable y i Local model parameter χ i Aggregate the local model parameters χ of all nodes i ;

[0022] Step S4: After aggregating the local model parameters of all nodes using the load-balanced sparse allreduce algorithm, update the global variable z.

[0023] Secondly, the present invention provides a privacy protection device based on efficient sparse communication, comprising:

[0024] Data reading and feature statistics module: This module is used by each node to read its own dataset, count its associated features, and add them to the set Ψ. i middle;

[0025] Weighting and Feature Value Partitioning Module: This module performs the allgather operation on all nodes, stores the associated features of all nodes in a two-dimensional vector ARI, uses the values ​​of ARI to calculate the weight W of each feature value, and partitions the feature values ​​according to the weights.

[0026] Local variable update module: After recording the association features, each node updates its local variable x sequentially using local data. i dual variable y i Local model parameter χ i Aggregate the local model parameters χ of all nodes i ;

[0027] The parameter aggregation and global variable update module is used to aggregate the local model parameters of all nodes using the load-balanced sparse allreduce algorithm, and then update the global variable z.

[0028] The beneficial effects of this invention are:

[0029] In this algorithm, each computation node first determines its correlation characteristics based on the distribution pattern of its processed dataset, and then uses these correlation characteristics to replace all other characteristics for computation and communication. That is, the algorithm uses sparse vectors for computation and communication, which can significantly reduce the computational cost and communication load of a single node.

[0030] To address the issue of high computational and communication overhead for machine learning models when processing large-scale sparse data, this paper proposes a general consensus distributed ADMM algorithm based on the LBSA communication model, building upon the distributed ADMM algorithm. This algorithm utilizes sparse vectors for computation and communication, effectively reducing the computational and communication load on a single node. Furthermore, this paper designs a sparse allreduce communication model—LBSA—based on the distribution characteristics of sparse data. This communication algorithm effectively balances the communication load between nodes, improving its scalability. To enhance the security of the LBSA-ADMM algorithm, this paper proposes a general consensus ADMM algorithm based on differential privacy—DP-LBSA-ADMM. This algorithm improves data security by adding a perturbation term to the penalty term parameter and dynamically adjusts the penalty term parameter to balance algorithm security and accuracy. Experimental results show that compared with other distributed ADMM algorithms, LBSA-ADMM and DP-LBSA-ADMM effectively reduce computational and communication overhead, while DP-LBSA-ADMM improves data security with minimal impact on algorithm accuracy.

[0031] The LBSA algorithm proposed in this paper reduces the communication time between nodes as the number of working nodes increases, demonstrating excellent scalability. This algorithm can also be extended to other distributed algorithms, and further research will be conducted to apply this communication model to other distributed machine learning algorithms.

[0032] The preferred embodiments of the present invention and their beneficial effects will be further described in detail with reference to specific implementation methods. Attached Figure Description

[0033] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but should not be construed as limiting the invention. In the drawings:

[0034] Figure 1 This is a frequency statistics of URL and WebSpam feature values;

[0035] Figure 2 This is a schematic diagram of sparse allreduce communication;

[0036] Figure 3 This is a flowchart of a privacy protection algorithm based on efficient sparse communication;

[0037] Figure 4 This is a schematic diagram of LBSA communication (N=4, p=24);

[0038] Figure 5 This is a comparison chart of the communication volume of different communication algorithms in a single communication session;

[0039] Figure 6 It shows the speedup and accuracy changes of different algorithms;

[0040] Figure 7 These are the convergence curves of different distributed ADMM algorithms;

[0041] Figure 8 This is a schematic diagram of a privacy protection device based on efficient sparse communication. Detailed Implementation

[0042] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0043] I. Algorithm Description

[0044] Please see Figure 3 This invention provides a privacy protection algorithm based on efficient sparse communication, including a distributed ADMM algorithm based on the LBSA communication model (abbreviated as LBSA-ADMM algorithm), which includes:

[0045] Step S1: Each node reads its own dataset, calculates its correlation features, and adds them to set Ψ. i This step involves preliminary data processing and feature extraction, laying the foundation for subsequent data processing.

[0046] Step S2: Perform the allgather operation on all nodes, store the associated features of all nodes in the two-dimensional vector ARI, use the values ​​of ARI to calculate the weight W of each feature value, and divide the feature values ​​according to the weights; this step achieves optimized selection of feature values ​​by calculating the weights of feature values, thus improving the efficiency of data processing.

[0047] Step S3: After recording the association features, each node updates its local variable x sequentially using local data. i dual variable y i Local model parameter χ iAggregate the local model parameters χ of all nodes i ;

[0048] Step S4: After aggregating the local model parameters of all nodes using the load-balanced sparse allreduce algorithm, update the global variable z. This step achieves distributed data processing and global optimization through local and global variable updates.

[0049] The detailed algorithm description is shown in Algorithm 1.

[0050]

[0051]

[0052] In this algorithm, features are evenly distributed according to their feature value distribution, ensuring a relatively balanced load for each node in each communication. Specifically, in the split-reduce phase, each node sends K / N sparse data points to other nodes. Therefore, the communication time is:

[0053] T balance-splitreduce = (N-1)α + 2Kβ(N-1) / N (16)

[0054] α represents the delay time of one communication, β represents the time required to transmit one byte of data, and K is the number of sparse data.

[0055] In the allgather phase, each node is responsible for distributing the reduced parameters to other nodes. This phase only requires sending the associated feature values ​​of each node. This phase can essentially be seen as the reverse process of the split-reduce phase, with the communication time being the same as the split-reduce phase.

[0056] T related-scatter =(N-1)α+2Kβ(N-1) / N (17)

[0057] Because each node sends its associated feature values ​​to other working nodes during the Allgather phase of this algorithm, each node needs to store the associated model parameters of other nodes. Compared with other ADMM algorithms, this algorithm requires an additional M... re Bytes of memory overhead.

[0058]

[0059] B key This indicates the number of bytes used to store sparse data index values.

[0060] In this algorithm, the update of local variables is only related to the associated parameters, therefore the computational complexity is O(K). i ).

[0061] In this algorithm, each computation node first determines its correlation characteristics based on the distribution pattern of its processed dataset, and then uses these correlation characteristics to replace all other characteristics for computation and communication. That is, the algorithm uses sparse vectors for computation and communication, which can significantly reduce the computational cost and communication load of a single node.

[0062] In step S3, when using distributed nodes to solve large-scale machine learning problems, the large-scale problem can usually be transformed into a consistency optimization problem as shown in equation (1):

[0063]

[0064] in Represents a dataset, Here, m represents the number of datasets, and p represents the number of features in the dataset.

[0065] Global consistency optimization introduces a local variable to replace the model parameters for each node, and ensures consistency among computational nodes by constructing constraint terms for both global and local variables. This transforms equation (1) into the global consistency problem shown in equation (2):

[0066]

[0067] This algorithm is used to solve the large-scale machine learning problem shown in equation (1). For application scenarios where the association characteristics of each node are fixed, such as text classification and social networks, equation (2) can be further transformed into solving the general consistency problem shown in equation (7):

[0068]

[0069] Based on the ADMM algorithm framework, the iterative formula for solving equation (7) is as follows:

[0070]

[0071] Where, x i Let y be a local variable of the i-th node. i For x i The dual variable, χ i These are intermediate variables, called local model parameters, Ψ i Θ represents the set of features associated with the i-th working node. j W represents the set of working nodes associated with the j-th feature. j Let represent the weight of the j-th feature, and ρ be the penalty term parameter. For x i Related global variables.

[0072] As shown in Equation (10), updating global variables involves two key steps: first, aggregating the features associated with the local model parameters; and second, calculating the weight W of each feature. j Clearly, the sparse allreduce algorithm can be used to aggregate the correlation features of each node. To calculate W... j First, when reading data, count the frequency of each feature and add features with a frequency greater than 0 to set Ψ. i In the middle; then for all Ψ i Performing an allgather operation yields W. j It should be noted that W j It only needs to be calculated once.

[0073] To address the two problems existing in the sparse allreduce algorithm, this invention proposes a privacy-preserving algorithm based on efficient sparse communication, which includes a load-balanced sparse allreduce algorithm (LBSA algorithm for short): In step S4, it is necessary to aggregate the local model parameters of all nodes. In the split-reduce stage, the feature values ​​of the local model parameters are divided according to the distribution of the associated feature values ​​of each working node. The division rule is shown in equation (13):

[0074]

[0075] Split[i] represents the starting value of the i-th interval, that is, the range of feature values ​​that the i-th (0≤i<N-1) working node is responsible for aggregating is [Split[i], Split[i+1]-1], and the range of feature values ​​that the (N-1)-th working node is responsible for aggregating is [Split[N-2], p]. This ensures that each node is responsible for roughly the same number of parameters for reduction, thus balancing the communication load of each node; during the allgather phase, each node broadcasts its associated feature values ​​to other nodes. Figure 4 The diagram shows the inter-node communication of the LBSA algorithm when the number of working nodes N is 4 and the number of feature values ​​p is 24.

[0076] In step S3, in the distributed ADMM algorithm for large-scale machine learning, updating local variables requires the participation of all features in the computation. However, in the general consistency ADMM algorithm, only the correlation features need to be used to update local variables. Therefore, sparse vector computation can be used when solving for local variables to reduce the computational load. The update formulas for local variable x and dual variable y can be further modified as shown in equations (14) and (15):

[0077]

[0078] The convergence of the distributed ADMM algorithm is highly sensitive to the value of the penalty term parameter. BSHe et al. proved that when the initial residual r...k Much larger than the dual residual s k When the dual variable updates slowly, the penalty term parameter needs to be increased to improve the algorithm speed, and vice versa. This invention's privacy-preserving algorithm based on efficient sparse communication includes a privacy-preserving general consensus ADMM algorithm (DP-LBSA-ADMM algorithm). In step S3, each node independently adds a perturbation term to the penalty term parameter. ( A random number greater than 1, whose value is determined by a distributed function. (Randomly generated), and the penalty term parameter is adjusted according to the values ​​of the dual residual and the initial residual. The update rules for the dual residual, penalty term parameter, local variables, local model parameters and initial residual are shown in equations (19)-(21), and the update formulas for other variables are the same as those for the distributed ADMM algorithm based on the LBSA communication model.

[0079]

[0080] in Let i be the dual residual of the i-th node. Let z be the penalty term parameter for the i-th node, and z be a global variable. The perturbation term for the i-th node Let be the initial residual of the i-th node.

[0081] The detailed algorithm description is shown in Algorithm 2.

[0082]

[0083]

[0084] II. Theoretical Analysis of DP-LBSA-ADMM Algorithm

[0085] In the DP-LBSA-ADMM algorithm, the penalty term parameters of each node change dynamically. This section proves the convergence of the algorithm. Symbol explanation: I p Denotes the p×p identity matrix. Let represent the diagonal matrix of the penalty term parameters at the k-th iteration. Let Define Y = (x; z) ∈ R 2p Z = (x; z; y) ∈ R 3p Y * and Z * Let ΔZ represent the solutions for the original variable and the dual variable, respectively. k =Z k+1 -Z k ,

[0086]

[0087] Based on the above definition, ΔZ k This can be expressed as equation (24). The update formula of the DP-LBSA-ADMM algorithm can be transformed into equations (25)-(29).

[0088]

[0089] r k =x k -z k (25)

[0090] s k =T k (z k -z k-1 (26)

[0091]

[0092] y k+1 :=y k +T k (z k+1 -x k+1 (29)

[0093] Since F(Z) is a monotonic function, then we have Z′, (ZZ′)(F(Z)-F(Z′))≥0, further we can obtain Z * It satisfies the expression shown in equation (30).

[0094]

[0095] Similarly, Y can be derived. k+1 Satisfy equations (31) and (32).

[0096]

[0097] From (31) and (32), we can obtain:

[0098]

[0099] Lemma 1: Optimal solution Z * =(x * ;z * ;y * ) and sequence Z k =(x k ;z k ;y k )satisfy

[0100] (zk+1 -z k ) T (y k -y k+1 )≥0 (34)

[0101]

[0102] Lemma 2: Sequence and sequence Z k =(x k ;z k ;u k )satisfy

[0103]

[0104] Equations (34) and (35) can be derived from equations (30), (32) and (33), equation (36) can be obtained from equation (35), and Lemma 2 can be obtained from equation (24).

[0105] Assumption 1: Assume a diagonal matrix Bounded, meaning it satisfies equations (38) and (39):

[0106]

[0107] Lemma 3: If Assumption 1 holds, then Z = (x; z; y) and Z′ = (x′; z′; y′) satisfy...

[0108]

[0109] Theorem 1: If Assumption 1 holds, then the sequence Z k =(x k ;z k ;y k )satisfy

[0110]

[0111] Proof of Theorem 1:

[0112] Let Z = Z k Z′=Z * Then, by Lemma 3, we can obtain:

[0113]

[0114] Combining equations (36) and (42), we can obtain

[0115]

[0116] Let k = 1, 2, ..., l, and summing up (43) yields...

[0117]

[0118] Right now

[0119]

[0120] As l→∞, we know from hypothesis 1 that Right now In other words

[0121] From equations (24), (25), (26) and (29), we can obtain

[0122]

[0123] From equation (46), it can be seen that only when the dual residuals and original residual When both are close to 0, The hypothesis holds true, meaning that the DP-LBSA-ADMM algorithm converges when hypothesis 1 holds true.

[0124] III. Experiments and Tests

[0125] This section uses the LBSA-ADMM and DP-LBSA-ADMM algorithms proposed in this paper to solve the logistic regression problem, which can be described as shown in Equation (47).

[0126]

[0127] Where m represents the number of samples in the dataset, b i ∈{-1,1} represents the sample label.

[0128] 3.1 Experimental Setup

[0129] Algorithm Comparison: This section selects the following two algorithms for comparison with the LBSA-ADMM and DP-LBSA-ADMM algorithms.

[0130] (1) GCADMM algorithm: Globally consistent ADMM algorithm, which uses the allreduce algorithm to aggregate local model parameters and solves subproblems based on dense data;

[0131] (2) PVP-ADMM: The ADMM algorithm based on differential privacy proposed by Shang F et al. adds perturbation terms to local variables.

[0132] To further analyze the performance of sparse LBSA, this section compares the LBSA algorithm with three other sparse allreduce algorithms: TOPK, SparCML, and Kylix. The difference between these three algorithms and LBSA-ADMM is that they employ different sparse allreduce algorithms.

[0133] (3) TopK-ADMM: The sparse model parameters are aggregated using the sparse allreduce algorithm proposed by Li S et al.

[0134] (4) Spar-ADMM: The dynamic sparse allreduce algorithm proposed by Renggli C et al. aggregates the sparse model parameters SparCML;

[0135] (5) Kylix-ADMM: Parameters are aggregated using the Kylix algorithm.

[0136] Platform and Algorithm Implementation: This paper implements the above algorithms based on the MPICH communication library and the C++ language. The experimental platform is Tianhe-2, and a total of 8 working nodes are used in the experiment.

[0137] Dataset: The test dataset uses RCV. 1 ,url 2 webspam 3 The relevant information for the three public datasets and the test dataset is shown in Table 2.

[0138] Table 1 Test Dataset Information Table

[0139]

[0140] 3.2 Performance Testing

[0141] This section compares the performance of different algorithms in terms of training time and accuracy. Training time includes computation time and communication time. Computation time refers to the update time of local variables, global variables, and local parameters, while communication time refers to the time for aggregating local model parameters χ between nodes. i The time includes synchronization time and data transmission time. Accuracy (Acc) is defined as shown in Equation (48). The number of working nodes for all algorithms is set to 64, and the penalty term parameter ρ is set to 1. Table 3 shows the training time and accuracy of different algorithms using different datasets.

[0142]

[0143] Acc=(TP+TN) / m (48)

[0144] Where TP represents the number of samples that were actually positive and predicted as positive, and TN represents the number of samples that were actually negative and predicted as negative.

[0145] As shown in Table 3: (1) The computation time of the GCADMM algorithm is longer than that of other algorithms, mainly because the other algorithms all use sparse vectors for computation, which reduces the computational load of a single node; (2) The communication time of the GCADMM algorithm is the longest, because the other algorithms all use the sparse allreduce algorithm instead of the dense allreduce algorithm to aggregate local model parameters, and the number of bytes that a single node needs to transmit is less, so the communication time is less. When the dataset is rcv, the DP-LBSA-ADMM algorithm can reduce the communication time by 59% compared with the GCADMM algorithm; (3) LBSA-ADMM has less communication time than Spar-ADMM, TopK-ADMM and Kylix-ADMM, mainly because the sparse allreduce algorithm proposed in this paper can better balance the load and reduce the synchronization time between nodes. Compared with Spar-ADMM, LBSA-ADMM can reduce communication time by 24%-44%; (4) compared with GCADMM algorithm, DP-LBSA-ADMM algorithm can reduce training time by up to 37.18% (when the dataset is URL); (5) compared with PVP-ADMM algorithm, DP-LBSA-ADMM algorithm has higher accuracy, mainly because the algorithm dynamically adjusts the penalty term parameter by combining the values ​​of dual residual and original residual. This also shows that DP-LBSA-ADMM does not lose algorithm accuracy.

[0146] Table 2 Training time and accuracy of different algorithms

[0147]

[0148]

[0149] 3.3 Traffic Volume Test

[0150] This paper proposes a sparse allreduce algorithm (LBSA) for sparse data. This section compares the LBSA algorithm with three other sparse allreduce algorithms: SparCML, TopK, and Kylix. The number of worker nodes is set to 8, 16, 32, and 64, respectively. Each worker node reads the same training dataset, and the communication volume of the node that transmits the most bytes is recorded. Figure 5 The figure shows the number of bytes that need to be transmitted in one communication for different sparse allreduce algorithms when using different numbers of worker nodes.

[0151] Depend on Figure 5It can be seen that: (1) In the split-reduce stage, LBSA and TopK algorithms transmit fewer bytes than the other two algorithms. This is because these two algorithms balance the transmission load among nodes by dividing the transmission parameters equally; (2) In the allgather stage, LBSA has less communication volume than the other algorithms. This is because LBSA only needs to transmit the correlation model parameters; (3) As the number of nodes increases, the number of bytes that LBSA needs to transmit in a single communication decreases. This is because, with a fixed dataset, as the number of nodes increases, the correlation model parameters of each node decrease, thus reducing the communication load. This also shows that the communication algorithm proposed in this paper has good scalability. The scalability of the algorithm will be further analyzed below.

[0152] 3.4 Scalability Testing

[0153] This section compares the scalability of different algorithms: the number of working nodes is set to 8, 16, 32, and 64 respectively. First, the training time and accuracy of different algorithms are tested. Then, the training time with 8 working nodes is used as the benchmark to analyze the speedup of different algorithms. Figure 6 The figure shows the speedup and accuracy curves of different algorithms. Adding a suffix 1 after the algorithm indicates the speedup of the corresponding algorithm, and adding a suffix 2 after the algorithm indicates the accuracy of the corresponding algorithm.

[0154] Figure 6 The results show that the accuracy of several algorithms decreases as the number of working nodes increases, which is due to the inherent characteristics of the ADMM algorithm. From... Figure 6 It can also be seen that the LBSA-ADMM algorithm has a higher speedup ratio than other algorithms. This is mainly because the LBSA-ADMM algorithm not only evenly distributes the model parameters, but also only requires the transmission of associated model parameters between nodes. As the number of nodes increases, the number of associated model parameters for each node decreases, thus reducing the communication volume and resulting in a better speedup ratio. This further proves that the LBSA-ADMM algorithm proposed in this paper has better scalability.

[0155] 3.5 Convergence Test

[0156] This section analyzes the impact of the initial value of the penalty term parameter ρ on the convergence of four algorithms: LBSA-ADMM, GCADMM, PVP-ADMM, and DP-LBSA-ADMM (since the four algorithms TopK-ADMM, Spar-ADMM, Kylix-ADMM, and LBSA-ADMM differ only in the communication algorithm used when aggregating the model parameters of each node, these algorithms have the same convergence). The number of working nodes for all four algorithms is set to 64, and the initial values ​​of the penalty term parameter are 1 and 10, respectively. The loss function is selected as the observation value. Figure 7The diagram shows how the loss function of different algorithms changes with the number of iterations.

[0157] Depend on Figure 7 It can be seen that the convergence of the GCADMM, PVP-ADMM, and LBSA-ADMM algorithms is significantly affected by the initial value of the penalty term parameter: when the number of iterations is the same, the loss function values ​​of these algorithms are larger when ρ=1 than when ρ=10. The convergence of the DP-LBSA-ADMM algorithm is less affected by the initial value of the penalty term parameter: when the algorithm converges, the values ​​of the loss function are very close. This is because DP-LBSA-ADMM dynamically adjusts the penalty term parameter; therefore, the initial value of the penalty term parameter only affects the algorithm in the early stages of training, and as the number of iterations increases, the penalty term parameter will fluctuate within a small range. Figure 7 It can also be seen that although the DP-LBSA-ADMM algorithm adds a perturbation term to the penalty term parameter, it does not affect the algorithm's convergence. However, when the dataset is a URL and ρ = 10, the loss function of the PVP-ADMM algorithm is not strictly convergent. This also demonstrates that DP-LBSA-ADMM has a smaller dependence on ρ than other algorithms.

[0158] IV. Summary

[0159] To address the issue of high computational and communication overhead for machine learning models when processing large-scale sparse data, this paper proposes a general consensus distributed ADMM algorithm based on the LBSA communication model, building upon the distributed ADMM algorithm. This algorithm utilizes sparse vectors for computation and communication, effectively reducing the computational and communication load on a single node. Furthermore, this paper designs a sparse allreduce communication model—LBSA—based on the distribution characteristics of sparse data. This communication algorithm effectively balances the communication load between nodes, improving its scalability. To enhance the security of the LBSA-ADMM algorithm, this paper proposes a general consensus ADMM algorithm based on differential privacy—DP-LBSA-ADMM. This algorithm improves data security by adding a perturbation term to the penalty term parameter and dynamically adjusts the penalty term parameter to balance algorithm security and accuracy. Experimental results show that compared with other distributed ADMM algorithms, LBSA-ADMM and DP-LBSA-ADMM effectively reduce computational and communication overhead, while DP-LBSA-ADMM improves data security with minimal impact on algorithm accuracy.

[0160] The LBSA algorithm proposed in this paper reduces the communication time between nodes as the number of working nodes increases, demonstrating excellent scalability. This algorithm can also be extended to other distributed algorithms, and further research will be conducted to apply this communication model to other distributed machine learning algorithms.

[0161] Based on the same inventive concept, this application also provides a privacy protection device based on efficient sparse communication, including modules for executing the steps in the embodiments corresponding to the privacy protection algorithm based on efficient sparse communication.

[0162] The methods described in the above-disclosed embodiments of this application are detailed in terms of their specific implementation methods. These methods can be implemented using various types of devices. Therefore, this application also discloses a privacy protection device based on efficient sparse communication. Specific embodiments are given below for detailed explanation.

[0163] Please see the appendix Figure 8 , Figure 8 This is a schematic diagram of a privacy protection device based on efficient sparse communication disclosed in an embodiment of this application. The device includes:

[0164] Data reading and feature statistics module: This module is used by each node to read its own dataset, count its associated features, and add them to the set Ψ. i middle;

[0165] Weighting and Feature Value Partitioning Module: This module performs the allgather operation on all nodes, stores the associated features of all nodes in a two-dimensional vector ARI, uses the values ​​of ARI to calculate the weight W of each feature value, and partitions the feature values ​​according to the weights.

[0166] Local variable update module: After recording the association features, each node updates its local variable x sequentially using local data. i dual variable y i Local model parameter χ i Aggregate the local model parameters χ of all nodes i ;

[0167] The parameter aggregation and global variable update module is used to aggregate the local model parameters of all nodes using the load-balanced sparse allreduce algorithm, and then update the global variable z.

[0168] In this device, each computing node first determines its correlation characteristics based on the distribution patterns of its processed datasets, and then uses these correlation characteristics to replace all other characteristics for computation and communication. That is, the algorithm uses sparse vectors for computation and communication, which can significantly reduce the computational cost and communication load of a single node.

[0169] The local variable update module specifically includes: When solving large-scale machine learning problems using distributed nodes, large-scale problems can usually be transformed into a consistency optimization problem as shown in equation (1):

[0170]

[0171] in Represents a dataset, Here, m represents the number of datasets, and p represents the number of features in the dataset.

[0172] Global consistency optimization introduces a local variable to replace the model parameters for each node, and ensures consistency among computational nodes by constructing constraint terms for both global and local variables. This transforms equation (1) into the global consistency problem shown in equation (2):

[0173]

[0174] This algorithm is used to solve the large-scale machine learning problem shown in equation (1). For application scenarios where the association characteristics of each node are fixed, such as text classification and social networks, equation (2) can be further transformed into solving the general consistency problem shown in equation (7):

[0175]

[0176] Based on the ADMM algorithm framework, the iterative formula for solving equation (7) is as follows:

[0177]

[0178]

[0179] Where, x i Let y be a local variable of the i-th node. i For x i The dual variable, χ i These are intermediate variables, called local model parameters, Ψ i Θ represents the set of features associated with the i-th working node. j W represents the set of working nodes associated with the j-th feature. j Let represent the weight of the j-th feature, and ρ be the penalty term parameter. For x i Related global variables.

[0180] As shown in equation (10), updating global variables involves two key steps: first, aggregating the features associated with the local model parameters; and second, calculating the weight W of each feature. j Clearly, the sparse allreduce algorithm can be used to aggregate the correlation features of each node. To calculate W... j First, when reading data, count the frequency of each feature and add features with a frequency greater than 0 to set Ψ. i In the middle; then for all Ψ i Performing an allgather operation yields W. j It should be noted that W j It only needs to be calculated once.

[0181] Aiming at the two problems existing in the existing sparse allreduce algorithm, the present invention adopts a load-balanced sparse allreduce algorithm (referred to as LBSA algorithm) based on a privacy protection device with efficient sparse communication. The parameter aggregation and global variable update module specifically includes: in the split-reduce phase, the feature values of the local model parameters are divided according to the distribution of the feature values associated with each worker node, and the division rule is shown in Equation (13):

[0182]

[0183] Split[i] represents the start value of the i-th interval, that is, the range of feature values that the i-th (0≤i<N-1) worker node is responsible for aggregating is [Split[i], Split[i+1]-1], and the range of feature values that the (N-1)-th worker node is responsible for aggregating is [Split[N-2], p]. That is, it is ensured that the number of parameters each node is responsible for reducing is basically the same, so as to balance the communication load of each node; in the allgather phase, each node broadcasts its associated feature values to other nodes. Figure 3 The figure shows a schematic diagram of inter-node communication of the LBSA algorithm when the number of worker nodes N is 4 and the number of feature values p is 24.

[0184] The local variable update module specifically includes: in the distributed ADMM algorithm for large-scale machine learning, all features need to participate in the calculation when updating local variables. However, in the general consensus ADMM algorithm, only the associated features need to be used to update local variables, so sparse vector calculation can be used when solving local variables to reduce the computational load. The update formulas for the local variable x and the dual variable y can be further modified as shown in Equations (14) and (15):

[0185]

[0186] The convergence of the distributed ADMM algorithm is sensitive to the value of the penalty term parameter. B.S. He et al. proved that when the initial residual r k is much larger than the dual residual s k , the dual variable updates slowly. In order to improve the speed of the algorithm, the penalty term parameter needs to be increased, otherwise the penalty term parameter should be decreased. The privacy protection device based on efficient sparse communication of the present invention includes a differentially private general consensus ADMM algorithm (referred to as DP-LBSA-ADMM algorithm). The local variable update module specifically includes: each node independently adds a perturbation term to the penalty term parameter ( is a random number greater than 1, whose value is determined by the distributed function (Randomly generated), and the penalty term parameter is adjusted according to the values ​​of the dual residual and the initial residual. The update rules for the dual residual, penalty term parameter, local variables, local model parameters and initial residual are shown in equations (19)-(21), and the update formulas for other variables are the same as those for the distributed ADMM algorithm based on the LBSA communication model.

[0187]

[0188] in Let i be the dual residual of the i-th node. Let z be the penalty term parameter for the i-th node, and z be a global variable. The perturbation term for the i-th node Let be the initial residual of the i-th node.

[0189] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0190] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0191] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A privacy-preserving algorithm based on efficient sparse communication, characterized in that, comprising a distributed ADMM algorithm based on the LBSA communication model, which includes: Step S1: Each node reads its own dataset, calculates its correlation features, and adds them to set Ψ. i middle; Step S2: All nodes perform an allgather operation, store the associated features of all nodes into a two-dimensional vector ARI, use the value of ARI to count the weight W of each feature value, and divide the feature values according to the weights; Step S3: After recording the association features, each node updates its local variable x sequentially using local data. i dual variable y i Local model parameter χ i Aggregate the local model parameters χ of all nodes i ; Step S4: Aggregate the local model parameters of all nodes by using the load-balanced sparse allreduce algorithm, and then update the global variable z.

2. The privacy protection algorithm based on efficient sparse communication according to claim 1, characterized in that, In step S3, when solving a large-scale machine learning problem using distributed nodes, the large-scale problem is converted into a consensus optimization problem represented by Equation (1) for solving: in Represents a dataset, Here are the model parameters, where m is the number of datasets and p represents the number of features in the dataset; Global consistency optimization introduces a local variable to replace the model parameters for each node, and constructs a global variable z and a local variable x. i The constraint terms ensure consistency among the computation nodes, thus transforming equation (1) into the global consistency problem shown in equation (2): This algorithm is used to solve the large-scale machine learning problem represented by Equation (1). For application scenarios where the association characteristic of each node is fixed, Equation (2) is further converted into a general consensus problem represented by Equation (7) for solving: Equation (7) is solved based on the ADMM algorithm framework, and its iterative formula is: Where, x i Let y be a local variable of the i-th node. i For x i The dual variable, χ i These are intermediate variables, called local model parameters, Ψ i Θ represents the set of features associated with the i-th working node. j W represents the set of working nodes associated with the j-th feature. j Let z represent the weight of the j-th feature, ρ be the penalty term parameter, and z be the weight of the j-th feature. gi For x i Related global variables.

3. The privacy protection algorithm based on efficient sparse communication according to claim 1, characterized in that, In step S4, it is necessary to aggregate the local model parameters of all nodes. In the split-reduce phase, the feature values of the local model parameters are divided according to the distribution of the associated feature values of each worker node, and the division rule is represented by Equation (13): Split[i] represents the start value of the i-th interval, that is, the range of feature values that the i-th (0≤i<N-1) worker node is responsible for aggregating is [Split[i], Split[i+1]-1], and the range of feature values that the (N-1)-th worker node is responsible for aggregating is [Split[N-2], p], which ensures that the number of parameters each node is responsible for reducing is substantially the same, so as to balance the communication load of each node; in the allgather phase, each node broadcasts its associated feature values to other nodes.

4. The privacy protection algorithm based on efficient sparse communication according to claim 2, characterized in that, In step S3, in the distributed ADMM algorithm for large-scale machine learning, all features need to participate in the calculation when updating local variables, while in the general consensus ADMM algorithm, only the associated features need to be used to update local variables. Therefore, sparse vector calculation can be used when solving local variables to reduce the computational load, and the update formulas for the local variable x and the dual variable y can be further modified as represented by Equations (14) and (15):

5. The privacy protection algorithm based on efficient sparse communication according to claim 2, characterized in that, This includes a privacy-preserving general consistency ADMM algorithm, in which each node independently adds a perturbation term to the penalty term parameter in step S3. A random number greater than 1, whose value is determined by a distributed function. The values ​​of the dual residual and the initial residual are randomly generated, and the penalty term parameter is adjusted according to the values ​​of the dual residual and the initial residual. The update rules for the dual residual, penalty term parameter, local variables, local model parameters and the initial residual are shown in equations (19)-(21). in Let i be the dual residual of the i-th node. Let z be the penalty term parameter for the i-th node, and z be a global variable. The perturbation term for the i-th node Let be the initial residual of the i-th node.

6. A privacy protection device based on efficient sparse communication, characterized in that, comprising: Data reading and feature statistics module: This module is used by each node to read its own dataset, count its associated features, and add them to the set Ψ. i middle; Weight statistics and feature value division module: configured for all nodes to perform an allgather operation, store the associated features of all nodes into a two-dimensional vector ARI, use the value of ARI to count the weight W of each feature value, and divide the feature values according to the weights; Local variable update module: After recording the association features, each node updates its local variable x sequentially using local data. i dual variable y i Local model parameter χ i Aggregate the local model parameters χ of all nodes i ; Parameter aggregation and global variable update module: configured to aggregate the local model parameters of all nodes by using the load-balanced sparse allreduce algorithm, and then update the global variable z.

7. The privacy protection apparatus based on efficient sparse communication according to claim 6, characterized in that, The local variable update module specifically comprises: when solving a large-scale machine learning problem using distributed nodes, converting the large-scale problem into a consensus optimization problem represented by Equation (1) for solving: in Represents a dataset, Here are the model parameters, where m is the number of datasets and p represents the number of features in the dataset; Global consistency optimization introduces a local variable to replace the model parameters for each node, and constructs a global variable z and a local variable x. i The constraint terms ensure consistency among the computation nodes, thus transforming equation (1) into the global consistency problem shown in equation (2): This algorithm is used to solve the large-scale machine learning problem shown in Equation (1). For application scenarios where the associated characteristics of each node are fixed, Equation (2) is further transformed to solve the general consensus problem shown in Equation (7): Equation (7) is solved based on the ADMM algorithm framework, and its iteration formula is as follows: Where, x i Let y be a local variable of the i-th node. i For x i The dual variable, χ i These are intermediate variables, called local model parameters, Ψ i Θ represents the set of features associated with the i-th working node. j W represents the set of working nodes associated with the j-th feature. j Let represent the weight of the j-th feature, and ρ be the penalty term parameter. For x i Related global variables.

8. The privacy protection apparatus based on efficient sparse communication according to claim 6, characterized in that The parameter aggregation and global variable updating module specifically comprises: in the split-reduce stage, dividing the feature values of local model parameters according to the distribution of the associated feature values of each worker node, wherein the division rule is shown in Equation (13): Split[i] represents the start value of the i-th interval, that is, the range of feature values that the i-th (0≤i<N-1) worker node is responsible for aggregating is [Split[i], Split[i+1]-1], and the range of feature values that the (N-1)-th worker node is responsible for aggregating is [Split[N-2], p], which ensures that the number of parameters each node is responsible for reducing is substantially the same, so as to balance the communication load of each node; in the allgather stage, each node broadcasts its associated feature values to other nodes.

9. The privacy protection apparatus based on efficient sparse communication according to claim 7, characterized in that The local variable updating module specifically comprises: in the distributed ADMM algorithm for large-scale machine learning, all features need to participate in the calculation when updating local variables, while in the general consensus ADMM algorithm, only the associated features are needed to update local variables. Therefore, sparse vector calculation can be utilized when solving local variables to reduce the calculation load, and the update formulas of the local variable x and the dual variable y can be further modified as shown in Equations (14) and (15):

10. The privacy protection apparatus based on efficient sparse communication according to claim 7, characterized in that The local variable update module specifically includes: each node independently adding a perturbation term to the penalty term parameter. A random number greater than 1, whose value is determined by a distributed function. The values ​​of the dual residual and the initial residual are randomly generated, and the penalty term parameter is adjusted according to the values ​​of the dual residual and the initial residual. The update rules for the dual residual, penalty term parameter, local variables, local model parameters and the initial residual are shown in equations (19)-(21): in Let i be the dual residual of the i-th node. Let z be the penalty term parameter for the i-th node, and z be a global variable. The perturbation term for the i-th node Let be the initial residual of the i-th node.

Images