Heterogeneous software defect prediction method based on top-k communication efficient federated learning

Abstract

The application is a heterogeneous software defect prediction algorithm based on top-k communication efficient federated learning; the method steps are as follows: all participants decode the received global parameters, and update the local model by using the decoded global model gradient parameters and the defect data of each participant based on a capsule neural network (CapsNet) for local training; each participant encrypts the local model gradient parameters by using Gaussian differential privacy; the encrypted gradient parameters are subjected to sparse binary compression; the relative distance between non-zero elements is subjected to Golomb coding and sent to the server end for aggregation; the server end decodes the received data and aggregates the data; and the server end sends back each participant after the aggregated parameters are subjected to sparse binary compression and Golomb coding. The application can effectively reduce the communication bit quantity and improve the communication efficiency under the condition that the model prediction performance is acceptable.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of software engineering. BACKGROUND

[0002] With the increasing size of software systems, their complexity is also increasing. The possibility of software system failure is increasing, so heterogeneous software defect prediction has emerged. Heterogeneous software defect prediction is to use a heterogeneous defect database of different metrics to construct a heterogeneous software defect prediction model, so that developers can benefit from the publicly available defect data set to build a prediction model with more defects and reduce software defects, thereby reducing software development costs.

[0003] The difficulty of heterogeneous defect prediction is that a large amount of defect data is protected by privacy and is kept by each party, and cannot be obtained at will, thereby forming a data island. As a new machine learning paradigm, federated learning has great advantages in alleviating the data island problem while protecting privacy. Federated learning uses the data sets of multiple participants, each participant uses machine learning methods to train their own private model, and through privacy protection technology, the model data information of multiple parties is fused to collaboratively build a global model to improve the prediction performance of a single model.

[0004] However, in the implementation of heterogeneous software defect prediction based on federated learning, each participant needs to send complete model parameters to the server side for updating in each global model training round. Since the heterogeneous software defect prediction model usually has millions of parameters, sending so many parameters to the server side will result in huge communication overhead, and with the increase in the number of participants and communication iteration rounds, the communication overhead will further increase. In order to reduce the communication overhead, a heterogeneous software defect prediction method based on top-k communication efficient federated learning is proposed, which can perform sparse binary compression on each participant under the premise of protecting data privacy, improve communication efficiency, and reduce the communication bit quantity of federated learning. SUMMARY

[0005] The application provides a top-k communication efficient federated learning heterogeneous software defect prediction algorithm; global parameters received by all participants are decoded, local training is performed based on a capsule neural network (CapsNet) using decoded global model gradient parameters and respective defect data, and a local model is updated; local model gradient parameters are encrypted by each participant using Gaussian differential privacy; sparse binary compression is performed on the encrypted gradient parameters; relative distances between non-zero elements are Golomb encoded and sent to a server end for aggregation; the server end decodes received data and aggregates the data; the server end performs sparse binary compression and Golomb encoding on the aggregated parameters and sends them back to each participant. When the maximum communication round is reached or the local capsule neural network model has converged, a prediction result is output.

[0006] The top-k communication efficient federated learning heterogeneous software defect prediction algorithm comprises the following steps:

[0007] Step a: all participants decode global parameters, perform local training based on a capsule neural network using decoded global model gradient parameters and defect data, and update a local model;

[0008] Step b: each participant encrypts local model gradient parameters using Gaussian differential privacy;

[0009] Step c: each participant performs sparse binary compression on the encrypted gradient parameters to form sparse gradients;

[0010] Step d: relative distances between non-zero elements of the sparse gradients are calculated, Golomb encoded, and sent to a server end for aggregation;

[0011] Step e: the server end decodes received data, aggregates the data, performs sparse binary compression and Golomb encoding on the aggregated parameters, and sends them back to each participant;

[0012] Step f: when the number of communications reaches the maximum communication round or the local capsule neural network model has converged, a prediction result is output.

[0013] The top-k communication efficient federated learning heterogeneous software defect prediction algorithm described above, step a is specifically:

[0014] The participants decode global parameters, perform local training based on a capsule neural network using decoded global model gradient parameters and defect data, and update a local model, which is specifically:

[0015] In this specific implementation, local training employs a capsule neural network. The first layer of the capsule neural network is a standard convolutional layer with ReLU activation. This layer extracts features, which are then used as input to the main capsule. The second layer is the main capsule layer, activated by the squash function. The third layer is the routing capsule, where the dynamic routing algorithm operates.

[0016] The data is input into a convolutional layer for feature extraction, and after ReLU activation, it is input into a capsule layer. Assume each capsule's input is (s1, s2, ..., s...). n ), s i Let v = (v1, v2, ..., v) represent the total input for the i-th capsule. m ) represents the capsule output, v i The output of the i-th capsule is obtained by applying a non-linear squash activation function, resulting in the following network output:

[0017]

[0018] The aforementioned heterogeneous software defect prediction algorithm based on top-k efficient federated learning, specifically step b, involves encrypting the local model gradient parameters using Gaussian difference privacy:

[0019] Given the local model gradient parameters D, a function f in an arbitrary domain, and a randomization mechanism. Achieving Gaussian difference privacy for local model parameters:

[0020]

[0021] A smaller ε indicates a stronger level of privacy protection. This represents the probability of violating differential privacy. We obfuscate f(·) by adding random noise:

[0022]

[0023] Among them, I M It is the identity matrix, N(ε,σ) 2 I M ) are the mean ε and variance σ 2 The multivariate Gaussian noise has the following variance:

[0024] σ 2 =2(Δf) 2 ln(1.25 / δ) / ε 2

[0025] The above-mentioned heterogeneous software defect prediction algorithm based on top-k efficient federated learning involves step c where each participant performs sparse binary compression on the encrypted gradient parameters to form sparse gradients. Specifically:

[0026] Let k-compression be: for a parameter 0 < k ≤ d, the operator of k-compression is comp: R d → R d , satisfying the compression property:

[0027]

[0028] SBC k is defined as - compression:

[0029]

[0030] Suppose the objective function f of training is L-smooth, μ-strongly convex, and E||Δω|| 2 ≤ G 2 For the client to upload to the server, the update rule is:

[0031]

[0032]

[0033] Then the convergence is:

[0034]

[0035] From the above formula, for SBC converges at a rate of O(G 2 / μT), which is the same as the conventional stochastic gradient descent method.

[0036] In the federated learning setting, the sparse binary compression algorithm needs to send the aggregated model parameters from the server side to the participant side.

[0037] According to the SBC algorithm, first define SBC k : R n → R n , Δω → Δω, which maps the (flattened) weight update Δω to the compressed operator of the sparse and binary weight update Δω. For the local weight update The SBC update rule can be written as:

[0038]

[0039]

[0040] All clients start from an empty residual value In the worst case, when the parameters sent from the client to the server are always sparse, the parameters Δω sent by the server to the client are also sparse. (t+1)The number of non-zero elements in increases linearly with the number of participating clients. If the participation rate exceeds the inverse sparsity rate 1 / p, the update parameters Δω (t+1) will become dense.

[0041] To avoid this problem, we adopt the following compression. First, we obtain the average sparse binary weight of each participant based on the SBC algorithm, and then add the current server-side residual value to obtain the sparse binary weight based on the SBC algorithm again. Therefore, the following compression is adopted:

[0042]

[0043] The client residual value update is respectively:

[0044]

[0045] The above top-k-based communication efficient federated learning heterogeneous software defect prediction algorithm, step d is specifically:

[0046] In order to transmit a set of sparse binary tensors generated by sparse binary compression, only the non-zero elements in the flattened tensor and the position of each average (μ + or μ - ) need to be transmitted, and only the distance between non-zero elements is transmitted instead of the absolute position of non-zero elements. In the random sparse mode, the distance is approximately geometrically distributed, and the success probability is about p, so Golomb coding is used for encoding. Golomb coding reduces the number of bits of the average position to:

[0047]

[0048] And is the golden ratio.

[0049] The relative distance between the non-zero elements of the sparse gradient is calculated, Golomb coding is performed, and sent to the server side for aggregation, which is specifically:

[0050] First, the relative distance d between the non-zero elements is calculated, and then:

[0051]

[0052]

[0053] The encoding result is:

[0054]

[0055] Finally, the Golomb coding is sent to the server side.

[0056] The heterogeneous software defect prediction algorithm based on top-k efficient federated learning of communication above, step e is specifically:

[0057] First, the participants upload the parameter i . The server side decodes the parameter:

[0058] Δω i = decode(Δω i )

[0059] Then calculate the model aggregation weight:

[0060]

[0061] The sparse binary compression algorithm also needs to be extended to the server side to send the aggregated model parameters to the participant side.

[0062] According to the SBC algorithm, first define SBC k :R n →R n , Δω → Δω, which maps the flattened weight Δω to the sparse and binary weight Δω. For local weight update The SBC update rule can be written as:

[0063]

[0064]

[0065] All clients start from an empty residual value . In the worst case, the parameters sent from the client to the server are always sparse, and the number of non-zero elements in the parameters Δω (t+1) sent by the server to the client increases linearly with the number of participating clients. If the participation rate exceeds the inverse sparsity rate 1 / p, the update parameter Δω (t+1) will become dense in essence.

[0066] To avoid this problem, we use the following compression:

[0067]

[0068] The server side residual value is updated as:

[0069]

[0070] Finally, the parameter is Golomb encoded and sent to the participants.

[0071] Advantages:

[0072] This invention employs a heterogeneous software defect prediction method based on top-k efficient federated learning. Based on the idea of ​​top-k efficient federated learning, it reduces the communication bit volume between the client and server without affecting model prediction performance. First, each participant performs local training using global parameters and its local dataset. Local training utilizes capsule networks to preserve the vector information of data features, resulting in good generalization performance. Then, each participant performs Gaussian difference privacy encryption on the trained model parameters to ensure the security of the model parameter information. Finally, the encrypted parameters are sparsely binary compressed, transforming dense computation into sparse computation. Then, Golomb encoding is used, and the data is sent to the server for aggregation. Finally, the server decodes the received data, performs model aggregation, and sends it back to each participant in the same way after sparse binary compression and encoding to update its local model. Experimental results demonstrate that this method can effectively reduce the communication bit volume while maintaining comparable model performance. Attached Figure Description

[0073] Figure 1 This is a flowchart of the efficient federated learning method based on top-k communication in this invention.

[0074] Figure 2 This is a diagram of the capsule neural network structure in the method of this invention.

[0075] Figure 3 This is a graph showing the prediction results of AEEEM and Relink as participating datasets in the method of this invention.

[0076] Figure 4 This is a graph showing the prediction results of NASA and Relink as participating datasets in the method of this invention.

[0077] Figure 5 This is a graph showing the prediction results of NASA and AEEEM datasets as participants in the method of this invention. Detailed Implementation

[0078] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0079] This specific implementation describes a heterogeneous software defect prediction method based on top-k efficient federated learning, with the flowchart shown below. Figure 1 As shown, it includes the following steps:

[0080] Step a: All participants use the decoded global model gradient parameters and defect data to perform local training based on the capsule neural network and update the local model;

[0081] In this specific embodiment, local training employs a capsule neural network. Figure 2The network structure of the capsule neural network is shown. One layer adopts a common convolution layer and a ReLU function activation, which is used for feature extraction and has an output dimension of (4, 4, 64). Then it is used as the input of the primary capsule. The second layer is the primary capsule layer and the squash function activation, in which the output dimension is two 3s representing the horizontal and vertical dimensions, 128 is the number of capsule channels, and 9 is the number of instantiation parameters of each primary capsule in the same dimension. The third layer is the routing capsule, and the dynamic routing algorithm works in this layer, and the output of this layer is a binary classification result. The three public databases of NASA, AEEEM and Relink are used as the local data set of the participants in the present application. For the three databases, X = {x1,..., x n} is used as the input data matrix, and the data set X includes test data and training data, x i represents the i-th sample of the training data, 1≤i≤n. The data is input into the convolution layer for feature extraction and then input into the capsule layer after ReLU activation.

[0082] It is assumed that the input of each capsule is (s1,s2,...,s n ), s i represents the total input of the i-th capsule, and v = (v1,v2,...,v m ) is the output of the capsule, v i is the output of the i-th capsule, which is obtained by a nonlinear squash activation function:

[0083]

[0084] The total input s i of the capsule i is obtained by the sum of the weights of all capsule outputs in the previous layer:

[0085]

[0086] U ij = ω ij I i

[0087] I i is the i-th input vector, and ω ij is the weight matrix of the i-th vector. In order to ensure that I i is aligned with the previous layer capsule s i , the transformation is performed using the weight matrix ω ij . c ij is the routing weight, which represents the contribution of the previous capsule to the upper layer capsule and is determined by the softmax function. c ij The calculation formula is:

[0088]

[0089] b ij is the log prior probability between the ith capsule and the jth capsule, which is usually used to reflect the consistency between U ij and v j , and the closer the values of U ij and v j are, the higher the correlation between the two capsules is. In order to make capsule i coupled to capsule j with equal probability, b ij will be initialized to 0.

[0090] In the iteration process, b ij is updated to b ij ← b ij + U ij v j , so that b ij can be more accurate. Finally, the output layer will replace each capsule with a module and perform the argmax function to obtain the predicted label, defective or non-defective.

[0091] Since the capsule neural network allows multiple classifications to exist at the same time, the Margin loss function is adopted instead of the traditional cross-entropy loss function, so as to reduce the error, and the loss function is:

[0092] L k = T k max(0, m + -||v k ||) 2 +λ(1-T k )max(0,||v k ||-m - ) 2

[0093] k is the class of classification, and if the correct classification corresponds to the class corresponding to the specific high-level capsule output, T k = 1, otherwise 0. m + is the upper boundary, generally taking the value 0.9, indicating that when the probability of the high-level capsule predicting the correct label is greater than or equal to 0.9, the value of the margin loss is 0; when the probability of predicting the correct label is less than 0.9, the value of the margin loss is not 0. m - is the lower boundary, generally taking the value 0.1, indicating that when the probability of the high-level capsule predicting the wrong label is less than or equal to 0.1, the value of the margin loss is 0; when the probability of predicting the wrong label is greater than 0.1, the value of the margin loss is not 0. λ is a hyperparameter, which is used to ensure the stability known in training, and takes a fixed value of 0.5.

[0094] Step b, each participant encrypts the local model gradient parameter using Gaussian differential privacy;

[0095] For local model gradient parameters D, and a function f of an arbitrary domain, and a randomization mechanism Achieve Gaussian differential privacy for local model parameters:

[0096]

[0097] The smaller the epsilon represents a stronger privacy protection level, Indicates the probability of breaking differential privacy. By adding random noise to blur f(·):

[0098]

[0099] Where I M is the identity matrix, N(epsilon, sigma 2 I M ) is a multivariate Gaussian noise with mean epsilon and variance sigma 2 The required noise variance is:

[0100] sigma 2 = 2(epsilon) 2 ln(1.25 / delta) / epsilon 2

[0101] Step c, each participant sparsely binarizes the encrypted gradient parameters to form sparse gradients;

[0102] Let k-compression be: for parameters 0 < k <= d, the compression operator of k-compression is comp: R d → R d , which satisfies the compression property:

[0103]

[0104] SBC k is defined as - Compression:

[0105]

[0106] Assume that the target function f of training is L-smooth, mu-strong convex, and E||Delta omega|| 2 <= G 2 For the client to upload to the server side, the update rule is:

[0107]

[0108]

[0109] Then the SBC algorithm converges to:

[0110]

[0111] From the above formula, it can be obtained that, for SBC converges at the rate of O(G 2 / μT) and the same as the conventional SGD.

[0112] In the federated learning setting, the sparse binary compression algorithm needs to send the aggregated model parameters from the server side to the participant side.

[0113] According to the SBC algorithm, first define SBC k :R n →R n ,Δω→Δω, map the flattened weight Δω to the sparse and binary weight Δω, for the local weight update The SBC update rule can be written as:

[0114]

[0115]

[0116] The client residual value is updated as:

[0117]

[0118] Step d, calculate the relative distance between the non-zero elements of the sparse gradient for Golomb coding and send to the server side for aggregation, and perform sparse binary compression and Golomb coding on the aggregated parameters and send back to each participant;

[0119] In order to pass a set of sparse binary tensors generated by sparse binary compression, only the non-zero elements in the flattened tensor and the position of each average value (μ + or μ - ) need to be passed, and only the distance between non-zero elements is transmitted instead of the absolute position of non-zero elements. In the random sparse mode, the distance is approximately geometrically distributed, with a success probability of about p, so Golomb coding is used for encoding. Golomb coding reduces the average position bit to:

[0120]

[0121] and is the golden ratio.

[0122] The relative distance between the non-zero elements of the sparse gradient is calculated for Golomb coding and sent to the server side for aggregation, which is:

[0123] First, calculate the relative distance d between the non-zero elements, and then calculate:

[0124]

[0125]

[0126] The encoding result is:

[0127]

[0128] Finally, the Golomb encoding is sent to the server side.

[0129] Step e, the server side decodes the received data and aggregates them, and sends back to each participant the sparsely binarized and Golomb encoded aggregated parameters.

[0130] First, the participant uploads the parameter i . The server side decodes the parameter:

[0131] Δω i = decode(Δω i )

[0132] Then the model aggregation weight is calculated:

[0133]

[0134] The SBC algorithm also needs to be extended to the server side to send the aggregated model parameters to the participant side.

[0135] According to the SBC algorithm, first define SBC k : R n → R n , Δω → Δω, which maps the (flattened) weight update Δω to the compressed operator of the sparsified and binarized weight update Δω. For the local weight update The SBC update rule can be written as:

[0136]

[0137]

[0138] All clients start from an empty residual value . In the worst case, when the parameters sent from the client to the server are always sparse, the number of non-zero elements in the parameters Δω (t+1) sent by the server to the client increases linearly with the number of participating clients. If the participation rate exceeds the inverse sparsity rate 1 / p, the update parameters Δω (t+1) will essentially become dense.

[0139] To avoid this problem, we use the following compression:

[0140]

[0141] The server-side residual value is updated as:

[0142]

[0143] Finally, the parameters are Golomb encoded and sent to the participants.

[0144] Step f, when the number of communications reaches the maximum communication round or the local capsule neural network model has converged, output the prediction result.

[0145] The participant's local data is divided into training data set and test data set according to 70% and 30% respectively, and when the maximum communication round is reached or the local model converges, the prediction result is output. Tables 1, 2 and 3 respectively AEEEM and Relink, NASA and Relink, NASA and AEEEM as data set, the relationship between average AUC and G-mean and different sparsity and the determination of the best sparsity of each experiment. In order to test the convergence of the method, Figure 3 、 4 and 5 show that as the number of communication rounds increases, most participants can converge, and their AUC and G-mean curves show stability, and their values are improved compared with those without participating in the federated learning process. Four federated learning related methods signSGD, DeepGradient Compression (DGC), FederatedAveraging (FedAvg) and Baseline are selected for comparison. Tables 4, 5 and 6 show that the data compression effect of our method is better than Baseline, signSGD, DGC and FedAvg, with less communication bit amount, and the model performance is comparable to Baseline. Compared with signSGD, DGC and FedAvg, it has better prediction performance.

[0146] Table 1 Prediction results of different sparsity rates when AEEEM and Relink are used as data sets

[0147]

[0148] Table 2 Prediction results of different sparsity rates when NASA and Relink are used as data sets

[0149]

[0150]

[0151] Table 3 Prediction results of different sparsity rates when NASA and AEEEM are used as data sets

[0152]

[0153] Table 4 Comparative experimental results of AEEEM and Relink as data sets

[0154]

[0155]

[0156] Table 5 Comparative experimental results of NASA and Relink as data sets

[0157]

[0158] Table 6 Comparative experimental results of NASA and AEEEM as data sets

[0159]

[0160]

Claims

1. A heterogeneous software defect prediction method based on top-k efficient federated learning, characterized in that, Includes the following steps: Step a: All participants decode the global parameters, use the decoded global model gradient parameters and defect data to perform local training based on the capsule neural network, and update the local model. Step b: Each participant uses Gaussian difference privacy to encrypt the gradient parameters of their local model; Step c: Each participant performs sparse binary compression on the encrypted gradient parameters to form a sparse gradient. Specifically, this involves processing the encrypted gradient parameters of each participant... First, let's exclude sparsity. All values ​​except the maximum and minimum weight update parameters are set to zero, and then the average of the remaining positive and negative weight update parameters is calculated locally: If the average value is positive Greater than the negative average The absolute value of the value is used to set all negative values ​​to zero and all positive values ​​to the positive mean, and vice versa; after each round of communication, the remaining value is accumulated as: Update the original values ​​for the weights. Update the weights after sparse binary representation; Step d: Calculate the relative distance between non-zero elements of the sparse gradient, perform Golomb encoding, and send it to the server to calculate the average value of the parameters for aggregation; Step e: The server decodes the received data, aggregates it, performs sparse binary compression and Golomb encoding on the aggregated parameters, and sends it back to each participant. Specifically: First, the participants upload parameters. The server decodes the parameters: Then calculate the model aggregate weights: The sparse binary compression method also needs to be extended to the server side to send aggregation model parameters to the participating parties. According to the sparse binary compression (SBC) method, first define , update the flattening weights Mapping to sparsification and binary weight updates The compression operator, for local weight updates , The update rule can be written as: Compression is performed using the following formula: The client's remaining value updates are as follows: Finally, the parameters Perform Golomb encoding and send it to the participants; Step f: When the number of communication rounds reaches the maximum or the local capsule neural network model has converged, output the prediction result.

2. The heterogeneous software defect prediction method based on top-k efficient federated learning for communication as described in claim 1, characterized in that, In step a, the participating party decodes the global parameters, uses the decoded global model gradient parameters and defect data to perform local training based on the capsule neural network, and updates the local model, specifically as follows: In this specific implementation, local training employs a capsule neural network. The first layer of the capsule neural network is a regular convolutional layer with ReLU activation, which is used to extract features and then use them as the input to the main capsule. The second layer is the main capsule layer with squash function activation. The third layer is a routing capsule, where the dynamic routing method operates. The data is input into a convolutional layer for feature extraction, and after ReLU activation, it is input into a capsule layer. Assume that the input to each capsule is (s1, s2, ..., s). n ), s i Let v represent the total input for the i-th capsule, where v = (v1, v2, ..., v...). m ) represents the capsule output, v i The output of the i-th capsule is obtained by applying a non-linear squash activation function, resulting in the following network output: 。 3. The heterogeneous software defect prediction method based on top-k efficient federated learning for communication as described in claim 1, characterized in that, In step b, each participant uses Gaussian difference privacy to encrypt the gradient parameters of the local model, specifically as follows: For local model gradient parameters and a function of arbitrary domain and a randomization mechanism , Achieving Gaussian difference privacy for local model parameters: Among them, the smaller This indicates a stronger level of privacy protection. The probability of violating differential privacy is represented by adding random noise to obscure it. : in, It is the identity matrix. It is the mean and variance The required noise variance for multivariate Gaussian noise is: 。 4. The heterogeneous software defect prediction method based on top-k efficient federated learning for communication according to claim 1, characterized in that, Step d calculates the relative distances between non-zero elements of the sparse gradient, performs Golomb encoding, and sends the results to the server for aggregation. Specifically: First, calculate the relative distances between non-zero elements. Then calculate: The encoding result is: Finally, the Golomb code is sent to the server.

Images