A data classification method and system based on fair federated learning algorithm

Abstract

The present invention proposes a data classification method and system based on a fair federated learning algorithm, which utilizes the local data set owned by each user terminal, determines the data quality score of each user terminal based on the entropy weight method, defines the aggregation weight based on the data quantity and data quality of the user terminal, and updates the global model according to the aggregation weight that takes both the data quantity and data quality into consideration, thereby solving the problem of unfair distribution of aggregation weights to each user in traditional federated learning that does not consider weight distribution or only considers the data quantity of the user terminal. Compared with the traditional federated learning scheme, the scheme of the present invention is fairer to each user terminal participating in the training, and can better promote the user terminals with high data quality scores to join the federated learning.

Description

Technical Field

[0001] The present invention relates to the field of data privacy technology, and in particular to a data classification method and system based on a fair federated learning algorithm. Background Art

[0002] The development of artificial intelligence requires a large amount of data, and a lot of high-quality data, but usually in the process of using data, it faces various constraints. First, it faces the constraints of various cybersecurity laws or data protection regulations; second, the interests between enterprises and institutions lead to constraints in the use of data; third, individuals are also constantly enhancing their awareness of privacy protection. These aspects are not conducive to data sharing, making data sharing a problem. There is a problem of "data islands" between enterprises and between institutions. Therefore, federated learning came into being to solve the problem of data islands in machine learning. Federated learning has attracted widespread attention from scholars since it was proposed by Google in 2017. Such as wireless communications and edge computing, smart medical care, environmental protection and other fields.

[0003] In the traditional federated learning training process, the aggregation weight of each user is the same during aggregation, or the proportion of user data to the total data volume is used as the weight during aggregation. This method is advantageous for users with large amounts of data, but for users with large amounts of data but poor data quality, allocating more weight to the user will affect the training of the global model, that is, making the computational cost of other users higher, that is, the number of iterations required for local model training iterations for other users and the number of iterations between the server and each user will be higher.

[0004] In response to the above problems, the present invention takes into account the impact of user data quality on model training during the federated learning training process, and proposes a data classification method and system based on a fair federated learning algorithm. Summary of the invention

[0005] The purpose of the present invention is to provide a data classification method and system based on a fair federated learning algorithm to solve the problem of unfair weight distribution to each user in the traditional federated learning training process.

[0006] To achieve the above object, the present invention provides a data classification method based on a fair federated learning algorithm, comprising the following steps:

[0007] Based on the entropy weight method, the data quality score of each user terminal is determined according to the entropy weight of each attribute in the local data set of the user terminal and the number of sample data corresponding to each attribute; the local data set contains a number of sample data and the attributes of each sample data;

[0008] Initializing a global model, and sending the initialized global model parameters to each user terminal;

[0009] According to the global model parameters, using the local data set owned by each user terminal, training is performed on each user terminal to obtain the local model parameters of each user terminal;

[0010] Determine the aggregation weight of each user terminal according to the data quantity of each user terminal and the data quality score of each user terminal;

[0011] Determining and updating the global model parameters according to the aggregation weight of each user terminal and the local model parameters of each user terminal;

[0012] Determine whether the condition for ending the iteration is met;

[0013] If not, jump to the step of "training each user terminal according to the parameters of the global model using the local data set owned by each user terminal to obtain the local model parameters of each user terminal";

[0014] If yes, the training is terminated and the global model classifies the data according to the trained global model parameters.

[0015] Optionally, based on the entropy weight method, the data quality score of each user terminal is determined according to the entropy weight of each attribute in the user terminal and the amount of data in each attribute, specifically including:

[0016] Preprocess the local data sets owned by each user terminal;

[0017] According to the preprocessed local data set, determine the probability of each sample of each user terminal appearing in the local data set;

[0018] Determine the information entropy of each attribute in each user terminal according to the probability of occurrence of each sample;

[0019] Determine the entropy weight of each attribute in each user terminal according to the information entropy of each attribute;

[0020] The data quality score of each user terminal is determined according to the entropy weight of each attribute in each user terminal and the amount of data in each attribute.

[0021] Optionally, the data quality score of each user terminal is determined using the following formula:

[0022]

[0023] in, Represents user u i The data quality score of k i represents the number of attributes of the i-th user, m represents the m-th attribute, and d ui Represents user u iThe number of samples, j represents the jth sample.

[0024] Optionally, obtaining the data quantity and data quality score of each user terminal and determining the aggregation weight of each user terminal specifically includes:

[0025] Determine the weight of each user's data quantity according to the proportion of each user's data quantity to the total data quantity;

[0026] Determine the data quality weight of each user according to the proportion of each user's data quality score to the total data quality score;

[0027] Determine the mixed weight of each user based on the data quantity weight and data quality weight of each user;

[0028] The aggregate weight of each user is determined according to the mixed weight of each user.

[0029] Optionally, the following formula is used to determine the mixed weight of each user:

[0030]

[0031] in, Represents user u i The mixing weights, Represents user u i The data quality weight, Represents user u i The data quantity weight is λ, and λ represents the proportion of data quality weight in the mixed weight.

[0032] Optionally, the global model is initialized using the following formula:

[0033]

[0034] Where p(y=1|x; θ) represents the probability that the classification result y is 1 when the attribute x is input; θ represents the weight of the attribute x, that is, the model parameter that needs to be optimized; e is the base of the natural logarithm; T represents the transpose operation of the matrix; g(θ T x) is the global model after initialization;

[0035] Its loss function is:

[0036]

[0037] Among them, m represents the number of samples participating in the training on the user side; p(x (i) ) represents the probability that the classification result y of the global model is 1 when the i-th attribute is input.

[0038] Optionally, the training is performed on each user terminal using a local data set owned by each user terminal according to the parameters of the global model, specifically including:

[0039] Generating a local model at each user terminal according to the global model parameters;

[0040] Using the local model to classify local data owned by each user terminal;

[0041] Perform a preset number of iterative trainings to update the parameters of the local model, as shown in the following formula:

[0042]

[0043] Among them, θ lt+1 represents the local model parameters of the lt+1th time, θ lt represents the local model parameter of the ltth time; η represents the learning rate of the user end; L represents the loss function of the global model.

[0044] Optionally, the parameters of the global model are updated using the following formula:

[0045]

[0046] Among them, θ gt+1 represents the global model parameters after the gt+1th update, represents the aggregation weight of the i-th user terminal, represents the local model parameters of the i-th client, and n is the number of clients.

[0047] Optionally, differential privacy is used to perform perturbations when determining and updating the global model parameters.

[0048] On the other hand, corresponding to the above-mentioned data classification method based on a fair federated learning algorithm, the present invention also provides a data classification system based on a fair federated learning algorithm, comprising:

[0049] The data quality determination module is used to determine the data quality score of each user terminal based on the entropy weight method according to the entropy weight of each attribute in the local data set of the user terminal and the number of sample data corresponding to each attribute;

[0050] An initialization module, which initializes the global model and sends the initialized parameters of the global model to each user terminal;

[0051] A local training module, which performs training on each user terminal according to the parameters of the global model and utilizes the local data set owned by each user terminal to obtain the local model parameters of each user terminal;

[0052] An aggregation weight determination module, used to determine the aggregation weight of each user terminal according to the data quantity of each user terminal and the data quality score of each user terminal;

[0053] A global model updating module, used to determine and update the parameters of the global model according to the aggregation weight of each user terminal and the local model parameters of each user terminal;

[0054] A judgment module is used to judge whether the condition for ending the iteration is met and obtain the judgment result;

[0055] An iterative update module, used for calling the local training module, the aggregation weight determination module and the global model update module when the judgment result is no;

[0056] The data classification module is used to end the iterative update when the judgment result is yes, and perform data classification according to the trained global model parameters.

[0057] According to the specific invention content provided by the present invention, the present invention discloses the following technical effects:

[0058] The present invention provides a data classification method and system based on a fair federated learning algorithm. The method uses the local data set owned by each user terminal to determine the data quality score of each user terminal based on the entropy weight method; after initializing the global model, the parameters of the global model are sent to each user terminal; according to the parameters of the global model, the local data set owned by each user terminal is used for training to obtain the local model parameters of each user terminal; the data quantity and data quality score of each user terminal are obtained to determine the aggregation weight of each user terminal; according to the aggregation weight of each user terminal and the local model parameters, the parameters of the global model are updated; a new global model is generated according to the parameters of the global model; if the global training times do not reach the upper limit of the global training times, the local data set owned by each user terminal is continued to be trained according to the parameters of the global model to obtain the local model parameters of each user terminal. The local model parameters of the terminal are obtained; the data quantity and data quality score of each user terminal are obtained, and the aggregation weight of each user terminal is determined; the parameters of the global model are updated according to the aggregation weight and local model parameters of each user terminal; when the global training times reach the upper limit of the global training times, the training is terminated, and the data is classified according to the global model; the present invention determines the data quality score of the user terminal based on the entropy weight method, and defines the aggregation weight based on the data quantity of the user terminal and the data quality of the user terminal, which solves the problem that the traditional federated learning only considers the data quantity of the user terminal and the aggregation weight distribution of each user is unfair. Compared with the traditional federated learning scheme, the scheme of the present invention is fairer to each user terminal participating in the training, and can better promote the user terminal with high data quality score to join the federated learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A flowchart of a data classification method based on a fair federated learning algorithm provided in Example 1 of the present invention;

[0061] Figure 2 A flowchart of step S1 in the data classification method provided in Example 1 of the present invention;

[0062] Figure 3 This is a flowchart of step S3 in the data classification method provided in Example 1 of the present invention;

[0063] Figure 4 A flowchart of step S4 in the data classification method provided in Example 1 of the present invention;

[0064] Figure 5 This is an overall framework diagram of the data classification method provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] The purpose of the present invention is to provide a data classification method and system based on a fair federated learning algorithm to solve the problem of unfair weight distribution to each user in the traditional federated learning training process.

[0067] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] Embodiment 1:

[0069] like Figure 1-4 As shown, the present invention provides a data classification method based on a fair federated learning algorithm, which is used to classify credit card fraud detection data, comprising the following steps:

[0070] In this embodiment, the classification data comes from UCI, a classification data set commonly used in machine learning, a credit card fraud detection data set creditcard, which performs a binary classification task. There are 30 attribute feature values ​​in the data set, all of which are numerical data, and there are 284,807 samples in total, as shown in Table 1:

[0071] Table 1 Creditcard original data set

[0072]

[0073] Divide the data in Table 1 into n users, each of which contains d i samples, each user’s sample is different, k i The data of each user after the division is shown in Table 2:

[0074] Table 2 User data table

[0075]

[0076] S1. Based on the entropy weight method, the data quality score of each user terminal is determined according to the entropy weight of each attribute in the local data set of the user terminal and the number of sample data corresponding to each attribute; the local data set contains a number of sample data and the attributes of each sample data;

[0077] like Figure 2 As shown, step S1 specifically includes:

[0078] S11. Preprocess the local data set owned by each user terminal; mainly including data normalization and data non-negativeness, that is, converting all data into the range of 0 to 1 and converting data containing negative numbers into positive numbers, and preprocessing each attribute in the user data according to the following formula:

[0079]

[0080] Among them, u i represents the i-th user, i = 1, 2, ..., n, i represents the number of the user terminal, n is the total number of user terminals; j = 1, 2, ..., d i , j represents the sample number, d i is the total number of samples; m=1,2,…,k i , m represents the number of the attribute, k i Indicates the total number of attributes; represents the first sample in attribute 1 of the first user, Representation sample The result after non-negative and normalized processing. 1The original data is shown in Table 3, and the normalized and non-negative data is shown in Table 4. The normalization and non-negative methods of other user data are similar.

[0081]

[0082] S12. According to the preprocessed local data set, determine the probability of each sample of each user terminal appearing in the local data set. The calculation formula is as follows:

[0083]

[0084] S13, determining the information entropy of each attribute in each user terminal according to the probability of occurrence of each sample; the calculation formula is as follows:

[0085]

[0086] S14, determining the entropy weight of each attribute in each user terminal according to the information entropy of each attribute; the calculation formula is as follows:

[0087]

[0088] S15. Determine the data quality score of each user terminal according to the entropy weight of each attribute in each user terminal and the amount of data in each attribute. The calculation formula is as follows:

[0089]

[0090] S2. Initialize the global model, and send the initialized parameters of the global model to each user terminal.

[0091] Use the following formula to initialize the global model:

[0092]

[0093] Where p(y=1|x; θ) represents the probability that the classification result y is 1 when the attribute x is input; θ represents the weight of the attribute x, that is, the model parameter that needs to be optimized; e is the base of the natural logarithm; T represents the transpose operation of the matrix; g(θ T x) is the global model after initialization.

[0094] Its loss function is:

[0095]

[0096] Among them, m represents the number of samples participating in the training on the user side; p(x (i) ) represents the probability that the classification result y of the global model is 1 when the i-th attribute is input.

[0097] S3, according to the parameters of the global model, using the local data set owned by each user terminal, training is performed on each user terminal to obtain the local model parameters of each user terminal;

[0098] S31, generating a local model at each user terminal according to the global model parameters;

[0099] S32, using the local model to classify the local data owned by each user terminal;

[0100] S33, perform a preset number of iterative trainings to update the parameters of the local model, as shown in the following formula:

[0101]

[0102] Among them, θ lt+1 represents the local model parameters of the lt+1th time, θ lt represents the local model parameter of the ltth time; η represents the learning rate of the user end; L represents the loss function of the global model.

[0103] S4. Determine the aggregation weight of each user terminal according to the data quantity of each user terminal and the data quality score of each user terminal;

[0104] S41. Determine the weight of the data quantity of each user according to the proportion of the data quantity of each user terminal to the total data quantity.

[0105]

[0106] S42. Determine the data quality weight of each user according to the proportion of each user's data quality score to the total data quality score.

[0107]

[0108] S43. Based on the weight of each user's data quantity and data quality weight The following formula is used to determine the mixed weight of each user

[0109]

[0110] in, Represents user u i The mixing weights, Represents user u i The data quality weight, Represents user u i The data quantity weight is λ, and λ represents the proportion of data quality weight in the mixed weight.

[0111] S44, based on the mixed weight of each user The following formula is used to determine the aggregation weight of each user

[0112]

[0113] S5. According to the aggregation weight of each user terminal and the local model parameters of each user terminal, the following formula is used to determine and update the parameters of the global model:

[0114]

[0115] Among them, θ gt+1 represents the global model parameters after the gt+1th update, represents the aggregation weight of the i-th user terminal, represents the local model parameters of the i-th client, and n is the number of clients.

[0116] When determining and updating the parameters of the global model, perturbations are performed through differential privacy, as shown in the following formula:

[0117]

[0118] Among them, N(0,σ 2 ) represents the added differential privacy. The role of differential privacy perturbation is to prevent information leakage during the training process.

[0119] S6. Determine whether the condition for ending the iteration is met; if not, jump to step S3; if yes, jump to step S7; the condition for ending the iteration may be: reaching a preset number of training times or the loss function L(θ) value converges, that is, tends to a stable value.

[0120] S7. End the training and perform data classification based on the trained global model.

[0121] The global model obtained by the data classification method based on the fair federated learning algorithm proposed in this embodiment needs to be compared with other solutions. The main indicator is fairness. There is currently no fairness definition standard, so according to the input contribution data, that is, the mixed weight of quantity and quality hw (u) , and proposes a fairness evaluation standard based on the idea that it should match the user's benefits. The user's benefits here refer to the accuracy of the model before the user joins the federated training. And the final optimal global model accuracy The difference between the two is the user's income (u) , which is formally expressed as follows:

[0122]

[0123] Equal Proportion-Fairness: Income received by users (u) With user contributions (u) Is proportional, that is: user contribution distribution and user revenue distribution The distance D kl The smaller the value, the fairer it is. The relative entropy knowledge can be used to calculate it as follows:

[0124]

[0125] Embodiment 2:

[0126] Corresponding to the data classification method based on a fair federated learning algorithm in Example 1, the present invention also provides a data classification system based on a fair federated learning algorithm, including:

[0127] The data quality determination module is used to use the local data set owned by each user terminal, based on the entropy weight method, to determine the data quality score of each user terminal according to the entropy weight of each attribute in the user terminal and the amount of data in each attribute;

[0128] An initialization module initializes the global model and sends the initialized global model parameters to each user terminal;

[0129] The local training module uses the local data set owned by each user terminal to perform training on each user terminal according to the global model parameters to obtain the local model parameters of each user terminal;

[0130] An aggregation weight determination module, used to determine the aggregation weight of each user terminal according to the data quantity of each user terminal and the data quality score of each user terminal;

[0131] A global model updating module, used to update global model parameters according to the aggregation weights of each user terminal and the local model parameters of each user terminal, and to update the global model according to the updated global model parameters;

[0132] A judgment module is used to judge whether the condition for ending the iteration is met and obtain the judgment result;

[0133] An iterative update module, used for calling the local training module, the aggregation weight determination module and the global model update module when the judgment result is no;

[0134] The data classification module is used to end the iterative update and perform data classification according to the trained global model when the judgment result is yes.

[0135] The program part of the technology can be considered as a "product" or "manufactured product" in the form of executable code and / or related data, which is participated in or realized by computer-readable media. Tangible and permanent storage media can include any memory or storage used by any computer, processor, or similar device or related module. For example, various semiconductor memories, tape drives, disk drives or any similar devices that can provide storage functions for software.

[0136] All or part of the software may sometimes communicate over a network, such as the Internet or other communications network. Such communications can load software from one computer device or processor to another. For example: from a server or host computer of a video target detection device to a hardware platform of a computer environment, or other computer environment that implements the system, or a system with similar functions related to providing information required for target detection. Therefore, another medium capable of transmitting software elements can also be used as a physical connection between local devices, such as light waves, radio waves, electromagnetic waves, etc., transmitted through cables, optical cables or air. Physical media used to carry carriers, such as cables, wireless connections or optical cables and the like, can also be considered as media that carry software. As used herein, unless limited to tangible "storage" media, other terms referring to computer or machine "readable media" refer to media involved in the process of the processor executing any instructions.

[0137] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core ideas of the present invention. Those skilled in the art should understand that the modules or steps of the present invention can be implemented by a general-purpose computer device. Alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into integrated circuit modules, or multiple modules or steps can be made into a single integrated circuit module. The present invention is not limited to any specific combination of hardware and software.

[0138] Meanwhile, for those skilled in the art, according to the concept of the present invention, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A data classification method based on a fair federated learning algorithm, characterized in that: The method comprises: Based on the entropy weight method, the data quality score of each user terminal is determined according to the entropy weight of each attribute in the local data set of the user terminal and the number of sample data corresponding to each attribute; the local data set contains a number of sample data and the attributes of each sample data; Initializing a global model, and sending the initialized parameters of the global model to each user terminal; According to the parameters of the global model, using the local data set owned by each user terminal, training is performed on each user terminal to obtain the local model parameters of each user terminal; Determine the aggregation weight of each user terminal according to the data quantity of each user terminal and the data quality score of each user terminal; Determine and update the parameters of the global model according to the aggregation weight of each user terminal and the local model parameters of each user terminal; Determine whether the conditions for ending the iteration are met; If not, jump to the step of "training each user terminal according to the parameters of the global model using the local data set owned by each user terminal to obtain the local model parameters of each user terminal"; If so, the training is terminated, and the global model performs data classification according to the trained global model parameters.

2. The data classification method according to claim 1, characterized in that: The entropy weight method is based on the entropy weight of each attribute in the user terminal and the amount of data in each attribute to determine the data quality score of each user terminal, specifically including: Preprocess the local data sets owned by each user terminal; According to the preprocessed local data set, determine the probability of each sample of each user terminal appearing in the local data set; Determine the information entropy of each attribute in each user terminal according to the probability of occurrence of each sample; Determine the entropy weight of each attribute in each user terminal according to the information entropy of each attribute; The data quality score of each user terminal is determined according to the entropy weight of each attribute in each user terminal and the amount of data in each attribute.

3. The data classification method according to claim 2, characterized in that: The data quality score of each user terminal is determined using the following formula: in, Represents user u i The data quality score of k i represents the number of attributes of the i-th user, m represents the m-th attribute, Represents user u i The number of samples, j represents the jth sample.

4. The data classification method according to claim 1, characterized in that: Determining the aggregation weight of each user terminal according to the data quantity and data quality score of each user terminal specifically includes: Determine the weight of each user's data quantity according to the proportion of each user's data quantity to the total data quantity; Determine the data quality weight of each user according to the proportion of each user's data quality score to the total data quality score; Determine the mixed weight of each user based on the data quantity weight and data quality weight of each user; According to the mixed weight of each user, the aggregate weight of each user is determined.

5. The data classification method according to claim 4, characterized in that: The following formula is used to determine the mixed weight of each user: in, Represents user u i The mixed weights, Represents user u i The data quality weight, Represents user u i The data quantity weight is λ, and λ represents the proportion of data quality weight in the mixed weight.

6. The data classification method according to claim 1, characterized in that: The global model is initialized using the following formula: Where p(y=1|x; θ) represents the probability that the classification result y is 1 when the attribute x is input; θ represents the weight of the attribute x, that is, the model parameter that needs to be optimized; e is the base of the natural logarithm; T represents the transpose operation of the matrix; g(θ T x) is the global model after initialization; Its loss function is: Among them, m represents the number of samples participating in the training on the user side; p(x (i) ) represents the probability that the classification result y of the global model is 1 when the i-th attribute is input.

7. The data classification method according to claim 6, characterized in that: The training is performed on each user terminal using the local data set owned by each user terminal according to the parameters of the global model, specifically including: Generating a local model at each user terminal according to the global model parameters; Using the local model to classify local data owned by each user terminal; Perform a preset number of iterative trainings to update the parameters of the local model, as shown in the following formula: Among them, θ lt+1 represents the local model parameters of the lt+1th time, θ lt represents the local model parameter of the ltth time; η represents the learning rate of the user end; L represents the loss function of the global model.

8. The data classification method according to claim 1, characterized in that: The parameters of the global model are updated using the following formula: Among them, θ gt+1 represents the global model parameters after the gt+1th update, represents the aggregation weight of the i-th user terminal, represents the local model parameters of the i-th client, and n is the number of clients.

9. The data classification method according to claim 1, characterized in that: Differential privacy is used to perform perturbations when determining and updating parameters of the global model.

10. A data classification system based on a fair federated learning algorithm, characterized in that: The data classification system comprises: The data quality determination module is used to determine the data quality score of each user terminal based on the entropy weight method according to the entropy weight of each attribute in the local data set of the user terminal and the number of sample data corresponding to each attribute; An initialization module, which initializes the global model and sends the initialized parameters of the global model to each user terminal; A local training module, which performs training on each user terminal according to the parameters of the global model and utilizes the local data set owned by each user terminal to obtain the local model parameters of each user terminal; An aggregation weight determination module, used to determine the aggregation weight of each user terminal according to the data quantity of each user terminal and the data quality score of each user terminal; A global model updating module, used to determine and update the parameters of the global model according to the aggregation weight of each user terminal and the local model parameters of each user terminal; A judgment module is used to judge whether the condition for ending the iteration is met and obtain the judgment result; An iterative update module, used for calling the local training module, the aggregation weight determination module and the global model update module when the judgment result is no; The data classification module is used to end the iterative update when the judgment result is yes, and perform data classification according to the trained global model parameters.

Images