A reinforcement learning-based federal system bias poisoning defense method and device

Abstract

This invention discloses a method and apparatus for preventing bias poisoning in a federated system based on reinforcement learning, comprising: a server calling a global model and distributing it to each client; each client receiving the global model from the central server and using local data to detect whether the global model is biased towards sensitive attributes based on an equal chance bias index; if bias is detected in the global model, a federated learning method based on reinforcement learning is used to remove the bias, resulting in a debiased model; the debiased model is trained using local data and uploaded to the central server; if no bias is detected in the global model, a global model is trained using local data and uploaded to the central server; the trained global model and the debiased model are aggregated to obtain a new global model, which is then distributed to the clients; the above steps are repeated until the federated learning based on reinforcement learning is completed.

Description

Technical Field

[0001] This invention belongs to the field of defense for federated learning, specifically relating to a method and apparatus for preventing bias poisoning in federated systems based on reinforcement learning. Background Technology

[0002] Federated learning has become a highly anticipated area of ​​artificial intelligence in recent years. With its rapid development and application, federated learning is now widely used in fields such as finance, healthcare, computer vision, and urban management. To ensure the safe application of federated learning in these fields, it is crucial to analyze and discover vulnerabilities in federated learning algorithms and models to prevent exploitation for illegal profit. Unlike other centralized deep learning methods, federated learning requires multiple distributed learners to learn collaboratively, with the learning process coordinated by a central server. Throughout the process, client data is stored locally, and only model parameters are communicated between clients via a parameter server. A typical training iteration works as follows: First, the central server sends the latest global model to each client. Then, each client updates its model locally using its local data and reports the updated model to the parameter server. Finally, the server aggregates all submitted local updates to form a new global model, which performs better than models trained using data from any single client. Therefore, the federated global model is highly correlated with each client.

[0003] However, federated learning systems are vulnerable to attacks from malicious clients. The central server cannot access client data and therefore cannot verify model updates from clients, especially when the system incorporates secure aggregation protocols to further protect client privacy. Theoretically, a malicious client can send any update to the server, and without effective protection to identify malicious updates to the neural network's learning weights, the server is easily compromised.

[0004] Sending malicious models to the server from a malicious client is one of the most common attacks in federated learning. Attackers can modify the gradients of individual neurons in their local model, increasing the model's bias towards specific attributes. These neurons are highly correlated with certain sensitive attributes. After the client modifies these neurons and sends them to the federated learning system for aggregation, the global model of the federated learning system will be poisoned with bias. Because the server cannot access the client's data, it is very difficult to distinguish malicious model updates.

[0005] This invention proposes a reinforcement learning-based method for preventing bias poisoning in horizontal federated learning, based on an anomaly detection mechanism of related neurons. This method performs global model update anomaly detection based on the difference between related neurons in the biased model and those in the normal model. Using only a small batch of data on the client side, it can determine the related neurons for biased attributes. By comparing the related neuron parameters of the unbiased model with those of the unbiased model, reinforcement learning is used to automatically fine-tune the biased model to an unbiased model, preserving the model updates obtained from federated learning while removing model bias introduced by malicious clients. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a method for preventing bias poisoning in federated systems based on reinforcement learning.

[0007] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows: A first aspect of the present invention provides a method for preventing bias poisoning in federated systems based on reinforcement learning, the method comprising:

[0008] The server calls the global model and distributes it to each client. The client receives the global model from the central server and uses local data to detect whether the global model is biased against sensitive attributes based on the equal chance bias index.

[0009] If the global detection model is biased, a federated learning method based on reinforcement learning is used to remove the bias, resulting in a debiased model. The debiased model is then trained using local data and uploaded to the central server.

[0010] If the global model is found to be unbiased, the global model is trained using local data and then uploaded to the central server.

[0011] The trained global model and the trained debiased model are aggregated to obtain a new global model, which is then sent to the client.

[0012] Repeat the above steps until the reinforcement learning-based federated learning ends.

[0013] Furthermore, if the global detection model is biased, a reinforcement learning-based federated learning method is used to remove the bias. The process of obtaining the debiased model includes:

[0014] If bias is detected in the global model, the global model is copied and trained using an unbiased dataset until the bias value of the global model is reduced to less than a custom equal chance bias metric threshold. The trained global model is then used as a low-bias model.

[0015] Find biased neurons: Input the trained low-bias model into a batch of samples, flip the data sensitivity attribute labels in the batch of samples, compare the difference in activation values ​​of neurons before and after the data sensitivity attribute flipping, and identify the neurons in the low-bias model that are related to the sensitivity attribute, i.e., biased neurons.

[0016] Referring to the bias neuron parameters in the low-bias model, the corresponding bias neuron parameters in the global model are modified through a Markov decision process. The bias neurons in the global model are fine-tuned until the bias value of the global model is less than the custom threshold for equal opportunity bias, thus obtaining the debiased model.

[0017] Furthermore, the bias value, also known as the equal opportunity bias index, is calculated using the following formula:

[0018]

[0019] Where Pr represents the probability of the model's predicted outcome. S represents the model's true prediction result, where S is the sensitive attribute of the sample and Y is the sample's true label.

[0020] Furthermore, the threshold for the custom equal opportunity bias indicator is set to 0.1.

[0021] Furthermore, the process of identifying bias neurons is as follows:

[0022] Identifying biased neurons: The trained low-bias model is input into the same batch of samples, where the activation values ​​of its neurons are the same. The data sensitivity attribute labels in a portion of the samples are flipped, i.e., the data sensitivity attribute label S=1 is changed to S=0. Due to the change in the data sensitivity attribute label, the activation values ​​of neurons in the two batches of samples differ. The change in activation value of some neurons is significantly greater than that of other neurons, showing a strong correlation with the sensitivity attribute. Neurons with a significantly greater change in activation value than other neurons are identified as biased neurons.

[0023] Furthermore, the Q-Learning algorithm is used in the process of modifying biased neurons through Markov decision processes.

[0024] A second aspect of the present invention provides a reinforcement learning-based federal system bias poisoning defense device, comprising one or more processors for implementing the reinforcement learning-based federal system bias poisoning defense method described above.

[0025] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-described reinforcement learning-based federated system bias poisoning defense method.

[0026] The beneficial effects of this invention are mainly reflected in: 1) using local datasets to detect bias in the global model, protecting the local model and improving fairness; 2) in the federated learning process, associating neuron activation with sensitive attributes to improve the stability of fairness poisoning detection; 3) this invention performs model bias detection and debiasing tasks in each federated learning session to ensure the fairness of the client model at all times. Attached Figure Description

[0027] Figure 1 This is a flowchart of the method of the present invention.

[0028] Figure 2 This is a block diagram of the reinforcement learning-based federal system bias poisoning defense method of the present invention.

[0029] Figure 3 A schematic diagram of a bias neuron;

[0030] Figure 4 This is a schematic diagram of the device of the present invention. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0033] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0034] Currently, federated learning is being applied in increasingly wider fields, but this brings with it various problems that impair model performance. For example, modifying the parameters of relevant neurons can introduce bias into the model, leading to classification errors on specific tasks. In response to this, this invention proposes a reinforcement learning-based method for defending against bias poisoning attacks in federated systems, protecting the model from such attacks.

[0035] The technical concept of this invention is as follows: the bias of the model is strongly correlated with some fixed neurons. Finding these neurons and making specific modifications can increase or decrease the bias. By fine-tuning these biased neurons, the bias can be significantly reduced without losing federated learning information.

[0036] Based on this situation, deep learning combined with neuron activation data is used to defend against federated learning bias poisoning attacks. First, after receiving an update from the server, the client checks if the model is biased. If the model shows significant bias on a certain sensitive attribute, the client flips the sensitive attribute labels on a batch of data and compares the difference in activation values ​​of the model's neurons before and after the attribute flip. This identifies a batch of neurons highly correlated with the sensitive attribute and defines them as biased neurons for that sensitive attribute. Then, the client continues to filter a batch of unbiased datasets for that attribute, backs up a biased global model, and trains this global model on the unbiased dataset until the model shows very low bias for that sensitive attribute. Finally, reinforcement learning is used to fine-tune the biased neurons of the biased global model, with the reference value being the global model retrained on unbiased data.

[0037] First, let me state the definition of bias and the method for measuring the degree of bias in the model in this invention:

[0038] The bias removal method of this invention is applied to data and models with demographic factors, such as age and gender. The bias primarily stems from unequal data quantities across attributes. For example, in a classification result where the number of male samples is significantly higher than female samples in the gender attribute, a model trained with such data will overemphasize the importance of gender while ignoring the influence of other attributes, leading to a biased classification that favors male samples. Therefore, in this example, equal opportunity bias is used. This is a posterior method that can simultaneously determine the fairness of the dataset and the model. For a dataset, directly comparing the proportion of classification labels under sensitive attributes indicates a lower bias for that attribute as the proportions become more consistent, and vice versa. For the model, the proportion of the model's classification accuracy on a fair test set, relative to the accuracy of classification of sensitive attributes, effectively reflects the model's fairness. The formula for calculating equal opportunity bias is:

[0039]

[0040] Where Pr represents the probability of the model's predicted outcome. S represents the model's true prediction result, where S is the sensitive attribute of the sample and Y is the sample's true label.

[0041] Reference Figures 1-3 This invention proposes a reinforcement learning-based method for defending against bias poisoning in federated systems, with the following steps:

[0042] (1) The server calls the global model and distributes it to each client. The client receives the global model A from the central server and uses local data to detect whether the global model A is biased against sensitive attributes based on the equal opportunity bias index. A custom threshold for the equal opportunity bias index is set. If the threshold is higher than the threshold, the global model is biased; if it is lower than the threshold, the global model is not biased.

[0043] (2) If the global model is biased, the federated learning method based on reinforcement learning is used to remove the bias; if the global model is not biased, the global model is trained using local data and the trained global model is uploaded to the central server.

[0044] The training objective of the federated learning is reduced to a finite optimization, and the formula for the training objective is as follows:

[0045]

[0046] Where N represents the existence of N participants each processing N local models f. i (w), where w are the parameters of the model, and each side is based on a private dataset. Using local target f i :R d → R is used for training, where a i =|D i | and This represents each data sample and its corresponding label. The goal of federated learning is to obtain a global model F(w), which is used to generalize the test data after aggregating the distributed training results from N parties.

[0047] Specifically, in round t, the central server will share the current model G. t Send to N selected parties, where [N] represents the set of integers {1, 2, ..., N}. Selected party i uses its own dataset D. i The optimization algorithm, with a learning rate lr, runs for E local epochs to locally compute the function f. i To obtain a new local model Then, the client updates the model. The data is sent to a central server, which averages all updates using its own learning rate η to generate a new global model G. t+1 :

[0048]

[0049] For attackers, bias poisoning aims to mislead a trained model, increasing its bias towards a specific attribute. In federated learning, the goal of a bias poisoning attack is to manipulate the bias of the local model, causing the global model to exhibit significant bias on the primary task, ultimately infecting all federated learning participants. Attacker i has local data D. i The objective in round t with target label τ is:

[0050]

[0051] in, For data samples, Data that represents the true labels of the samples and is biased. With clean data satisfy and Function P is the corresponding training optimization function, and function R uses a set of parameters φ to transform normal data in any class into biased data with attacker selection. Therefore, the normal model w i By maximizing this formula, it can be transformed into a bias model.

[0052] The method of removing bias using reinforcement learning specifically includes the following sub-steps:

[0053] (2.1) If bias is detected in the model, copy the global model and train it using the unbiased dataset until the bias value of the model is reduced to EOD < 0.1, thus obtaining the low-bias model B;

[0054] (2.2) Flip the data-sensitive attribute labels and compare the difference in activation values ​​of neurons before and after attribute flipping to identify a group of neurons in the model that are highly correlated with the sensitive attributes, namely biased neurons;

[0055] Regarding the identification of biased neurons, when a trained model is input with the same batch of samples, the activation values ​​of its neurons are the same. However, if the sensitivity attribute S=1 of one batch of samples is changed to S=0, the activation values ​​of neurons in the two batches of samples will differ. These differences are caused by the change in these sensitivity attributes. In fully connected deep learning models, the activation value of each neuron changes, but the change in some neurons is significantly greater than that of other neurons, showing a strong correlation with the sensitivity attribute. Therefore, these neurons are defined as biased neurons, such as... Figure 3 As shown.

[0056] (2.3) Referring to the bias neuron parameters in the low-bias model B trained in step (2.1), the corresponding bias neuron parameters in the global model are modified through a Markov decision process. Reinforcement learning is then used to fine-tune the bias neurons in the high-bias global model A (Equal Opportunity Bias EOD > 0.1) until the bias value EOD of global model A is < 0.1, enabling global model A to achieve both high accuracy and debiasing. The specific method is as follows:

[0057] Once the biased neuron from step (2.2) is determined, manually modifying it can affect the fairness of the model for that sensitive attribute, increasing or decreasing the bias. Therefore, reinforcement learning is used to automatically modify neuron parameters. Reinforcement learning can be summarized as a Markov decision process, simply put, a cyclical process in which an agent takes action to change its state, obtains a reward, and interacts with the environment. Among the many reinforcement learning algorithms, the Q-Learning algorithm is most suitable for the scheme of this invention. The idea of ​​Q-Learning is to first select action A using a greedy method based on the current state S, then obtain a reward R, and update to a new state S′. Based on state S′, the optimal A′ is directly selected from all actions using a greedy method, summarized as the following formula:

[0058] Q(S,A)=Q(S,A)+α(R+γQ(S′,A′)-Q(S,A))

[0059] The Q-Learning algorithm works by creating a Q-Table to store states S and all possible actions A. In each round, a first state is randomly initialized. Then, for each step in the round, an action A is selected from the Q-Table based on the current state S using a greedy algorithm. Action A is executed, resulting in a new state S′ and the current reward R. Simultaneously, the value of Q(S,A) in the table is updated, and the loop continues until the end. Here, α is the learning rate and γ is the reward decay coefficient. The entire algorithm continuously updates the values ​​in the Q-Table and determines the best action to take in a given state based on the updated values. In the debiasing scheme of this invention, the goal of reinforcement learning is to minimize bias while ensuring the main performance of the model. Amplifying or reducing the parameters of biased neurons is used as the reinforcement learning action A. The modified model's bias and model accuracy are used to design the reward R. Four combinations of increasing or decreasing bias and increasing or decreasing model accuracy are used as states S to construct the reinforcement learning. It is worth noting that each round of federated learning includes one or more reinforcement learning processes, depending on the number of sensitive attributes with bias.

[0060] (3) Use local data to train the debiased model C obtained in step (2), and upload the trained debiased model C to the central server;

[0061] (4) Aggregate the trained global model A obtained in step (2) and the debiased model C obtained in step (3) to obtain a new global model, and send the new global model to the client.

[0062] (5) Repeat steps (2) to (4) above until the federated learning ends.

[0063] The method of this invention is an organic combination of neural networks, federated learning and reinforcement learning. It uses reinforcement learning to adaptively fine-tune biased neurons, thereby reducing model bias while ensuring the stability of main performance.

[0064] The method of this invention does not require detecting which client is malicious. When applied to the client, it only performs fairness checks on the global model and performs model debiasing in each round of federated learning, ensuring the fairness of the client model to the greatest extent possible. The descriptions in the embodiments of this specification are merely examples of implementations of the inventive concept. The scope of protection of this invention should not be considered limited to the specific forms stated in the embodiments, and the scope of protection of this invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

[0065] Corresponding to the aforementioned embodiments of the reinforcement learning-based federal system bias poisoning defense method, the present invention also provides embodiments of a reinforcement learning-based federal system bias poisoning defense device.

[0066] See Figure 4 The present invention provides a reinforcement learning-based federal system bias poisoning defense device, which includes one or more processors for implementing the reinforcement learning-based federal system bias poisoning defense method in the above embodiments.

[0067] The embodiment of the reinforcement learning-based federated system bias poisoning defense device of this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any data processing-capable device in which the reinforcement learning-based federated system bias poisoning defense device of this invention is located. (Except for...) Figure 4In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0068] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0069] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0070] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the reinforcement learning-based federated system bias poisoning defense method described in the above embodiments.

[0071] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for preventing bias poisoning in a federated system based on reinforcement learning, characterized in that, The method includes: The server calls the global model and distributes it to each client. The client receives the global model from the central server and uses local data to detect whether the global model is biased against sensitive attributes based on the equal chance bias index. If the global detection model is biased, a federated learning method based on reinforcement learning is used to remove the bias, resulting in a debiased model. The debiased model is then trained using local data and uploaded to the central server. If the global model is found to be unbiased, the global model is trained using local data and then uploaded to the central server. The trained global model and the trained debiased model are aggregated to obtain a new global model, which is then sent to the client. Repeat the above steps until the reinforcement learning-based federated learning ends; If the global detection model is biased, a reinforcement learning-based federated learning method is used to remove the bias. The process of obtaining the debiased model includes: If bias is detected in the global model, the global model is copied and trained using an unbiased dataset until the bias value of the global model is reduced to less than a custom equal chance bias metric threshold. The trained global model is then used as a low-bias model. Find biased neurons: Input the trained low-bias model into a batch of samples, flip the data sensitivity attribute labels in the batch of samples, compare the difference in activation values ​​of neurons before and after the data sensitivity attribute flipping, and identify the neurons in the low-bias model that are related to the sensitivity attribute, i.e., biased neurons. Referring to the bias neuron parameters in the low-bias model, the corresponding bias neuron parameters in the global model are modified through a Markov decision process. The bias neurons in the global model are fine-tuned until the bias value of the global model is less than the custom threshold of equal opportunity bias index, thus obtaining the debiased model. The bias value, or equal opportunity bias indicator, is calculated using the following formula: ； in, This represents the probability of the model's predicted outcome. This represents the model's true prediction results. For the sensitive attributes of the sample, The true label of the sample; The process of finding bias neurons specifically involves: Identifying biased neurons: The trained low-bias model is input into the same batch of samples, where the activation values ​​of its neurons are the same. The data sensitivity attribute labels in a portion of the samples are flipped, i.e., the data sensitivity attribute label S=1 is changed to S=0. Due to the change in the data sensitivity attribute label, the activation values ​​of neurons in the two batches of samples are different. The change in activation value of some neurons is significantly greater than that of other neurons, showing a strong correlation with the sensitivity attribute. Neurons with a significantly greater change in activation value than other neurons are identified as biased neurons. In the process of modifying biased neurons through Markov decision process, the Q-Learning algorithm is used. The amplification or reduction of biased neuron parameters is taken as the action A of reinforcement learning. The modified model's bias and model accuracy are designed as rewards R. The four combinations of increasing or decreasing bias and increasing or decreasing model accuracy are used as states S to build reinforcement learning.

2. The method for preventing bias poisoning in a federated system based on reinforcement learning according to claim 1, characterized in that, The custom threshold for equal opportunity bias is set to 0.

1.

3. A reinforcement learning-based defense device for preventing bias poisoning in a federated system, characterized in that, It includes one or more processors for implementing the reinforcement learning-based federal system bias poisoning defense method as described in any one of claims 1-2.

4. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program is used to implement the reinforcement learning-based federal system bias poisoning defense method as described in any one of claims 1-2.

Images