Attention-based federated learning method and device, equipment and storage medium

Abstract

The application relates to an attention-activated federated learning method and device, equipment and a storage medium, and belongs to the field of artificial intelligence security in the computer field. The application optimizes a traditional federated learning framework, uses attention instead of the concept of local model communication, proposes an attention-activated Byzantine robust federated learning method, enhances the robustness of the federated learning framework, greatly reduces the bandwidth of the federated learning in the communication process through data compression and other means, and designs a safer aggregation algorithm on the server, so that the simultaneous defense of the directed attack and the non-directed attack is achieved.

Description

Technical Field

[0001] This invention relates to the fields of federated learning and knowledge transfer technology, and in particular to a federated learning method, apparatus, device and storage medium based on activation attention. Background Technology

[0002] Federated learning is a decentralized machine learning framework designed to address data bastion and data privacy issues. In a federated learning framework, there is one service provider and multiple clients. The server only needs to collect the model parameters trained by each client to aggregate the global model, eliminating the need for local private data.

[0003] However, the existence of Byzantine attacks threatens the security of federated learning systems. For example, a malicious server attacker can receive local model updates and tamper with the global model, while a malicious client attacker can manipulate local updates and affect the global model. Current mainstream defense methods focus on designing evaluation criteria and optimization aggregation rules for local model updates during federated learning training to defend against data poisoning and model poisoning. Examples include loss function-based filtering (LFR), error rate-based filtering (ERR), and Krum, Bulyan, and Trimmed mean methods. Therefore, these defense methods ignore the limitations of federated learning: locally updated models can expose their network structure and detailed parameters, which attackers can easily exploit to create Byzantine attack techniques. Summary of the Invention

[0004] The problem to be solved by the present invention is to provide a federated learning method, apparatus, device and storage medium based on activation attention, so as to overcome the limitation that the local update model of federated learning can reveal its network structure and detailed parameters.

[0005] The present invention adopts the following technical solution:

[0006] A federated learning method based on activation attention includes the following steps:

[0007] S1. Privacy-respecting local attention extraction: Based on the federated learning method, a local attention is defined. The local attention is extracted from the activation layer of the convolutional neural network along the channel dimension and is used to replace the local model trained by the local client using privacy data for updating and participate in the aggregation rules of federated learning.

[0008] S2. Scoring and clustering local attention: The distribution distance between local attentions is calculated using the Wasserstein bulldozer distance method as the score of the local attention. The score is then used to cluster similar local attentions using the k-median algorithm and aggregated into global attention.

[0009] S3. Transferring shared knowledge from global attention to local attention: Train local attention using the FedNAT loss function to transfer global attention.

[0010] Specifically, in step S1, the local attention is extracted from the activation layers of the convolutional neural network along the channel dimension and calculated using the following formula:

[0011]

[0012] Among them, F i This represents the i-th channel of the output feature map of the convolutional activation layer, where p represents the norm value, |F i | represents F i The absolute value; This represents the distribution of features across all channels of the Feature Map, and the p-norm value is used to highlight the probability distribution of these features on the image.

[0013] Furthermore, step S2 includes the following sub-steps:

[0014] S2.1 Calculate the distribution distance between the i-th local attention and other local attention using Wasserstein distance, as shown in the following formula:

[0015]

[0016] in, This represents the Wasserstein distance between the i-th local Attention and the j-th local Attention. This represents the i-th local attention. Let represent the j-th local Attention, and vec() represent the vectorized form of the local Attention;

[0017] S2.2 The score of the i-th local attention is calculated using the k-median algorithm, with the following relationship:

[0018]

[0019] in, Let κMedian[] represent the score of the i-th local attention in the aggregation rule, κMedian[] represent the k-median algorithm, and N represent the total number of local attentions. This represents any value that variable j can take among N numbers.

[0020] S2.3 Calculate the scores of all local attention points sequentially, and then use the calculated local attention scores... The k-median algorithm is used to select the NF clean local attention points with the most compact distribution, as shown in the following formula:

[0021]

[0022] Where G represents the filtered clean local Attention set, g i Let G represent the i-th local attention selected, and stop selecting when |G| = NF. Let N represent the total number of local attentions, and F represent the number of potential polluting attentions.

[0023] S2.4. Aggregate the clean local attention set G mentioned above using the FedAvg algorithm to obtain the global attention, as shown in the following formula:

[0024]

[0025] Among them, D n express Size of the local client dataset in the k-th round of communication This represents the nth local attention in the k-th round of communication. This represents the global attention obtained from aggregation.

[0026] Furthermore, in step S3, the local attention is trained using the FedNAT loss function, as shown in the following formula:

[0027]

[0028] in This represents the standard cross-entropy of the local model θ on the test set. Indicates that global Attention is at the th layer The first and local attention layer The L2 norm is normalized to represent their difference loss; α, β, and ρ represent the activation vectors of the i-th fully connected layer for global Attention and local Attention, respectively, and the bias loss is calculated using the Kullback-Leibler algorithm; α, β, and ρ are hyperparameters that control the degree of Attention transfer.

[0029] Furthermore, the present invention also proposes a computing device based on a federated learning method using activation attention, comprising:

[0030] The acquisition module is used by the server to obtain the models trained by all clients using local data in federated learning, as well as the size of the dataset during the training process.

[0031] The calculation module is used to calculate the local attention of the corresponding client based on the model trained by all clients using local data, and to calculate the distribution score of the local attention on the server side, and to cluster multiple clean local attentions from the local attention scores to calculate the global attention.

[0032] In the update module, the server distributes the global Attention to all clients, and the clients update their local models according to the FedNAT loss function.

[0033] Furthermore, the present invention also includes an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the computer program in the memory is executed by the processor, and the computer program is used to implement the steps in the federated learning method based on activation attention described above.

[0034] Furthermore, the present invention also includes a computer-readable storage medium for storing the computer program, characterized in that, when the computer-readable storage medium can read the computer program into a memory and be executed by a processor, it can implement the steps in any of the above-described federated learning methods based on activation attention.

[0035] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0036] 1. This invention proposes a method for applying knowledge transfer to federated learning, which is a novel Byzantine robust federated learning framework that eliminates the possibility of attackers spying on client models and network structures, and can fundamentally prevent most attacks against federated models.

[0037] 2. In high-level non-IID datasets, where attackers can obtain the server's aggregation rules and all clients' update parameters, this invention demonstrates good robustness against both targeted and non-targeted attacks. It also significantly reduces the bandwidth required during communication and improves the convergence efficiency of federated learning. Attached Figure Description

[0038] Figure 1 This is a flowchart of the federated learning method based on activation attention in this invention;

[0039] Figure 2 This is a software flowchart of the federated learning method based on activated attention, as described in this invention.

[0040] Figure 3 This is a block diagram of the computing device module for the federated learning method based on activation attention of the present invention;

[0041] Figure 4 This is a basic structural block diagram of the electronic device of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are merely illustrative of the invention and constitute only a part of the embodiments involved in this invention, and are not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concept of the invention. All non-innovative embodiments implemented in this way by other researchers in the art are within the protection scope of this invention.

[0043] This invention provides a federated learning method based on activation attention, such as... Figure 1 The steps shown are as follows:

[0044] S1. Privacy-respecting local attention extraction: Based on the federated learning method, a local attention is defined. The local attention is extracted from the activation layer of the convolutional neural network along the channel dimension and is used to replace the local model trained by the local client using privacy data for updating and participate in the aggregation rules of federated learning.

[0045] S2. Scoring and clustering local attention: The distribution distance between local attentions is calculated using the Wasserstein bulldozer distance method as the score of the local attention. The score is then used to cluster similar local attentions using the k-median algorithm and aggregated into global attention.

[0046] S3. Transferring shared knowledge from global attention to local attention: Train local attention using the FedNAT loss function to transfer global attention.

[0047] As a specific embodiment of the present invention, such as Figure 2 As shown,

[0048] First, the local client extracts the Attention from the local model trained on private data. The Attention is extracted from the convolutional activation layers along the channel dimension of the local model, and the Attention is used to replace the local model update to participate in the aggregation rules of federated learning.

[0049] Then: Based on the definition of Attention above, it is specifically calculated using the following relationship:

[0050]

[0051] Among them, F i This represents the i-th channel of the output feature map of the convolutional activation layer, where p represents the norm value, |F i | represents F i The absolute value; This represents the distribution of features across all channels of the Feature Map, and the p-norm value is used to highlight the probability distribution of these features on the image.

[0052] Next, the server obtains the client's local attention and uses the Wasserstein method to calculate the distribution distance between the local attentions as its score.

[0053] Specifically, calculating the score of local attention as described in step 3 includes the following steps:

[0054] (1) The distribution distance between the i-th local attention and other local attentions is calculated using the Wasserstein distance, which is calculated by the following relationship:

[0055]

[0056] in, This represents the Wasserstein distance between the i-th local Attention and the j-th local Attention. This represents the i-th local attention. Let represent the j-th local Attention, and vec() represent the vectorized form of the local Attention;

[0057] (2) The score of the i-th local attention is calculated using the k-median algorithm, and is calculated using the following formula:

[0058]

[0059] in, Let κMedian[] represent the score of the i-th local attention in the aggregation rule, κMedian[] represent the k-median algorithm, and N represent the total number of local attentions. This represents any value that variable j can take among N numbers.

[0060] (3) The server uses the scores of the local attention mentioned above, uses the k-median algorithm to cluster the local attention with similar scores, and uses it to aggregate the global attention.

[0061] Based on the scoring of the local attention The k-median algorithm is used to select the NF clean local attention nodes with the most compact distribution, which are calculated using the following formula:

[0062]

[0063] Where G represents the filtered clean local Attention set, g i Let G represent the i-th local attention selected, and stop selecting when |G| = NF. Let N represent the total number of local attentions, and F represent the number of potential polluting attentions.

[0064] (4) G is aggregated using the FedAvg algorithm to obtain the global Attention, which is calculated using the following relationship:

[0065]

[0066] Where G represents the selected clean local Attention set. Let g represent the nth local attention in the kth round of communication. i Let G represent the i-th local attention selected, and stop selecting when |G| = NF. Let N represent the total number of local attentions, and F represent the number of potential polluting attentions.

[0067] Next, the server distributes the aforementioned global attention to all clients.

[0068] The client uses the global attention mechanism to calculate the FedNAT loss function to train a new round of local model. Calculated using the following formula:

[0069]

[0070] in, This represents the standard cross-entropy of the local model θ on the test set. Indicates that global Attention is at the th layer The first and local attention layer The l2 norm is normalized to represent their difference loss. Let represent the activation vectors of the i-th fully connected layer for global Attention and local Attention, respectively. The bias loss is calculated using the Kullback-Leibler algorithm. Furthermore, α, β, and ρ are hyperparameters controlling the degree of Attention transfer.

[0071] It should be noted that, for example Figure 3 As shown, the present invention also provides a computing device for a Byzantine-robust federated learning method based on activation attention. The computing device of the present invention is used to perform the steps described above, specifically including:

[0072] The acquisition module is used by the server to obtain the models trained by all clients using local data in federated learning, as well as the size of the dataset during the training process.

[0073] The calculation module is used to calculate the local attention of the corresponding client based on the model trained by all clients using local data, and to calculate the distribution score of the local attention on the server side, and to cluster multiple clean local attentions from the local attention scores to calculate the global attention.

[0074] The update module is used by the server to send the global Attention to all clients, and the clients update their local models according to the FedNAT loss function.

[0075] like Figure 4As shown, the present invention also provides an electronic device. It should be understood that the above-described computing device can be applied to any electronic device with data processing capabilities, such as a computer or other similar device. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data-processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0076] From a hardware perspective, Figure 4 The diagram shown is a hardware structure diagram of any electronic device with data processing capabilities, which can be used to implement the Byzantine robust federated learning method based on activation attention provided by this invention. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of that data processing device, which will not be elaborated further. The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0077] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0078] The present invention also provides a computer-readable storage medium for storing the computer program, wherein the computer-readable storage medium can read the computer program into a memory and execute it by a processor to implement the aforementioned Byzantine-robust federated learning method based on activation attention.

[0079] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., mounted on the device.

[0080] Furthermore, the computer-readable storage medium may include both internal storage units of any device with data processing capabilities and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the device with data processing capabilities, and may also be used to temporarily store data that has been output or will be output.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions or improvements made by those skilled in the art without departing from the principle of the present invention should be included within the protection scope of the present invention.

Claims

1. A federated learning method based on activation attention, characterized in that, Specifically, the following steps are included: S1. Privacy-respecting local attention extraction: Based on the federated learning method, a local attention is defined. The local attention is extracted from the activation layer of the convolutional neural network along the channel dimension and used to update the local model trained by the local client using privacy data and participate in the aggregation rules of federated learning. S2. Scoring and clustering local attention: The distribution distance between local attentions is calculated using the Wasserstein bulldozer distance method as the score of the local attention. The score is then used to cluster similar local attentions using the k-median algorithm and aggregated into global attention. S3. Transferring shared knowledge from global attention to local attention: Train local attention using the FedNAT loss function to transfer global attention.

2. The federated learning method based on activation attention according to claim 1, characterized in that, In step S1, the local attention is extracted from the activation layers of the convolutional neural network along the channel dimension and calculated using the following formula: Among them, F i This represents the i-th channel of the output feature map of the convolutional activation layer, where p represents the norm value, |F i | represents F i The absolute value; This represents the distribution of features across all channels of the Feature Map, and the p-norm value is used to highlight the probability distribution of these features on the image.

3. The federated learning method based on activation attention according to claim 2, characterized in that, Step S2 includes the following sub-steps: S2.1 Calculate the distribution distance between the i-th local attention and other local attention using Wasserstein distance, as shown in the following formula: in, This represents the Wasserstein distance between the i-th local Attention and the j-th local Attention. This represents the i-th local attention. Let represent the j-th local Attention, and vec() represent the vectorized form of the local Attention; S2.2 The score of the i-th local attention is calculated using the k-median algorithm, with the following relationship: in, Let N represent the score of the i-th local attention in the aggregation rule, kMedian[] denotes the k-median algorithm, and N represent the total number of local attentions. This represents any value that variable j can take from N numbers; S2.3 Calculate the scores of all local attention points sequentially, and then use the calculated local attention scores... The k-median algorithm is used to select the NF clean local attention points with the most compact distribution, as shown in the following formula: Where G represents the filtered clean local Attention set, g i Let G represent the i-th local attention selected, and stop selecting when |G| = NF. Let N represent the total number of local attentions, and F represent the number of potential polluting attentions. S2.

4. Aggregate the clean local attention set G mentioned above using the FedAvg algorithm to obtain the global attention, as shown in the following formula: Among them, D n express Size of the local client dataset in the k-th round of communication This represents the nth local attention in the k-th round of communication. This represents the global attention obtained from aggregation.

4. The federated learning method based on activation attention according to claim 1, characterized in that, In step S3, the local attention mechanism is trained using the FedNAT loss function, as shown in the following formula: in, This represents the standard cross-entropy of the local model θ on the test set. Indicates that global Attention is at the th layer The first and local attention layer The L2 norm is normalized to represent their difference loss; α, β, and ρ represent the activation vectors of the i-th fully connected layer for global Attention and local Attention, respectively, and the bias loss is calculated using the Kullback-Leibler algorithm; α, β, and ρ are hyperparameters that control the degree of Attention transfer.

5. A computing device for implementing the method of any one of claims 1 to 4, characterized in that, include: The acquisition module is used by the server to obtain the models trained by all clients using local data in federated learning, as well as the size of the dataset during the training process. We perform privacy-respecting local attention extraction, define local attention based on federated learning methods, extract it from the activation layers of the convolutional neural network along the channel dimension, and use it to update the local model trained on the local client using privacy data, and participate in the aggregation rules of federated learning. The calculation module is used to calculate the local attention of the corresponding client based on the model trained by all clients using local data, and to calculate the distribution score of the local attention on the server side, and to cluster multiple clean local attentions from the local attention scores to calculate the global attention; the local attention is scored and clustered, and the distribution distance between local attentions is calculated using the Wasserstein bulldozer distance method as the score of the local attention, and the score is used to cluster similar local attentions using the k-median algorithm and then to aggregate the global attention; In the update module, the server transfers shared knowledge from the global attention to the local attention, trains the local attention using the FedNAT loss function, and uses this local attention to transfer the global attention. The global attention is then distributed to all clients, and the clients update their local models according to the FedNAT loss function.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program in the memory is executed by the processor, and the computer program is used to implement the steps in the federated learning method based on activation attention as described in any one of claims 1 to 4.

7. A computer-readable storage medium for storing a computer program, characterized in that, When the computer-readable storage medium can read the computer program into memory and execute it by a processor, it can implement the steps of the federated learning method based on activation attention as described in any one of claims 1 to 4.

Images