A robust federated learning method based on convolution gate mask residual

Abstract

The application discloses a robust federated learning method based on convolution gate mask residual, comprising: configuring trainable gate parameters for the convolution layers of the last round global model and each restored local model; and performing distillation training by using clean samples; constructing a two-dimensional topological mask according to the gate vectors of adjacent convolution layers; constructing a two-dimensional residual mask according to the difference between each local gate vector and the global gate vector; filtering the local model update gradient by using the two-dimensional topological mask and determining an important feature alignment score; determining a historical residual offset score by using the two-dimensional residual mask; fusing the important feature alignment score and the historical residual offset score, and performing abnormal screening on the local model update gradient according to the comprehensive abnormal score and a preset abnormal score threshold, and updating the current global model by using the screened local model update gradient. The application can effectively resist hidden backdoor attacks and improve the robustness and security of the global model of federated learning.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and specifically to a robust federated learning method based on convolutional gated mask residuals. Background Technology

[0002] With the rapid development of artificial intelligence technology and the increasing sophistication of data privacy protection regulations, federated learning has emerged as a privacy-preserving computing paradigm. Federated learning allows participants to collaboratively train a global model by exchanging model parameters or gradients with a central server, without leaving their local data domain, effectively breaking down "data silos." However, this highly decentralized, distributed nature, lacking visibility into the local training process on clients, inevitably exposes federated learning systems to serious security threats.

[0003] Among numerous security threats, covert backdoor attacks are particularly serious. These attacks typically operate on the basis of data poisoning: a malicious client implants specific triggers (such as tiny pixel blocks or noise patterns) into local training samples and alters the sample labels to target labels specified by the attacker. By training on this contaminated data, the malicious client generates "poisoned gradients" containing backdoor characteristics and uploads them to a central server. Once these poisoned gradients are aggregated, a covert backdoor is implanted into the global model—meaning the model performs perfectly normally when processing clean samples, but immediately produces the attacker's expected erroneous output when inputting samples containing specific triggers. This attack method, which is highly covert under normal circumstances and extremely destructive when triggered, seriously threatens applications with extremely high security and robustness requirements, such as autonomous driving and medical diagnostics.

[0004] Currently, the most representative method for defending against such attacks in related technologies is the robust aggregation method based on the comparison of distance or similarity of all parameters, which has the following drawbacks:

[0005] 1. The difficulty in eliminating differences caused by data heterogeneity leads to a significant decrease in defense performance. In real-world federated learning scenarios, the local data distribution of normal clients is typically highly heterogeneous (not independent and identically distributed), resulting in substantial differences in the gradients calculated by normal clients. Traditional methods, relying solely on macroscopic distance metrics, are highly susceptible to misclassifying normal clients with unique data distributions as malicious nodes, severely impacting the generalization ability of the global model.

[0006] 2. The feature dilution effect at the global dimension makes it difficult to defend against covert backdoor attacks. Modern covert backdoor attacks typically implant triggers by fine-tuning only a very small number of specific parameter channels. When similarity or distance is calculated based on all parameters, extremely weak local anomalous features are completely "diluted" by a massive amount of normal parameters, making the malicious gradient highly homogeneous with the normal gradient in macroscopic performance, ultimately causing traditional defense mechanisms to fail.

[0007] In summary, there is an urgent need for a federated learning method that can effectively defend against covert backdoor attacks. Summary of the Invention

[0008] This invention provides a robust federated learning method based on convolutional gated mask residuals to effectively resist hidden backdoor attacks and improve the robustness and security of the global federated learning model.

[0009] This invention provides a robust federated learning method based on convolutional gated mask residuals, the method comprising:

[0010] Get the local model update gradients uploaded by multiple clients in the current communication round, and update the gradients based on the global model and each local model in the previous round to restore each local model;

[0011] Trainable gating parameters are configured for the convolutional layers of the previous global model and each restored local model; and clean samples are used to distill the configured trainable gating parameters to obtain the global gating vector and the local gating vector respectively.

[0012] A two-dimensional topological mask is constructed based on the gate vectors of adjacent convolutional layers; and a two-dimensional residual mask is constructed based on the difference between each local gate vector and the global gate vector.

[0013] The local model update gradient is filtered using a two-dimensional topological mask to obtain important update gradients. Based on the important update gradients, the important feature alignment scores between the local model and the previous global model are determined.

[0014] Using a two-dimensional residual mask, the single-round residual gradient is extracted from the local model update gradient and the historical mean residual gradient is updated; based on the historical mean residual gradient, the historical residual offset score is determined.

[0015] The alignment score of important features and the offset score of historical residuals are fused to obtain a comprehensive anomaly score. Based on the comprehensive anomaly score and a preset anomaly score threshold, the local model update gradient is used to filter out anomalies. The current global model is then updated using the filtered local model update gradient.

[0016] In some embodiments of the present invention, clean samples are used to perform distillation training on the configured trainable gating parameters to obtain a global gating vector and a local gating vector, including:

[0017] The previous global model and each local model are used as the target model, and the target model without trainable gating parameters is used as the teacher model, while the target model with trainable gating parameters is used as the student model.

[0018] Clean samples are input into the teacher model and the student model respectively to obtain the first prediction result and the second prediction result; and the first loss is determined based on the mean square error between the first prediction result and the second prediction result.

[0019] Based on the configured trainable gating parameters Regularization determines the second loss;

[0020] Based on the first loss and the second loss, the joint loss is determined; and the configured trainable gating parameters are iteratively optimized based on the joint loss to obtain the optimized gating vector, which includes the global gating vector and the local gating vector.

[0021] In some embodiments of the present invention, a two-dimensional topological mask is constructed based on the gating vectors of adjacent convolutional layers, including:

[0022] For the previous global model and each restored local model, obtain the gate vector of the i-th convolutional layer and the gate vector of the (i-1)-th convolutional layer;

[0023] Perform an outer product operation on the gate vector of the i-th convolutional layer and the gate vector of the (i-1)-th convolutional layer to obtain the two-dimensional topological mask corresponding to the i-th convolutional layer.

[0024] In some embodiments of the present invention, a two-dimensional residual mask is constructed based on the difference between each local gating vector and the global gating vector, including:

[0025] The difference between each local gating vector and the global gating vector is truncated using the ReLU activation function to obtain the offset gating parameters;

[0026] Perform an outer product operation on the offset gating parameters of the i-th convolutional layer and the offset gating parameters of the (i-1)-th convolutional layer to obtain the two-dimensional residual mask corresponding to the i-th convolutional layer.

[0027] In some embodiments of the present invention, a two-dimensional topological mask is used to filter the local model update gradient to obtain important update gradients, and based on the important update gradients, an important feature alignment score between the local model and the previous round global model is determined, including:

[0028] For each convolutional layer in the restored local model, the two-dimensional topological mask is broadcast and expanded in the spatial dimension so that the expanded two-dimensional topological mask matches the shape of the local model's updated gradient.

[0029] The extended two-dimensional topological mask is multiplied element-wise with the local model update gradient to obtain the filtered update tensor.

[0030] The filtered update tensors of each convolutional layer are flattened and stitched together to obtain the important update gradients.

[0031] Calculate the cosine similarity between the important update gradients of each restored local model and the important update gradients of the previous global model;

[0032] Based on the median and absolute median difference of each cosine similarity, the cosine similarity is standardized to obtain the important feature alignment score.

[0033] In some embodiments of the present invention, a two-dimensional residual mask is used to extract the single-round residual gradient from the local model update gradient and update the historical mean residual gradient; based on the historical mean residual gradient, the historical residual offset score is determined, including:

[0034] For each convolutional layer in the restored local model, the two-dimensional residual mask is broadcast and expanded in the spatial dimension so that the expanded two-dimensional residual mask matches the shape of the local model's update gradient.

[0035] The extended two-dimensional residual mask is multiplied element-wise with the local model update gradient to obtain the single-round residual gradient;

[0036] Based on the number of aggregation rounds participated in by the client, the residual gradient of a single round is accumulated to obtain the accumulated residual gradient and the corresponding historical mean residual gradient;

[0037] The historical residual offset score is determined based on the absolute peak value of the historical mean residual gradient of the current client and the maximum cosine similarity between the historical mean residual gradient of the current client and the historical mean residual gradient of other clients.

[0038] In some embodiments of the present invention, the alignment score of important features and the historical residual offset score are fused to obtain a comprehensive anomaly score, including:

[0039] The root mean square fusion method is used to fuse the alignment scores of important features and the historical residual offset scores to obtain a comprehensive anomaly score.

[0040] In the robust federated learning method based on convolutional gate mask residuals provided in this invention, clean samples are used to distill the configured trainable gate parameters to obtain global gate vectors and local gate vectors. A two-dimensional topological mask is constructed based on the gate vectors of adjacent convolutional layers. A two-dimensional residual mask is constructed based on the difference between each local gate vector and the global gate vector. The local model update gradient is filtered using the two-dimensional topological mask to obtain important update gradients. Based on the important update gradients, the important feature alignment score between the local model and the previous round global model is determined. The single-round residual gradient is extracted from the local model update gradient using the two-dimensional residual mask, and the historical mean residual gradient is updated. The historical residual offset score is determined based on the historical mean residual gradient. The important feature alignment score and the historical residual offset score are fused to obtain a comprehensive anomaly score. Anomaly screening is performed on the local model update gradient based on the comprehensive anomaly score and a preset anomaly score threshold. The current global model is updated using the screened local model update gradient. This invention effectively addresses the shortcomings of existing defense methods where local anomaly features are easily diluted by massive parameters by introducing two-dimensional convolutional gating and L1 sparsity constraints to filter redundant weight channels, thus achieving accurate capture of subtle features of hidden backdoors. Simultaneously, by utilizing global and local gating with the ReLU activation function to obtain gating residuals and combining historical mean to smooth out single-sample fluctuations, it effectively removes heterogeneous interference from data in real non-independent, identically distributed scenarios, significantly improving the federated learning system's ability to intercept hidden poisoned gradients. Through asymmetric truncation and historical state tracking mechanisms, it eliminates differences caused by data heterogeneity, effectively resisting hidden backdoor attacks in federated learning and significantly improving the robustness of the global model. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the robust federated learning method based on convolutional gated mask residuals provided in an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram illustrating the generation of a two-dimensional topological mask and a two-dimensional residual mask provided in an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0046] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0047] The use of "applies to" or "configured to" in this invention implies an open and inclusive language, which does not exclude the applicability to or configuration to devices performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, because processes, steps, calculations, or other actions "based on" one or more conditions or values ​​may in practice be based on additional conditions or values ​​beyond those conditions.

[0048] In this invention, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.

[0049] Currently, the most representative method for defending against this type of attack is a robust aggregation method based on distance or similarity comparison of all parameters, which includes the following steps:

[0050] 1. The server collects data from each client in the current round. The uploaded local model is updated and flattened into a one-dimensional full parameter vector. .

[0051] 2. Calculate spatial distance scores between each client instance, or between a client instance and the global model, using all parameters. Commonly used metrics include full cosine similarity or Euclidean distance.

[0052] Cosine similarity: ;

[0053] Euclidean distance: .

[0054] 3. Based on the assumption that "malicious gradients will necessarily differ from most normal gradients in spatial distribution," the central server uses sorting or clustering algorithms to remove client sets with abnormal score performance (such as excessively large distances or excessively low similarities). And for the remaining set of benign clients Perform average aggregation to update the global model.

[0055] However, the most representative methods for defending against such attacks, namely robust aggregation methods based on distance or similarity comparison of all parameters, are ineffective against covert backdoor attacks and have poor defense performance. To address these issues, this invention provides a robust federated learning method based on convolutional gated mask residuals. The following description, in conjunction with the accompanying drawings, illustrates this robust federated learning method based on convolutional gated mask residuals.

[0056] like Figure 1 As shown, this embodiment of the invention provides a robust federated learning method based on convolutional gated mask residuals. This method is applied to the server and includes the following steps:

[0057] S101: Obtain the local model update gradients uploaded by multiple clients in the current communication round, and restore each local model based on the global model and the local model update gradients from the previous round.

[0058] S102 consists of convolutional layers of the previous global model and each restored local model, configured with trainable gating parameters; and clean samples are used to distill and train the configured trainable gating parameters to obtain the global gating vector and the local gating vector respectively.

[0059] S103, construct a two-dimensional topological mask based on the gating vectors of adjacent convolutional layers; and construct a two-dimensional residual mask for capturing cross-channel connectivity based on the difference between each local gating vector and the global gating vector.

[0060] This step aims to up-map the one-dimensional gating parameters to capture the topological relationships of cross-channel feature connections between adjacent convolutional layers.

[0061] S104 uses a two-dimensional topological mask to filter the local model update gradient to obtain important update gradients, and based on the important update gradients, determines the important feature alignment score between the local model and the previous round global model.

[0062] S105: Using a two-dimensional residual mask, extract the single-round residual gradient from the local model update gradient and update the historical mean residual gradient; determine the historical residual offset score based on the historical mean residual gradient.

[0063] S106: The alignment score of important features and the offset score of historical residuals are fused to obtain a comprehensive anomaly score; and the local model update gradient is filtered out based on the comprehensive anomaly score and a preset anomaly score threshold. The current global model is then updated using the filtered local model update gradient.

[0064] The robust federated learning method based on convolutional gate mask residuals provided in this invention uses clean samples to distill and train the configured trainable gate parameters, obtaining global and local gate vectors. A two-dimensional topological mask is constructed based on the gate vectors of adjacent convolutional layers. A two-dimensional residual mask is constructed based on the differences between each local gate vector and the global gate vector. The two-dimensional topological mask is used to filter the local model update gradient to obtain important update gradients, and based on these important update gradients, the important feature alignment score between the local model and the previous-round global model is determined. Using the two-dimensional residual mask, the single-round residual gradient is extracted from the local model update gradient, and the historical mean residual gradient is updated. The historical mean residual gradient is used to determine the historical residual offset score. The important feature alignment score and the historical residual offset score are fused to obtain a comprehensive anomaly score. Based on the comprehensive anomaly score and a preset anomaly score threshold, the local model update gradient is anomaly filtered out, and the filtered local model update gradient is used to update and obtain the current global model. This invention effectively addresses the shortcomings of existing defense methods where local anomaly features are easily diluted by massive parameters by introducing two-dimensional convolutional gating and L1 sparsity constraints to filter redundant weight channels, thus achieving accurate capture of subtle features of hidden backdoors. Simultaneously, by utilizing global and local gating with the ReLU activation function to obtain gating residuals and combining historical mean to smooth out single-sample fluctuations, it effectively removes heterogeneous interference from data in real non-independent, identically distributed scenarios, significantly improving the federated learning system's ability to intercept hidden poisoned gradients. Through asymmetric truncation and historical state tracking mechanisms, it eliminates differences caused by data heterogeneity, effectively resisting hidden backdoor attacks in federated learning and significantly improving the robustness of the global model.

[0065] In some embodiments of the present invention, clean samples are used to perform distillation training on the configured trainable gating parameters to obtain a global gating vector and a local gating vector, including:

[0066] The previous global model and each local model are used as the target model, and the target model without trainable gating parameters is used as the teacher model, while the target model with trainable gating parameters is used as the student model.

[0067] Clean samples are input into the teacher model and the student model respectively to obtain the first prediction result and the second prediction result; and the first loss is determined based on the mean square error between the first prediction result and the second prediction result.

[0068] Based on the configured trainable gating parameters Regularization determines the second loss;

[0069] Based on the first loss and the second loss, the joint loss is determined; and the configured trainable gating parameters are iteratively optimized based on the joint loss to obtain the optimized gating vector, which includes the global gating vector and the local gating vector.

[0070] In some examples, the previous global model and each restored local model are used as the target model, and trainable gating parameters matching the output channel dimension are configured for the convolutional layers in each target model. n is the number of convolutional layers. Let the symbol be... This represents the convolution operation. Let represent the activation function, then the target model's _____ Convolutional output It can be represented as:

[0071] .

[0072] To optimize the configured trainable gating parameters, the original target model without gating parameters is used as the teacher model (i.e., weights frozen), and the target model with configured trainable gating parameters is used as the student model. Distillation is performed using a small number of clean samples from the server (i.e., normal samples without backdoor trigger injection), and the joint loss function is used. The definition is as follows:

[0073] ;

[0074] in, These represent the output logic values ​​of the student model after a clean sample is input, i.e., the first prediction result. It is the output logic value of the teacher model after a clean sample is input into the teacher model, which is the second prediction result; To ensure accuracy, the mean square error loss, This is the first loss; To drive the sparsity of gating parameters The regularization penalty term, also known as the second loss; and This is a hyperparameter.

[0075] It is understandable that in the distillation process of the aforementioned teacher and student models, the joint loss function is relied upon. The student model is iteratively optimized. Specifically, while keeping the original network weights of the student model frozen, the goal is to minimize the joint loss function. To optimize the objective, the backpropagation algorithm is used to update only the trainable gating parameters in the student model. This allows the student model to approximate the teacher model's predicted output while simultaneously driving the gating parameters to become sparsity. When the joint loss function... Upon convergence, the trainable gating parameters obtained during the final training are used as the optimized gating vector.

[0076] It should be noted that in this embodiment, considering the time and computational overhead of the federated learning system, only a small number of clean samples are used for training, and excellent defense performance can still be achieved under these conditions. Of course, in other embodiments of the present invention, to further improve the overall performance and robustness of the model, a large number of clean samples can be introduced for sufficient training. The present invention does not impose a specific limit on the number of clean samples used for training.

[0077] In some embodiments of the present invention, due to the first in the target model The channel importance of the input features in a convolutional layer is determined by its preceding connected layer (i.e., the first convolutional layer). The gate parameter vector of a convolutional layer The channel importance of its output features is determined by the gating parameter vector of the current layer; and the channel importance of its output features is determined by the gating parameter vector of the current layer. Decide.

[0078] Accordingly, such as Figure 2 As shown, a two-dimensional topological mask is constructed based on the gating vectors of adjacent convolutional layers, including:

[0079] For the previous global model and each restored local model, obtain the gate vector of the i-th convolutional layer and the gate vector of the (i-1)-th convolutional layer;

[0080] Perform an outer product operation on the gate vector of the i-th convolutional layer and the gate vector of the (i-1)-th convolutional layer to obtain the two-dimensional topological mask corresponding to the i-th convolutional layer.

[0081] In some examples, the target model (including the previous global model and each restored local model) is extracted. Gating vectors corresponding to convolution layers and the input gating vector of the previous layer ,and .

[0082] If the first The first layer of the target model is the convolutional layer, i.e. At that time, the input gating parameter vector will be... The default setting is a vector of all 1s with the same dimension as the number of channels in the input data. (Calculation) and The tensor outer product is used to obtain the two-dimensional topological mask of the first convolutional layer.

[0083] If the first If the convolutional layer is not the first layer, then the current layer's gate vector is calculated. With the input gate vector of the previous layer The tensor outer product is used as a two-dimensional topological mask for the feature importance distribution of convolutional layer i. Indicatively, , This represents the tensor outer product operation. In a physical sense, it refers to a two-dimensional topological mask. This forms a two-dimensional matrix, where each element reflects the joint activation weights of the cross-channel connections between a specific input and output channel.

[0084] By iterating through each convolutional layer of each target model and repeating the outer product operation described above, a two-dimensional topological mask set for each target model is obtained. .

[0085] In some embodiments of the present invention, the microscopic mechanism of backdoor attacks lies in the fact that, in order for the model to respond accurately to specific triggers, attackers typically need to forcibly activate redundant channels that are originally dormant in the neural network, or abnormally amplify the weights of certain specific channels. Therefore, the positive shift of the client feature importance distribution relative to the global consensus is a key perspective for detecting covert attack behavior. Furthermore, by calculating the positive shift of the client feature importance distribution relative to the global consensus to remove the interference of data heterogeneity, backdoor attacks can be accurately identified.

[0086] Continue reading Figure 2 Based on the differences between each local gating vector and the global gating vector, a two-dimensional residual mask is constructed, including:

[0087] The difference between each local gating vector and the global gating vector is truncated using the ReLU activation function to obtain the offset gating parameters;

[0088] Perform an outer product operation on the offset gating parameters of the i-th convolutional layer and the offset gating parameters of the (i-1)-th convolutional layer to obtain the two-dimensional residual mask corresponding to the i-th convolutional layer.

[0089] In some examples, the local gating vector is computed. With global gate vector The difference is calculated, and the ReLU activation function is used to perform asymmetric truncation on the difference to obtain the offset gating parameters characterizing the positive offset. Indicatively, The ReLU activation function forces the channel weights that are less than or equal to the global gating vector to zero, and only retains the positive deviation values ​​that are greater than the global gating vector, thereby extracting the excess activation degree of the client on each corresponding channel relative to the global consensus.

[0090] Extract the offset gating parameters of the current convolutional layer and the offset gating parameters corresponding to the previous convolutional layer Perform a tensor outer product operation on the two, and use the result of the tensor outer product operation as a two-dimensional residual mask to characterize the distribution of the feature importance offset of the current convolutional layer. .

[0091] By iterating through each convolutional layer of each restored local model, we obtain the set of two-dimensional residual masks corresponding to each restored local model. .

[0092] In some embodiments of the present invention, a two-dimensional topological mask is used to filter the local model update gradient to obtain important update gradients, and based on the important update gradients, an important feature alignment score between the local model and the previous round global model is determined, including:

[0093] For each convolutional layer in the restored local model, the 2D topological mask is broadcast-expanded in the spatial dimension to match the shape of the expanded 2D topological mask with the shape of the gradient update in the local model. In some examples, a function is used... Two-dimensional topological mask Perform broadcast extension.

[0094] The expanded 2D topological mask is updated along with the gradient of the local model. Element-wise multiplication is performed, and the result is used as the filtered update tensor. It is understandable that multiplying the expanded 2D topological mask element-wise with the local model update gradient can filter out low-importance gradient noise.

[0095] The filtered update tensors of each convolutional layer are flattened and concatenated to obtain the important update gradients. .

[0096] In some examples, utilizing The function flattens the filtered update tensors from each convolutional layer and concatenates them according to the network topology into a high-dimensional one-dimensional core feature vector, which is then used as an important update gradient. The specific calculation formula is as follows:

[0097]

[0098] In the formula, This indicates an element-wise multiplication operation.

[0099] Calculate the important update gradients for each restored local model. Important update gradients corresponding to the previous global model Cosine similarity between them.

[0100] Based on the median and absolute median difference of each cosine similarity, the cosine similarity is standardized to obtain the important feature alignment score.

[0101] In some examples, to effectively filter out anomalies and address similarity distribution shifts caused by extreme non-independent identically distributed (Non-IID) data, the median absolute difference (MAD) is used to standardize the cosine similarity. Specifically, the set of observations is defined as the cosine similarities calculated by all participating clients in the current communication round. ( (To the total number of clients participating in the aggregation), extract the median of the observation set. And calculate its absolute median difference. :

[0102] ;

[0103] Cosine similarity of any client The formula for calculating the standard score is:

[0104] ;

[0105] The server will calculate the standard score. The alignment score serves as the final key feature for each corresponding client in this round.

[0106] The robust federated learning method based on convolutional gated mask residuals provided in this invention utilizes two-dimensional topological masks to accurately extract the core gradients that truly play a role in the main task during the model updates of each participant. By calculating similarity in the core feature space rather than the full parameter space, malicious perturbations hidden under a large number of redundant parameters in the poisoned gradients can be effectively removed, thereby capturing abnormal behaviors that deviate from the global optimization direction.

[0107] In some embodiments of the present invention, a two-dimensional residual mask is used to extract the single-round residual gradient from the local model update gradient and update the historical mean residual gradient; based on the historical mean residual gradient, the historical residual offset score is determined, including:

[0108] For each convolutional layer in the restored local model, the two-dimensional residual mask is broadcast and expanded in the spatial dimension so that the shape of the expanded two-dimensional residual mask matches the shape of the gradient update of the local model. That is, the matrix shape of the expanded two-dimensional residual mask is exactly the same as the matrix shape of the gradient update of the local model.

[0109] The extended two-dimensional residual mask is multiplied element-wise with the local model update gradient to obtain the single-round residual gradient.

[0110] Based on the number of aggregation rounds participated in by the client, the residual gradient of each round is accumulated to obtain the accumulated residual gradient and the corresponding historical mean residual gradient. Indicatively, c represents the aggregation round in which the client participates. For clients in historical rounds The single-round residual gradient extracted from it. This is the set of all communication rounds that the client has participated in up to the current moment.

[0111] The historical residual offset score is determined based on the absolute peak value of the historical mean residual gradient of the current client and the maximum cosine similarity between the historical mean residual gradient of the current client and the historical mean residual gradient of other clients.

[0112] In some examples, functions are used Extract the absolute peak value of the historical residual mean gradient of the current client to measure the degree to which local parameters are abnormally amplified. Simultaneously, utilize the function... Calculate the maximum cosine similarity between the historical mean residual gradient of the current client and the historical mean residual gradients of other participating clients to measure the anomaly of their feature offset directions.

[0113] Based on the observation set composed of the absolute peak and maximum cosine similarity of all clients in the current round, the standard scores corresponding to the absolute peak and maximum cosine similarity are calculated using the median absolute difference (MAD) normalization method. The server then sums the squares of these two standard scores and takes the average to obtain the final historical residual offset score for each client. The specific calculation formula is as follows:

[0114] .

[0115] The robust federated learning method based on convolutional gated mask residuals provided in this invention can effectively smooth out the random bias caused by single-round data sampling through historical residual accumulation and averaging across rounds. This allows for the accurate capture of the stubborn feature importance distribution shifts exposed by malicious clients during long-term training. It overcomes the problem of potential failure in important feature alignment detection in the later stages of target model convergence or under extreme data heterogeneity conditions. It can also overcome the problem of random bias in single-round calculation results caused by the characteristics of hidden backdoor implantation (i.e., continuous optimization across rounds).

[0116] In some embodiments of the present invention, the alignment score of important features and the historical residual offset score are fused to obtain a comprehensive anomaly score, including:

[0117] The root mean square fusion method is used to fuse the alignment scores of important features and the historical residual offset scores to obtain a comprehensive anomaly score.

[0118] In some examples, a root mean square nonlinear fusion strategy is used to combine the important feature alignment scores and historical residual offset scores to obtain the final comprehensive anomaly score for each client.

[0119] After removing the local model update gradients of clients whose overall anomaly scores exceed a preset anomaly score threshold, the server performs a weighted average aggregation (such as the FedAvg algorithm) on the local model update gradients of the remaining benign clients. The aggregated gradients are then used to update the previous global model, thereby obtaining the current global model.

[0120] The server distributes the current global model to each client, and repeats steps S101 to S106 until the global model reaches the convergence condition or the maximum number of iterations, thus completing the federated learning training.

[0121] The robust federated learning method based on convolutional gated mask residuals provided in this invention effectively addresses the deficiency in existing defense methods where local anomalies are easily diluted by the full set of parameters by constructing a dual filtering mechanism based on a two-dimensional topological mask and a two-dimensional residual mask using convolutional gates. This achieves precise capture of minute features of hidden backdoors at the kernel level. Simultaneously, this scheme utilizes the ReLU asymmetric truncation mechanism to extract the gated residuals and combines cross-round historical state tracking and mean-based processing to effectively remove heterogeneous interference from data in real non-independent and identically distributed scenarios. Compared to existing technologies, the method in this invention maintains good defense performance even in relatively extreme non-independent and identically distributed scenarios, significantly improving the federated learning system's ability to intercept hidden poisoned gradients and the overall model security.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0123] The robust federated learning method based on convolutional gated mask residuals provided by the embodiments of the present invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A robust federated learning method based on convolutional gated mask residuals, characterized in that, The method includes: Obtain the local model update gradients uploaded by multiple clients in the current communication round, and restore each local model based on the global model of the previous round and the update gradients of each local model. Trainable gating parameters are configured for the convolutional layers of the previous global model and each restored local model; and clean samples are used to distill the configured trainable gating parameters to obtain the global gating vector and the local gating vector respectively. A two-dimensional topological mask is constructed based on the gate vectors of adjacent convolutional layers; and a two-dimensional residual mask is constructed based on the difference between each local gate vector and the global gate vector. The local model update gradient is filtered using the two-dimensional topological mask to obtain important update gradients, and based on the important update gradients, the important feature alignment score between the local model and the previous round global model is determined. Using the two-dimensional residual mask, the single-round residual gradient is extracted from the local model update gradient and the historical mean residual gradient is updated; based on the historical mean residual gradient, the historical residual offset score is determined. The important feature alignment score and the historical residual offset score are fused to obtain a comprehensive anomaly score; and the local model update gradient is anomaly filtered out based on the comprehensive anomaly score and a preset anomaly score threshold, and the current global model is obtained by updating the local model update gradient after the anomaly is filtered out.

2. The robust federated learning method based on convolutional gated mask residuals according to claim 1, characterized in that, The process of distilling and training the configured trainable gating parameters using clean samples to obtain global and local gating vectors includes: The previous global model and each of the local models are used as target models, and the target model without trainable gating parameters is used as the teacher model, and the target model with trainable gating parameters is used as the student model. The clean samples are input into the teacher model and the student model respectively to obtain the first prediction result and the second prediction result; and the first loss is determined based on the mean square error between the first prediction result and the second prediction result. According to the configured trainable gating parameters Regularization determines the second loss; Based on the first loss and the second loss, a joint loss is determined; and based on the joint loss, the configured trainable gating parameters are iteratively optimized to obtain an optimized gating vector, which includes the global gating vector and the local gating vector.

3. The robust federated learning method based on convolutional gated mask residuals according to claim 1, characterized in that, Based on the gate vectors of adjacent convolutional layers, a two-dimensional topological mask is constructed, including: For the previous global model and each restored local model, obtain the gate vector of the i-th convolutional layer and the gate vector of the (i-1)-th convolutional layer; Perform an outer product operation on the gate vector of the i-th convolutional layer and the gate vector of the (i-1)-th convolutional layer to obtain the two-dimensional topological mask corresponding to the i-th convolutional layer.

4. The robust federated learning method based on convolutional gated mask residuals according to claim 1, characterized in that, The step of constructing a two-dimensional residual mask based on the difference between each of the local gating vectors and the global gating vector includes: The difference between each local gating vector and the global gating vector is truncated using the ReLU activation function to obtain the offset gating parameters; Perform an outer product operation on the offset gating parameters of the i-th convolutional layer and the offset gating parameters of the (i-1)-th convolutional layer to obtain the two-dimensional residual mask corresponding to the i-th convolutional layer.

5. The robust federated learning method based on convolutional gated mask residuals according to claim 1, characterized in that, The step of filtering the local model update gradient using the two-dimensional topological mask to obtain important update gradients, and determining the important feature alignment score between the local model and the previous round global model based on the important update gradients, includes: For each convolutional layer in the restored local model, the two-dimensional topological mask is broadcast and expanded in the spatial dimension so that the expanded two-dimensional topological mask matches the shape of the local model's update gradient. The extended two-dimensional topological mask is multiplied element-wise with the local model update gradient to obtain the filtered update tensor. The filtered update tensors of each convolutional layer are flattened and concatenated to obtain the important update gradients. Calculate the cosine similarity between the important update gradients corresponding to the restored local models and the important update gradients corresponding to the previous round global models; The cosine similarities are standardized based on the median and absolute median difference of each cosine similarity to obtain the important feature alignment score.

6. The robust federated learning method based on convolutional gated mask residuals according to claim 1, characterized in that, The step of extracting single-round residual gradients from the local model update gradients using the two-dimensional residual mask and updating the historical mean residual gradients; determining the historical residual offset score based on the historical mean residual gradients, including: For each convolutional layer in the restored local model, the two-dimensional residual mask is broadcast and expanded in the spatial dimension so that the expanded two-dimensional residual mask matches the shape of the local model's update gradient. The extended two-dimensional residual mask is multiplied element-wise with the local model update gradient to obtain the single-round residual gradient; Based on the aggregation rounds participated in by the client, the residual gradient of the single round is accumulated to obtain the accumulated residual gradient and the corresponding historical mean residual gradient; The historical residual offset score is determined based on the absolute peak value of the historical mean residual gradient of the current client and the maximum cosine similarity between the historical mean residual gradient of the current client and the historical mean residual gradient of other clients.

7. The robust federated learning method based on convolutional gated mask residuals according to claim 1, characterized in that, The fusion of the important feature alignment score and the historical residual offset score to obtain a comprehensive anomaly score includes: The root mean square fusion method is used to fuse the important feature alignment score and the historical residual offset score to obtain the comprehensive anomaly score.

Images