A robustness-enhanced decentralized federated learning method and system

Abstract

The application discloses a robust decentralized federated learning method and system, and an embodiment of the application comprises: obtaining local iteration training performed by a target client in each communication round, wherein the local iteration training comprises: receiving model parameters of neighbor models from neighbor nodes; calculating parameter differences between a local model and each neighbor model based on multi-dimensional consistency metrics; selecting a set of trusted nodes from the neighbor nodes according to the relationship between the parameter differences and a dynamic threshold; calculating a local gradient based on a local data set to obtain a dual variable; determining an update direction based on the local gradient and a current dual constraint term; determining the momentum of the current round based on the update direction and a historical momentum accumulation value; and updating the local model parameters according to the momentum and a learning rate. In the presence of malicious nodes or abnormal updates, the convergence stability and robustness of the decentralized federated learning system are improved.

Description

Technical Field

[0001] This invention relates to the field of computing technology, and in particular to a robust decentralized federated learning method and system. Background Technology

[0002] Machine learning is a method of automatically building models by analyzing data. Its core goal is to extract potential patterns and regularities from data to make predictions, classifications, or decisions. With the rapid development of artificial intelligence technology and the continuous expansion of its application scenarios, machine learning has shown wide application value in fields such as natural language processing, image recognition, object detection, and voice intelligence. However, it also faces a series of problems, such as data privacy issues and model illusion problems.

[0003] Training machine learning models typically relies on large-scale, high-quality data. However, traditional methods require centralized training on servers, where data often contains sensitive information, and centralized storage can pose privacy risks. To address this issue, in 2016, McMahan et al. first proposed the Federated Learning (FL) framework, which enables multiple participants to collaboratively train models without sharing the original data, thus balancing privacy protection with multi-party data utilization.

[0004] A typical federated learning framework consists of a central parameter server and multiple distributed clients. In each training round, the central server distributes the global model to each client. The clients update their models based on their local data and then upload their parameters. The server then aggregates the parameters from all clients to update the global model. However, this architecture relies on the central server for both global model distribution and parameter aggregation, making the central server a single point of failure and a potential source of unreliability. If the server fails or is maliciously interfered with in an attempt to corrupt the global model or infer user data, unpredictable losses can occur.

[0005] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this invention is to provide a robust decentralized federated learning method and system. By designing the update mechanism of each client model in the decentralized network, the consistency and convergence speed of model training are guaranteed under the conditions of data heterogeneity and topological sparsity. At the same time, it provides a robust mechanism to resist potential abnormal nodes or malicious attacks, thereby mitigating the model illusion problem caused by them.

[0007] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a robust decentralized federated learning method applied to a centralized federated learning system comprising multiple directly communicating clients. The method includes: in each communication round, the target client performs local iterative training, wherein the local iterative training includes: receiving model parameters from multiple neighboring models from neighboring nodes; calculating the parameter differences between the local model and each neighboring model based on a multi-dimensional consistency metric, wherein the multi-dimensional consistency metric includes a distance metric between parameter vectors, a directional consistency metric, and a norm relative deviation metric; and calculating the parameter differences based on the parameter differences and dynamic... The dynamic threshold is used to select a set of trustworthy nodes from neighboring nodes. The dynamic threshold is determined based on the statistical distribution characteristics of the differences in all parameters in the current round. The local gradient is calculated based on the local dataset, and the dual variable is obtained by applying regularized decay processing to the dual variable of the previous round. The update direction is determined based on the local gradient and the current dual constraint term, wherein the dual constraint term is used to constrain the difference between the local model and the trustworthy nodes, and the dual constraint term is generated based on the dual variable. The momentum of the current round is determined based on the update direction and the historical momentum accumulation value. The local model parameters are updated according to the momentum and the learning rate.

[0008] In some embodiments, the step of selecting a set of trustworthy nodes from neighboring nodes based on the relationship between the parameter differences and the dynamic threshold includes: calculating the mean and standard deviation of the parameter differences corresponding to all neighboring nodes in the current round; determining the dynamic threshold based on the mean and standard deviation, wherein the dynamic threshold is the product of the mean and the threshold adjustment coefficient plus the standard deviation; and including neighboring nodes whose parameter differences are less than or equal to the dynamic threshold into the set of trustworthy nodes.

[0009] In some embodiments, updating the local model parameters based on the momentum and learning rate includes: according to the following formula

[0010] Update local model parameters;

[0011] in, Indicates that the target client n is in the th order. Round of global iteration, the first Model parameters during the next local iteration Indicates the learning rate; Indicates the target client at the 1st Round of global iteration, the first Momentum during sublocal iterations.

[0012] In some embodiments, determining the momentum of the current cycle based on the update direction and the historical momentum accumulation value includes: according to the following formula

[0013] Calculate the momentum of the current round;

[0014] Historical momentum accumulation value, used to represent the target client's momentum accumulation value at the [number]th [time]. Round of global iteration, the first Momentum during the next local iteration; where... Indicates the target client at the 1st Round of global iteration, the first Update direction during sublocal iteration.

[0015] In some embodiments, determining the update direction based on the local gradient and the current dual constraint term includes: determining the update direction based on the following formula:

[0016]

[0017] in, Indicates the target client at the 1st Round of global iteration, the first Update direction during sublocal iteration Representing the trusted neighbor model The dual constraint term of the difference. This indicates that the target client's current local model parameters are compared with the trusted neighbor model in the 1st... The differences between model parameters after each round of global iteration This indicates division by the penalty parameter. The cumulative historical model difference parameter after scaling.

[0018] In some embodiments, the method further includes: for each trusted neighbor model in the trusted node set, calculating the reciprocal of the parameter difference between it and the local model as an initial weight; normalizing all initial weights to obtain the aggregate weight of each trusted neighbor model; and performing a weighted summation of all trusted neighbor models in the trusted node set based on the aggregate weight to obtain an aggregate model.

[0019] In some embodiments, the update formula for the dual variable is:

[0020]

[0021] in, Let m represent the dual variables between the target client n and the trusted neighbor model m in the k-th round. This represents the penalty parameter, where i represents the attenuation coefficient. and These represent the updated local model and the trusted neighbor model, respectively.

[0022] In some embodiments, calculating the parameter difference between the local model and each trusted neighbor model based on a multi-dimensional consistency metric includes: calculating the following metrics for each layer of the model parameters: calculating the absolute difference between parameter vectors based on Euclidean distance; calculating the directional deviation between parameter vectors based on cosine similarity; calculating the scale relative difference between parameter vectors based on norm ratio; and weighting and fusing the absolute difference, directional deviation, and scale relative difference to obtain the parameter difference.

[0023] In some embodiments, the formula for weighted fusion is:

[0024]

[0025] Where Vn represents the local model, Vm represents the trusted neighbor model, L represents the total number of model layers, and W represents the total number of model layers. l This represents the weight coefficient of the l-th layer. Represents the Euclidean distance of the l-th layer. Represents the cosine distance of the l-th layer. λ1, λ2, and λ3 represent the relative difference in norms of the l-th layer, and λ1, λ2, and λ3 are fusion coefficients.

[0026] To achieve the above objectives, in a second aspect, embodiments of the present invention provide a robust decentralized federated learning system, comprising multiple directly communicating clients; each client performs local iterative training, wherein the local iterative training includes: receiving model parameters from multiple neighboring models from neighboring nodes; calculating the parameter differences between the local model and each neighboring model based on a multi-dimensional consistency metric, wherein the multi-dimensional consistency metric includes a distance metric between parameter vectors, a directional consistency metric, and a norm relative deviation metric; selecting a set of trustworthy nodes from the neighboring nodes according to the relationship between the parameter differences and a dynamic threshold, wherein the dynamic threshold is determined based on the statistical distribution characteristics of all parameter differences in the current round; calculating the local gradient based on the local dataset, and obtaining the dual variable obtained by performing regularized decay processing on the dual variable of the previous round; determining the update direction based on the local gradient and the current dual constraint term, wherein the dual constraint term is used to constrain the differences between the local model and the trustworthy nodes, and the dual constraint term is generated based on the dual variable; determining the momentum of the current round based on the update direction and the historical momentum accumulation value; and updating the local model parameters according to the momentum and the learning rate.

[0027] In some embodiments, a client in a robust decentralized federated learning system of the present invention can perform the method as shown in any of the first aspects.

[0028] Compared with existing technologies, the robust decentralized federated learning method and system of the present invention, through multi-dimensional consistency measurement and dynamic threshold screening, eliminates Byzantine nodes and anomalous updates, ensuring that only trusted neighbors participate in constraints. The dual variable, after regularization decay, retains necessary historical memory to constrain long-term consistency while preventing the propagation of early malicious interference across rounds. The momentum mechanism transforms hard dual constraints into smooth inertial motion, suppressing parameter oscillations caused by Non-IID data. The overall technical solution eliminates the single point of failure risk of the central server and achieves model training stability and convergence in complex environments with malicious nodes and non-independent homomorphic data distribution, significantly improving the robustness and practicality of decentralized federated learning. Attached Figure Description

[0029] Figure 1 This is a flowchart of an embodiment of a robust decentralized federated learning method according to the present invention;

[0030] Figure 2 This is a flowchart of another embodiment of a robust decentralized federated learning method according to the present invention;

[0031] Figure 3 This illustrates the technical effects of one or more embodiments of the present invention;

[0032] Figure 4 An exemplary system architecture of an embodiment of the decentralized federated learning system of the present invention is shown. Detailed Implementation

[0033] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but it should be understood that the scope of protection of the present invention is not limited to the specific embodiments.

[0034] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprises" shall be understood to include the stated elements or components without excluding other elements or other components.

[0035] First, we will explain some concepts that may be involved in this invention.

[0036] Federated Learning (FL): A distributed machine learning paradigm that allows multiple participants to collaboratively train a model without sharing the original data, thus balancing privacy protection and multi-party data utilization.

[0037] Decentralized Federated Learning (DFL): A variant of federated learning in which clients exchange and aggregate model parameters by communicating directly with neighboring nodes, without the need for a central server for coordination.

[0038] ADMM (Alternating Direction Method of Multipliers) is an algorithmic framework for solving distributed optimization problems. It addresses consistency constraints by introducing dual variables.

[0039] Euclidean distance: The straight-line distance between two points in Euclidean space, used to measure the degree of difference between two vectors in terms of numerical values.

[0040] Cosine Similarity: The cosine of the angle between two vectors, used to measure the similarity between the two vectors in direction, with a value range of [-1, 1].

[0041] Dual variable: In an optimization problem, a Lagrange multiplier associated with constraints is used to transform a constrained optimization problem into an unconstrained optimization problem.

[0042] Byzantine attack: A malicious attack pattern in distributed systems in which the attacking node can send arbitrary error messages to disrupt the normal operation of the system.

[0043] Data heterogeneity: In federated learning scenarios, the local data distribution of different clients differs, i.e., the data is not independently and identically distributed (Non-IID).

[0044] Momentum: An acceleration technique in gradient descent optimization algorithms that accelerates convergence and reduces oscillations by accumulating historical gradient directions.

[0045] like Figure 1 As shown, a robust decentralized federated learning method according to a preferred embodiment of the present invention can be applied to a robust decentralized federated learning system. The method includes the following steps:

[0046] Please refer to Figure 1 This illustrates the flow of an embodiment of the robust-enhanced decentralized federated learning method according to the present invention. Figure 1 As shown, this robust-enhanced decentralized federated learning method includes the following steps:

[0047] Step 101: In each communication round, the target client receives model parameters of multiple neighbor models from neighbor nodes.

[0048] In this embodiment, the implementer of the robust decentralized federated learning method (e.g., a client device in a decentralized network) can exchange current local model parameters with neighboring nodes in each communication round to obtain the neighboring model.

[0049] Here, the decentralized federated learning system consists of multiple client nodes, each communicating only with its directly connected neighbors. In each round of communication, the client exchanges current model parameters with all its neighbors, constructing a decentralized communication graph. The target client can be any client in the robust decentralized federated learning system.

[0050] Step 102: Calculate the parameter differences between the local model and each neighboring model based on a multi-dimensional consistency metric.

[0051] In this embodiment, for each received neighbor model, the executing entity uses a multi-dimensional consistency measurement method to calculate the difference between the local model and the neighbor model as a reliability indicator.

[0052] Step 103: Based on the relationship between the parameter differences and the dynamic threshold, select a set of trustworthy nodes from the neighboring nodes.

[0053] Step 104 calculates the local gradient based on the local dataset and obtains the dual variable obtained by performing regularization decay on the dual variable of the previous round.

[0054] Step 105: Determine the update direction based on the local gradient and the current dual constraint term.

[0055] In this embodiment, the execution entity calculates the local gradient based on the local dataset and updates the local model parameters by introducing a distributed optimization framework with dual variables, in conjunction with a robust aggregation model.

[0056] Step 106: Determine the momentum of the current round based on the update direction and the historical momentum accumulation value.

[0057] In this embodiment, the executing entity applies regularization decay processing to the dual variable to reduce the impact of historical accumulated bias on the current update.

[0058] Step 107: Update the local model parameters based on the momentum and learning rate.

[0059] Multidimensional consistency metrics: A parameter difference evaluation system that integrates three-dimensional geometric features, including: Distance metric (Euclidean distance): Calculates the L2 norm difference between parameter vectors, reflecting the absolute deviation at the numerical level; Directional consistency metric (cosine distance): Measures the consistency of the update direction by the cosine value of the angle between vectors, identifying malicious attacks with opposite directions; Norm relative deviation metric: Calculates the relative difference of the Frobenius norm of parameter vectors, detecting overall scaling anomalies in model weights.

[0060] Dynamic threshold, the screening boundary is adaptively determined based on the statistical distribution of parameter differences in this round.

[0061] Dual variables, introduced in the ADMM distributed optimization framework, are Lagrange multipliers used to accumulate the deviation between the local model and the neighboring models in historical rounds, forming a cross-round memory mechanism.

[0062] Regularized decay is a mechanism that applies exponential decay to the historical values ​​of the dual variable to prevent the infinite accumulation of historical outliers.

[0063] The dual constraint term, a consistency penalty term constructed based on the dual variables, is used to quantify the deviation penalty between the local model and the neighbor model, forcing correction towards the consensus direction.

[0064] The update direction, taking into account both local data optimization (local gradient) and neighborhood consensus (dual constraint term), determines the evolutionary trend of the model parameters by adjusting the parameter vector un.

[0065] Momentum, based on the historical update direction accumulated by the exponential moving average (EMA), is used to smooth high-frequency oscillations during training.

[0066] In this embodiment, the execution entity performs local iterative training based on the updated local model parameters. The above steps are repeated until the number of communication rounds reaches a preset upper limit K or a convergence condition is met (such as the change in model parameters being less than a threshold or the loss function converging), and the finally trained local model is output.

[0067] It should be noted that the method provided in this embodiment achieves robust training under a decentralized architecture. Multi-dimensional consistency measurement and dynamic threshold filtering eliminate Byzantine nodes and anomalous updates, ensuring that only trusted neighbors participate in constraints. After regularization and decay, the dual variable retains necessary historical memory to constrain long-term consistency while preventing the cross-round propagation of early malicious interference. The momentum mechanism transforms hard dual constraints into smooth inertial motion, suppressing parameter oscillations caused by Non-IID data. The overall technical solution eliminates the single point of failure risk of the central server and achieves model training stability and convergence in complex environments with malicious nodes and non-independent homomorphic data distributions, significantly improving the robustness and practicality of decentralized federated learning. To verify the effectiveness of the robustness-enhanced decentralized federated learning method proposed in this invention, experiments were conducted in environments with heterogeneous data (Dirichlet distribution parameter set to 0.3) and malicious interference (label flipping attack, Byzantine ratio 0.2), and compared with decentralized federated learning methods without robust filtering aggregation and dual variable regularization.

[0068] like Figure 3 As shown, experimental results demonstrate that the proposed method significantly improves accuracy, convergence, and stability. Specifically, under the same number of training epochs, the model accuracy of this method is approximately 8% to 12% higher than the comparative method; in terms of convergence speed, the number of communication epochs required to reach the target accuracy is reduced by approximately 30% to 40%; and in terms of stability, the training curve of this method is smoother, with significantly reduced fluctuations. These experimental results prove that the proposed method can effectively improve the robustness and convergence stability of decentralized federated learning systems even in complex environments with malicious nodes and abnormal updates.

[0069] In some embodiments, the step of selecting a set of trustworthy nodes from neighboring nodes based on the relationship between the parameter differences and the dynamic threshold includes: calculating the mean and standard deviation of the parameter differences corresponding to all neighboring nodes in the current round; determining the dynamic threshold based on the mean and standard deviation, wherein the dynamic threshold is the product of the mean and the threshold adjustment coefficient plus the standard deviation; and including neighboring nodes whose parameter differences are less than or equal to the dynamic threshold into the set of trustworthy nodes.

[0070] The mean of parameter differences refers to the arithmetic mean of the set of parameter differences calculated between the client and all neighboring nodes in the current communication round k, reflecting the overall deviation level of the neighbor model in this round.

[0071] The standard deviation of parameter differences refers to the standard deviation of the set of parameter differences mentioned above. It reflects the degree of dispersion of differences between neighboring nodes and is used to identify outliers.

[0072] Dynamic threshold refers to the screening boundary adaptively determined based on statistical distribution characteristics. The threshold adjustment coefficient is a hyperparameter used to control the strictness of screening, and its value is usually in the range of [0.5, 2.0]. When α is large (e.g., 1.5), the system tolerates more differences and has enhanced inclusiveness; when α is small (e.g., 0.8), the screening is more stringent, and only highly similar models are accepted.

[0073] Therefore, a dynamic threshold mechanism based on statistical distribution characteristics enables adaptive identification of trustworthy nodes. Compared to a fixed threshold, this mechanism can automatically adjust the screening criteria according to real-time changes in the data distribution of the current round: when the Non-IID characteristic leads to generally large differences among neighbors (increased standard deviation), the threshold is increased accordingly to avoid mistakenly identifying honest nodes; when extreme outliers occur (deviation from the mean by more than α times the standard deviation), even if the overall level of difference is high (large mean), outliers can still be accurately identified. This adaptive capability allows the system to adapt to dynamically changing data distributions and attack patterns in real-world environments without pre-setting attack prior knowledge, significantly improving the accuracy and environmental adaptability of screening.

[0074] In some embodiments, updating the local model parameters based on the momentum and learning rate includes:

[0075] According to the following formula

[0076] Update local model parameters;

[0077] in, Indicates that the target client n is in the th order. Round of global iteration, the first Model parameters during the next local iteration Indicates the learning rate; Indicates the target client at the 1st Round of global iteration, the first Momentum during sublocal iterations.

[0078] The learning rate is a hyperparameter used to control the step size of parameter updates. This determines the adjustment range of model parameters along the momentum direction in each iteration, balancing convergence speed and stability.

[0079] Therefore, stable iteration of local model training is achieved through the above parameter update formula. Combined with the momentum mechanism, this update strategy avoids parameter oscillations that may result from directly using the original gradient or dual constraints, ensuring that the model parameters converge towards the optimal solution along a smooth inertial trajectory. The introduction of the learning rate provides a means to adjust the convergence speed. While ensuring training stability, reasonable step size control improves convergence efficiency and prevents model divergence due to excessively large step sizes or convergence lag due to excessively slow step sizes.

[0080] In some embodiments, determining the momentum of the current cycle based on the update direction and the historical momentum accumulation value includes:

[0081] According to the following formula

[0082] Calculate the momentum of the current round;

[0083] Historical momentum accumulation value, used to represent the target client's momentum accumulation value at the [number]th [time]. Round of global iteration, the first Momentum during sublocal iteration;

[0084] in, Indicates the target client at the 1st Round of global iteration, the first Update direction during sublocal iteration.

[0085] Historical momentum accumulation is a weighted historical value calculated using an exponential moving average, representing the accumulated trends in the past. Current momentum is a new momentum value calculated using an exponential moving average, combining historical trends with the current trend.

[0086] Thus, the momentum calculation effectively smooths out high-frequency oscillations during training through the exponential moving average mechanism. In decentralized federated learning, dual constraints may fluctuate drastically in Non-IID data environments. The momentum mechanism filters out instantaneous disturbances into stable trends by preserving historical trends, preventing model parameters from oscillating between neighbor constraints. Simultaneously, momentum accumulation provides an inertial propulsion effect; even if there is a delay in a round of communication or a temporary absence of individual neighbors, historical momentum can still maintain a stable propagation towards group consensus, significantly accelerating the convergence speed of cross-node consensus and improving training stability.

[0087] In some embodiments, determining the update direction based on the local gradient and the current dual constraint term includes:

[0088] The update direction is determined based on the following formula:

[0089]

[0090] in, Indicates the target client at the 1st Round of global iteration, the first Update direction during sublocal iteration Representing the trusted neighbor model The dual constraint term of the difference. This indicates that the target client's current local model parameters are compared with the trusted neighbor model in the 1st... The differences between model parameters after each round of global iteration This indicates division by the penalty parameter. The cumulative historical model difference parameter after scaling.

[0091] The penalty parameter is used to quantify the strength of the penalty for inconsistent behavior in the model, controlling the trade-off between local optimization (gradient descent) and neighbor consensus (consistency constraint). The larger the penalty parameter, the stronger the forced consistency constraint.

[0092] The difference between the current local model parameters and the trusted neighbor model is used for real-time calculation of parameter deviation, reflecting the immediate degree of deviation after this round of communication.

[0093] The cumulative historical model difference parameter is used to characterize the value of the dual variable after scaling by the penalty parameter, representing the contribution weight of the cumulative deviation of historical rounds to the current constraint.

[0094] Therefore, by explicitly introducing a dual constraint term, a collaborative update mechanism for consistency constraints and local optimization is constructed. The penalty parameter provides a means to adjust the constraint strength, forcing the local model to converge towards the consensus direction of trusted neighbors, effectively preventing the divergence of models at each node in a Non-IID data environment. The historical accumulation term of the dual variable introduces a long-term memory mechanism, ensuring that even if an attacker employs a latent strategy of small deviations in each round, the accumulation of historical deviations will still generate sufficient penalty force in subsequent rounds, forcing a correction back to the consensus direction. This mechanism, while safeguarding the local data optimization space, maintains the global consistency of the decentralized network, significantly improving the convergence stability of the model in a distributed heterogeneous environment.

[0095] In some embodiments, the method further includes: for each trusted neighbor model in the trusted node set, calculating the reciprocal of the parameter difference between it and the local model as an initial weight; normalizing all initial weights to obtain the aggregate weight of each trusted neighbor model; and performing a weighted summation of all trusted neighbor models in the trusted node set based on the aggregate weight to obtain an aggregate model.

[0096] As an example, the following formula is used to perform model aggregation based on the filtered set of trusted nodes.

[0097]

[0098] Therefore, by using distance-inverse weighting and normalization, differentiated trust assignments are achieved for trusted neighbors. This mechanism grants higher aggregation weights to high-quality neighbors (those with small parameter differences), while the weights of potentially malicious nodes are significantly reduced due to their large differences, even if they pass the screening. The consensus model generated by weighted aggregation provides a benchmark for collective intelligence, which can be used to assist in the convergence monitoring of dual constraints and as a reference benchmark for the next round of communication. This ensures that even in environments with a high proportion of Byzantine faults, the collective intelligence of honest nodes still dominates the overall model direction, effectively suppressing the impact of malicious interference on the system and improving the aggregation quality and the accuracy of the final model.

[0099] In some embodiments, the update formula for the dual variable is:

[0100]

[0101] in, Let m represent the dual variables between the target client n and the trusted neighbor model m in the k-th round. This represents the penalty parameter, where i represents the attenuation coefficient. and These represent the updated local model and the trusted neighbor model, respectively.

[0102] By implementing regularized updates of the dual variable through an explicit mathematical formula, a quantifiable historical control mechanism is provided. This formula can "regularize the propagation of biases and accelerate convergence" by precisely adjusting the half-life of historical memory through the parameter i, enabling the system to achieve a balance between "rapidly forgetting outdated information" and "retaining necessary historical memory." In malicious interference environments, this formula ensures that early attack signals do not indefinitely affect current decisions, while retaining reasonable memories of recent normal collaborations, thereby smoothing the training curve, avoiding parameter oscillations, and significantly improving the convergence speed and final model accuracy of the algorithm in non-independent homomorphic data environments.

[0103] In some embodiments, calculating the parameter difference between the local model and each neighboring model based on a multi-dimensional consistency metric includes: calculating the following metrics for each layer of the model parameters: calculating the absolute difference between parameter vectors based on Euclidean distance; calculating the directional deviation between parameter vectors based on cosine similarity; calculating the scale relative difference between parameter vectors based on norm ratio; and weighting and fusing the absolute difference, directional deviation, and scale relative difference to obtain the parameter difference.

[0104] Euclidean distance refers to the L2 norm of the difference between corresponding elements of two model parameter vectors in an L-dimensional parameter space. The formula is as follows: This metric reflects the absolute deviation of the model weights at the numerical level and is used to detect anomalous jumps in parameter values.

[0105] Cosine similarity measures directional consistency by calculating the cosine of the angle between two parameter vectors. Its distance form is defined as... The value is 0 when the two vectors are in the same direction and 1 when they are in opposite directions. This metric specifically captures the adversarial nature of the model's update direction.

[0106] The relative deviation of the norm refers to the relative difference in the overall scale (Frobenius norm) of two parameter vectors, and is calculated using the following formula: This metric detects overall scaling anomalies in model weights, serving as a defense against model scaling attacks or norm bombing attacks.

[0107] By employing a three-dimensional collaborative evaluation of Euclidean distance, cosine similarity, and norm relative deviation, the deviation characteristics between model parameters can be comprehensively characterized, rather than relying solely on single numerical differences. This mechanism can comprehensively assess model differences, improve anomaly detection accuracy, and effectively distinguish between reasonable offsets caused by data heterogeneity and anomalous deviations caused by malicious attacks in various scenarios. Even if attackers employ covert scaling or sign-flipping attacks, orientation and scale metrics can still expose their malicious nature, preventing honest nodes from being misclassified as malicious nodes. This ensures that only genuine anomalous updates are filtered in decentralized networks, improving the robustness and accuracy of federated learning systems in the face of unknown attack types.

[0108] In some embodiments, the formula for weighted fusion is:

[0109]

[0110] Where Vn represents the local model, Vm represents the neighbor model, L represents the total number of model layers, and W represents the total number of layers. l This represents the weight coefficient of the l-th layer. Represents the Euclidean distance of the l-th layer. Represents the cosine distance of the l-th layer. λ1, λ2, and λ3 represent the relative difference in norms of the l-th layer, and λ1, λ2, and λ3 are fusion coefficients.

[0111] The total weight normalization factor (Wtotal) refers to the sum of the weight coefficients of all network layers. It is used to normalize the weighted differences of each layer to a global average difference, ensuring the comparability between networks of different depths.

[0112] It should be noted that by configuring hierarchical weights and adjusting fusion coefficients, a refined quantitative evaluation of model differences is achieved. This allows for the quantification of parameter differences, supports hierarchical weight configuration, and enables the system to differentiate the importance of deviations in each layer based on network structural characteristics (e.g., shallow layers are sensitive, deep layers are stable). This improves the accuracy of difference evaluation for complex neural network models and provides a reliable quantitative basis for subsequent screening and aggregation.

[0113] In some embodiments, the adaptive weighted aggregation of the neighbor model in the set of trusted nodes based on the parameter differences includes:

[0114] For each neighbor model in the set of trusted nodes, calculate the reciprocal of the parameter difference between it and the local model, and use it as the initial weight;

[0115] All initial weights are normalized to obtain the aggregate weights for each neighbor model;

[0116] The robust aggregation model is obtained by weighted summation of all neighbor models in the set of trusted nodes based on the aggregation weight.

[0117] By employing distance-inverse weighting and normalization, an adaptive optimization allocation of aggregation weights is achieved. This mechanism reduces the weight of malicious nodes, ensuring aggregation quality. This weight allocation strategy effectively prevents the global model from drifting towards malicious update directions, ensuring the accuracy and security of industrial prediction models.

[0118] For example, please refer to Figure 2 , Figure 2 This paper presents an exemplary flow of a robust ADMM decentralized federated learning method proposed in this invention, introducing a robust aggregation mechanism with multi-dimensional difference measurement and a dual variable regularization decay mechanism. By performing consistency measurement, dynamic filtering, and adaptive weighted aggregation on neighbor models, combined with the dual variable regularization decay mechanism, stable consensus among honest nodes is achieved. This algorithm not only improves the consistency and final accuracy among local models, but also effectively suppresses the interference of malicious nodes on the training process in unknown attack environments, ensuring the stability and robustness of the algorithm in distributed, non-IID data environments. The specific key points and steps of the invention are as follows:

[0119] Step 1: Algorithm Input and Initialization. Input parameters include: initial model. , indicating each client Initialization model; dual variables , representing the Lagrange multipliers introduced in the ADMM algorithm to handle consistency constraints in distributed optimization, where the constraint condition is the neighboring clients. and The model parameters must be consistent; the learning rate The penalty parameter determines the magnitude of each local model update. Quantify the penalty for differences in models between adjacent nodes; momentum parameter Control the historical gradient weights.

[0120] The second step is model communication and robust aggregation.

[0121] In each round of communication, the client From all neighboring nodes Swap current model parameters A decentralized communication graph is constructed. For the received neighbor models, a model consistency metric is used. To calculate the difference between the local model and the neighboring models, where , , And this gap is used as a credibility indicator.

[0122] Based on dynamic thresholds, i.e., conditions Filter the set of trusted nodes ,in This represents the average value of the difference. The variance represents the difference.

[0123] The model aggregation operation based on the filtered set of trustworthy nodes is as follows:

[0124]

[0125] The third step is local training.

[0126] Each client is based on the formula Calculate the local gradient. Indicates the client In the Round of global iteration, the first Local gradient during the next local iteration Indicates the client The gradient of the loss function is calculated locally. This represents the gradient operator. Indicates the client In the Round of global iteration, the first Model parameters during the next local iteration.

[0127] To ensure that the deviation between the local client model and the neighbor client model is controlled by the dual variable To constrain parameters and prevent divergence, each client needs to combine its local gradient with consistency constraints from neighboring clients, according to the formula. Calculate the update direction. Among them, Indicates the client In the Round of global iteration, the first Update direction during sublocal iteration The dual constraint term represents the difference between adjacent client models. Indicates the client Current local model parameters and neighboring clients In the The differences between model parameters after each round of global iteration This indicates division by the penalty parameter. The cumulative historical model difference parameter after scaling.

[0128] Each client accumulates historical update directions, according to the formula.

[0129] Calculate momentum. Among them, Indicates the client In the Round of global iteration, the first Momentum during sublocal iterations.

[0130] According to the formula Update the local model. Among these, Indicates the client In the Round of global iteration, the first Model parameters during the next local iteration. Indicates learning rate

[0131] Finish After the next local iteration, in order to pass the final state of the local optimization between rounds, set , .

[0132] Step 4: Regularized decay dual variable update

[0133] The dual variable is subject to regularized decay in each update round, by applying exponential decay to the historical dual variable. Suppressing abnormal gradient accumulation improves training stability under malicious interference environments.

[0134] The fifth step is to iteratively update the communication.

[0135] Repeat the above steps until the number of communication rounds reaches the preset upper limit K or convergence is achieved, and output the final trained local model.

[0136] Please refer to Figure 3 , Figure 3 The technical effects of one or more embodiments of the present invention are illustrated. Experiments were conducted in an environment with heterogeneous data (Dirichlet distribution parameter set to 0.3) and malicious interference (label flipping attack, Byzantine ratio 0.2). The results were compared with a decentralized federated learning method without robust filtering aggregation and dual variable regularization. As can be seen from the experimental figures above, the method proposed in this invention significantly improves accuracy, convergence, and stability, proving the effectiveness of the proposed method.

[0137] Please refer to Figure 4 , Figure 4 A schematic diagram of an embodiment of a robust decentralized federated learning system is shown.

[0138] exist Figure 4 The system shown may include multiple clients that communicate directly: each client executes the method of any other embodiment or combination of embodiments of the present invention to perform local iterative training.

[0139] In this embodiment, each client performs local iterative training, which includes: receiving model parameters from multiple neighboring models from neighboring nodes; calculating the parameter differences between the local model and each neighboring model based on a multi-dimensional consistency metric, wherein the multi-dimensional consistency metric includes a distance metric between parameter vectors, a directional consistency metric, and a norm relative deviation metric; selecting a set of trustworthy nodes from the neighboring nodes based on the relationship between the parameter differences and a dynamic threshold, wherein the dynamic threshold is determined based on the statistical distribution characteristics of all parameter differences in the current round; calculating the local gradient based on the local dataset, and obtaining the dual variable obtained by performing regularized decay processing on the dual variable of the previous round; determining the update direction based on the local gradient and the current dual constraint term, wherein the dual constraint term is used to constrain the differences between the local model and the trustworthy nodes, and the dual constraint term is generated based on the dual variable; determining the momentum of the current round based on the update direction and the historical momentum accumulation value; and updating the local model parameters based on the momentum and the learning rate.

[0140] exist Figure 4 Please refer to the Chinese text. Figure 4 , Figure 4 An exemplary system architecture of an embodiment of the decentralized federated learning system of the present invention is shown.

[0141] like Figure 4 As shown, the system architecture may include clients 401, 402, 404, and 405, and network 403. Network 403 is used as the medium to provide a communication link between clients 401, 402, 404, and 405. Network 403 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0142] Clients 401, 402, 404, and 405 can be either hardware or software. When clients 401, 402, 404, and 405 are hardware, they include, but are not limited to, servers, smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc. When clients 401, 402, 404, and 405 are software, they can be installed on the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0143] It should be noted that the decentralized federated learning method provided in this embodiment can be executed by a client.

[0144] It should be understood that Figure 4 The number of clients and networks shown is merely illustrative. Depending on implementation needs, there can be any number of clients and networks.

[0145] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A robust decentralized federated learning method, characterized in that, The method, applied to a centralized federated learning system comprising multiple clients communicating directly, includes: In each communication round, the target client receives model parameters of multiple neighbor models from neighbor nodes; Based on a multi-dimensional consistency metric, the parameter differences between the local model and each neighboring model are calculated. The multi-dimensional consistency metric includes a distance metric between parameter vectors, a directional consistency metric, and a norm relative deviation metric. Based on the relationship between the parameter differences and the dynamic threshold, a set of trustworthy nodes is selected from the neighboring nodes, wherein the dynamic threshold is determined based on the statistical distribution characteristics of all parameter differences in this round; Calculate the local gradient based on the local dataset, and obtain the dual variable obtained by applying regularized decay processing to the dual variable of the previous round; The update direction is determined based on the local gradient and the current dual constraint term, wherein the dual constraint term is used to constrain the difference between the local model and the trusted node, and the dual constraint term is generated based on the dual variable; Based on the update direction and the historical momentum accumulation value, the momentum of the current round is determined; Update the local model parameters based on the momentum and learning rate.

2. The method according to claim 1, characterized in that, The step of selecting a set of trustworthy nodes from neighboring nodes based on the relationship between the parameter differences and the dynamic threshold includes: Calculate the mean and standard deviation of the parameter differences for all neighboring nodes in this round; The dynamic threshold is determined based on the mean and standard deviation, wherein the dynamic threshold is the product of the mean and the threshold adjustment coefficient plus the standard deviation; Neighboring nodes whose parameter differences are less than or equal to the dynamic threshold are included in the trusted node set.

3. The method according to claim 1, characterized in that, The step of updating the local model parameters based on the momentum and learning rate includes: according to the following formula Update local model parameters; in, Indicates that the target client n is in the th order. Round of global iteration, the first Model parameters during the next local iteration Indicates the learning rate; Indicates the target client at the 1st Round of global iteration, the first Momentum during sublocal iterations.

4. The method according to claim 3, characterized in that, The determination of the momentum for the current round based on the update direction and historical momentum accumulation includes: according to the following formula Calculate the momentum of the current round; Historical momentum accumulation value, used to represent the target client's momentum accumulation value at the [number]th [time]. Round of global iteration, the first Momentum during the next local iteration; where... Indicates the target client at the 1st Round of global iteration, the first Update direction during sublocal iteration.

5. The method according to claim 4, characterized in that, The step of determining the update direction based on the local gradient and the current dual constraint term includes: determining the update direction based on the following formula: ; in, Indicates the target client at the 1st Round of global iteration, the first Update direction during sublocal iteration Representing the trusted neighbor model The dual constraint term of the difference. This indicates that the target client's current local model parameters are compared with the trusted neighbor model in the 1st... The differences between model parameters after each round of global iteration This indicates division by the penalty parameter. The cumulative historical model difference parameter after scaling.

6. The method according to claim 3, characterized in that, The method further includes: For each trusted neighbor model in the set of trusted nodes, calculate the reciprocal of the parameter difference between it and the local model, and use it as the initial weight; All initial weights are normalized to obtain the aggregate weights for each trusted neighbor model; Based on the aggregation weight, a weighted summation is performed on all trusted neighbor models in the trusted node set to obtain the aggregated model.

7. The method according to claim 1, characterized in that, The update formula for the dual variable is: ; in, Let m represent the dual variables between the target client n and the trusted neighbor model m in the k-th round. This represents the penalty parameter, where i represents the attenuation coefficient. and These represent the updated local model and the trusted neighbor model, respectively.

8. The method according to claim 1, characterized in that, The calculation of parameter differences between the local model and each trusted neighbor model based on a multi-dimensional consistency metric includes: For each layer of the model parameters, calculate the following metrics: Calculate the absolute difference between parameter vectors based on Euclidean distance; Calculate the directional deviation between parameter vectors based on cosine similarity; Calculate the scale relative difference value of the parameter vector based on the norm ratio; The parameter difference is obtained by weighting and fusing the absolute difference value, the directional deviation value, and the scale relative difference value.

9. The method according to claim 8, characterized in that, The formula for the weighted fusion is: ; Where Vn represents the local model, Vm represents the trusted neighbor model, L represents the total number of model layers, and W represents the total number of model layers. l This represents the weight coefficient of the l-th layer. Represents the Euclidean distance of the l-th layer. Represents the cosine distance of the l-th layer. λ1, λ2, and λ3 represent the relative difference in norms of the l-th layer, and λ1, λ2, and λ3 are fusion coefficients.

10. A robust decentralized federated learning system, characterized in that, It includes multiple clients that communicate directly; each client performs local iterative training, wherein the local iterative training includes Receive model parameters from multiple neighbor models from neighbor nodes; Based on a multi-dimensional consistency metric, the parameter differences between the local model and each neighboring model are calculated. The multi-dimensional consistency metric includes a distance metric between parameter vectors, a directional consistency metric, and a norm relative deviation metric. Based on the relationship between the parameter differences and the dynamic threshold, a set of trustworthy nodes is selected from the neighboring nodes, wherein the dynamic threshold is determined based on the statistical distribution characteristics of all parameter differences in this round; Calculate the local gradient based on the local dataset, and obtain the dual variable obtained by applying regularized decay processing to the dual variable of the previous round; The update direction is determined based on the local gradient and the current dual constraint term, wherein the dual constraint term is used to constrain the difference between the local model and the trusted node, and the dual constraint term is generated based on the dual variable; Based on the update direction and the historical momentum accumulation value, the momentum of the current round is determined; Update the local model parameters based on the momentum and learning rate.

Images