An Adaptive Distributed Support Vector Regression Method to Combat Byzantine Fault
By employing an adaptive distributed support vector regression method, utilizing a smooth objective function and MOM aggregation mechanism, the robustness and asymptotic efficiency issues of distributed regression methods under Byzantine faults are addressed, achieving efficient prediction and stability in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- QINGDAO UNIV
- Filing Date
- 2026-02-12
- Publication Date
- 2026-05-26
AI Technical Summary
Existing distributed support vector regression methods suffer from performance degradation under Byzantine fault conditions, making it difficult to achieve robust estimations, and traditional improved methods have low asymptotic efficiency.
An adaptive distributed support vector regression method is adopted. By introducing the smoothing concept to approximate the objective function, and combining the MOM aggregation mechanism and robust gradient estimation, a robust global gradient update mechanism is designed to suppress the influence of abnormal machines and improve the asymptotic efficiency of global gradient estimation.
In the presence of Byzantine faults, it improves the robustness and prediction accuracy of distributed regression methods, making it applicable to a variety of real-world distributed systems. It shows significant advantages, especially in scenarios with potentially malicious machines or unreliable devices, and possesses good scalability and computational efficiency.
Smart Images
Figure CN122087769A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of distributed machine learning optimization technology, and mainly relates to an adaptive distributed support vector regression method to combat Byzantine faults. Background Technology
[0002] With the continuous expansion of data volume, single-machine storage and computing models can no longer meet the needs of modern data analysis. Even if data can be stored completely on a single machine, its computational efficiency is often severely limited. Therefore, adopting a distributed architecture for data storage and processing has become an inevitable trend. However, the introduction of a distributed environment brings new theoretical and practical challenges to traditional statistical inference methods. In a typical distributed computing framework, data is distributed across multiple worker machines, each machine independently performs local computations, and transmits the results to the host machine through communication. The host machine is responsible for maintaining and updating global parameters. However, in practical applications, distributed statistical inference still faces many problems, the most critical of which is the Byzantine problem.
[0003] The Byzantine problem is one of the core challenges in distributed systems. Due to hardware or software failures, data corruption, or communication anomalies, worker machines may transmit incorrect information, leading to a significant degradation in the overall system performance. This type of failure is known as a Byzantine failure. To address this risk, researchers have proposed various robust distributed algorithms designed to maintain the convergence and accuracy of the system even in the presence of malicious or malfunctioning machines.
[0004] In distributed regression problems, traditional Support Vector Regression (SVR) methods typically aggregate the local results from each machine using a weighted average. Traditional SVR has several significant advantages: it is insensitive to outliers, its computational complexity is independent of the input space dimension, and it boasts strong generalization ability and high prediction accuracy. However, in Byzantine environments, its performance often degrades sharply, making robust estimation difficult. Existing improvements, such as Mean aggregation and Median-Of-Means (MOM), while enhancing robustness to some extent, generally suffer from low asymptotic efficiency. Therefore, there is an urgent need to construct a novel framework that maintains distributed convergence, high robustness, and good asymptotic efficiency even in Byzantine environments, to further improve the robustness and predictive performance of distributed SVR in complex environments. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and to design an adaptive distributed support vector regression method to combat Byzantine faults. This method enhances robustness under data anomalies by introducing an SVR model and employs a variance-reduced MOM aggregation mechanism to effectively suppress the influence of anomalous machines while improving the asymptotic efficiency of global gradient estimation.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] An adaptive distributed support vector regression method to combat Byzantine faults includes the following steps:
[0008] (1) Let the sample set be ,in Indicates the first The feature vector of each sample For its response variable, , for Victorious space, It is a one-dimensional Euclidean space.
[0009] Based on the classic SVR, let the regression hyperplane be... ,in, For the weight vector, This is a bias term. For uniform representation, let... And record the residuals The empirical risk minimization objective function for SVR is written as:
[0010] (1)
[0011] in, , For insensitive loss degree, This is the regularization parameter.
[0012] To obtain a differentiable approximation, the smoothing approach of Pang et al. is adopted. A smooth approximation is performed. First, a sign function is introduced.
[0013]
[0014] but Further ,but For smoothing Define a smooth approximation function Among them, bandwidth parameters and take For the indicator function A smooth approximation, specifically defined as
[0015]
[0016] when hour, ,in ,thereby Therefore, the objective function for minimizing the empirical risk of SVR can be approximated by the following smooth form.
[0017]
[0018] The parameter is obtained according to equation (2). The gradient is
[0019]
[0020] in, for The first derivative of .
[0021] (2) The index set of samples Divided into A subset that does not overlap with each other The agreement Each machine holds an index set For the corresponding sample, let t=1, the host machine obtains the initial parameter estimates on its local data. .
[0022] (3) During the t-th iteration, the MOM-based aggregation method obtains the MOM estimate and uses it as the global gradient estimate. , , , Indicates the first The local gradient of the d-th dimension of the machine.
[0023] (4) In global gradient estimation To obtain robust global gradients ,
[0024]
[0025] in, It is the variance of the gradient in the d-th dimension. For quantile partition numbers, , , It is the inverse function of the standard normal distribution. For indicator functions, This represents the probability density function of the standard normal distribution.
[0026] (5) The host machine obtains a robust global gradient. Then, the global parameters are updated using gradient descent. ,in, This is the learning rate.
[0027] (6) When When t = t + 1, Repeat steps (3)-(5), or conversely, when At this point, convergence is considered complete, and the global optimal parameter estimate is finally obtained. .
[0028] Step (3) specifically involves the host machine broadcasting parameters in the t-th iteration. To each working machine. The machine is receiving Then, calculations are performed based on the local sample set. The local gradient vector of dimension 1, the first dimension 2, is... Uploaded by machine dimensional local gradient vector The global gradient of L working machines is denoted as
[0029]
[0030] The median of each dimension is used as the global gradient estimate. , , .
[0031] Step (4) specifically involves the following: Since the system may be susceptible to Byzantine attacks, directly aggregating a global gradient may lead to deviations in model updates, or even complete deviations from the optimal solution. Consider the following stochastic optimization problem.
[0032] (3)
[0033] in, As expected, The parameters to be optimized are... This represents the quantile loss function. For quantile partition numbers, , , It is the variance (theoretical value) of the gradient in the d-th dimension. , It is the inverse function of the standard normal distribution.
[0034] The first derivative of equation (3) is
[0035]
[0036] in, This is an indicator function.
[0037] The second derivative (Hessian matrix) of equation (3) is:
[0038]
[0039] in, For the first The expected value (theoretical value) of the gradient in dimension . , The probability density function of the standard normal distribution is used here. As The initial value is updated to the estimated value.
[0040]
[0041] This yields an explicit approximation of the robust global gradient on the host machine.
[0042]
[0043] parameter It can be estimated as
[0044] , for In the Dimension value,
[0045] when ,therefore And because Further approximations yield Finally, a one-step estimator is obtained.
[0046]
[0047] Thus, the host machine can obtain asymptotically unbiased and variance-controlled robust gradient estimates in the presence of a Byzantine machine. .
[0048] Compared with existing technologies, this invention offers the following advantages: By introducing a robust aggregation mechanism based on the MOM method, it effectively suppresses the influence of anomalous machines, significantly improving system stability and robustness while ensuring prediction accuracy. It can be widely applied to various practical distributed systems, especially in scenarios with potentially malicious machines or unreliable devices. This method is suitable for typical "master-slave" distributed computing architectures: the master machine is responsible for maintaining the global model and gradient aggregation, while each worker machine independently executes the SVR training task on its local dataset and periodically uploads local gradients to the master machine for robust aggregation. No original data needs to be transmitted during the entire process, effectively protecting data privacy, making it particularly suitable for distributed learning scenarios such as federated learning where data does not leave the local machine. This method possesses good scalability and computational efficiency, adapting to large-scale machine deployments with low communication costs, making it suitable for efficient operation in edge computing or resource-constrained environments. In summary, the proposed algorithm achieves a good balance between theoretical robustness and engineering feasibility, providing an efficient and reliable solution for distributed regression learning. Attached Figure Description
[0049] Figure 1 This is a flowchart of the adaptive distributed support vector regression method against Byzantine faults involved in Example 1.
[0050] Figure 2 This is a framework diagram of the adaptive distributed support vector regression method for combating Byzantine faults involved in Example 1.
[0051] Figure 3 The graph shows a comparison of MSE (Mean Sequence Equation) on the same dataset using three methods—MOM, VRMOM, and MEAN—based on the SVR (Smoothing Objective Function) model, on a machine without Byzantine faults.
[0052] Figure 4 The graph shows the MSE comparison of three methods based on the SVR smoothing objective function: MOM, VRMOM, and MEAN, on the same dataset with a Byzantine fault machine ratio of 20%.
[0053] Figure 5 The graph shows the MSE comparison of three methods based on the SVR smoothing objective function: MOM, VRMOM, and MEAN, on the same dataset with a Byzantine fault machine ratio of 40%. Detailed Implementation
[0054] The present invention will be further described below through specific embodiments.
[0055] Example 1
[0056] The adaptive distributed support vector regression method against Byzantine faults involved in this embodiment specifically includes the following steps:
[0057] (1) First, the objective function of the SVR model is smoothed to construct the empirical risk minimization objective function of SVR.
[0058] Let the sample set be ,in Indicates the first The feature vector of each sample For its response variable, , for Victorious space, It is a one-dimensional Euclidean space.
[0059] Based on the classic SVR, let the regression hyperplane be... ,in, For the weight vector, This is a bias term. For uniform representation, let... And record the residuals The empirical risk minimization objective function for SVR is written as:
[0060] (1)
[0061] in, , This is the insensitive loss, used to control the model's tolerance for small errors; This is a regularization parameter used to suppress model complexity and alleviate overfitting. This is an optimization parameter. and This invention designs an adaptive parameter adjustment strategy that integrates the comprehensiveness of grid search with the robustness of cross-validation.
[0062] Equation (1) objective function in No detail should be overlooked, and Since it is not differentiable at zero, it is difficult to directly use gradient-based optimization methods. To obtain a differentiable approximation, the smoothing idea of Pang et al. is adopted. Perform a smooth approximation.
[0063] First, let's introduce the symbolic function.
[0064]
[0065] but Further ,but For smoothing Define a smooth approximation function Among them, bandwidth parameters and take For the indicator function A smooth approximation (which can be viewed as an integral form of a kernel function) is specifically defined as follows:
[0066]
[0067] when hour, ,in ,thereby Therefore, the objective function for minimizing the empirical risk of SVR can be approximated by the following smooth form.
[0068]
[0069] The parameter is obtained according to equation (2). The gradient is
[0070]
[0071] in, for The first derivative of .
[0072] (2) The index set of samples Divided into A subset that does not overlap with each other The agreement Each machine holds an index set The corresponding samples, Each machine contains One sample, .
[0073] Assume that a certain percentage of the machines in the system have Byzantine faults, the number of which is... ,in This indicates the proportion of faulty machines, meaning the number of Byzantine machines is less than half the total number of machines in the system.
[0074] Let t=1, the host machine obtains initial parameter estimates on its local data using a standard linear regression solver (such as the linear model fitting function in R). .
[0075] (3) During the t-th iteration, the MOM-based aggregation method obtains the MOM estimate and uses it as the global gradient estimate. ;
[0076] In the t-th iteration, the host machine first broadcasts the parameters. To each working machine. The machine is receiving Then, calculations are performed based on the local sample set. The local gradient vector of dimension 1; if the 1st dimension is 160 ... If a machine is a Byzantine machine, then any value can be uploaded (denoted as ). ).but dimensional local gradient vector Updated to
[0077]
[0078] in This indicates that the objective function has parameters and sample data The value below, Indicates the objective function relative to The first derivative of . The global gradient of L working machines is denoted as .
[0079]
[0080] Then each The gradients are passed back to the master machine for initial aggregation. Traditional mean aggregation is sensitive to extreme values; even a few Byzantine machines can cause outliers that significantly affect the overall gradient aggregation result, thus reducing the accuracy and stability of model training. Therefore, the median of the d-th dimension of the global gradients from L working machines is used. , , The median of each dimension (i.e., the MOM estimate) is used as the global gradient estimate. .
[0081] (4) In global gradient estimation To obtain robust global gradients ;
[0082] While MOM estimation exhibits some robustness to Byzantine faults, multiquantile composite estimation, which fully extracts more distributional information from the gradient vector, typically performs better and yields better asymptotic results compared to methods that only use median aggregation. Details are as follows.
[0083] Because the system may be susceptible to Byzantine attacks, directly aggregating a global gradient may lead to biased model updates, or even completely deviate from the optimal solution. Consider the following stochastic optimization problem.
[0084] (3)
[0085] in, As expected, The parameters to be optimized are... This represents the quantile loss function. For quantile partition numbers, , , It is the variance (theoretical value) of the gradient in the d-th dimension. , It is the inverse function of the standard normal distribution.
[0086] To avoid directly solving complex optimization problems, a Taylor expansion of the objective function and one-step estimation are used to simplify the calculation process. The first derivative of equation (3) is...
[0087]
[0088] in, Let be the indicator function. The second derivative (Hessian matrix) of equation (3) is:
[0089]
[0090] in, For the first The expected value (theoretical value) of the gradient in dimension . , The probability density function of the standard normal distribution is used here. As The initial value is updated to the estimated value.
[0091]
[0092] This yields an explicit approximation of the robust global gradient on the host machine.
[0093]
[0094] parameter It can be estimated as
[0095] ,
[0096] for In the The value of the dimension. Minsker's research shows... It has asymptotic normality, that is, when Utilizing this property, there are And because Further approximations yield Finally, a one-step estimator is obtained.
[0097]
[0098] Thus, the host machine can obtain asymptotically unbiased and variance-controlled robust gradient estimates in the presence of a Byzantine machine. .
[0099] (5) The host machine obtains a robust global gradient. Then, the global parameters are updated using gradient descent.
[0100]
[0101] in, The learning rate is used. In each iteration, the process performs small-step updates based on robust gradients, thereby ensuring stable convergence of the method in a distributed environment with Byzantine machines.
[0102] (6) When When t = t + 1, Repeat steps (3)-(5), or conversely, when At this point, convergence is considered complete, and the global optimal parameter estimate is finally obtained. ,generally Set as .
[0103] Comparative Example 1
[0104] This comparative example only uses the MOM method to estimate the global gradient. Compared with Example 1, step (4) is omitted. Specifically, it includes the following steps:
[0105] (1)-(3) Same as Example 1.
[0106] (4) The host machine obtains the global gradient estimate. Then, the global parameters are updated using the classic gradient descent method.
[0107]
[0108] in, The learning rate is used. In each iteration, the process performs small-step updates based on robust gradients, thereby ensuring stable convergence of the method in a distributed environment with Byzantine machines.
[0109] (5) When Let t = t + 1. Repeat steps (3)-(5), or conversely, when At this point, convergence is considered complete, and the global optimal parameter estimate is finally obtained. ,generally Set as .
[0110] Figure 3-4The chart compares the mean squared error (MSE) of three methods—VRMOM (Example 1), MOM (Comparative Example 1), and MEAN—based on the smoothing objective function of SVR, on Byzantine fault machines with different proportions (0%, 20%, and 40%). Mean squared error (MSE) is a standard method for evaluating model prediction error, and the formula is as follows:
[0111] .
[0112] in, For the true parameters of the d-th dimension, For global optimal parameter estimation .exist and Construct a two-dimensional mesh in a coordinate system, where each node of the mesh represents a combination of parameters, i.e., by... and The parameter pairs are then used to split the original dataset into k-fold cross-validation methods. Each subset is used in turn as the validation set, and the remaining subsets are used as the validation set. A subset is used to train the model, and so on. After training and validation, the mean squared error (MSE) is calculated for each validation result, and this result is taken as the mean squared error. The mean of the test results is used as the model's performance metric. For each pair... and The parameter values are calculated, and their corresponding mean squared errors are determined. Finally, after k-fold cross-validation, the parameter combination with the smallest mean squared error (MSE) is selected as the optimal parameter of the model at the intersection of all parameter combinations in the grid. Figure 3-4 The ordinate in the graph represents the MSE under optimal parameters. This allows us to understand the MSE at a scale of 0% (…). When the MSE of all three methods gradually decreases and stabilizes with increasing iteration count, at proportions of 20% and 40%, MOM-SSVR and VRMOM-SSVR methods show a relatively stable convergence trend with increasing T value, while MEAN-SSVR exhibits fluctuations, indicating that it failed to maintain convergence well after long-term training. It is evident that the VRMOM-SSVR method demonstrates relatively better stability throughout the entire training process, ultimately achieving the smallest MSE distance.
Claims
1. An adaptive distributed support vector regression method to combat Byzantine faults, characterized in that, Specifically, the following steps are included: (1) Let the sample set be ,in Indicates the first The feature vector of each sample For its response variable, , for Victorious space, For one-dimensional Euclidean space, Based on the classic SVR, let the regression hyperplane be... ,in, For the weight vector, For the bias term, and for a unified representation, let And record the residuals The empirical risk minimization objective function for SVR is written as: （1） in, , For insensitive loss degree, For regularization parameters, To obtain a differentiable approximation, the smoothing approach of Pang et al. is adopted. To achieve a smooth approximation, we first introduce a sign function. but Further ,but For smoothing Define a smooth approximation function Among them, bandwidth parameters and take For the indicator function A smooth approximation, specifically defined as when hour, ,in ,thereby Therefore, the objective function for minimizing the empirical risk of SVR can be approximated by the following smooth form. The parameter is obtained according to equation (2). The gradient is in, for The first derivative of ; (2) The index set of samples Divided into A subset that does not overlap with each other The agreement Each machine holds an index set For the corresponding sample, let t=1, the host machine obtains the initial parameter estimates on its local data. ; (3) During the t-th iteration, the MOM-based aggregation method obtains the MOM estimate and uses it as the global gradient estimate. , , , Indicates the first The local gradient of the d-th dimension of the machine; (4) In global gradient estimation To obtain robust global gradients ; in, It is the variance of the gradient in the d-th dimension. For the number of quantile divisions, , , It is the inverse function of the standard normal distribution. For indicator functions, The probability density function representing the standard normal distribution; (5) The host machine obtains a robust global gradient. Then, the global parameters are updated using gradient descent. ,in, The learning rate; (6) When When t = t + 1, Repeat steps (3)-(5), or conversely, when At this point, convergence is considered complete, and the global optimal parameter estimate is finally obtained. .
2. The adaptive distributed support vector regression method for combating Byzantine faults according to claim 1, characterized in that, Step (3) specifically involves the host machine broadcasting parameters in the t-th iteration. To each working machine, the first The machine is receiving Then, calculations are performed based on the local sample set. The local gradient vector of dimension 1, the first dimension 2, is... Uploaded by machine dimensional local gradient vector The global gradient of L working machines is denoted as The median of each dimension is used as the global gradient estimate. , , .
3. The adaptive distributed support vector regression method for combating Byzantine faults according to claim 1, characterized in that, Step (4) specifically involves the following: Since the system may be susceptible to Byzantine attacks, directly aggregating a global gradient may lead to deviations in model updates, or even complete deviations from the optimal solution. Consider the following stochastic optimization problem. （3） in, As expected, The parameters to be optimized are... This represents the quantile loss function. For quantile partition numbers, , , It is the variance (theoretical value) of the gradient in the d-th dimension. , It is the inverse function of the standard normal distribution. The first derivative of equation (3) is in, For indicator functions, The second derivative (Hessian matrix) of equation (3) is: in, For the first The expected value (theoretical value) of the gradient in dimension . , The probability density function of the standard normal distribution is used here. As The initial value is updated to the estimated value. This yields an explicit approximation of the robust global gradient on the host machine. parameter It can be estimated as , for In the Dimension value, when ,therefore And because Further approximations yield Finally, a one-step estimator is obtained. Thus, the host machine can obtain asymptotically unbiased and variance-controlled robust gradient estimates in the presence of a Byzantine machine. .