Heterogeneous federated learning method based on contribution weighted aggregation

Abstract

The invention discloses a heterogeneous federated learning method based on contribution weighted aggregation, and the method comprises the steps: evaluating the distribution difference and quantity of data provided by heterogeneous equipment to obtain a contribution degree distribution weight, employing the weighted aggregation based on the contribution degree, a personalized method based on regularization, and a differential privacy protection technology, the method is realized through a data contribution evaluation module, a regular optimization module, a differential privacy module and a weighted aggregation module. According to the method, data sharing and model training among different medical institutions can be promoted while the privacy of the patient is protected, the accuracy of disease diagnosis and the formulation of a personalized treatment scheme are improved, and cross-institution collaborative learning is realized.

Description

Technical Field

[0001] The present invention belongs to the field of medical care technology, and in particular relates to a heterogeneous federated learning method based on contribution weighted aggregation. Background Art

[0002] In the era of big data, the massive data owned by various organizations and institutions has great potential application value, and data security and privacy protection have become the focus of global attention. Especially in the field of machine learning, how to effectively use distributed data for model training while ensuring that data privacy is not leaked has become an urgent problem to be solved. In order to cope with the privacy leakage risks faced by traditional machine learning during data transmission or centralized training, federated learning technology came into being. Federated learning trains models on local devices and only shares the aggregated information of model parameters without sharing the original data, thereby achieving model training and updating under the premise of protecting data privacy. However, in actual large-scale scenarios, there are large differences in data distribution, model structure, computing power and network environment among participating clients, which brings challenges to achieving federated collaboration. Heterogeneous federated learning, as an important branch of federated learning, aims to solve the problems caused by these differences. Heterogeneous federated learning overcomes the problems of insufficient data privacy protection and high data transmission requirements by designing a training mechanism that can cope with data heterogeneity, model personalization and privacy protection. It plays an important role in achieving effective collaborative training to improve prediction performance, especially in government services, financial insurance, smart medical care, smart cities and edge computing. However, the challenges faced by heterogeneous federated learning mainly include data heterogeneity, computing resource differences, model generalization ability, and security issues. Data heterogeneity refers to the inconsistent data distribution of each client, which causes the model to be offset when training locally, reducing the convergence speed and accuracy of the model. Computing resource differences mean that each client device has different computing and network resources, which requires service providers to balance between model complexity and the amount of accessible data. Model generalization ability is affected by heterogeneity because the model needs to remain effective in diverse data and computing environments. Security issues involve protecting the data generated on each device when sharing model updates to prevent sensitive information leakage. Solving these problems is critical to improving the performance and reliability of federated learning systems.

[0003] Through the above analysis, the problems and defects of the prior art are as follows:

[0004] The existing technology has made some progress in improving the generalization ability and adaptability of models in heterogeneous federated learning environments, but it still faces convergence challenges under heterogeneous data, personalized processing, communication efficiency, privacy protection and model performance balance. In order to enhance the generalization, adaptability and security of heterogeneous federated learning, the present invention provides a heterogeneous federated learning method based on contribution weighted aggregation. Summary of the invention

[0005] The purpose of the present invention is to provide a heterogeneous federated learning method based on contribution weighted aggregation, which can promote data sharing and model training among different medical institutions while protecting patient privacy, improve the accuracy of disease diagnosis and the formulation of personalized treatment plans, and realize cross-institutional collaborative learning.

[0006] The technical solution adopted in the present invention is a heterogeneous federated learning method based on contribution weighted aggregation. The contribution weights are obtained by evaluating the distribution differences and quantity of data provided by heterogeneous devices. It adopts contribution-based weighted aggregation, regularization-based personalization method, and differential privacy protection technology. It is implemented through a data contribution evaluation module, a regularization optimization module, a differential privacy module, and a weighted aggregation module.

[0007] The present invention is also characterized in that:

[0008] The data contribution evaluation module is used to measure data contribution. Data distribution difference and data volume are selected as measurement indicators, and a data contribution evaluation model is constructed by calculating the KL divergence value between the indicators.

[0009] The data contribution evaluation module can quantify the contribution of each device data by calculating the KL divergence value, and use the quantification result for subsequent model training and weight allocation. Specifically,

[0010] In the federated learning system, it is assumed that there are a total of K clients, each of which performs stochastic gradient descent SGD optimization locally and performs server synchronization after each T SGD iteration. represents the weight of client k after the SGD update in round t, It represents the global model after the nth aggregation, assuming a probability distribution T as the target distribution, and represents the weight set when the two indicators are indifferent, that is, the factors affecting the data distribution difference and data volume impact factor The data contribution evaluation model is C k , as shown in formula (1):

[0011] C k =ρ u ·u k +ρ v ·v k (1)

[0012] C-based k Calculate the model aggregation weight for client k The server collects all locally trained models from the client and aggregates them, and updates the global model according to their corresponding weights. The global model of the nth round is shown in formula (2):

[0013]

[0014] The next time the model is aggregated, the server will send ω to all heterogeneous devices participating in the training. n ( T F) , the client will use Initialize the local model and perform SGD iterations locally.

[0015] The regularization optimization module is used to formulate regularization-based optimization strategies and train personalized models according to data contribution, ensuring the fairness and consistency of weight distribution so that each device can participate in the overall task in the best way.

[0016] The regularization optimization module can adjust the regularization parameters according to the data contribution of each device, as follows:

[0017] The regularization formulas are shown in (3) and (4):

[0018] L(ω)=MSE(ω)+λR(ω) (3)

[0019]

[0020] Where ω represents the parameters that need to be trained in the network, L(ω) is the loss function, MSE is the mean square error, R(ω) is a function of ω, called the regularization term, λ∈[0,∞) is a constant that controls the degree of regularization and is used to measure the proportion of the regularization term and the loss function. When λ=0, there is no regularization. The larger λ is, the more important the regularization term is. The training goal is to minimize the loss value. There is an intermediate value in the process of λ increasing from 0, so that the training can obtain a set of suitable ω values ​​so that both MSE and λR(ω) are small enough. Choosing different forms of R(ω) will have different solutions. In the personalized optimization strategy based on regularization, L2 regularization is selected and the regularization coefficient is adjusted. By limiting the Euclidean norm of ω, many coefficients of ω can be made smaller, but not zero, thereby obtaining a smaller variance. Each eigenvalue has some contribution to the predicted value, and finally a dense solution can be obtained. The loss function J2 is shown in (5):

[0021]

[0022] Train the model independently on each device while applying personalized regularization strategies.

[0023] The differential privacy module can adopt the differential privacy technology of Gaussian mechanism in the federated learning training process by introducing perturbation noise that satisfies Gaussian distribution into the data results.

[0024] The differential privacy module can protect data privacy by introducing Gaussian noise without sacrificing model performance, as follows:

[0025] Assume that there is a finite data space x of size |x|, r represents a record with d attributes, and the data set D is an unordered set of n records sampled from the finite data space x. If two data sets D and D' have only one record r different, then the two data sets are called adjacent data sets. M is a random algorithm that accesses the database and implements certain functions. Algorithm M is used to implement differential privacy. In order to enhance data privacy protection and reduce the risk of overfitting during model training, for numerical data, according to the characteristics of the data and the privacy protection requirements, the Gaussian mechanism is used to introduce random dynamic disturbance noise into the data results. The Gaussian mechanism provides a relaxed (ε, δ)-differential privacy, defined as (6):

[0026] Pr[M(D)∈S]≤e ε Pr[M(D′)∈S]+δ (6)

[0027] Among them, Pr(M(D)∈S) represents the probability that the output of the random algorithm M belongs to the set S; the parameter ε represents the privacy budget, which represents the degree of privacy protection achieved by the differential privacy technology. The smaller the ε value, the higher the degree of privacy protection, but the worse the data availability. Conversely, the larger the ε value, the lower the degree of privacy protection and the better the data availability. The control algorithm M implements the privacy guarantee level. Differential privacy is actually a trade-off between the degree of privacy protection and data availability. δ represents the probability of allowing violation of strict differential privacy.

[0028] The weighted aggregation module can dynamically adjust the aggregation weight according to the contribution of the device, and adopts the weighted aggregation method based on contribution to adjust the aggregation weight, so that the devices that contribute more to the overall learning process have more influence in the model update.

[0029] The beneficial effect of the present invention is that the heterogeneous federated learning method based on contribution weighted aggregation assigns weights according to the distribution differences and quantity of heterogeneous device data, and proposes a weighted aggregation method based on contribution, which can dynamically adjust the weights, balance data heterogeneity, and make devices with higher contribution more influential in model updates. Secondly, a personalized strategy based on regularization is formulated for heterogeneous devices assigned different weights, so that each device can participate in the overall task in the best way. At the same time, differential privacy methods are used in federated learning training to ensure the security and privacy of participating device data. Finally, the data heterogeneity scenario of Dirichlet distribution is simulated, and experiments are carried out on MNIST and CIFAR10 datasets. The results show that the present invention can effectively improve the model's generalization ability and adaptability to heterogeneous data, and improve the overall efficiency and performance of the heterogeneous federated learning system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a model framework diagram of the heterogeneous federated learning method based on contribution weighted aggregation of the present invention;

[0031] Figure 2 It is a weighted aggregation flow chart of the heterogeneous federated learning method based on contribution weighted aggregation of the present invention;

[0032] FIG3( a ) is a Laplace distribution curve of the heterogeneous federated learning method based on contribution weighted aggregation of the present invention;

[0033] FIG3( b ) is a Gaussian distribution curve of the heterogeneous federated learning method based on contribution weighted aggregation of the present invention;

[0034] Figure 4 A diagram showing the influence of the regularization strength of the heterogeneous federated learning method based on contribution weighted aggregation of the present invention on the model accuracy;

[0035] FIG5( a ) shows the influence of σ and C on the accuracy in the heterogeneous federated learning method based on contribution weighted aggregation of the present invention;

[0036] FIG5( b ) shows the influence of σ and C on ε in the heterogeneous federated learning method based on contribution weighted aggregation of the present invention;

[0037] FIG6( a ) is a schematic diagram of the accuracy of the heterogeneous federated learning method based on contribution weighted aggregation in the present invention under MNIST experiment;

[0038] FIG6( b ) is a schematic diagram of the accuracy of the heterogeneous federated learning method based on contribution weighted aggregation of the present invention in the experiment under CIFAR10. DETAILED DESCRIPTION

[0039] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] The heterogeneous federated learning method based on contribution weighted aggregation in this invention is combined with Figure 1 and Figure 2 ,The contribution allocation weights are obtained by evaluating the distribution differences and quantity of data provided by heterogeneous devices, and contribution-based weighted aggregation, regularization-based personalization method, and differential privacy protection technology are adopted. It is implemented through data contribution evaluation module, regularization optimization module, differential privacy module, and weighted aggregation module.

[0041] The data contribution evaluation module is used to measure data contribution. Data distribution difference and data volume are selected as measurement indicators, and a data contribution evaluation model is constructed by calculating the KL (Kullback-Leibler) divergence value between indicators to accurately reflect the unique value of each device data and its contribution to the overall model training.

[0042] The data contribution evaluation module can quantify the contribution of each device data by calculating the KL divergence value, and use the quantification result for subsequent model training and weight allocation. Specifically,

[0043] In the federated learning system, it is assumed that there are a total of K clients, each of which performs stochastic gradient descent SGD optimization locally, and performs server synchronization after each T SGD iteration. represents the weight of client k after the SGD update in round t, represents the global model after the nth aggregation, the data distribution difference weight and data volume weight are u and v respectively, and the influencing factors are ρ u and ρ v , KL divergence provides an effective tool for quantifying the differences in data from different clients for the contribution evaluation model, that is, measuring the difference between data distribution difference u and data volume v, so as to adjust the aggregation strategy of the model. Assume a probability distribution T as the target distribution, which represents the weight set when the two indicators are indifferent, that is, there is a data distribution difference influencing factor and data volume impact factor The data contribution evaluation model is C k , as shown in formula (1):

[0044] C k =ρ u ·u k +ρ v ·v k (1)

[0045] C-based k Calculate the model aggregation weight for client k The server collects all locally trained models from the client and aggregates them, and updates the global model according to their corresponding weights. The global model of the nth round is shown in formula (2):

[0046]

[0047] The next time the model is aggregated, the server will send a The client will use Initialize the local model and perform SGD iterations locally.

[0048] The regularization optimization module is used to formulate regularization-based optimization strategies and train personalized models according to data contribution, ensuring the fairness and consistency of weight distribution, so that each device can participate in the overall task in the best way and improve the training efficiency of the model.

[0049] The regularization optimization module can adjust the regularization parameters according to the data contribution of each device to achieve personalized model training. The details are as follows:

[0050] HFedCWA designs a regularized personalized optimization strategy for the weights assigned to heterogeneous devices based on a weighted aggregation method based on contribution, and further customizes it through regularization technology. As part of the loss function, the regularization term imposes a penalty on the size of the model parameters, encouraging the model to learn a simpler representation, thereby improving the generalization ability of the model. The regularization formulas are shown in (3) and (4):

[0051] L(ω)=MSE(ω)+λR(ω) (3)

[0052]

[0053] Where ω represents the parameters that need to be trained in the network, L(ω) is the loss function, MSE is the mean square error, R(ω) is a function of ω, called the regularization term, λ∈[0,∞) is a constant that controls the degree of regularization and is used to measure the proportion of the regularization term and the loss function. When λ=0, there is no regularization. The larger λ is, the more important the regularization term is. The training goal is to minimize the loss value. There is an intermediate value in the process of λ increasing from 0, so that the training can obtain a set of suitable ω values ​​so that both MSE and λR(ω) are small enough. Choosing different forms of R(ω) will have different solutions. In the personalized optimization strategy based on regularization, L2 regularization is selected and the regularization coefficient is adjusted. By limiting the Euclidean norm of ω, many coefficients of ω can be made smaller, but not zero, thereby obtaining a smaller variance. Each eigenvalue has some contribution to the predicted value, and finally a dense solution can be obtained. The loss function J2 is shown in (5):

[0054]

[0055] Train the model independently on each device while applying personalized regularization strategies.

[0056] Combined with Figure 3(a) and Figure 3(b), the differential privacy module can adopt the differential privacy technology of the Gaussian mechanism in the federated learning training process. By introducing perturbation noise that satisfies the Gaussian distribution into the data results, the privacy protection capability of the data is enhanced, ensuring the security of the system when processing sensitive information.

[0057] The differential privacy module can protect data privacy by introducing Gaussian noise without sacrificing model performance, as follows:

[0058] Assume that there is a finite data space x of size |x|, r represents a record with d attributes, and the data set D is an unordered set of n records sampled from the finite data space x. If two data sets D and D' have only one record r different, then the two data sets are called adjacent data sets. M is a random algorithm that accesses the database and implements certain functions. Algorithm M is used to implement differential privacy. In order to enhance data privacy protection and reduce the risk of overfitting during model training, for numerical data, according to the characteristics of the data and the privacy protection requirements, the Gaussian mechanism is used to introduce random dynamic disturbance noise into the data results. The Gaussian mechanism provides a relaxed (ε, δ)-differential privacy, defined as (6):

[0059] Pr[M(D)∈S]≤e ε Pr[M(D′)∈S]+δ (6)

[0060] Among them, Pr(M(D)∈S) represents the probability that the output of the random algorithm M belongs to the set S; the parameter ε represents the privacy budget, which represents the degree of privacy protection achieved by the differential privacy technology. The smaller the ε value, the higher the degree of privacy protection, but the worse the data availability. Conversely, the larger the ε value, the lower the degree of privacy protection and the better the data availability. The control algorithm M achieves the privacy assurance level. Differential privacy is actually the trade-off between the degree of privacy protection and data availability. δ represents the probability of violating strict differential privacy. The differential privacy of the Gaussian mechanism is used in the HFedCWA method to obtain smooth weights, improve training efficiency, and adapt to different data sets.

[0061] The weighted aggregation module can dynamically adjust the aggregation weight according to the contribution of the device, and adopts the contribution-based weighted aggregation method to adjust the aggregation weight, so that the devices that contribute more to the overall learning process have more influence in the model update, improve the training effect of the model, and enhance the model's adaptability and generalization ability to heterogeneous data, so as to achieve more effective model updates.

[0062] Example 1

[0063] In the specific experiment, HFedCWA is based on the PFLlib (Personalized Federated Learning Algorithm Library) framework, which can flexibly integrate and implement a variety of federated learning algorithms. It can not only compare the performance of different algorithms in heterogeneous environments, but also expand and customize functions according to specific application scenarios through modular design. The experiment selected Python 3.8 as the programming language, CUDA 11.7, and PyTorch 2.0.0 as the deep learning framework. Detailed hardware and software equipment parameters, including but not limited to CPU model, memory size, GPU model and video memory capacity, are shown in Table 1 to ensure the reproducibility of the experimental results.

[0064] Table 1 Experimental environment

[0065] Modules parameter Graphics NVIDIA GeForce RTX 3060,1 CPU Intel(R)Xeon(R)CPU E5-2682v4 RAM 15GB Ubuntu Windows 10 Experimental Tools Python 3.8 Pytorch 2.0

[0066] The MNIST and CIFAR10 used in the present invention are data sets for image classification tasks with 10 output classes. MNIST contains 70,000 grayscale images of handwritten 0-9 digits, of which 60,000 are used for training and 10,000 are used for testing, and the pixels of each image are 28*28; CIFAR10 contains 60,000 images of 10 categories of airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks, of which 50,000 are used for training and 10,000 are used for testing, and each category has about 6,000 images, and the pixels of each image are 32*32.

[0067] For the MNIST dataset, HFedCWA uses a convolutional neural network (CNN) as the local training model, which is described as follows: its network architecture includes convolutional layers, pooling layers, and fully connected layers. The first convolutional layer (conv1) uses a Conv2d convolutional layer with 1 input channel, 32 output channels, a convolution kernel size of 5x5, and a step size of 1, followed by a ReLU activation function, and then a maximum pooling layer MaxPool2d with a pooling window size of 2x2 and a step size of 2; the second convolutional layer (conv2) uses a Conv2d convolutional layer with 32 input channels and 32 output channels. The number of channels is 64, the convolution kernel size is 5x5, the stride is 1, followed by a ReLU activation function, and then a max pooling layer MaxPool2d, the pooling window size is 2x2, and the stride is 2; the pre-processing of the fully connected layer (fc1), first a Linear fully connected layer, the input feature number is 1024, the output feature number is 512, and the bias is used, followed by a ReLU activation function; the output fully connected layer (fc), and finally a Linear fully connected layer, the input feature number is 512, the output feature number is 10, and the bias is used. In the HFedCWA-CNN architecture, the data is first passed through two layers of convolutional layers for feature extraction, and each convolution layer is followed by a ReLU activation function and a max pooling layer to reduce the feature dimension and extract important features. After convolution and pooling, the data is flattened and passed through a fully connected layer for further feature transformation, and finally the final classification result is obtained through the output fully connected layer.

[0068] For the CIFAR10 dataset, HFedCWA uses a deep neural network (DNN) as a local training model, which is described as follows: Its network architecture contains two main fully connected layers (Linear layers). The first fully connected layer (fc1) has an input feature number (in_features) of 3072, an output feature number (out_features) of 100, and uses a bias (bias = True); the second fully connected layer (fc) has an input feature number (in_features) of 100, an output feature number (out_features) of 10, and uses a bias (bias = True). In the HFedCWA-DNN architecture, the data is first transformed by the first fully connected layer, mapping the input 3072-dimensional features to a 100-dimensional space. Then, the second fully connected layer further maps these 100-dimensional features to the 10-dimensional space of the output layer, which usually corresponds to the 10 categories in the classification task. Each fully connected layer is usually followed by an activation function to introduce nonlinearity and perform classification.

[0069] In a heterogeneous federated learning environment, the two architectures use a contribution weighted aggregation method to enable models trained on different devices to perform effective information fusion based on the contribution of their data.

[0070] Example 2

[0071] HFedCWA aims to solve the image classification problem, and selects AUPR, accuracy (ACC), precision (P), recall (R) and F-measure (F1) indicators to evaluate the effect of the model. ACC represents the percentage of correctly classified samples in the total number of samples among all sample data, P represents the percentage of correctly classified normal samples among the number of samples predicted to be normal, R represents the percentage of correctly predicted normal samples among the number of samples that are actually normal, AUPR represents the area under the PR curve, that is, the average value of precision under different recall rates, focusing on the prediction accuracy of positive samples, acting on unbalanced data sets, F1 is the harmonic mean of P and R, and the calculation formulas are shown in (7)-(10) respectively.

[0072]

[0073]

[0074] The ACC and F1 indicators can better reflect the training effect of the model, so ACC and F1 are selected as evaluation indicators to verify the performance of HFedCWA.

[0075] Example 3

[0076] When processing diverse data from different heterogeneous devices, HFedCWA uses a weighted aggregation method to assign weights in model training based on the distribution differences and quantity of each device's data, so that devices that contribute more to the overall learning process have greater influence in model updates; and then formulate regularization-based personalized strategies for heterogeneous devices based on the assigned weights, so that each device can participate in the overall task in the best way. Combining personalized algorithms and weighted aggregation methods can capture device data characteristics, generate customized models, improve model generalization capabilities and adaptability to heterogeneous data, thereby improving the efficiency and performance of the entire federated learning system; at the same time, differential privacy technology is introduced to ensure the security and privacy of device data participating in the learning process.

[0077] Next, the effectiveness of HFedCWA will be comprehensively evaluated from four aspects: regularization strength sensitivity analysis, privacy protection parameter optimization and balance, algorithm performance analysis under heterogeneous data, and algorithm performance comparison. First, in the regularization strength sensitivity analysis, by adjusting the regularization strength, the generalization ability and overfitting risk of the model are balanced, ensuring that the model can maintain stable performance when facing data with different distributions and quantities. Secondly, in the privacy protection parameter optimization and balance, differential privacy technology is introduced. By accurately setting the parameters of Gaussian noise and reasonably allocating the privacy budget, the security and privacy of the device data participating in the learning process are protected, and the model performance is maximized. Furthermore, in the algorithm performance analysis under heterogeneous data, a weighted aggregation method is adopted to give each device data a weight in model training according to its distribution difference and quantity, so that the device with higher contribution has greater influence in model updating. Finally, by comparing a series of key performance indicators of multiple advanced algorithms in heterogeneous data environments in detail, it is proved that effective processing and efficient learning of diversified data can be achieved.

[0078] Regularization Strength Sensitivity Analysis

[0079] The regularization-based personalization strategy adjusts the regularization strength (λ value) to adapt to the data distribution and data volume on different devices. The choice of λ value is crucial to the generalization ability of the model in heterogeneous data environments. For devices with low contribution, the data distribution of these devices is usually very different from the global data distribution, or the data volume is small. In these cases, by choosing a higher λ value, the model can be trained on the same device. λ value, helping prevent the model from overfitting to these limited or biased data. This approach forces the model to learn more generalized features, thereby improving the model's generalization ability on unknown data. On the contrary, for devices with higher contribution, whose data distribution is less different from the global data distribution, or whose data volume is larger, by selecting a lower λ value, the complexity of the model during training is increased. Regarding the sensitivity analysis of the λ value, a series of λ values ​​(0.0001, 0.001, 0.01, 0.1) are selected to train the model to evaluate the impact of different λ values ​​on the model accuracy, so as to set the optimal λ value for MNIST and CIFAR10 to achieve a balance between generalization ability and training performance. By adjusting the λ value, the accuracy of the model changes as follows Figure 4 shown.

[0080] Depend on Figure 4It can be seen that the λ values ​​corresponding to the best accuracy of the model on the MNIST and CIFAR10 datasets are 0.01 and 0.001 respectively. The higher λ value on MNIST is because the data homogeneity is high and the model is more likely to overfit, while the lower λ value on the CIFAR10 dataset is used to adapt to its higher heterogeneity and complexity, further confirming the importance of regularization parameter selection for model performance. The results show that appropriate regularization strength helps the model better capture subtle features and patterns in the data, make full use of the diversity of data, and train personalized models according to the needs of each device, thereby maintaining a balance between generalization ability and fit, and adapting to the data distribution and data volume on different devices, which is particularly important for model training in a federated learning environment. Through experiments on two widely used representative datasets, MNIST and CIFAR10, the data heterogeneity problem that the client may encounter in a federated learning environment can be well simulated, proving the applicability of HFedCWA to devices with extreme data heterogeneity.

[0081] Example 4

[0082] Privacy parameter optimization and balance

[0083] In HFedCWA, the optimal accuracy is achieved by adding Gaussian noise to the gradient information and accurately setting the relevant parameter values ​​of Gaussian noise. This calculation determines the required privacy budget to balance privacy protection and model performance. The standard deviation σ of the Gaussian distribution is usually related to the noise level, which affects the stability and generalization ability of the model. The clipping threshold C is used to control the amplitude of the gradient update to maintain the stability of the model training and avoid the gradient explosion phenomenon. The privacy budget ε is negatively correlated with the noise. The relaxation term δ can balance the accuracy of the model and the strength of privacy protection. When the value of δ increases, it means that greater privacy leakage is allowed, which reduces the amount of noise injected into the model update, thereby improving the accuracy of the model. However, this improvement is at the expense of privacy protection, making the model more vulnerable to attacks. On the contrary, when the value of δ decreases, it means that the strength of privacy protection is increased, which increases the amount of noise injected into the model update, thereby reducing the accuracy of the model, but provides stronger privacy protection and improves the model's resistance to attacks. δ is set to 10, and C represents the gradient clipping threshold. The experimental ranges of parameters σ and C are σ∈[0.1, 0.2, 0.3, 0.4, 0.5] and C∈[0.1, 0.3, 0.5, 0.7, 0.9], respectively. Figure 5(a) shows the impact of σ and C on the accuracy, and Figure 5(b) shows the impact of σ and C on ε.

[0084] As shown in Figure 5(a), the larger σ is, the more noise is added, the greater the disturbance to the model parameters, and the lower the model accuracy; the smaller σ is, the less the model parameters are affected by noise, the better they can capture data features, the higher the model accuracy, and the lower the model loss function value, indicating that the model performance is better; but if σ is too small, the model will overfit the training data and lose generalization ability. Larger C values ​​may make the model too complex and increase the risk of overfitting, while smaller C values ​​may limit the learning ability of the model. When σ=0.3 and C=0.5, the model achieves the best balance between gradient stability, control complexity and generalization ability, and achieves the highest accuracy. This shows that the model can effectively learn data features while maintaining generalization ability, and strikes a balance between controlling model complexity and preventing overfitting, avoiding the instability of excessive gradients without over-restricting gradient updates. As shown in Figure 5(b), the privacy budget is proportional to model performance and inversely proportional to privacy protection. Using the formula based on the optimal values ​​of σ and C Further calculations show that the optimal value of ε is 0.8. Therefore, in the Gaussian mechanism of differential privacy, the optimal values ​​of parameters δ, σ, C, and ε are 10 -5 , 0.3, 0.5 and 0.8. MNIST and CIFAR10 are standard datasets that are widely used in similar methods. They are not much different from real datasets and have high feasibility. Extensive tests are conducted on the optimization of differential privacy parameters in HFedCWA on the two datasets to ensure the stability of its performance under different parameter settings, thereby verifying that the influence of σ and C on classification accuracy is effective. Therefore, the influence of HFedCWA on σ and C on real datasets is not very large, and a balance can be achieved between privacy protection and model performance.

[0085] Example 5

[0086] Algorithm performance under heterogeneous data

[0087] On the MNIST and CIFAR10 datasets, Dirichlet distribution is used to evaluate the performance of the algorithm under heterogeneous data of varying degrees. Weighted aggregation experiments are performed based on the contribution of heterogeneous devices to evaluate the impact of different global training rounds and client number settings on model performance.

[0088] The experimental design details are as follows: the learning rate of weighted weights and the learning rate of local update process are both set to 0.005; the client participation rate of each round of training is set to 1 to better evaluate the performance of each client; the range of global training rounds is set to {100, 200, 300, 400, 500} to observe the impact of different training cycles on model performance and ensure that the algorithm has no accuracy gain when more rounds of communication are performed; the Dirichlet parameter values ​​on MNIST and CIFAR10 are set to 0.1 and 0.3 respectively, and the regularization parameter values ​​in local personalized updates are set to 0.01 and 0.001 respectively; the range of the number of clients is set to {5, 10, 15, 20} to simulate the federated learning environment with different data heterogeneity. The heterogeneity of client devices is taken into account in the experiment, and the corresponding weights are assigned according to the distribution difference and quantity of each client data. The experimental results are shown in Table 2 and Figures 6(a) and 6(b), which record the accuracy achieved in different heterogeneous situations in detail. These data not only reveal the importance of weighted aggregation strategy for improving model performance, but also reflect the effectiveness of HFedCWA-CNN and HFedCWA-DNN algorithms in processing heterogeneous data.

[0089] Table 2 Accuracy of HFedCWA for different global training rounds and client numbers under MNIST and CIFAR10

[0090]

[0091] As shown in Table 2 and the data and diagrams in Figures 6(a) and 6(b), as the number of clients participating in training increases, the data heterogeneity between clients also increases. In this context, the accuracy of HFedCWA on the MNIST and CIFAR10 datasets is not only significantly improved, but also the accuracy of the model when it reaches convergence in different heterogeneous situations is close to the optimal level. When the number of global training rounds increases, the accuracy of HFedCWA on both datasets shows an upward trend, indicating that more training data helps improve the generalization ability of the model. The experiment reveals the important impact of client data heterogeneity on the performance of the federated learning model and proves the effectiveness of HFedCWA in dealing with data heterogeneity. HFedCWA adopts a weighted aggregation method based on contribution for clients with different data distributions. Even in extreme data heterogeneity, it can effectively balance global consistency and individual differences through a regularization-based personalization strategy. This design can adapt to the data characteristics of different clients and achieve accurate personalized adjustments during the model training process, so that HFedCWA can effectively cope with the challenges of data heterogeneity and maintain a high accuracy while protecting user privacy. Its algorithm performance has been fully verified.

[0092] Example 6

[0093] Performance comparison analysis

[0094] In the algorithm performance comparison experiment, the aim is to evaluate the performance of HFedCWA and the current leading federated learning algorithms when processing heterogeneous data, with the goal of measuring the effectiveness and efficiency of these algorithms when facing different data distributions. These algorithms include FedAvg, FML, MOON, FedBN, GPFL, EmbracingFL, FedAF, FedCPD, CroSSHeteroFL, FEDACS and RFCSC. In order to make a fair comparison, the experiment selected two widely recognized datasets, MNIST and CIFAR10, which represent different image recognition tasks and data complexity. On these datasets, HFedCWA was compared with 11 other algorithms. By evaluating four key performance indicators: AUPR (Area Under the Precision-Recall Curve), accuracy (ACC), F-measure (F1) and average time per iteration (PAT), the classification performance, accuracy, balance and training efficiency of the model can be comprehensively reflected. In order to ensure the fairness and comparability of the experimental results, unified experimental parameters are set for all algorithms involved in the comparison, including the number of clients (num_clients=20), the number of global training rounds (global_rounds=500) and the local learning rate (local_learning_rate=0.005), which helps to accurately evaluate the performance of each algorithm under the same conditions. The experimental results are recorded in detail in Table 3.

[0095] Table 3 AUPR, ACC, F1 and PAT of 12 algorithms under different data heterogeneity conditions

[0096]

[0097]

[0098] From the comparative experimental results in Table 3, it can be seen that in the heterogeneous data scenario simulated by Dirichlet distribution, HFedCWA shows its advantages in many aspects. On MNIST, HFedCWA has an ACC of 88.77% and an F1 of 71.58%, which is higher than most other algorithms. In particular, compared with FedAvg, FML, MOON and FedBN, HFedCWA has significant improvements in ACC and F1. On CIFAR10, HFedCWA has an ACC of 86.89% and an F1 of 69.5%, which also shows high performance, especially when compared with algorithms such as FedAvg, FedCPD and FEDACS. This not only proves its model generalization ability in heterogeneous data environments, but also highlights its effectiveness under different data distributions. On MNIST and CIFAR10, HFedCWA has an AUPR of 94.58% and 92.61% respectively, the highest among all algorithms, indicating that HFedCWA has extremely high recognition ability when processing data; HFedCWA has a PAT of 2.35s and 2.41s respectively, which shows that HFedCWA can effectively control the risk of privacy leakage in the iteration process while maintaining high efficiency. Considering that the communication overhead will affect the scalability of the algorithm as the number of clients increases. In order to meet this challenge, combined with the real-time and dynamic nature of the clients participating in the training, a group periodicity strategy is adopted to allow the client to perform more local training before synchronizing with the central server, thereby reducing the number of communications between the central server and the client, reducing the overall communication cost of the training model, and optimizing the communication efficiency. Considering AUPR, ACC, F1 and PAT comprehensively, HFedCWA not only performs well in model performance, but also has obvious advantages in security and efficiency, which fully proves its feasibility, effectiveness and pertinence in application scenarios that require processing heterogeneous data and protecting user privacy. These results not only provide new research directions and optimization strategies for the field of federated learning, but also have important significance for guiding future research directions and algorithm optimization.

[0099] HFedCWA is not only innovative in technology, but also shows wide applicability and high efficiency in practical applications, especially in protecting privacy, improving model generalization ability, and improving prediction accuracy. Significant progress has been achieved in the field of federated learning through several key steps. First, the data contribution is evaluated by measuring the data distribution difference and data volume, and weights are assigned to heterogeneous devices accordingly. This approach makes model updates more effective because it performs weighted aggregation according to the contribution of each device in the overall learning process. Second, a personalized optimization strategy based on regularization is adopted, allowing devices to train customized models according to specific needs, thereby improving the generalization ability of the model. In addition, differential privacy technology is integrated to ensure collaborative training without leaking privacy information, ensuring data security during the learning process. In a heterogeneous data environment, HFedCWA outperforms other methods in AUPR, ACC, and F1, which proves its advantages in model generalization and precision-recall balance. At the same time, PAT is effectively controlled while maintaining efficient performance, significantly improving iteration efficiency. Comparative experiments on MNIST and CIFAR10 further verified the feasibility and effectiveness of HFedCWA, provided a new research perspective and optimization strategy for the field of federated learning, and had important guiding significance for promoting future algorithm optimization.

[0100] HFedCWA has practical value in processing heterogeneous data, protecting privacy, improving model generalization ability, and improving prediction accuracy, making it show wide application potential in multiple fields. In the field of healthcare, it can promote data sharing and model training between different medical institutions while protecting patient privacy, so as to improve the accuracy of disease diagnosis and the formulation of personalized treatment plans, and realize cross-institutional collaborative learning.

[0101] In the field of federated learning, the construction of heterogeneous models is an important and complex task. In order to improve the performance of the model at the global and personalized levels, the present invention proposes HFedCWA, which combines the weighted aggregation method based on contribution, the personalized optimization strategy based on regularization, and the differential privacy technology. It can not only reflect the unique contribution of each device to the overall learning process, but also enhance the adaptability of the model to data heterogeneity, more accurately and efficiently identify and utilize the data characteristics of each device, significantly improve the security, efficiency and comprehensive performance of the model, and show great potential in the field of heterogeneous federated learning. In future work, it can be combined with technologies such as model compression and communication efficiency optimization, and the algorithm performance can be further improved by continuously improving and optimizing. HFedCWA will provide strong support for building more intelligent, secure and efficient personalized models.

Claims

1. A heterogeneous federated learning method based on contribution weighted aggregation, characterized by: The contribution allocation weights are obtained by evaluating the distribution differences and quantity of data provided by heterogeneous devices. Contribution-based weighted aggregation, regularization-based personalization methods, and differential privacy protection technology are adopted. This is implemented through a data contribution evaluation module, a regularization optimization module, a differential privacy module, and a weighted aggregation module.

2. The heterogeneous federated learning method based on contribution weighted aggregation according to claim 1 is characterized in that: The data contribution evaluation module is used to measure data contribution, wherein data distribution difference and data volume are selected as measurement indicators, and a data contribution evaluation model is constructed by calculating the KL divergence value between the indicators.

3. The heterogeneous federated learning method based on contribution weighted aggregation according to claim 2 is characterized in that: The data contribution evaluation module can quantify the contribution of each device data by calculating the KL divergence value, and use the quantification result for subsequent model training and weight allocation. Specifically, In the federated learning system, it is assumed that there are a total of K clients, each of which performs stochastic gradient descent SGD optimization locally and performs server synchronization after each T SGD iteration. represents the weight of client k after the SGD update in round t, It represents the global model after the nth aggregation, assuming a probability distribution T as the target distribution, and represents the weight set when the two indicators are indifferent, that is, there are factors affecting data distribution differences. and data volume impact factor The data contribution evaluation model is C k , as shown in formula (1): C k =ρ u ·u k +r v ·v k (1) C-based k Calculate the model aggregation weight for client k The server collects all locally trained models from the client and aggregates them, and updates the global model according to their corresponding weights. The global model of the nth round is shown in formula (2): The next time the model is aggregated, the server will send a The client will use Initialize the local model and perform SGD iterations locally.

4. The heterogeneous federated learning method based on contribution weighted aggregation according to claim 3 is characterized in that: The regularization optimization module is used to formulate a regularization-based optimization strategy according to data contribution and train a personalized model to ensure the fairness and consistency of weight distribution so that each device can participate in the overall task in the best way.

5. The heterogeneous federated learning method based on contribution weighted aggregation according to claim 4 is characterized in that: The regularization optimization module can adjust the regularization parameters according to the data contribution of each device, as follows: The regularization formulas are shown in (3) and (4): L(ω)=MSE(ω)+λR(ω) (3) Where ω represents the parameters that need to be trained in the network, L(ω) is the loss function, MSE is the mean square error, R(ω) is a function of ω, called the regularization term, λ∈[0,∞) is a constant that controls the degree of regularization and is used to measure the proportion of the regularization term and the loss function. When λ=0, there is no regularization. The larger λ is, the more important the regularization term is. The training goal is to minimize the loss value. There is an intermediate value in the process of λ increasing from 0, so that the training can obtain a set of suitable ω values ​​so that both MSE and λR(ω) are small enough. Choosing different forms of R(ω) will have different solutions. In the personalized optimization strategy based on regularization, L2 regularization is selected and the regularization coefficient is adjusted. By limiting the Euclidean norm of ω, many coefficients of ω can be made smaller, but not zero, thereby obtaining a smaller variance. Each eigenvalue has some contribution to the predicted value, and finally a dense solution can be obtained. The loss function J2 is shown in (5): Train the model independently on each device while applying personalized regularization strategies.

6. The heterogeneous federated learning method based on contribution weighted aggregation according to claim 5 is characterized in that: The differential privacy module can adopt the differential privacy technology of Gaussian mechanism in the federated learning training process by introducing disturbance noise that satisfies Gaussian distribution into the data results.

7. The heterogeneous federated learning method based on contribution weighted aggregation according to claim 6 is characterized in that: The differential privacy module can protect data privacy by introducing Gaussian noise without sacrificing model performance, as follows: Assume that there is a finite data space x of size |x|, r represents a record with d attributes, and the data set D is an unordered set of n records sampled from the finite data space x. If two data sets D and D' have only one record r different, then the two data sets are called adjacent data sets. M is a random algorithm that accesses the database and implements certain functions. Algorithm M is used to implement differential privacy. In order to enhance data privacy protection and reduce the risk of overfitting during model training, for numerical data, according to the characteristics of the data and the privacy protection requirements, the Gaussian mechanism is used to introduce random dynamic disturbance noise into the data results. The Gaussian mechanism provides a relaxed (ε, δ)-differential privacy, defined as (6): Pr[M(D)∈S]≤e ε (6) Among them, Pr(M(D)∈S) represents the probability that the output of the random algorithm M belongs to the set S; the parameter ε represents the privacy budget, which represents the degree of privacy protection achieved by the differential privacy technology. The smaller the ε value, the higher the degree of privacy protection, but the worse the data availability. Conversely, the larger the ε value, the lower the degree of privacy protection and the better the data availability. The control algorithm M implements the privacy guarantee level. Differential privacy is actually a trade-off between the degree of privacy protection and data availability. δ represents the probability of allowing violation of strict differential privacy.

8. The heterogeneous federated learning method based on contribution weighted aggregation according to claim 7, characterized in that: The weighted aggregation module can dynamically adjust the aggregation weight according to the contribution of the device, and adopts a contribution-based weighted aggregation method to adjust the aggregation weight, so that the device with a higher contribution to the overall learning process has more influence in the model update.

Images