Weighted aggregation federated learning method and system based on cosine parameter packet

Abstract

The invention discloses a weighted aggregation federal learning method and system based on a cosine parameter packet, a parameter server distributes a global model to a client participating in training, and the client obtains the latest global model and completes local fusion; performing model training on each client, flattening the model and calculating a total threshold value, ensuring that each client maintains a unique local model according to local data distribution, and effectively utilizing limited system resources; each client divides model parameters into packets based on a similarity relationship between a global model and a local model, and performs selection and sharing of sparse similarity parameter packets, thereby reducing communication overhead for training and transmitting the parameter packets, and helping resource-limited equipment to overcome computing and communication bottlenecks; a server completes alignment and dual-weighted aggregation of sparse parameter packets by using a mask position shared by a client, the ability of a model to quickly absorb new knowledge is enhanced, the aggregation speed of training is accelerated, and the influence of heterogeneity of the client on the training progress is reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of federated learning, and mainly relates to a weighted aggregation federated learning method and system based on a cosine parameter bag. BACKGROUND

[0002] Federated learning is concerned because it can utilize the massive data of clients, and at the same time, it protects privacy by sharing model parameters among clients without exchanging raw data for collaborative training of the model. In federated learning, the data generated and collected by edge client devices located in different geographical locations is a large amount of non-independent and identically distributed data, and the computation, storage, bandwidth and other resources are also different, which leads to the heterogeneity of clients in data and system resources. In this case, a single global model is difficult to generalize all local data sets of clients, and the traditional method using a single global model often leads to significant deviation of the local data distribution of a single client, resulting in slow convergence speed and poor inference performance, and even unable to be deployed in a resource-limited environment. In order to solve the problem of heterogeneity in federated learning, some researchers have proposed personalized federated learning, which customizes a local model for each client according to the local data distribution, system resources and model structure requirements of each client, in order to enhance the ability of the model to solve the problem of heterogeneity.

[0003] Existing methods show that the intrinsic performance degradation of the model is caused by client drift due to data heterogeneity. When there is a significant aggregation difference between local model updates and global model updates, the aggregated global model cannot fully adapt to the heterogeneous data distribution among clients. In order to reduce the client drift, Karimireddy et al. [P. Karimireddy, S. Kale, M. Mohri, et. al. “SCAFFOLD: stochastic controlled averaging for federated learning”. In Proceedings of the 37th International Conference on Machine Learning, 2020. pp 5132-5143.] use the difference between the last round of global update and the current round of local update to compensate for the current round of local update. Li et al. [T. Li, A. K. Sahu, M. Zaheer, et. al. “Federated Optimization in Heterogeneous Networks”. Proceedings of the Third Conference on Machine Learning and Systems, MLSys 2020.], Acar et al. [U. Acar, Y. Zhao, R. M. Navarro, et. al. “Federated learning based on dynamic regularization”. In 9th International Conference on Learning Representations, ICLR 2021, 2021.] and Zhang et al. [J. Zhang, Zq. Li, B. Li, et. al. “Federated learning with label distribution skew via logits calibration. In International Conference on Machine Learning, ICML, 2022, pages 26311-26329] and other researchers add a regularization term in the local loss function to promote the local update to approach the global update. In addition, there are also some model aggregation strategies and knowledge distillation techniques. Although these methods have made great achievements, they ignore the consideration of limited client resources.

[0004] Clients in federated learning are usually composed of physical devices with different processors, memories, and bandwidths, which results in resource differences among devices (system resource heterogeneity). Since federated learning relies on frequent model interaction, resource-constrained clients often struggle to keep pace with the collaborative training pace of complex global models. This can lead to communication bottlenecks, computational delays, and client participation imbalance, i.e., the laggard effect. Existing methods alleviate resource constraints and improve the interaction efficiency and overall performance of both ends of the model by sharing sparse model parameters, optimizing model architecture, and model splitting. However, these works only consider the inconsistency of limited system resources and lack consideration of client data heterogeneity. In local joint collaborative training with heterogeneous data and limited resources, local client models only share part of the entire model, which reduces communication costs and computational overhead, but limits the performance bottleneck of the model in a heterogeneous data environment. Thus, in reality, the heterogeneity of data needs to be considered, and client resources need to be limited in all aspects, not just one. SUMMARY

[0005] The present application is exactly aimed at the problems existing in the prior art, and provides a cosine parameter package-based weighted aggregation federated learning method and system. A parameter server distributes a global model to clients participating in training, and the clients obtain the latest global model and complete local fusion. Each client performs model training, flattens the model, and calculates a total threshold, ensuring that each client maintains a unique local model according to its local data distribution while effectively utilizing limited system resources. Each client divides model parameters into packages based on the similarity relationship between the global model and the local model, and performs sparse similarity parameter package selection and sharing, reducing the communication overhead of training and transmitting parameter packages and helping resource-constrained devices overcome computational and communication bottlenecks. The server uses the mask positions shared by the clients to complete sparse parameter package alignment and double-weighted aggregation, enhancing the model's ability to quickly absorb new knowledge, accelerating the condensation speed of training, and reducing the impact of client heterogeneity on training progress. Continuous training is performed until the model converges, and the optimal model is input with the heterogeneous data to be processed to obtain the final prediction result.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is: a cosine parameter package-based weighted aggregation federated learning method, comprising the following steps:

[0007] S1, federated preparation: a parameter server distributes a global model to clients participating in training, and the clients obtain the latest global model and complete local fusion. The number of clients participating in training is less than the total number of clients in the current client set;

[0008] S2, client training: each client performs model training, flattens the model and calculates the total threshold, packages the model and calculates the package threshold, selects the shared parameter package, and simultaneously generates a mask matrix and completes the double weight assignment of the mask value;

[0009] S3, server-side aggregation: for the sparse parameter package and the mask matrix shared by the client, the server forms a new global matrix according to the mask matrix of the client, completes the effective index and double weight aggregation of the client parameter package, and then forms a new global model;

[0010] S4, iteration: iterate steps S1 to S3 until the method converges or reaches the maximum iteration period, obtain the final weighted aggregation federated learning model based on the cosine parameter package, and input the heterogeneous data to be processed on the model to obtain the final prediction result.

[0011] As an improvement of the present application, the client i performs a flattening operation on the global model and the local model , and calculates the overall similarity threshold between the current local model and the global model of the last round of federated training using a cosine similarity measure function :

[0012] ;

[0013] wherein represents the flattened local model, and represents the flattened global model.

[0014] As another improvement of the present application, in step S2, the client divides the parameter package according to the size of the flattened global model and the local model, and calculates the similarity threshold between each global model parameter package and the local model parameter package using a cosine similarity measure function and KL divergence and the parameter distance :

[0015] ;

[0016] ;

[0017] wherein represents the jth parameter package of the client i, represents the jth parameter package of the global model, and represents the number of parameter packages.

[0018] As another improvement of the present application, in step S2, the specific method for selecting the shared parameter package is that the client i generates a mask matrix , and selects the parameter package smaller than the overall threshold K parameter package selection:

[0019] ;

[0020] The client i performs the following assignment operation on the corresponding mask position:

[0021] .

[0022] As another improvement of the application, the step S3 specifically comprises the following steps:

[0023] S31, after receiving the sparse parameter package shared by the client and the mask matrix, the parameter server accumulates and sums the client mask matrices of the same size according to the index position, and assigns the final mask matrix result to the global mask matrix; then, the server flattens the multi-dimensional global model into a one-dimensional global model, and packs the one-dimensional global model into a plurality of parameter package subsets of the same size according to the fixed size;

[0024] S32, under the new global mask, the server forms an aggregated weight according to the client mask and the global mask , and simultaneously aggregates the parameter package of the global model and the parameter package uploaded by the client one by one according to the weighting weight, to obtain a new global model, and the aggregation process task is shown in the following formula:

[0025] .

[0026] In order to achieve the above purpose, the technical scheme adopted by the application is: a cosine parameter package based weighted aggregation federated learning system, which is divided into a client and a server:

[0027] The number of clients is not unique, and each client is provided with a client training module, in which the latest global model is obtained and local fusion is completed, the global model and the local model are flattened, and the total threshold value for selecting the model parameter package is calculated, each client packs the flattened two models, calculates the package threshold value of each parameter package, selects the parameter package to be shared according to the total threshold value and the package threshold value, generates a mask matrix, and completes double weight assignment of the mask value;

[0028] The server end is provided with a double weight aggregation module, based on the sparse parameter package and the mask matrix shared by the client, a new global matrix is formed according to the mask matrix of the client, and the effective index and double weight aggregation of the client parameter package are completed under the guidance of the matrix, to form a new global model;

[0029] The server and the clients obtain a target prediction model through joint training under a preset loop condition, and realize target prediction

[0030] To achieve the above object, the technical scheme adopted by the present application is: an electronic device, comprising:

[0031] at least one processor, and a memory coupled to the at least one processor;

[0032] The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method of any one of claims 1-5.

[0033] To achieve the above object, the technical scheme adopted by the present application is: a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed, the method of any one of claims 1-5 can be implemented.

[0034] Compared with the prior art, the present application has the beneficial effects:

[0035] (1) The client in the system of the present application is allowed to generate individualized masks according to the respective local models, and the cosine similarity is used to calculate the shared partial parameter package, effectively alleviating the demand of the client for bandwidth resources; the double weight value of the parameter package is calculated using the KL divergence and the cosine similarity, which is used as the position mark of the effective parameter package, so as to reduce the influence of data heterogeneity on the accuracy of the model.

[0036] (2) The system disclosed in the present application can be used in a horizontal federated learning scene with model isomorphism and data distribution heterogeneity, and through cosine sparse parameter packaging and server-side double weight aggregation, a balance between limited device resources and data heterogeneity is achieved. The resource-limited client uses the cosine similarity to calculate the local parameter package that needs to be shared, effectively reducing the communication overhead and the calculation time, and ensuring the feasibility in practical application. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The workflow diagram of the federated learning method based on the cosine parameter package weight aggregation of the present application;

[0038] Figure 2 The framework principle schematic diagram of the federated learning method based on the cosine parameter package weight aggregation of the present application;

[0039] Figure 3 The data distribution schematic diagram in the test example of the present application;

[0040] Figure 4A performance comparison diagram of the cosine parameter package-based weighted aggregation federated learning on the client in the test example of the present application;

[0041] Figure 5 A diagram showing the influence of the number of limited resource clients on model performance in the test example of the present application. DETAILED DESCRIPTION

[0042] The present application will be further illustrated below in conjunction with the accompanying drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0043] Example 1

[0044] In order to reduce the common influence of heterogeneous data and limited system resources on federated learning, the present application provides a cosine parameter package-based weighted aggregation federated learning method, as shown in Figure 1 The method comprises the following steps:

[0045] Step S1, federated preparation: the parameter server distributes the global model to a plurality of clients participating in training, and the clients obtain the latest global model and complete local fusion.

[0046] A plurality of clients are randomly selected from the local client set to participate in the current round of federated learning using the randomly selected plurality of clients, and the global model is distributed to the selected clients, and the number of randomly selected clients is less than the total number of clients in the local client set , wherein represents the selected client, and N represents the number of all clients.

[0047] Step S2, client training: in federated learning, frequent model interaction between clients and servers poses a major challenge to resource-constrained client devices, because frequent interaction of a large number of model parameters not only reduces interaction efficiency but also places a heavy communication burden on resource-constrained clients, so in this step, the model is flattened and the total threshold is calculated, the model is packaged and the package threshold is calculated, the shared parameter package is selected, and the mask matrix is generated and the double weight assignment of the mask value is completed, so as to avoid the interference of the heterogeneity of the remaining clients and maximize the utility of each round of communication.

[0048] Step S21: the client i performs flattening operation on the global model and the local model , and calculates the overall similarity threshold of the current local model and the global model of the last round of federated training using the cosine similarity measure function , the formula is as follows:

[0049]

[0050] wherein Represents a flattened local model. Represents a flattened global model;

[0051] Step S22: The client divides the flattened global model and local model into parameter packs according to their size (PACK), and calculates the similarity threshold between each global model parameter pack and local model parameter pack using the cosine similarity metric and KL divergence. Distance from parameters :

[0052]

[0053]

[0054] in This represents the j-th parameter packet of client i. This represents the j-th parameter pack of the global model. This represents the number of parameter packets;

[0055] Step S23: Client i generates a mask matrix. It will be less than the overall threshold. The K parameter packages are selected, and the selection mechanism is as follows:

[0056]

[0057] Step S24: Client i assigns values ​​to the corresponding mask positions as follows:

[0058]

[0059] Cosine similarity enhances the generalization ability of a model to some extent, but it only captures directional consistency while ignoring differences in update magnitude. This insensitivity to magnitude may lead to... Overweighting can lead to biases in the model aggregation process, thus requiring a next step to better utilize information-rich local updates and improve the adaptability of the global model.

[0060] Step S3, Server-side aggregation: For the sparse parameter packets and mask matrix shared by the clients, the server forms a new global matrix based on the client's mask matrix. Under the guidance of this matrix, the server completes the effective indexing and dual-weight aggregation of the client's parameter packets, thereby forming a new global model.

[0061] Step S31: After receiving the parameter package and mask shared by the client, the parameter server updates the global mask, then flattens the global model and completes the parameter packaging;

[0062] Step S32: Perform the double-weighted aggregation task shown in the following formula under the guidance of the new global mask, and then obtain a new global model:

[0063]

[0064] Step S4, iterative calculation: iterate steps S1 to S3 until the method converges or reaches the maximum iteration period, forming a final cosine parameter package-based weighted aggregation federated learning method for processing client data heterogeneity.

[0065] Figure 2 The framework principle diagram of the cosine parameter package-based weighted aggregation federated learning method of the application, the method of the application is based on the FedCSPACK framework, under the condition of keeping general, the server completes the aggregation process of the global model and the global mask , and then broadcasts them to all clients. Under the guidance of the global mask , the client i fuses the global model into the local model based on the local data . Before sharing the local model, the client i converts , into a one-dimensional vector , , and uses cosine similarity to calculate the overall similarity between the two models . Then, according to the size of the PACK, pack , into parameter packages , , and calculate the similarity threshold and the discrete value of each , . At the same time, use Top-k to select K , that are less than as the parameter package shared by the client i. Finally, in order to effectively align the of the client i, let the effective weight value of the global mask .

[0066] The server collects the local model and the local mask shared by the client i, aggregates to form a new global mask , and completes the alignment and weighted aggregation task of the new global model according to its effective weight value.

[0067] The above steps are repeated until a specified training period is reached or the local model of the client converges. Finally, the personalized local model of each client will be used for its inference training process.

[0068] Test Example

[0069] In order to further verify the effectiveness of the cosine parameter package-based weighted aggregation federated learning method and system proposed in the application, the method of the application is compared with the classic federated learning methods FedAVG, FedProx, FedNova, FedALA, MOON, FedDBE, FedAS, FedNTD and FedPAC on a real data set.

[0070] FedAvg[B.McMahan, E.Moore, D.Ramage, S.Hampson, et.al,“Communication-efficient learning of deep networks from decentralized data,”in Artificial intelligence and statistics.PMLR,2017,pp.1273–1282] is the most popular federated learning solution at present, its purpose is to learn a single global model collaboratively with clients, and the model only needs to collect the shared model of the client, without storing the original data of the client in the central server, and reducing the communication cost through local iterative update. Although this learning method breaks the legal and regulatory restrictions on data exchange and data sharing, the data heterogeneity caused by data silos and system resource limitations seriously affect the model performance and communication efficiency. FedProx[B.Luo, P. Han, P. Sun, et.al, "Optimization Design for Federated Learning in Heterogeneous 6G Networks," in IEEE Network, 2024, vol. 37, no. 2, pp. 38-43] by introducing L2 to the local objective function, to constrain the deviation between the local model and the global model, but the size of L2 will affect the degree of model personalization and convergence speed. FedDyn[C. Jin, X. Chen, Y. Gu, Q. Li. “FedDyn: Adynamic and efficient federated distillation approach on Recommender System”.2022 IEEE 28th International Conference on Parallel and Distributed Systems(ICPADS), 786-793.] is a dynamic regularization federated learning method, in each round of model training, the objective function of each local device is dynamically updated to ensure that the optimal value direction of the local objective function and the global objective function is consistent.SCAFFOLD[P. Karimireddy, S. Kale, M. Mohri, et.al. “SCAFFOLD: stochastic controlled averaging for federated learning”. In Proceedings of the 37thInternational Conference on Machine Learning, 2020. pp 5132–5143.] corrects the optimization direction of the client model through variance reduction techniques to improve the convergence of the global model, but the additional parameters increase the communication overhead. FedNova[J. Wang, Q. Liu, H. Liang, et.al, “Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization,” Neural Information Processing Systems, Neural Information Processing Systems, 2020] normalizes the global model according to the local steps of each party. FedProto[Tan, Y., Long, G., LIU, L., Zhou, T., Lu, Q., Jiang, J., & Zhang, C. (2022). FedProto: Federated Prototype Learning across Heterogeneous Clients. Proceedings of the AAAI Conference on Artificial Intelligence,36(8), 8432-8440.] regulates local model training through abstract class prototypes of clients and servers, while making the generated local prototypes similar enough to the global prototype, but consumes too much time in the training process.FedALA [J. Zhang, Y. Hua, H. Wang, et.al, “FedALA:Adaptive Local Aggregation for Personalized Federated Learning,” Proceedings of the AAAI Conference on Artificial Intelligence, 2023,37(9), 11237-11244] fuses the global model of each round with the local model of the last round to preserve local personalized information. MOON [Qb Li,Bs He, D Song, “Model-Contrastive Federated Learning,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021,pp.10713-10722] corrects the optimization direction of the user model by comparing the loss of the global model in two rounds of training. FedGH [Yi, L., Wang, G., Liu, X., Shi, Z., & Yu, H. FedGH: Heterogeneous Federated Learning with Generalized Global Header.Proceedings of the 31st ACM International Conference on Multimedia,2023.] uploads the calculated local average representation and class label to the server, and the server trains the global prediction head and broadcasts it to the client to replace the local prediction head. FedDBE [Jq Zhang, H Yang, J Cao, et.al, “Eliminating Domain Bias for Federated Learning in Representation Space,” in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, 2023] reduces the domain difference between the server and the client in the representation space by promoting bidirectional knowledge transfer between them.FedAS [Xy Yang, Wk Huang, M Ye. “FedAS: Bridging Inconsistency in Personalized Federated Learning,” in IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp: 11986-11995] enhances the positioning of global parameters by combining them with local insights. However, these solutions do not deeply consider the division between model parameters, which can easily lead to a bottleneck in model performance. FedNTD [Gh Lee, M Jeong, Yj Shin, et. al, “Preservation of the Global Knowledge by Not-True Distillation in Federated Learning,” in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022] uses global and historical models as teachers to facilitate the distillation of comprehensive knowledge and complete the transfer of global general knowledge and historical personalized knowledge to local models, thereby alleviating catastrophic forgetting. FedPAC [J Xu, Xy Tong, S L Huang, “Personalized Federated Learning with Feature Alignment and Classifier Collaboration,” in The Eleventh International Conference on Learning Representations, 2023] improves the overall accuracy and robustness of the model through feature alignment and classifier collaboration, but for resource-constrained clients, implementing and optimizing this model can be challenging. Although these methods effectively address multiple heterogeneous problems in federated learning, they all consider heterogeneous problems from a single perspective, lacking overall consideration of data heterogeneity and limited system resources.

[0071] Four public available datasets, i.e. Fashion-MNIST (FMNIST), EMNIST, CIFAR-10 and CIFAR-100, are used as experimental data in the present test example. The FMNIST consists of 60,000 training data and 10,000 test data. The CIFAR10 and CIFAR100 use 50,000 and 10,000 images as training data and test samples, respectively. The EMNIST uses 700,000 data as the training set and 110,000 data as the test set. In order to fully verify the performance of the method FedCSPACK, the Dirichlet distribution method is used to divide the data distribution of the client data in a non-independent and identically distributed manner. As shown in FIG. 5, the discrete distribution of the data owned by the five clients is shown. Figure 3 Figure 3 The data categories and data quantities owned by the clients are shown from two aspects of the clients and the data categories. In the 10 data categories, Figure 3 a shows the data categories and total quantities owned by each client, and different colors of the clients represent the data quantities of each data category owned by the clients; Figure 3 b shows how many clients each category has, and the color in each category represents the data quantity of the category owned by the clients.

[0072] The experimental evaluation of all schemes is carried out in three heterogeneous data environments, and the overall performance of the global model is as follows Table 1. Table 1 shows the test accuracy (%) of each model when the Dirichlet distribution parameters are 0.3, 0.6 and 1.0 distribution on four data sets:

[0073] Table 1

[0074]

[0075] ​As can be seen from Table 1 above, on FMNIST, as the data distribution changes from Dir(0.3) to Dir(1.0), the accuracy of the model generally improves. The method FedCSPACK of the present application performs well under all data distributions, especially when Dir(1.0), it achieves an accuracy of 90.73%, which is significantly higher than other methods. For more complex CIFAR-10 and CIFAR-100, the method FedCSPACK of the present application still performs well. On CIFAR-10 and CIFAR-100, the method FedCSPACK of the present application achieves 78.71% and 43.20% respectively when Dir(1.0). On the large data set EMNIST, the accuracy of the model is generally high. This may be because the increase in data volume helps the model to better learn the features. However, when the data distribution changes from Dir(1.0) to Dir(0.3), the accuracy of the model decreases. This may be due to the homogenization of the data distribution, which leads to a decrease in the number of samples of certain classes, thereby affecting the generalization ability of the model of other schemes. However, the method FedCSPACK of the present application can still ensure the high accuracy of the model through the double weight packet aggregation method, especially when Dir(0.6) achieves 86.26%.

[0076] Table 2

[0077]

[0078] Table 2 is the test accuracy (%) of the data pieces of each method on the four data sets. As can be seen from Table 2, the method FedCSPACK of the present application is superior to all the comparison methods in all evaluation data sets. It achieves the highest performance index in the four data sets, and has a great advantage compared with the competitors. On FMNIST, FedCSPACK reaches 70.28%, which is about 2.2% higher than the second-ranked FedNova of 68.06%. On EMNIST, it scores 49.00%, which is about 2.3% higher than the second-ranked FedAvg and FedALA methods of 46.67%. Even in the CIFAR-10 subset with generally lower performance, the method FedCSPACK of the present application still maintains a clear leading advantage of 13.08%, which is about 2.4% higher than MOON of 10.71%. Although other methods show serious performance degradation on specific data sets, for example, FedAS scores only 9.55% on EMNIST, the method FedCSPACK of the present application always provides leading results in different data sets. Compared with traditional methods such as FedAvg and FedProx, FedAvg and FedProx show stagnant performance in data sets, for example, 68.03% on FMNIST, but the method FedCSPACK of the present application achieves a systematic breakthrough, and its advantage is particularly obvious on more complex data sets such as CIFAR-10 and EMNIST. In summary, through double-weight aggregation, the method FedCSPACK of the present application maintains high model accuracy and shows effective adaptability to data sets from simple to complex and from small scale to large scale.

[0079] Figure 4 is a schematic diagram of the generalization ability of the global model on the client. Figure 4 It is shown that the average accuracy maintained by the method FedCSPACK of the present application on all clients is 68.63%, 67.2%, 58.63% and 21.31% respectively, which shows that the cosine top-k parameter packing technology can effectively enhance the knowledge absorption ability of the client and improve the generalization ability of the global model. In summary, FedCSPACK has a significant advantage in global-to-local generalization under Dirichlet and sharded partitioning. Its benefit is particularly obvious in the highly heterogeneous sharded scenario, proving that sparse parameter packing and double-weight aggregation effectively train personalized models while maintaining global adaptability to data heterogeneity.

[0080] To quantitatively evaluate the computational efficiency improvement and bandwidth resource reduction of FedCSPACK for limited resource clients, Table 3 provides specific computational efficiency and bandwidth requirement results, which are the comparison of total training time (hours) and transmission parameter size (GB) for T = 100. Traditional: FedProx and FedNova. Split: MOON, FedDBE and FedAS. Distillation: FedNTD and FedPAC. Table 3 shows the improved computational performance and optimal resource utilization.

[0081] Table 3

[0082]

[0083] As can be seen in the above Table 3, in FMNIST, the communication traffic and time of FedCSPACK are the lowest, which are 0.13 GB and 0.65 hours, respectively. In CIFAR-10 and CIFAR-100, the computational time of FedCSPACK is 0.68 hours and 0.39 hours, respectively, which indicates that FedCSPACK can improve the computational efficiency of limited resource clients when dealing with complex data sets. In the large data set EMNIST, the communication traffic of FedCSPACK is 6.93 GB, which is about 3 times lower than FedAvg, Tradition, Split, Distillation and FedALA. In terms of computational time consumption, the time consumption of FedCSPACK is 2.80H, and the time consumption of other schemes is 11.16H, 7.74H, 5.63H, 8.72H and 8.41H, respectively, which is the least among all schemes. These results show that FedCSPACK can effectively control the computational time and communication traffic by sharing some parameter packs, which is particularly important for resource-limited devices to deal with large-scale data. In summary, the ability of FedCSPACK to provide efficient and balanced performance when dealing with simple, complex and large data sets shows its potential and advantage in practical applications.

[0084] Considering that different clients have different data heterogeneity distributions, which in turn affect the model performance and model generalization ability, random client experiments were conducted to verify the generalization performance of the global model generated by the scheme in the presence of client data heterogeneity. Figure 5 It is illustrated that the global model can still achieve good model performance, model stability and model generalization when dealing with client data heterogeneity under the random participation of limited resource clients.

[0085] In Figure 5In particular, the model performance fluctuates significantly with the decrease of client participation and the increase of dataset heterogeneity. This indicates that data heterogeneity has a more significant impact when fewer clients are involved, which is particularly evident in the complex CIFAR-100. However, the FedCSPACK method still maintains model stability in this case. Under different Dirichlet distributions, FedCSPACK outperforms other methods with an average accuracy of 49%, 87%, and 91%. This advantage is particularly evident under Dir(0.3), highlighting its excellent adaptability to data heterogeneity. This also indicates that the FedCSPACK method maintains high performance in four data distributions, confirming its effectiveness in adapting to different data scenarios. In summary, the FedCSPACK method performs well under different client numbers and data heterogeneity conditions. Its strong adaptability, stability, and efficiency make it a significant advantage in FL, especially when facing uneven data distribution and limited clients.

[0086] Therefore, based on the above experimental data, it can be further shown that the present case can solve the problem of coexistence of resource constraints and data heterogeneity in federated learning. The key insight is to pack the client model parameters, use cosine similarity and Top-k to calculate the shared parameter package, and reduce the demand for limited resources by the client. Through the mask matrix, KL divergence, and double-masked weight, the FedCSPACK method can ensure that the server efficiently aggregates the sparse parameter package of the client model, ensuring the robustness and generalization ability of the global model. A large number of experiments on various data sets show that the FedCSPACK method achieves SOTA performance. Before sharing the local model, the client packs the model and uses cosine similarity to calculate the shared partial parameter package, effectively alleviating the demand for bandwidth resources by the client. In order to ensure that the server can effectively aggregate the sparse local parameter package and enhance the generalization ability of the model, the client generates a mask matrix and assigns a double weight to the matrix to ensure the index position of the model parameter package. Finally, the parameter service completes the efficient aggregation of all client parameter packages according to the effective index of the mask matrix and the weight ratio, thereby improving the robustness and generalization ability of the global model. The present application calculates the similarity relationship between model parameter packages based on the implicit relationship between the global model and the local model, and uses the similarity threshold of the parameter package to complete the selection of the parameter package, relying on the double weight ratio of the mask matrix to complete the effective aggregation of the global model. While ensuring the model performance and generalization ability, it effectively balances data heterogeneity and system heterogeneity.

[0087] Unless specifically stated otherwise, the relative steps, numerical expressions, and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0088] The various embodiments described in the specification are progressive in nature, and each embodiment highlights the differences from other embodiments. The same or similar parts between embodiments can be mutually referred to. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0089] The units and method steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been described in the above description in general terms. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art can use different methods to implement the described functions for each specific application, but such implementation does not exceed the scope of the present application.

[0090] A person of ordinary skill in the art can understand that all or part of the steps in the above method can be instructed by a program to complete the relevant hardware, and the program can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk, etc. Alternatively, all or part of the steps of the above embodiments can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiments can be implemented in the form of hardware or in the form of a software function module. The present application is not limited to any specific form of combination of hardware and software.

Claims

1. A weighted aggregation federated learning method based on cosine parameter packages, characterized in that... It includes the following steps: S1, Federated Preparation: The parameter server distributes the global model to the clients participating in the training. The clients obtain the latest global model and complete local fusion. The number of clients participating in the training is less than the total number of clients in the current client set. S2, Client Training: Each client trains the model, flattens the model and calculates the total threshold, packages the model and calculates the package threshold, selects the shared parameter package, and generates the mask matrix and completes the dual weight assignment of the mask values. S3, Server-side aggregation: For the sparse parameter packets and mask matrix shared by the clients, the server forms a new global matrix based on the client's mask matrix, completes the effective indexing and double-weight aggregation of the client's parameter packets, and then forms a new global model; S4, Iteration: Iterate from step S1 to step S3 until the model converges or reaches the maximum iteration period, obtain the optimal model, and obtain the prediction result.

2. The weighted aggregation federated learning method based on cosine parameter packages as described in claim 1, characterized in that: In step S2, client i accesses the global model. With local model Perform a flattening operation and use the cosine similarity metric function to calculate the overall similarity threshold between the current local model and the global model from the previous round of federated training. : ； in Represents a flattened local model. This represents a flattened global model.

3. The weighted aggregation federated learning method based on cosine parameter packages as described in claim 2, characterized in that: In step S2, the client divides the flattened global model and local model into parameter packages according to their size, and simultaneously calculates the similarity threshold between each global model parameter package and local model parameter package using the cosine similarity metric and KL divergence. Distance from parameters ,: ； ； in This represents the j-th parameter packet of client i. This represents the j-th parameter pack of the global model. This represents the number of parameter packets.

4. The weighted aggregation federated learning method based on cosine parameter packages as described in claim 3, characterized in that: In step S2, the specific method for selecting the shared parameter packet is as follows: Client i generates a mask matrix. It will be less than the overall threshold. Select from the K parameter packages: ； Client i assigns a value to the corresponding mask position in the following manner: 。 5. The weighted aggregation federated learning method based on cosine parameter packages as described in claim 4, characterized in that: Step S3 specifically includes the following steps: S31, after receiving the sparse parameter packet and mask matrix shared by the client, the parameter server sums the client mask matrices of the same size according to the index position, and assigns the final mask matrix result to the global mask matrix. The server flattens the multidimensional global model into a one-dimensional global model, and packages the one-dimensional global model into multiple parameter packages of the same size according to a fixed size. S32, under the new global mask, the server uses the client mask. and global mask The aggregate weights are formed, and the server simultaneously sends the global model's parameter package. Parameter packets uploaded by the client The weighted aggregates are performed one by one to obtain a new global model. The aggregation process is shown in the following equation: 。 6. A weighted aggregation federated learning system based on cosine parameter packages, consisting of a client and a server, characterized by: The number of clients is not unique, and each client has a client training module. In the client training module, the latest global model is obtained and local fusion is completed. The global model and the local model are flattened, and the total threshold for selecting model parameter packages is calculated. Each client packages the two flattened models, calculates the package threshold for each parameter package, selects the parameter packages to be shared based on the total threshold and the package threshold, and generates a mask matrix and completes the dual weight assignment of the mask values. The server-side is equipped with a dual-weight aggregation module, which forms a new global matrix based on the sparse parameter package and mask matrix shared by the client and the mask matrix of the client. Under the guidance of this matrix, the effective indexing and dual-weight aggregation of the client parameter package are completed, thereby forming a new global model. The server and each client obtain a target prediction model through joint training under preset cyclic conditions, thereby achieving target prediction.

7. An electronic device, characterized in that, include: At least one processor, and a memory coupled to said at least one processor; The memory stores a computer program that can be executed by the at least one processor to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, enables the implementation of the method as described in any one of claims 1-5.

Images