Personalized federated learning method and device based on integrated model

Abstract

The application discloses an individualized federated learning method and device based on an integrated model, wherein the method comprises the following steps: each participant constructs a new single model to fit the residual of a global model on a local data set, and uploads a trained single model of the current round to a server; the server performs first weighting on the model according to the data volume, and distributes the integrated model with determined weights of the current round to each participant; each participant performs a model simplification operation on the local data of the current round, and uploads the final selection result in the form of binary coding again; the server determines the weights of the model of the current round again according to the model simplification result, and distributes the global model with the weights to each participant; after reaching a predetermined round, each participant obtains an individualized model; thus, by integrating the integrated model into the federated learning framework and constructing a corresponding model individualization method and a communication cost saving scheme, the prediction performance of the federated learning on structured data in a non-independent and identically distributed manner is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a personalized federated learning method based on an ensemble model and a personalized federated learning device based on an ensemble model. BACKGROUND

[0002] Federated learning (FL) is one of the important technologies of the privacy computing system, and its essence is a machine learning framework. In the medical application scenario, federated learning breaks the data island phenomenon from a technical point of view, on the one hand, ensuring the privacy and security protection of personal patient data of medical institutions, and on the other hand, improving the accuracy of medical data model application analysis.

[0003] In the related art, in view of the data heterogeneity problem in the actual application scenario, the personalized federated learning algorithm has become an important research topic. However, most of the current federated learning frameworks are for parameter transmission and model aggregation of neural networks, and there is still less research on federated model aggregation algorithms for traditional machine learning models and their integration. On the one hand, traditional machine learning models are difficult to integrate at the parameter level; on the other hand, how to realize model personalization through traditional machine learning algorithms has not been thoroughly discussed. SUMMARY

[0004] The present application aims to at least solve one of the above technical problems to some extent. To this end, one object of the present application is to propose a personalized federated learning method based on an ensemble model, which improves the prediction performance of federated learning on structured data with non-independent and identically distributed by integrating an ensemble model into a federated learning framework and constructing a corresponding model personalization method and communication cost saving scheme.

[0005] A second object of the present application is to propose a personalized federated learning device based on an ensemble model.

[0006] To achieve the above object, the first aspect of the present application proposes a personalized federated learning method based on an integrated model, comprising the following steps: sending a current global model to each participant so that each participant establishes a corresponding personalized model according to the global model; obtaining the personalized model corresponding to each participant, and performing first weighting processing on the personalized model corresponding to each participant to obtain an integrated model; sending the integrated model to each participant so that each participant splits the integrated model into multiple personalized models, and performs promotion integration of the multiple personalized models with the global model respectively to obtain multiple single integrated models, and simplifies the multiple single integrated models to obtain a simplified result; obtaining the simplified result corresponding to each participant, and aggregating the simplified result corresponding to each participant to obtain a global model selection result, and performing second weighting processing on the personalized model corresponding to each participant according to the global model selection result to obtain a new integrated model, and adding the new integrated model to the global model to obtain an updated global model; sending the updated global model to each participant so as to perform the next round of model update until the target federated model is obtained after the global round.

[0007] According to the personalized federated learning method based on the integrated model, first, the current global model is sent to each participant so that each participant establishes a corresponding personalized model according to the global model; then, the personalized model corresponding to each participant is obtained, and first weighting processing is performed on the personalized model corresponding to each participant to obtain an integrated model; then, the integrated model is sent to each participant so that each participant splits the integrated model into multiple personalized models, and performs promotion integration of the multiple personalized models with the global model respectively to obtain multiple single integrated models, and simplifies the multiple single integrated models to obtain a simplified result; then, the simplified result corresponding to each participant is obtained, and the simplified result corresponding to each participant is aggregated to obtain a global model selection result, and second weighting processing is performed on the personalized model corresponding to each participant according to the global model selection result to obtain a new integrated model, and the new integrated model is added to the global model to obtain an updated global model; finally, the updated global model is sent to each participant so as to perform the next round of model update until the target federated model is obtained after the global round; thereby, the federated model aggregation of heterogeneous data can be performed under the premise of guaranteeing data privacy, and the personalized model is generated for the participants according to the data distribution to obtain higher performance and lower communication cost.

[0008] In addition, the personalized federated learning method based on the integrated model according to the above-mentioned embodiments of the present application can have the following additional technical features:

[0009] Optionally, before the federated learning method based on the integrated model is performed, the method further comprises: each participant respectively acquires a local data set; server-side initialization, statistics of the data amount of each participant, and determination of the global parameters, wherein the global parameters include the global round, the number of participants, the data amount of each participant, and the base model hyperparameters.

[0010] Optionally, each participant establishes a corresponding personalized model according to the global model, comprising: predicting the local data according to the global model to obtain a prediction result; calculating the residual of the global model and the true label according to the prediction result; and establishing a personalized model according to the residual.

[0011] Optionally, the personalized model corresponding to each participant is subjected to a first weighting process to obtain an integrated model, comprising: subjecting the personalized model to a first weighting process according to the data amount of each participant to obtain a corresponding weight; and combining the personalized model corresponding to each participant according to the corresponding weight to obtain an integrated model.

[0012] Optionally, the plurality of single integrated models are simplified to obtain a simplified result, comprising: testing the plurality of single integrated models, and discarding combinations in which the loss does not decrease, so as to simplify the plurality of single integrated models; and adding the plurality of single integrated models after simplification to the personalized model, so as to update the personalized model.

[0013] Optionally, the participant uploads the respective simplified result in the form of a binary vector to the server side, so that the server side acquires the simplified result corresponding to each participant.

[0014] To achieve the above object, the second aspect of the present application proposes a personalized federated learning device based on an integrated model, comprising: a first sending module, configured to send a current global model to each participant, so that each participant establishes a corresponding personalized model according to the global model; a processing module, configured to obtain the personalized model corresponding to each participant, and perform first weighting processing on the personalized model corresponding to each participant to obtain an integrated model; a second sending module, configured to send the integrated model to each participant, so that each participant splits the integrated model into a plurality of personalized models, and performs promotion integration of the plurality of personalized models with the global model respectively to obtain a plurality of single integrated models, and simplifies the plurality of single integrated models to obtain a simplified result; a model updating module, configured to obtain the simplified result corresponding to each participant, and aggregate the simplified result corresponding to each participant to obtain a global model selection result, and perform second weighting processing on the personalized model corresponding to each participant according to the global model selection result to obtain a new integrated model, and add the new integrated model to the global model to obtain an updated global model; the first sending module sends the updated global model to each participant, so as to perform the next round of model updating until the target federated model is obtained after the global round.

[0015] According to the personalized federated learning device based on the integrated model, the current global model is sent to each participant by the first sending module, so that each participant establishes a corresponding personalized model according to the global model; the processing module obtains the personalized model corresponding to each participant, and performs first weighting processing on the personalized model corresponding to each participant to obtain an integrated model; the second sending module sends the integrated model to each participant, so that each participant splits the integrated model into a plurality of personalized models, and performs promotion integration of the plurality of personalized models with the global model respectively to obtain a plurality of single integrated models, and simplifies the plurality of single integrated models to obtain a simplified result; the model updating module obtains the simplified result corresponding to each participant, and aggregates the simplified result corresponding to each participant to obtain a global model selection result, and performs second weighting processing on the personalized model corresponding to each participant according to the global model selection result to obtain a new integrated model, and adds the new integrated model to the global model to obtain an updated global model; the first sending module sends the updated global model to each participant, so as to perform the next round of model updating until the target federated model is obtained after the global round; thereby, the federated model aggregation of heterogeneous data can be performed under the premise of guaranteeing data privacy, and the personalized model is generated for the participants according to the data distribution to obtain higher performance and lower communication cost.

[0016] In addition, the personalized federated learning device based on the integrated model according to the above-mentioned embodiments of the present application can further have the following additional technical features:

[0017] Optionally, the device further comprises a model establishing module configured to predict local data according to the global model to obtain a prediction result, calculate a residual error between the global model and a true label according to the prediction result, and establish a personalized model according to the residual error.

[0018] Optionally, the processing module is further configured to perform first weighting processing on the personalized models according to the amount of data of each participant to obtain corresponding weights, and combine the personalized models corresponding to each participant according to the corresponding weights to obtain an integrated model.

[0019] Optionally, the device further comprises a simplifying module configured to test the plurality of single integrated models, discard combinations with no loss reduction, and simplify the plurality of single integrated models; and add the simplified plurality of single integrated models to the personalized models to update the personalized models. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 FIG. 1 is a flowchart of a personalized federated learning method based on an integrated model according to an embodiment of the present application;

[0021] Figure 2 FIG. 2 is a flowchart of a personalized federated learning method based on an integrated model according to an embodiment of the present application;

[0022] Figure 3 FIG. 3 is a schematic diagram of the overall architecture of a personalized federated learning method based on an integrated model according to an embodiment of the present application;

[0023] Figure 4 FIG. 4 is a line graph showing the influence of data heterogeneity on model accuracy according to an embodiment of the present application;

[0024] Figure 5 FIG. 5 shows a comparison of model performance according to an embodiment of the present application;

[0025] Figure 6 FIG. 6 is a block diagram of a personalized federated learning device based on an integrated model according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] Embodiments of the present application are described below in the detailed description and illustrated in the accompanying drawings by using examples, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present application, and should not be understood as limiting the present application.

[0027] In the related art, since the present application considers the actual scene of federated learning of big data, each participant represents a medical institution in different regions, and they independently and irrelevantly collect local data. Due to the differences in the characteristics of data in different regions, the local data sampled by each participant has different data distribution. This data heterogeneity problem brings great challenges to the prediction accuracy. Therefore, the present application proposes a personalized federated learning method based on an integrated model. First, the current global model is sent to each participant, so that each participant establishes a corresponding personalized model according to the global model. Then, the personalized model corresponding to each participant is obtained, and the personalized model corresponding to each participant is subjected to first weighting processing to obtain an integrated model. Then, the integrated model is sent to each participant, so that each participant splits the integrated model into multiple personalized models, and respectively integrates the multiple personalized models with the global model to obtain multiple single integrated models, and simplifies the multiple single integrated models to obtain a simplified result. Then, the simplified result corresponding to each participant is obtained, and the simplified result corresponding to each participant is aggregated to obtain a global model selection result, and the personalized model corresponding to each participant is subjected to second weighting processing according to the global model selection result to obtain a new integrated model, and the new integrated model is added to the global model to obtain an updated global model. Finally, the updated global model is sent to each participant, so that the next round of model updating is performed until the target federated model is obtained after the global round. Thus, the federated model aggregation of heterogeneous data can be performed under the premise of ensuring data privacy, and personalized models are generated for participants according to data distribution to obtain higher performance and lower communication cost.

[0028] In order to better understand the above technical solutions, the exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0029] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings in the specification and specific embodiments.

[0030] Figure 1This is a flowchart illustrating the personalized federated learning method based on an ensemble model according to an embodiment of this application, as shown below. Figure 1 As shown, this personalized federated learning method based on an ensemble model includes the following steps:

[0031] S101, Send the current global model to each participant so that each participant can build a corresponding personalized model based on the global model.

[0032] As an example, each participant builds a corresponding personalized model based on the global model, including: predicting local data based on the global model to obtain prediction results; calculating the residual between the global model and the true label based on the prediction results; and building a personalized model based on the residual.

[0033] As a specific example, in the t-th round, the prediction result of the global model on the local data sample i is:

[0034]

[0035] in, The weight of the personalized model built by the k-th participant in the j-th round in the global model; This is the prediction result of the corresponding feature vector X of the personalized model constructed by the k-th participant in the j-th round.

[0036] Participants calculate the residuals between the global model and the true labels based on the predictions of the global model, and then build a new personalized model based on these residuals. The loss of the new personalized model is as follows:

[0037]

[0038] Where Ω is the regularization term and l is the loss function.

[0039] In addition, each participant uploads their personalized model to the server after it is created.

[0040] As an example, such as Figure 2 As shown, before implementing the personalized federated learning method based on the ensemble model, the process also includes: each participant acquiring their own local dataset; server-side initialization, counting the amount of data for each participant, and determining global parameters, which include the global round, the number of participants, the amount of data for each participant, and the hyperparameters of the base model.

[0041] In other words, data preparation is performed before federated learning begins, whereby each participant obtains its own local dataset; then the server initializes, counts the amount of data for each participant, and determines global parameters, which include the global round, the number of participants, the amount of data for each participant, and the hyperparameters of the basic model.

[0042] As an embodiment, each participant determines local data, and the data of different participants has the same feature group and label; the data set of the kth participant is D k = {X k ,y k}, where 1≤k≤m, m is the number of participants; the server end determines global parameters: the global round N, the number of participants m, the data amount of each participant [Size1, Size2, …, Size m ], and the base model hyperparameter ω; the server sends the global parameters to each participant; each participant will locally maintain two ensemble models composed of m×N base models, including a global model with a shared structure and a personalized local model.

[0043] In the present application, the global model shared by each participant and the server end is obtained by aggregating the training results of each participant, and the global model will have good generalization ability; at the same time, each participant also obtains a local personalized model, which has better performance on the local data of the participant than the global model; the personalized model is obtained by model simplification operation of each participant, and the personalized models of different participants have different structures.

[0044] Since it is difficult for traditional machine learning models to directly fuse model parameters, the present application realizes model aggregation by model integration and fitting the global residual error by local models.

[0045] S102, obtaining the personalized model corresponding to each participant, and performing first weighting processing on the personalized model corresponding to each participant to obtain an ensemble model.

[0046] As an embodiment, the first weighting processing on the personalized model corresponding to each participant to obtain an ensemble model includes: performing first weighting processing on the personalized model according to the data amount of each participant to obtain a corresponding weight; and combining the personalized model corresponding to each participant according to the corresponding weight to obtain an ensemble model.

[0047] It should be noted that in actual application scenarios, the data amount owned by different participants has considerable differences, and the data amount of some participants is insufficient to meet the modeling requirements; therefore, the present application performs first weighting on the model according to the data amount owned by each participant; the participant with larger data amount has data closer to the global distribution, and increasing the weight of this part can make the global model have better generalization ability and be more suitable for the global data distribution; the model trained by less data may overfit to part of the data and reduce the performance of the global model, and reducing the weight of them can to some extent avoid this problem.

[0048] As a specific embodiment, the server side performs the first weighting on the personalized model generated in the current round according to the data volume of each participant:

[0049]

[0050] The server side combines the personalized models uploaded by each participant into an integrated model in the current round:

[0051]

[0052] The server side distributes the weighted integrated model in the current round to each participant.

[0053] S103, send the integrated model to each participant, so that each participant splits the integrated model into multiple personalized models, and respectively performs boosting integration of the multiple personalized models and the global model to obtain multiple single integrated models, and simplifies the multiple single integrated models to obtain a simplified result.

[0054] That is, after obtaining the integrated model in the current round distributed by the server side, each participant respectively performs boosting integration of each single model in the bagging integrated model and the global model; the participants respectively test these temporary integrated models, and discard combinations whose loss does not decrease; then, each participant adds the simplified model in the current round to the personalized model, and completes the update of the personalized model in the current round; wherein, the boosting integration means that each single model established in each global round can be regarded as progressive iteration of the model in the last round, and new models are continuously added to the global model to obtain a high-performance integrated model, in addition, this integration method can improve the accuracy of any given learning algorithm, which means that the model algorithm of each participant does not need to be fixed, and a machine learning algorithm more suitable for local data can be selected; the bagging integration means that each participant independently uses different data to construct new models locally, and the new models constructed by different participants have no dependency relationship; the server side performs parallel integration on the models from different participants to aggregate global data information and improve the generalization ability of the model.

[0055] It should be noted that the integrated model after the first weighting has good generalization ability, but due to the difference in the distribution of the data of each participant, each participant still needs to simplify the model on this basis to obtain a personalized model that performs better on local data.

[0056] As an embodiment, the simplification of the multiple single integrated models to obtain the simplified result includes: testing the multiple single integrated models, and discarding combinations whose loss does not decrease, so as to simplify the multiple single integrated models; adding the multiple single integrated models after simplification to the personalized model, so as to update the personalized model.

[0057] As a specific embodiment, each participant splits the received integrated model again into a plurality of single models, and respectively performs boosting integration with the global model, respectively tests the models obtained after the boosting integration, and discards the combinations in which the loss does not decrease; then, each participant adds the simplified model of the current round to the personalized model, and completes the update of the personalized model of the current round.

[0058] The participants upload the respective simplified results in the form of binary vectors to the server side:

[0059]

[0060] Among them,

[0061] In the present application, each participant verifies the model of the current round trained by each participant The performance on the local data is used for model screening, and the single models that do not contribute to the loss reduction are deleted, thereby obtaining a certain performance improvement.

[0062] S104, obtaining the simplified result corresponding to each participant, and aggregating the simplified result corresponding to each participant to obtain a global model selection result, and performing second weighting processing on the personalized model corresponding to each participant according to the global model selection result to obtain a new integrated model, and adding the new integrated model to the global model to obtain an updated global model.

[0063] That is, the present application realizes the iteration and update of the model by continuously performing the integration of the new model and the new model and the integration of the new model and the global model; therefore, in each iteration, the server side does not change the previous global model; in the present application, each participant maintains a global model same as the server side while having a personalized model to save the communication cost required for transmitting the global model, and the global model is synchronized with the server side at the end of each round.

[0064] As an embodiment, the server side aggregates the received model selection results to obtain a global model selection result:

[0065] V=[v1,v2,…,v m ] T

[0066] Among them,

[0067] According to the data amount of each participant and the global model selection result, the server side performs second weighting on each personalized model obtained in the current round:

[0068]

[0069] The server end distributes the weighted completed bagging model to each participant and adds it to the global model, completing the server end's update of the global model in this round.

[0070] It should be noted that in this application, the server will perform a second weighting on the single model in this round according to the personalized results of each participant; the weight obtained from the model test results can further reduce the influence of some over-fitting models; in addition, compared with one-time personalized operation, combining the personalized process with the model aggregation process can obtain a better personalized model.

[0071] S105, the updated global model is sent to each participant for the next round of model update until the target federated model is obtained after the global round.

[0072] That is, each participant adds the received integrated model to the global model to complete the synchronization of the global model in this round; after the end, each participant starts a new round of model update until the predetermined number of rounds is completed.

[0073] The server end of the present application integrates the single models trained by different participants in the same round, and improves the integration of bagging models in different rounds, and on this basis, performs federated learning according to the method in Figure 3 , that is, by integrating the integrated model into the federated learning framework and constructing the corresponding model personalization method and communication cost saving scheme, the prediction performance of federated learning on non-independent and identically distributed structured data is improved; the following will verify the improvement of the prediction performance of the method proposed in this application by designing experiments to simulate real scenarios.

[0074] The experiment uses the public IST dataset (The International Stroke Trial database) as experimental data. The data contains stroke prognosis data from different countries and regions. The model task in the embodiment is to predict the recovery condition of patients at discharge by collecting feature data during the patient's hospitalization. In the embodiment, a decision tree model is used as the base model, Figure 4 The influence of data heterogeneity on model accuracy is given in

[0075] The FedAvg variant algorithm weighted by data volume is used as the baseline model; as shown in Figure 5 , the method provided in this application can obtain a prediction model with higher accuracy, which indicates that the designed personalized scheme and the method of integrating the integrated model are superior to the baseline model.

[0076] In summary, the present application firstly performs data preparation, i.e., each participant prepares a local data set respectively; then initialization, i.e., server-side initialization, statistics of the data amount of each participant, determination of global parameters, and distribution to each participant; then, each participant tests the current global model on the local data, and builds a new single model to fit the residual of the global model on its own data set, and uploads the trained single model of the current round to the server side; the server side performs the first weighting on the received model according to the data amount owned by each participant; the server side distributes the integrated model with the determined weight to each participant; each participant tests the model generated in the current round on the local data; each participant combines each single model generated in the current round with the global model to form a new integrated model; and checks the loss on the local data, if the loss does not decrease, the model is deleted to simplify the integrated model of the current round; then, each participant uploads the final selection result of the model of the current round in the form of binary coding again; the server side determines the weight of each single model in the model of the current round according to the model simplification result of each participant; the server side adds the integrated model of the current round to the global model, and distributes the final weight to each participant; each participant adds the integrated model of the current round to the local global model according to the received weight; and updates the model simplification result on the local data to the local personalized model; then, the current round is ended. Thus, the model aggregation and update of federated learning are realized through model integration, and the model simplification method is provided to realize model personalization, so that each participant of federated learning obtains a personalized integrated model, and the final personalized model is used as an integrated model adapted to the local data distribution, which has better prediction performance than the traditional method; in addition, the global model and the personalized model are maintained on the participant side at the same time, which reduces the communication cost of the foregoing federated learning algorithm, so that the present application has a wider application scenario.

[0077] To realize the above-mentioned embodiments, the present application further provides an integrated model-based personalized federated learning device, as shown in the accompanying drawings. Figure 6 The integrated model-based personalized federated learning device comprises a first sending module 10, a processing module 20, a second sending module 30, and a model updating module 40.

[0078] The first sending module 10 is configured to send the current global model to each participant, so that each participant establishes a corresponding personalized model according to the global model.

[0079] The processing module 20 is configured to obtain the personalized model corresponding to each participant, and perform the first weighting processing on the personalized model corresponding to each participant to obtain an integrated model.

[0080] The second sending module 30 is configured to send the integrated model to each participant, so that each participant splits the integrated model into a plurality of personalized models, and performs boosting integration of the plurality of personalized models and the global model respectively to obtain a plurality of single integrated models, simplifies the plurality of single integrated models to obtain a simplified result;

[0081] The model updating module 40 is configured to obtain the simplified result corresponding to each participant, aggregates the simplified result corresponding to each participant to obtain a global model selection result, and performs second weighting processing on the personalized model corresponding to each participant according to the global model selection result to obtain a new integrated model, and adds the new integrated model to the global model to obtain an updated global model;

[0082] The first sending module 10 sends the updated global model to each participant, so as to perform the next round of model updating until the target federated model is obtained after the global round.

[0083] As an embodiment, the model establishing module is further configured to predict the local data according to the global model to obtain a prediction result, calculate a residual error between the global model and a true label according to the prediction result, and establish the personalized model according to the residual error.

[0084] As an embodiment, the processing module 20 is further configured to perform first weighting processing on the personalized model according to the data amount of each participant to obtain a corresponding weight, and combine the personalized model corresponding to each participant according to the corresponding weight to obtain the integrated model.

[0085] As an embodiment, the simplification module is further configured to test the plurality of single integrated models, and discard a combination in which a loss does not decrease, so as to simplify the plurality of single integrated models, and add the plurality of single integrated models after simplification to the personalized model, so as to update the personalized model.

[0086] As an embodiment, the participant uploads the respective simplified result in the form of a binary vector to the server side, so that the server side obtains the simplified result corresponding to each participant.

[0087] As an embodiment, the first data processing module and the second data processing module are further included, the first data processing module is arranged at each participant to obtain respective local data sets, and the second processing module is arranged at the server side to count the data amount of each participant and determine a global parameter, wherein the global parameter includes a global round, a number of participants, the data amount of each participant, and a base model hyperparameter.

[0088] It should be noted that the foregoing explanation of the embodiment of the personalized federated learning method based on the integrated model is also applicable to the personalized federated learning device based on the integrated model of the present embodiment, which will not be described here.

[0089] In summary, according to the personalized federated learning device based on the integrated model of the present embodiment, the first sending module sends the current global model to each participant, so that each participant establishes a corresponding personalized model according to the global model; the processing module obtains the personalized model corresponding to each participant, and performs first weighting processing on the personalized model corresponding to each participant to obtain an integrated model; the second sending module sends the integrated model to each participant, so that each participant splits the integrated model into multiple personalized models, and performs promotion integration of the multiple personalized models and the global model respectively to obtain multiple single integrated models, and performs simplification on the multiple single integrated models to obtain a simplified result; the model updating module obtains the simplified result corresponding to each participant, and aggregates the simplified result corresponding to each participant to obtain a global model selection result, and performs second weighting processing on the personalized model corresponding to each participant according to the global model selection result to obtain a new integrated model, and adds the new integrated model to the global model to obtain an updated global model; the first sending module sends the updated global model to each participant, so as to perform the next round of model updating until the target federated model is obtained after the global round is reached; thereby, the federated model aggregation of heterogeneous data can be performed under the premise of ensuring data privacy, and the personalized model is generated for the participant according to the data distribution to obtain higher performance and lower communication cost.

[0090] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system), and computer program product of the present embodiment. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the block or blocks.

[0092] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 one or more processes and / or blocks Figure 1 one or more blocks that can, for example, be implemented as computer program instructions.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the Figure 1 one or more processes and / or blocks Figure 1 one or more blocks that can, for example, be implemented as computer program instructions.

[0094] It is noted that in the claims the word "comprising" does not exclude not having other parts than those listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. It is further noted that claims which include an element having a predefined property can also include other elements without the predefined property. The application can be implemented by means of hardware comprising several distinct elements, and by means of a programmed computer. In the unitary claim drawn to several devices, the terms "means" or "apparatus" can refer to one or more instances of the device. The use of the term "about" with a given value of an element denotes the value of use of the element and the value of the element. The use of the term "first", "second" and "third" etc. does not connote any order, but rather are used for naming purposes only.

[0095] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to cover all such modifications and variations as fall within the true spirit and scope of the application.

[0096] It is apparent that a person skilled in the art can make various changes and modifications to the application without departing from the spirit and scope thereof. The application therefore intends to include all such changes and modifications as fall within the true spirit and scope of the application. The above specification, examples and data provide a complete description of the manufacture and use of the composition of the application. Since many embodiments of the application can be made without departing from the spirit and scope of the application, the application should not be limited to the examples contemplated.

[0097] In the description of the application, it should be understood that the terms "first", "second" are used only for descriptive purposes, and cannot be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second" can be explicitly or implicitly included one or more of the features. In the description of the application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified and limited.

[0098] In this application, unless otherwise expressly specified and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integrated; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through intermediate medium, or the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0099] In this application, unless otherwise expressly specified and limited, the first feature is "on" or "under" the second feature, which can be direct contact between the first and second features, or indirect contact between the first and second features through intermediate medium. Moreover, the first feature "above", "above" and "above" the second feature can be the first feature directly above or obliquely above the second feature, or only indicate that the first feature is higher than the second feature in horizontal height. The first feature "below", "below" and "below" the second feature can be the first feature directly below or obliquely below the second feature, or only indicate that the first feature is lower than the second feature in horizontal height.

[0100] In the description of the application, the description of the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the application. In this specification, the illustrative description of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in the specification and the features of different embodiments or examples without contradiction.

[0101] Although the embodiments of the application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be construed as limiting the application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the application.

Claims

1. An integrated model-based personalized federated learning method, characterized in that, The method comprises the following steps: sending a current global model to each participant so that each participant establishes a corresponding individualized model according to the global model; obtaining the individualized model corresponding to each participant and performing first weighting processing on the individualized model corresponding to each participant to obtain an integrated model; sending the integrated model to each participant so that each participant splits the integrated model into a plurality of individualized models and performs boosting integration on the plurality of individualized models and the global model respectively to obtain a plurality of single integrated models, simplifies the plurality of single integrated models to obtain a simplified result; obtaining the simplified result corresponding to each participant, aggregating the simplified result corresponding to each participant to obtain a global model selection result, and performing second weighting processing on the individualized model corresponding to each participant according to the global model selection result to obtain a new integrated model, and adding the new integrated model to the global model to obtain an updated global model; sending the updated global model to each participant so as to perform the next round of model updating until a target federated model is obtained after a global round; wherein each participant establishes a corresponding individualized model according to the global model, comprising: performing prediction on local data according to the global model to obtain a prediction result; calculating the residual of the global model and the true label according to the prediction result; establishing an individualized model according to the residual; wherein the plurality of single integrated models are simplified to obtain a simplified result, comprising: testing the plurality of single integrated models and discarding combinations with no loss reduction so as to simplify the plurality of single integrated models; adding the plurality of single integrated models after simplification to the individualized model so as to update the individualized model. Wherein, the participants upload their respective simplified results in the form of binary vectors to the server side so that the server side obtains the simplified result corresponding to each participant.

2. The integrated model-based personalized federated learning method of claim 1, wherein, Before performing the individualized federated learning method based on the integrated model, it further comprises: each participant respectively obtains a local data set; the server side is initialized, the data amount of each participant is counted, and the global parameters are determined, wherein the global parameters include the global round, the number of participants, the data amount of each participant and the basic model hyperparameters.

3. The integrated model based personalized federated learning method of claim 2, wherein, The first weighting processing is performed on the individualized model corresponding to each participant to obtain an integrated model, comprising: performing first weighting processing on the individualized model according to the data amount of each participant to obtain a corresponding weight; combining the individualized model corresponding to each participant according to the corresponding weight to obtain an integrated model.

4. An apparatus for personalized federated learning based on integrated model, characterized in that, comprising: a first sending module, the first sending module is used for sending a current global model to each participant so that each participant establishes a corresponding individualized model according to the global model; a processing module, the processing module is used for obtaining the individualized model corresponding to each participant and performing first weighting processing on the individualized model corresponding to each participant to obtain an integrated model; The second sending module is configured to send the integrated model to each participant, so that each participant splits the integrated model into a plurality of personalized models, and performs boosting integration of the plurality of personalized models and the global model respectively to obtain a plurality of single integrated models, and simplifies the plurality of single integrated models to obtain a simplified result; The model updating module is configured to obtain the simplified result corresponding to each participant, aggregate the simplified result corresponding to each participant to obtain a global model selection result, perform second weighting processing on the personalized model corresponding to each participant according to the global model selection result to obtain a new integrated model, and add the new integrated model to the global model to obtain an updated global model; The first sending module sends the updated global model to each participant to perform next round of model updating until a target federated model is obtained after global rounds are reached; The model establishing module is configured to perform prediction on local data according to the global model to obtain a prediction result, calculate a residual error between the global model and a real label according to the prediction result, and establish a personalized model according to the residual error; The simplification module is configured to test the plurality of single integrated models, discard combinations in which loss does not decrease, so as to simplify the plurality of single integrated models, and add the plurality of single integrated models after simplification to the personalized model to update the personalized model. The participant uploads the respective simplified result to the server in the form of a binary vector, so that the server obtains the simplified result corresponding to each participant.

5. The integrated model based personalized federated learning apparatus of claim 4, wherein, The processing module is further configured to perform first weighting processing on the personalized model according to a data amount of each participant to obtain a corresponding weight, and combine the personalized model corresponding to each participant according to the corresponding weight to obtain an integrated model.

Images