Federal learning contribution degree evaluation method and device, electronic equipment, medium and product

Abstract

The invention discloses a federated learning contribution degree evaluation method and device, electronic equipment, a medium and a product, and relates to the technical field of federated learning, and the federated learning contribution degree evaluation method comprises the steps: obtaining an initial global model, and sending the initial global model to each client in a preset client set; performing federal iteration training on the initial global model based on a received local model uploaded by each client in a preset client set, and determining an overall global model; performing federal iteration training on the initial global model based on received local models uploaded by clients except the to-be-evaluated client in a preset client set, and determining partial global models; and determining the contribution degree of the to-be-evaluated client according to a performance difference value between the overall global model and the partial global model. According to the federal learning contribution degree evaluation method provided by the invention, the calculation complexity of the contribution degree is reduced, and the time consumption and the calculation resource demand are reduced.

Description

Technical Field

[0001] The present application relates to the technical field of federated learning, and in particular to a method, device, electronic device, computer-readable storage medium, and computer program product for evaluating the contribution of federated learning. Background Art

[0002] At present, federated learning is used in finance, medical care and other fields involving privacy data or isolated data. In the process of federated learning, the contribution of each participant or client is an important parameter for establishing an effective incentive mechanism. The current mainstream solution is to determine the data contribution of each client by calculating the Shapley value, which requires a complex game theory foundation and high time complexity. However, this calculation method needs to consider all possible combinations of participants when there are a large number of participants, resulting in high time complexity, too long time consumption, and more computing resources. For edge devices or mobile devices with limited resources, executing the task of calculating the Shapley value will generate excessive performance overhead and power consumption.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a method, device, electronic device, computer-readable storage medium and computer program product for evaluating the contribution of federated learning, aiming to solve the technical problem that the current method for evaluating the contribution of federated learning is too complex.

[0005] To achieve the above objectives, the present application proposes a federated learning contribution evaluation method applied to a server, and the federated learning contribution evaluation method includes:

[0006] Acquire an initial global model, and send the initial global model to each client in a preset client set;

[0007] Iteratively updating the initial global model based on the received local models uploaded by each client in the preset client set to determine the overall global model;

[0008] Based on the received local models uploaded by clients other than the client to be evaluated in the preset client set, multiple model aggregation and updating are performed to determine a partial global model;

[0009] The contribution of the client to be evaluated is determined according to the performance difference between the overall global model and the partial global model.

[0010] In one embodiment, the step of performing multiple model aggregation and updating based on the received local models uploaded by clients other than the client to be evaluated in the preset client set to determine a partial global model includes:

[0011] Receiving the local models uploaded by each of the clients, and then aggregating the local models corresponding to the clients in the preset client set except the client to be evaluated to obtain an updated global model;

[0012] The updated global model is sent to the clients in the preset client set except the client to be evaluated, so that each client can iteratively train the updated global model, and a part of the global model is determined according to the local model uploaded by each client after training.

[0013] In one embodiment, the step of sending the updated global model to clients other than the client to be evaluated in the preset client set so that each client can iteratively train the updated global model, and determining part of the global model according to the local model uploaded by each client after training includes:

[0014] Sending the updated global model to the clients in the preset client set except the client to be evaluated, so that the clients in the preset client set except the client to be evaluated can train the updated global model to obtain local models corresponding to the clients respectively;

[0015] Receive and aggregate the local models sent by each of the clients to obtain an updated global model;

[0016] Determine whether the number of iterative trainings for updating the global model reaches a preset number;

[0017] If not, return to the execution step: send the updated global model to the clients in the preset client set except the client to be evaluated;

[0018] If so, the updated global model is set as a partial global model.

[0019] In one embodiment, the performance difference is an accuracy difference, and the step of determining the contribution of the client to be evaluated according to the performance difference between the overall global model and the partial global model includes:

[0020] Acquire preset test sample data, input the test sample data into the overall global model and the partial global model respectively, and obtain prediction results corresponding to the overall global model and the partial global model respectively;

[0021] Calculate the accuracy rates of the overall global model and the partial global model respectively according to the true labels of the test sample data and the prediction results of the overall global model and the partial global model respectively;

[0022] The difference between the accuracy of the overall global model and the accuracy of the partial global model is calculated to obtain the contribution of the client to be evaluated.

[0023] In one embodiment, the performance difference is a recall rate difference, and the step of determining the contribution of the client to be evaluated according to the performance difference between the overall global model and the partial global model includes:

[0024] Obtaining preset test sample data, inputting the test sample data into the overall global model and the partial global model respectively, and determining the confusion matrices corresponding to the overall global model and the partial global model respectively;

[0025] Calculate the recall rates corresponding to the overall global model and the partial global model respectively according to the confusion matrices corresponding to the overall global model and the partial global model respectively;

[0026] The difference between the recall rate of the overall global model and the recall rate of the partial global model is calculated to obtain the contribution of the client to be evaluated.

[0027] In addition, the present application also proposes a method for evaluating the contribution of federated learning applied to a client, wherein the method for evaluating the contribution of federated learning includes:

[0028] When receiving the initial global model sent by the server, training the initial global model according to the local data to obtain a trained local model;

[0029] The local model is uploaded to the server so that the server can iteratively update the initial global model based on the local models uploaded by each client in the preset client set to determine the overall global model, and perform multiple model aggregation and updates based on the local models uploaded by clients other than the client to be evaluated in the preset client set to determine the partial global model, and determine the contribution of the client to be evaluated based on the performance difference between the overall global model and the partial global model.

[0030] In addition, to achieve the above purpose, the present application also proposes a federated learning contribution evaluation device, which includes:

[0031] An initialization module, used for obtaining an initial global model, and sending the initial global model to each client in a preset client set;

[0032] An overall global training module, used for iteratively updating the initial global model based on the local models uploaded by each client in the preset client set, to determine the overall global model;

[0033] A partial global training module, used to perform multiple model aggregation and updates based on the local models uploaded by the clients other than the client to be evaluated in the preset client set, to determine a partial global model;

[0034] The contribution calculation module is used to determine the contribution of the client to be evaluated according to the performance difference between the overall global model and the partial global model.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the federated learning contribution evaluation method as described above.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the federated learning contribution evaluation method described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the federated learning contribution evaluation method described above.

[0038] The present application proposes a method for evaluating the contribution of federated learning. In the method, an initial global model is first obtained, and the initial global model is sent to each client in a preset client set to initialize and preliminarily train the global model. Then, the initial global model is iteratively updated based on the local models uploaded by each client in the preset client set to determine the overall global model, wherein the overall global model is obtained by federal training of all clients including the client to be evaluated. Then, based on the local models uploaded by clients other than the client to be evaluated in the preset client set, multiple model aggregation and updates are performed to determine the partial global model, wherein the partial global model is obtained by training other clients excluding the client to be evaluated. Finally, the contribution of the client to be evaluated is determined based on the performance difference between the overall global model and the partial global model. Because the difference between the overall global model and the partial global model is whether the client to be evaluated participates in federated learning, the performance difference between the two can reflect the contribution made by the client to be evaluated to the performance of the overall global model. Compared with the traditional method of using Shapley value to determine the data contribution of each client, the technical solution of the present application provides a contribution evaluation based on the idea of ​​forgetting learning. It does not require complex game theory foundations. The contribution calculation is carried out simultaneously with the federated learning training process. The overall global model and partial global model are trained at the same time, with low time complexity. Finally, the performance difference between the overall global model and the partial global model is calculated to determine the client's contribution. The overall calculation complexity is lower, less computing resources are consumed, and the performance overhead and power consumption of the device are also lower. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 A flowchart of the first embodiment of the method for evaluating the contribution of federated learning in this application;

[0042] Figure 2 This is a schematic diagram of the principle of federated learning between the server and the client in the embodiment of the federated learning contribution evaluation method of this application;

[0043] Figure 3 This is a schematic diagram of the process of training part of the global model in the embodiment of the federated learning contribution evaluation method of this application;

[0044] Figure 4 This is a schematic diagram of the structure of the federated learning contribution evaluation device in the embodiment of the present application;

[0045] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the federated learning contribution evaluation method in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] The execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone or a server, etc., or an electronic device or control device capable of realizing the above functions. The following takes the server as an execution subject as an example to illustrate this embodiment and the following embodiments.

[0050] This application embodiment provides a method for evaluating the contribution of federated learning. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the federated learning contribution evaluation method of the present application. The federated learning contribution evaluation method is applied to the server side. The federated learning contribution evaluation method includes:

[0051] Step S10, obtaining an initial global model, and sending the initial global model to each client in a preset client set;

[0052] It should be noted that the server is a hardware device provided by a participant or a third party for federated learning and model aggregation during the process of federated learning contribution evaluation. It can be a local server or a cloud server. The server has a communication connection with the client of each participant to transmit the initialized model or the trained model. By combining the distributed computing resources and local data of each client, a global model that can be shared by all participants is trained.

[0053] Among them, obtaining the initial global model means initializing the global model, that is, building the hierarchical architecture of the model and setting the initial parameters of the model according to the needs of the actual scenario, so as to obtain the initial global model to be trained. After obtaining the initial global model, it is sent to all clients included in the preset client set, so that each client can use the corresponding local data for local training and return the trained local model to the server. Among them, the preset client set includes the clients (such as computer devices) provided by all participants involved in this federated learning.

[0054] Exemplarily, the initial global model can be an anti-fraud model applied in the financial field, which is used to predict the user's financial risk based on the user's financial data to determine whether the user meets the corresponding qualification requirements (such as loan qualifications). The type of the initial global model can be one of logistic regression, decision tree, random forest, support vector machine or neural network.

[0055] like Figure 2 As shown, in the federated learning contribution evaluation method of the embodiment of the present application, one server corresponds to multiple clients (for example, client 1, client i to client N), and each client respectively uses a local data set (for example, D1, D i , D N wait).

[0056] Step S20, iteratively updating the initial global model based on the local models uploaded by each client in the preset client set, to determine the overall global model;

[0057] The initial global model is sent to each client through the server, so that each client can perform federated iterative training on the initial global model. Specifically, during the federated iterative training process, each client trains the sent initial global model according to local data and then transmits it back to the server. The server aggregates the model and then sends it back to the client. This cycle is repeated multiple times to iteratively update the initial global model multiple times, and finally a trained overall global model is obtained.

[0058] Step S30, performing multiple model aggregation and updating based on the local models uploaded by the clients other than the client to be evaluated in the preset client set, and determining a partial global model;

[0059] It should be noted that when the initial global model is sent for the first time in step S10, it needs to be sent to all clients. After receiving the local models of all clients, the server excludes the local model of the client to be evaluated and aggregates the local models of other clients, thereby obtaining a partial global model that excludes the client to be evaluated, which can be used to evaluate the contribution of the client to be evaluated.

[0060] It should be noted that step S20 and step S30 can be executed simultaneously, that is, after the initial global model is sent to each client, each client can perform iterative training simultaneously to obtain the overall global model and the partial global model.

[0061] Furthermore, while each client is training a partial global model, it can train N-1 partial global models simultaneously or in batches. That is, each client needs to participate in the federated iterative training process of N-1 partial global models for evaluating the contribution of other clients.

[0062] Step S40, determining the contribution of the client to be evaluated according to the performance difference between the overall global model and the partial global model.

[0063] After the overall global model and the partial global model corresponding to the client to be evaluated are obtained, the performance parameters corresponding to the overall global model and the partial global model are calculated respectively.

[0064] The choice of performance parameters depends on the task type (such as classification, regression, clustering, etc.) of the overall global model and the partial global model, the characteristics of the dataset, and the business requirements. For example, the performance parameters of a model suitable for supervised learning tasks, especially classification problems, can be accuracy, recall, or sensitivity.

[0065] In actual application scenarios, the performance difference between the overall global model and the partial global model may be negative, that is, the overall global model obtained when the client to be evaluated participates in federated learning is lower in performance than the partial global model obtained when the client to be evaluated does not participate in federated learning. In this case, the contribution of the client to be evaluated is also negative, which means that the client to be evaluated and its local data have a negative effect in federated learning.

[0066] In federated learning, the client's contribution plays an important role in establishing an effective incentive mechanism. According to the contribution of each client, appropriate incentive measures can be taken for the participants corresponding to each client, so as to ensure that the participants are willing to invest more resources (such as electricity, computing resources, communication bandwidth, etc.) for local model training and communication, thereby improving the training effect of the overall global model, and encouraging high-quality data providers to actively participate in federated learning, while reducing the participation of low-quality data participants to avoid low-quality data providers affecting model performance. Moreover, the continuous incentive mechanism can ensure that the participants remain motivated throughout the training process and continue to contribute data and computing resources.

[0067] In the federated learning contribution evaluation method of the embodiment of the present application, an initial global model is first obtained, and the initial global model is sent to each client in a preset client set to initialize and preliminarily train the global model. Then, the initial global model is iteratively updated based on the local models uploaded by each client in the preset client set to determine the overall global model, wherein the overall global model is obtained by federal training of all clients including the client to be evaluated. Then, based on the local models uploaded by clients other than the client to be evaluated in the preset client set, multiple model aggregation and updates are performed to determine the partial global model, wherein the partial global model is obtained by training other clients excluding the client to be evaluated. Finally, the contribution of the client to be evaluated is determined based on the performance difference between the overall global model and the partial global model. Because the difference between the overall global model and the partial global model is whether the client to be evaluated participates in federated learning, the performance difference between the two can reflect the contribution of the client to be evaluated to the performance of the overall global model. Compared with the traditional method of using Shapley value to determine the data contribution of each client, the technical solution of the present application provides a contribution evaluation based on the idea of ​​forgetting learning. It does not require complex game theory foundations. The contribution calculation is carried out simultaneously with the federated learning training process. The overall global model and partial global model are trained at the same time, with low time complexity. Finally, the performance difference between the overall global model and the partial global model is calculated to determine the client's contribution. The overall calculation complexity is lower, less computing resources are consumed, and the performance overhead and power consumption of the device are also lower.

[0068] In addition, the technical solution of the embodiment of the present application is based on forgetting learning to calculate the contribution, which is essentially to compare whether a certain client participates and obtain the performance difference between different global models. Therefore, the federated learning contribution evaluation method of the embodiment of the present application is a fair and transparent calculation solution.

[0069] Further, in a feasible implementation manner, the step of performing multiple model aggregation and updating based on the received local models uploaded by clients other than the client to be evaluated in the preset client set to determine a partial global model includes:

[0070] Step S11, receiving the local models uploaded by each client, and then aggregating the local models corresponding to the clients in the preset client set except the client to be evaluated to obtain an updated global model;

[0071] Step S12, sending the updated global model to the clients in the preset client set except the client to be evaluated, so that each client can iteratively train the updated global model, and determine part of the global model according to the local model uploaded by each client after training.

[0072] The embodiment of the present application provides a method for training a partial global model. First, after the server initializes the global model, the initial global model is recorded as f0 and sent to each client in the preset client set. In t=1 rounds of iterative training, client i first initializes the received initial global model f0 as a local model, recorded as Among them, client i can refer to each client; client i then passes the local data set D i Perform local training to make the local model Updated to And update the local model Upload to the server.

[0073] After receiving the local models sent by all clients, the server aggregates the local models of the preset client set except the client to be evaluated, and obtains an updated global model after the first round of iterative training, wherein the updated global model is a global model that has undergone one training, and is obtained by aggregating the local models provided by the clients except the client to be evaluated.

[0074] Since there are many participants in federated learning, when the contribution of each client needs to be evaluated, multiple partial global models need to be trained. For example, if the preset client set includes N clients, N partial global models need to be determined to determine the contribution of each of the N clients. After the server receives the local models of each client, it can aggregate them to obtain N partial global models to evaluate the contribution of each of the N clients.

[0075] Specifically, in model aggregation, the server can use some aggregation strategy (such as the federated weighted average strategy FedAvg) to aggregate these updated local models to form a new global model. Usually, each client's local model is weighted according to its data volume. Finally, the server uses the aggregated model update to adjust the parameters of the initial global model to obtain an updated global model.

[0076] Taking client i as an example, after excluding client i, the updated global model obtained by aggregating the remaining clients is recorded as

[0077] In addition, after obtaining the updated model after the first round of training, further iterative training is required, so the server can send the updated model to the clients in the preset client set except the client to be evaluated, so that each client can use local data to train again, and return the trained updated model to the server. After executing this cycle for a preset number of times, a partial global model is finally obtained.

[0078] Since Shapley-based contribution calculation requires calculating the model aggregation under different subsets in each round, N clients have a total of 2 N There are many different combinations. Therefore, solving the contribution based on Shapley value has an exponential time complexity of O(2 N ); On the contrary, it can be seen from the above embodiment that the contribution calculation based on forgetting learning is directly aggregated in each round, and the time complexity is O(1). Therefore, the federated learning contribution evaluation method of the embodiment of the present application has lower time complexity than the traditional method of calculating the Shapley value, takes less time, consumes less computing resources, reduces the performance overhead and power consumption of hardware devices such as edge devices or mobile devices participating in federated learning, and reduces the hardware requirements in federated learning.

[0079] Further, in a feasible embodiment, referring to Figure 3 The step of sending the updated global model to the clients other than the client to be evaluated in the preset client set so that each client can iteratively train the updated global model, and determining part of the global model according to the local model uploaded by each client after training includes:

[0080] Step S131, sending the updated global model to the clients other than the client to be evaluated in the preset client set so that the clients other than the client to be evaluated in the preset client set train the updated global model to obtain local models corresponding to each client;

[0081] Step S132, receiving and aggregating the local models sent by each client to obtain an updated global model;

[0082] Step S133, determining whether the number of iterative training for updating the global model reaches a preset number;

[0083] If not, the process returns to step S131: sending the updated global model to the clients in the preset client set except the client to be evaluated;

[0084] Step S134: If yes, the updated global model is set as a partial global model.

[0085] An embodiment of the present application provides a method for iteratively training a partial global model, wherein the partial global model refers to a global model obtained by training clients other than the client to be evaluated in a preset client set. Compared with the overall global model obtained by training all clients, the partial global model lacks the participation of the client to be evaluated, which makes it easier to measure the contribution of the client to be evaluated based on the performance difference between the overall global model and the partial global model.

[0086] After completing the first round of iterative training, and obtaining an updated global model through model aggregation by the server based on the local models corresponding to the clients other than the client to be evaluated in the preset client set, the updated global model is again sent to the clients other than the client to be evaluated in the preset client set, so that the clients can iteratively train the local models corresponding to the updated global model. After repeating this for a preset number of times, the training is stopped, and the finally obtained updated global model is determined as the partial global model.

[0087] Exemplarily, starting from the second round of iterative training, the following training steps are executed cyclically: client i updates the received global model f t-1 Reinitialize to the local model, denoted as It should also be noted that when client i simultaneously trains multiple updated global models, the remaining N-1 global models will also be updated It is also initialized as a local model, denoted as These N models all use the local dataset D i Perform local training. After the training is completed, Update to f t i , Updated to Finally, client i updates the local model Upload to the server; after receiving the local model updated by the client, the server aggregates the local models provided by other clients excluding client i, and mentions the updated global model i used to evaluate the contribution of client i; it should also be noted that the client processing to be evaluated includes client i and may also include other clients, so the server can also aggregate the local models corresponding to the clients excluding other clients to be evaluated, obtain other updated global models, and finally obtain N updated global models; finally, the server sends the updated updated global model to the remaining N-1 clients except client i, so that each client can carry out the next round of training.

[0088] In another feasible embodiment, in step S20, the step of training the overall global model is similar to the step of step S10, except that the training of the overall global model requires the participation of all clients, and the first round of training can be the same as the step of step S31. After the server receives the local models sent by all clients respectively, all local models are aggregated to obtain an updated initial global model, and then the updated initial global model is sent to each client, so that each client can iteratively train the updated initial global model and repeat a preset number of times, and finally obtain the overall global model after iterative training, which is recorded as

[0089] It can be understood that after step S20 and step S30 are executed, when there are N clients to be evaluated, the server stores the following N+1 models: first, the global model f obtained by training all the clients of the participants in the preset client set T , and the other is the global model obtained by training with the rest of the clients except client i

[0090] Further, in a feasible embodiment, the performance difference is a recall rate difference, and the step of determining the contribution of the client to be evaluated according to the performance difference between the overall global model and the partial global model includes:

[0091] Step S41, obtaining preset test sample data, inputting the test sample data into the overall global model and the partial global model respectively, and determining the confusion matrices corresponding to the overall global model and the partial global model respectively;

[0092] Step S42, calculating the recall rates corresponding to the overall global model and the partial global model respectively according to the confusion matrices corresponding to the overall global model and the partial global model respectively;

[0093] Step S43, calculating the difference between the recall rate of the overall global model and the recall rate of the partial global model to obtain the contribution of the client to be evaluated.

[0094] In the embodiment of the present application, when the overall global model is a model type that focuses on prediction accuracy (such as a classification model), accuracy is a relatively important parameter for measuring model performance. Therefore, in the embodiment of the present application, the accuracy difference between the overall global model and the partial global model is used to determine the contribution of the client to be evaluated.

[0095] When determining the accuracy of the overall global model and the partial global model, it is necessary to obtain preset test sample data, and the test sample data is consistent with the type of the overall global model. For example, when the overall global model is a financial anti-fraud model, the test sample data can be the user's financial data. The overall global model can predict whether the user has financial risks based on the input user financial data, and the prediction result is one of yes or no. In addition, the test sample data carries a true label, and the true label represents whether each user has risks. After the overall global model and the partial global model output the prediction results corresponding to multiple users respectively, they are compared with the true labels of each user. If they are consistent, the prediction is correct. If they are inconsistent, the prediction is wrong. The accuracy of the overall global model and the partial global model can be obtained by calculating the ratio of the number of correct predictions to the total number of predictions.

[0096] Finally, the difference between the accuracy of the overall global model and the accuracy of the partial global model is calculated, which can be used as the contribution of the client to be evaluated, and is used to characterize the improvement in accuracy before and after the client to be evaluated participates in federated training. The greater the improvement, the higher the contribution. If the difference is a negative value, it means that the client to be evaluated has a negative effect and the contribution is negative.

[0097] In another feasible embodiment, the performance difference is a recall rate difference, and the step of determining the contribution of the client to be evaluated according to the performance difference between the overall global model and the partial global model includes:

[0098] Step S44, obtaining preset test sample data, inputting the test sample data into the overall global model and the partial global model respectively, and determining the confusion matrices corresponding to the overall global model and the partial global model respectively;

[0099] Step S45, calculating the recall rates corresponding to the overall global model and the partial global model respectively according to the confusion matrices corresponding to the overall global model and the partial global model respectively;

[0100] Step S46, calculating the difference between the recall rate of the overall global model and the recall rate of the partial global model to obtain the contribution of the client to be evaluated.

[0101] The embodiment of the present application provides another method for measuring the contribution of the client to be evaluated, that is, using the recall rate to determine the contribution of the client to be evaluated. The recall rate can also be called sensitivity, which is suitable for binary classification problems. The recall rate characterizes the ability of the model to correctly identify all actual positive examples, that is, it reflects the ratio of the model correctly predicting the positive class among the samples that are actually positive. This is an important evaluation indicator, especially in application scenarios (such as medical scenarios) with fewer positive examples or higher misdiagnosis costs.

[0102] Specifically, when calculating the recall rate of the overall global model and the partial global model, it is necessary to predefine positive and negative examples, clearly distinguish positive examples (target events) from negative examples (non-target events), and then generate confusion matrices based on the prediction results output by the overall global model and the partial global model respectively: According to the prediction results of the model and the actual labels, the corresponding confusion matrix is ​​constructed. The confusion matrix is ​​a table that shows the comparison between the model prediction results and the actual results, and contains the following four elements: True Positives (TP) is the number of samples that are actually positive and correctly predicted as positive by the model; False Positives (FP) is the number of samples that are actually negative but incorrectly predicted as positive by the model; True Negatives (TN) is the number of samples that are actually negative and correctly predicted as negative by the model; False Negatives (FN) is the number of samples that are actually positive but incorrectly predicted as negative by the model. The calculation formula for the recall rate is: Sensitivity = TP / (TP+FN). The higher the recall rate, the stronger the model's ability to identify positive examples.

[0103] Similarly, the difference between the accuracy of the overall global model and the recall of the partial global model is calculated at the end, which can be used as the contribution of the client to be evaluated, and is used to characterize the improvement of the recall rate of the client to be evaluated before and after participating in the federated training. The greater the improvement, the higher the contribution. If the difference is a negative value, it means that the client to be evaluated has a negative effect and the contribution is negative.

[0104] In a feasible embodiment, the overall global model can be applied to the financial field, the overall global model and the partial global model are both financial risk prediction models (such as financial anti-fraud models), and the client is a computer corresponding to the financial institution, which is used to store the user's financial data and train the initial global model.

[0105] It should also be noted that the initial global model is a pre-built financial risk prediction model, and the updated global model is a financial risk prediction model under training. The participants in federated learning can be financial institutions, the client is a computer, server or other electronic device provided by the financial institution, and the server can be a computer, server or other electronic device provided by a participant, or a computer, server or other electronic device provided by a third party that organizes federated learning.

[0106] Exemplarily, each client stores the financial data of the financial institution's corresponding customers or users, such as personal information, consumption data, loan data, repayment data or other financial information data. In order to train a financial risk prediction model with better performance and higher accuracy, it is necessary to expand the amount of training data and combine the user financial data of multiple financial institutions. However, privacy is involved. In order to avoid user privacy leakage, the federated learning method is adopted to train the model. In addition, in order to better evaluate the contribution of each financial institution in federated learning (including financial data, computing resources, power consumption, etc.), the federated learning contribution evaluation method provided in the embodiment of this application is adopted to determine the contribution of the financial institution.

[0107] For example, the overall global model is a financial risk prediction model trained by the clients of financial institution A and the clients provided by other financial institutions, while the partial global model is a financial risk prediction model trained by the clients provided by other financial institutions except the clients of financial institution A. The difference in accuracy between the two models can reflect the contribution of the clients of financial institution A. For example, the accuracy of the overall global model is 96.5%, and the accuracy of the partial global model is 93.5%. It can be seen that the accuracy of the overall global model has increased by 3% due to the addition of the clients of financial institution A. It can be considered that the contribution of the clients of financial institution A in federated learning is equivalent to 3%.

[0108] After determining the contribution of the clients of each financial institution, the organizers of this federated learning can give corresponding incentives based on the size of the contribution, so that all participants can ensure higher enthusiasm and continue to contribute data and computing resources to make the overall global model more perfect.

[0109] In addition, the present application also provides a method for evaluating the contribution of federated learning applied to a client, wherein the method for evaluating the contribution of federated learning includes:

[0110] Step A10, when the initial global model sent by the server is received, the initial global model is trained according to the local data to obtain a trained local model;

[0111] Step A20, uploading the local model to the server, so that the server can iteratively update the initial global model based on the local models uploaded by each client in the preset client set, determine the overall global model, and perform multiple model aggregation and updates based on the local models uploaded by clients other than the client to be evaluated in the preset client set, determine the partial global model, and determine the contribution of the client to be evaluated based on the performance difference between the overall global model and the partial global model.

[0112] Specifically, after the initial global model is sent down by the server, the client is used to initialize it as a local model, then train the local model based on its own stored local data, and upload the local model to the server after the training is completed.

[0113] Furthermore, after the client receives the updated global model after the initial aggregation sent by the server, it is also initialized as a local model, and then the local model is trained based on its own stored local data, and the local model is uploaded to the server after the training is completed. The updated global model in this case is used to evaluate the model before the training of some global models of other clients in the preset client set is completed.

[0114] It should be noted that the client is mainly used to perform local training steps. No matter it is the initial global model sent by the server for training the overall global model, or the updated global model or other intermediate step model for training the partial global model, it is trained according to the local data stored locally and uploaded to the server after completing the local training for further aggregation and updating by the server, and finally the overall global model and the partial global model are determined for further contribution evaluation.

[0115] The specific execution steps of the federated learning contribution evaluation method applied to the client provided in the embodiment of the present application are consistent with the above-mentioned federated learning contribution evaluation method applied to the server. The specific steps executed by the client can refer to the content of the previous embodiment, which will not be repeated here. In addition, the federated learning contribution evaluation method applied to the client can solve the technical problem of the high complexity of the current federated learning contribution evaluation method. Compared with the prior art, the beneficial effects of the federated learning contribution evaluation method provided in the present application are the same as the beneficial effects of the federated learning contribution evaluation method provided in the previous embodiment, and the other technical features of the federated learning contribution evaluation method applied to the client are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0116] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the federated learning contribution evaluation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0117] In addition, this application also provides a federated learning contribution evaluation device, referring to Figure 4 , the federated learning contribution evaluation device comprises:

[0118] An initialization module 10 is used to obtain an initial global model and send the initial global model to each client in a preset client set;

[0119] An overall global training module 20, configured to iteratively update the initial global model based on the local models uploaded by each client in the preset client set, and determine an overall global model;

[0120] A partial global training module 30 is used to control the clients in the preset client set except the client to be evaluated to iteratively update the initial global model to determine a partial global model;

[0121] The contribution calculation module 40 is used to perform multiple model aggregation and updates based on the local models uploaded by the clients other than the client to be evaluated in the preset client set, so as to determine a part of the global model.

[0122] In one embodiment, the partial global training module 30 is further used for:

[0123] Receiving the local models uploaded by each of the clients, and then aggregating the local models corresponding to the clients in the preset client set except the client to be evaluated to obtain an updated global model;

[0124] The updated global model is sent to the clients in the preset client set except the client to be evaluated, so that each client can iteratively train the updated global model, and a part of the global model is determined according to the local model uploaded by each client after training.

[0125] In one embodiment, the partial global training module 30 is further used for:

[0126] Sending the updated global model to clients other than the client to be evaluated in the preset client set, so that the clients other than the client to be evaluated in the preset client set can train the updated global model to obtain local models corresponding to the clients;

[0127] Receive and aggregate the local models sent by each of the clients to obtain an updated global model;

[0128] Determine whether the number of iterative trainings for updating the global model reaches a preset number;

[0129] If not, return to the execution step: send the updated global model to the clients in the preset client set except the client to be evaluated;

[0130] If so, the updated global model is set as a partial global model.

[0131] In one embodiment, the performance difference is an accuracy difference, and the contribution calculation module 40 is further used to:

[0132] Acquire preset test sample data, input the test sample data into the overall global model and the partial global model respectively, and obtain prediction results corresponding to the overall global model and the partial global model respectively;

[0133] Calculate the accuracy rates of the overall global model and the partial global model respectively according to the true labels of the test sample data and the prediction results of the overall global model and the partial global model respectively;

[0134] The difference between the accuracy of the overall global model and the accuracy of the partial global model is calculated to obtain the contribution of the client to be evaluated.

[0135] In one embodiment, the performance difference is a recall rate difference, and the contribution calculation module 40 is further used for:

[0136] Obtaining preset test sample data, inputting the test sample data into the overall global model and the partial global model respectively, and determining the confusion matrices corresponding to the overall global model and the partial global model respectively;

[0137] Calculate the recall rates corresponding to the overall global model and the partial global model respectively according to the confusion matrices corresponding to the overall global model and the partial global model respectively;

[0138] The difference between the recall rate of the overall global model and the recall rate of the partial global model is calculated to obtain the contribution of the client to be evaluated.

[0139] The federated learning contribution evaluation device provided by the present application adopts the federated learning contribution evaluation method in the above embodiment, which can solve the technical problem of the high complexity of the current federated learning contribution evaluation method. Compared with the prior art, the beneficial effects of the federated learning contribution evaluation device provided by the present application are the same as the beneficial effects of the federated learning contribution evaluation method provided by the above embodiment, and the other technical features in the federated learning contribution evaluation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0140] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the federated learning contribution evaluation method in the above-mentioned embodiment 1.

[0141] Reference below Figure 5, which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0142] like Figure 5 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0143] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0144] The electronic device provided by this application adopts the federated learning contribution evaluation method in the above embodiment, which can solve the technical problem of the high complexity of the current federated learning contribution evaluation method. Compared with the prior art, the beneficial effects of the electronic device provided by this application are the same as the beneficial effects of the federated learning contribution evaluation method provided by the above embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0145] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0146] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0147] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the federated learning contribution evaluation method in the above-mentioned embodiment.

[0148] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0149] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0150] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device executes: obtaining an initial global model and sending the initial global model to each client in a preset client set; iteratively updating the initial global model based on the local models uploaded by each client in the preset client set to determine the overall global model; performing multiple model aggregation and updates based on the local models uploaded by clients other than the client to be evaluated in the preset client set to determine the partial global model; and determining the contribution of the client to be evaluated based on the performance difference between the overall global model and the partial global model.

[0151] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0153] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0154] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned federated learning contribution evaluation method, and can solve the technical problem that the current federated learning contribution evaluation method is too complex. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the federated learning contribution evaluation method provided in the above-mentioned embodiment, and will not be repeated here.

[0155] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned federated learning contribution evaluation method.

[0156] The computer program product provided by this application can solve the technical problem that the current federated learning contribution evaluation method is too complex. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the federated learning contribution evaluation method provided by the above embodiment, which will not be repeated here.

[0157] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for evaluating the contribution of federated learning, characterized in that: Applied to the server, the federated learning contribution evaluation method includes: Acquire an initial global model, and send the initial global model to each client in a preset client set; Iteratively updating the initial global model based on the received local models uploaded by each client in the preset client set to determine the overall global model; Based on the received local models uploaded by clients other than the client to be evaluated in the preset client set, multiple model aggregation and updating are performed to determine a partial global model; The contribution of the client to be evaluated is determined according to the performance difference between the overall global model and the partial global model.

2. The method for evaluating the contribution of federated learning according to claim 1, wherein: The step of performing multiple model aggregation and updating based on the received local models uploaded by clients other than the client to be evaluated in the preset client set to determine a partial global model comprises: Receiving the local models uploaded by each of the clients, and then aggregating the local models corresponding to the clients in the preset client set except the client to be evaluated to obtain an updated global model; The updated global model is sent to the clients in the preset client set except the client to be evaluated, so that each client can iteratively train the updated global model, and a part of the global model is determined according to the local model uploaded by each client after training.

3. The method for evaluating the contribution of federated learning according to claim 2, wherein: The step of sending the updated global model to the clients in the preset client set except the client to be evaluated so that each client can iteratively train the updated global model, and determining part of the global model according to the local model uploaded by each client after training includes: Sending the updated global model to the clients in the preset client set except the client to be evaluated, so that the clients in the preset client set except the client to be evaluated can train the updated global model to obtain local models corresponding to the clients respectively; Receiving and aggregating the local models sent by each of the clients to obtain an updated global model; Determine whether the number of iterative trainings for updating the global model reaches a preset number; If not, return to the execution step: send the updated global model to the clients in the preset client set except the client to be evaluated; If so, the updated global model is set as a partial global model.

4. The method for evaluating the contribution of federated learning according to claim 1, wherein: The performance difference is an accuracy difference, and the step of determining the contribution of the client to be evaluated according to the performance difference between the overall global model and the partial global model comprises: Acquire preset test sample data, input the test sample data into the overall global model and the partial global model respectively, and obtain prediction results corresponding to the overall global model and the partial global model respectively; Calculate the accuracy rates of the overall global model and the partial global model respectively according to the true labels of the test sample data and the prediction results of the overall global model and the partial global model respectively; The difference between the accuracy of the overall global model and the accuracy of the partial global model is calculated to obtain the contribution of the client to be evaluated.

5. The method for evaluating the contribution of federated learning as claimed in claim 1, characterized in that: The performance difference is a recall rate difference, and the step of determining the contribution of the client to be evaluated according to the performance difference between the overall global model and the partial global model includes: Obtaining preset test sample data, inputting the test sample data into the overall global model and the partial global model respectively, and determining the confusion matrices corresponding to the overall global model and the partial global model respectively; Calculate the recall rates corresponding to the overall global model and the partial global model respectively according to the confusion matrices corresponding to the overall global model and the partial global model respectively; The difference between the recall rate of the overall global model and the recall rate of the partial global model is calculated to obtain the contribution of the client to be evaluated.

6. A method for evaluating the contribution of federated learning, characterized in that: Applied to the client, the federated learning contribution evaluation method includes: When receiving the initial global model sent by the server, training the initial global model according to the local data to obtain a trained local model; The local model is uploaded to the server so that the server can iteratively update the initial global model based on the local models uploaded by each client in the preset client set to determine the overall global model, and perform multiple model aggregation and updates based on the local models uploaded by clients other than the client to be evaluated in the preset client set to determine the partial global model, and determine the contribution of the client to be evaluated based on the performance difference between the overall global model and the partial global model.

7. A federated learning contribution evaluation device, characterized in that: The federated learning contribution evaluation device comprises: An initialization module, used for obtaining an initial global model, and sending the initial global model to each client in a preset client set; An overall global training module, used for iteratively updating the initial global model based on the local models uploaded by each client in the preset client set, to determine the overall global model; A partial global training module, used to perform multiple model aggregation and updates based on the local models uploaded by the clients other than the client to be evaluated in the preset client set, to determine a partial global model; The contribution calculation module is used to determine the contribution of the client to be evaluated according to the performance difference between the overall global model and the partial global model.

8. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the federated learning contribution evaluation method as described in any one of claims 1 to 5 or claim 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the federated learning contribution evaluation method according to any one of claims 1 to 5 or claim 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 or claim 6 is implemented.

Images