Federal learning method and device based on knowledge sharing and computer equipment

Abstract

According to the federated learning method and device based on knowledge sharing and the computer equipment provided by the invention, in one communication round, the assistant server determines the shared client in the plurality of clients and determines the shared data set; and the sharing client constructs a local knowledge tensor according to the local model and the sharing data set and broadcasts the local knowledge tensor to other sharing clients. Next, a knowledge matrix is generated according to local knowledge tensors of other shared clients, then a reputation weight matrix is determined in the knowledge matrix based on data contribution, the knowledge matrix is weighted based on the reputation weight matrix, and the local model is updated through the weighted knowledge matrix; according to the updating mode, global aggregation operation does not need to be carried out, dependence on communication resources is reduced, the determined sharing client side can carry out knowledge sharing based on the sharing data set, available knowledge of other devices can be learned to the maximum degree so as to improve the local model of the client side, and therefore the adaptability of the local model and the application client side of the local model is improved.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a federated learning method, apparatus and computer equipment based on knowledge sharing. Background Art

[0002] In a federated learning (Heterogeneous Federated Learning, HetFL) environment consisting of heterogeneous devices, there are huge differences in device resources and data distribution. In order to fully utilize the resources on these diverse devices and improve the overall performance of artificial intelligence applications, traditional federated learning methods usually send local model updates of participating devices to a central server, which is responsible for aggregating these updates and improving the global model, and sometimes also involves point-to-point communication to directly update the global model.

[0003] However, in HetFL scenarios, individual devices (such as smartphones, sensors, etc.) often have limited resources, are sensitive to communication overhead, and have large differences in data distribution between different devices. These characteristics make the existing federated learning methods less adaptable to the models generated in actual applications and their application devices, making it difficult to guarantee the performance and accuracy of the generated models. Summary of the invention

[0004] The purpose of this application is to solve at least one of the above technical defects, especially the technical defects that a single device in the prior art often has limited resources, is sensitive to communication overhead, and the data distribution between different devices also varies greatly. These characteristics make the existing federated learning method poorly adaptable between the model generated in actual application and its application device, and thus it is difficult to ensure the performance and accuracy of the generated model.

[0005] In a first aspect, the present application provides a federated learning method based on knowledge sharing, which is applied to a collaborative training system, wherein the system includes multiple clients and an assistant server, and the method includes:

[0006] The assistant server determines a shared client from among the multiple clients, and determines a shared data set from among the preset public data sets;

[0007] The shared client constructs a local knowledge tensor according to its local model and the shared data set, and broadcasts the local knowledge tensor to other shared clients;

[0008] The sharing client generates a knowledge matrix according to the received local knowledge tensors of other sharing clients, and calculates the knowledge coefficient and data distribution difference with other sharing clients according to the data contribution of each sharing client in the knowledge matrix to determine the reputation weight matrix;

[0009] The shared client weights the knowledge matrix with the reputation weight matrix, updates the local model of the shared client according to the weighted knowledge matrix, and returns to the assistant server to continue the step of determining the shared client among multiple clients until the preset communication conditions are met.

[0010] In one embodiment, the assistant server determines a shared client from a plurality of clients, and determines a shared data set from a preset public data set, including:

[0011] The assistant server records the communication delay of each client when communicating with each client, and performs cluster analysis on each client according to the communication delay to obtain multiple client sets;

[0012] The assistant server selects a client set with the lowest communication delay from multiple client sets, and determines each client corresponding to the client set as a shared client;

[0013] The assistant server performs unbiased sampling in a preset public data set according to a preset sampling ratio, and generates a shared data set based on the sampled data samples.

[0014] In one embodiment, the sharing client constructs a local knowledge tensor according to its local model and the shared data set, including:

[0015] The sharing client inputs the shared data set into its local model, obtains the contribution of the sharing client to each data sample in the shared data set, and generates a local knowledge tensor according to the contribution of the sharing client to each data sample in the shared data set.

[0016] In one embodiment, the sharing client calculates the knowledge coefficient and data distribution difference with other sharing clients in the knowledge matrix according to the data contribution of each sharing client to determine the reputation weight matrix, including:

[0017] The sharing client quantifies the data distribution difference between the sharing client and other sharing clients by using KL divergence according to the data contribution of each sharing client in the knowledge matrix;

[0018] The sharing client quantifies the degree of knowledge difference between the sharing client and other sharing clients according to the data contribution of each sharing client in the knowledge matrix, so as to determine the knowledge coefficient between the client and other sharing clients;

[0019] The sharing client constructs an objective function according to the data distribution difference and the knowledge coefficient, and calculates a reputation weight matrix of the sharing client when the objective function is minimized.

[0020] In one embodiment, the objective function is expressed as:

[0021]

[0022] In the formula, Indicates a shared client The reputation weight matrix of shared clients Indicates the current shared client, shared client Indicates other shared clients, represents the knowledge coefficient, i.e., the shared client and shared clients For the data sample knowledge differences, represents the temperature coefficient, Represents the data difference distribution, Indicates a shared client For the data sample Contribution Representing a shared client For the data sample Contribution Indicates the number of shared clients, Represents the number of data samples.

[0023] In one embodiment, the sharing client weights the knowledge matrix with the reputation weight matrix, and updates the local model of the sharing client according to the weighted knowledge matrix, including:

[0024] The sharing client performs a multiplication operation on the reputation weight matrix and the knowledge matrix to obtain a weighted knowledge matrix, then uses the weighted knowledge matrix as a soft target, and uses knowledge distillation to update the local model of the sharing client.

[0025] In one embodiment, the process of the client constructing a local model includes:

[0026] The client obtains a preset initial model, and generates a plurality of mask matrices satisfying preset conditions for the initial model;

[0027] The client multiplies the initial model by the parameters of each mask matrix respectively to obtain multiple candidate models, and calculates the L2 norm between each candidate model and the initial model to determine the target candidate model with the minimum L2 norm;

[0028] The client performs a loss value test on the target candidate model. If the loss value is less than a preset threshold, the target candidate model is determined as a local model. Otherwise, the target candidate model is fine-tuned and the fine-tuned target candidate model is determined as a local model.

[0029] In a second aspect, the present application provides a federated learning device based on knowledge sharing, which is applied to a collaborative training system. The system includes multiple clients and assistant servers. The device includes:

[0030] An information determination module, used for the assistant server to determine a shared client from a plurality of clients, and to determine a shared data set from a preset public data set;

[0031] A tensor construction module, used for the shared client to construct a local knowledge tensor according to its local model and the shared data set, and broadcast the local knowledge tensor to other shared clients;

[0032] A matrix determination module, used for the sharing client to generate a knowledge matrix according to the received local knowledge tensors of other sharing clients, and calculate the knowledge coefficient and data distribution difference with other sharing clients according to the data contribution of each sharing client in the knowledge matrix to determine the reputation weight matrix;

[0033] A model updating module is used for the shared client to weight the knowledge matrix with the reputation weight matrix, update the local model of the shared client according to the weighted knowledge matrix, and return to the assistant server to continue the step of determining the shared client among multiple clients until the preset communication conditions are met.

[0034] In a third aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the federated learning method based on knowledge sharing as described in any of the above embodiments.

[0035] In a fourth aspect, the present application provides a computer device, comprising: one or more processors, and a memory;

[0036] The memory stores computer-readable instructions, and when the one or more processors execute the computer-readable instructions, the steps of the federated learning method based on knowledge sharing as described in any one of the above embodiments are performed.

[0037] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0038] The federated learning method, device and computer equipment based on knowledge sharing provided by the present application, in a communication round of a collaborative training system composed of heterogeneous devices, the assistant server determines the shared client among multiple clients, and determines the shared data set in the public data set. The shared client constructs a local knowledge tensor based on its local model and shared data set, and broadcasts the constructed local knowledge tensor to other shared clients to achieve knowledge sharing. Then, the shared client generates a knowledge matrix based on the local knowledge tensors of other shared clients, and then calculates the knowledge coefficient and data distribution difference between other shared clients based on data contribution in the knowledge matrix, and then determines the reputation weight matrix. The knowledge matrix is ​​weighted based on the reputation weight matrix, and the local model is updated with the weighted knowledge matrix. This model update method does not require global aggregation operations, and only some clients participate in federated learning each time the communication is carried out to reduce dependence on communication resources, so that devices with limited resources can also participate in federated learning. At the same time, the shared client can determine the knowledge coefficient and data distribution difference based on the shared data set for knowledge sharing. In this way, the differences in data distribution and knowledge between different clients can be fully considered, and the available knowledge of other devices can be learned to the greatest extent to improve its own local model, thereby improving the adaptability of the local model to its application client and ensuring the performance and accuracy of the local models finally obtained by different clients. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A flowchart of a federated learning method based on knowledge sharing provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of a process for determining a reputation weight matrix provided in an embodiment of the present application;

[0042] Figure 3 A flowchart of a process for a client to construct a local model provided in an embodiment of the present application;

[0043] Figure 4 An example diagram of a client constructing a local model provided in an embodiment of the present application;

[0044] Figure 5 An example diagram of a federated learning method based on knowledge sharing provided in an embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of a federated learning device based on knowledge sharing provided in an embodiment of the present application;

[0046] Figure 7 An internal structure diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] In one embodiment, the present application provides a federated learning method based on knowledge sharing. The following embodiment is described by applying the method to a collaborative training system. The system includes an assistant server and multiple clients. It can be understood that the clients in the collaborative training system can be composed of different types of devices, wherein the clients can be mobile phones, sensors, computers, etc., which have certain computing resources. The present application does not make specific restrictions on this.

[0049] like Figure 1 As shown, the present application provides a federated learning method based on knowledge sharing, which is applied to a collaborative training system, the system including multiple clients and assistant servers, and the method includes:

[0050] S101: The assistant server determines a shared client from among multiple clients, and determines a shared data set from among preset public data sets.

[0051] In this step, when there are multiple different devices participating in federated learning and the resources of the devices are limited, a collaborative training system consisting of multiple clients and assistant servers can be determined. Then each client can build a local model according to its own situation. When the local model is built, multiple communications will begin. During each communication, the assistant server will determine the shared clients participating in knowledge sharing in this communication, and determine the shared data set from the public data set.

[0052] Specifically, when the assistant server determines a shared client among multiple clients, it can analyze and determine from one or more dimensions such as the client's resource availability, communication efficiency, data quality, client reputation, etc., and this application does not impose specific restrictions on this.

[0053] The assistant server is a node in the collaborative training system, which is responsible for selecting shared clients and determining shared data sets in each round of communication. A shared client refers to a client responsible for knowledge sharing among multiple clients in this communication. A shared data set refers to a set of data shared between various shared clients in this communication. In addition, to ensure the reliability of the shared data set, the public data set can be regularly maintained and updated.

[0054] S102: The shared client constructs a local knowledge tensor according to its local model and shared dataset, and broadcasts the local knowledge tensor to other shared clients.

[0055] In this step, each sharing client can build a local knowledge tensor based on its local model and the shared data determined in the previous step, and then pass the local knowledge tensor built by itself to other sharing clients. Based on this, each sharing client can obtain the local knowledge tensors of all other sharing clients, so as to subsequently build a knowledge matrix and realize knowledge sharing. It can be understood that other sharing clients refer to all sharing clients except the current sharing client in the sharing clients determined in this communication.

[0056] Among them, the local knowledge tensor is a structured representation of the logits output of the local model for each data sample in the shared data set, and the local knowledge tensor can be used to measure the contribution of the corresponding local model to each data sample in the shared data set. In addition, in this application, the logits output of the local model for each data sample in the shared data set can be referred to as knowledge.

[0057] S103: The sharing client generates a knowledge matrix according to the received local knowledge tensors of other sharing clients, and calculates the knowledge coefficient and data distribution difference with other sharing clients according to the data contribution of each sharing client in the knowledge matrix to determine the reputation weight matrix.

[0058] In this step, when a sharing client receives the local knowledge tensor broadcast by other sharing clients, it combines the received local knowledge tensor with its own local knowledge tensor to obtain a knowledge matrix, and then calculates the knowledge coefficient and data distribution difference between the current sharing client and other sharing clients based on the data contribution of each sharing client measured in the knowledge matrix, and then determines the reputation weight matrix based on the knowledge coefficient and data distribution difference.

[0059] Among them, data contribution refers to the contribution of the sharing client to each data sample in the shared data set. The knowledge coefficient is used to quantify the degree of knowledge difference between the sharing client and other sharing clients. It can measure the effect of the knowledge shared by other sharing clients on the performance improvement of the local model of the sharing client itself. The data distribution difference is used to measure the degree of local data difference between different clients. The reputation weight matrix can evaluate the quality of the knowledge shared by other sharing clients. It is used to screen the knowledge shared by other sharing clients to ensure the quality and reliability of knowledge transfer.

[0060] S104: The shared client weights the knowledge matrix with the reputation weight matrix, updates the local model of the shared client according to the weighted knowledge matrix, and returns to the assistant server to continue the step of determining the shared client among multiple clients until the preset communication conditions are met.

[0061] The preset communication condition refers to the condition for stopping the communication iteration. Specifically, the preset communication condition can be set to reach the preset number of communications, or can be set to the difference between the parameters of the current model of each client and the local model of the previous communication is within a preset range. The preset communication condition can be flexibly set according to the actual situation of each client, and this application does not impose specific restrictions on this.

[0062] In this step, the sharing client can weight the knowledge matrix with its determined reputation weight matrix, and then use the weighted knowledge matrix to update the local model of the sharing client. This does not require global aggregation operations, reduces communication costs, and is more conducive to enhancing the privacy security of local data.

[0063] It is understandable that since the reputation weight matrix fully considers the knowledge differences and data distribution differences of different clients when determining it, the knowledge differences are mainly reflected in the knowledge relevance, and the influence of irrelevant knowledge on the local model update is avoided as much as possible. The weighted knowledge matrix is ​​obtained by weighting the knowledge matrix through such a reputation weight matrix. The knowledge matrix does not contain knowledge that is irrelevant to the current shared client and data with large differences. In this way, only knowledge that is closely related to the local model of the current shared client can be absorbed, which improves the convergence speed and the performance of the final model.

[0064] Furthermore, due to the diversity of local data distribution of different sharing clients, the knowledge matrix may also include some malicious information, such as false information that is intentionally shared. Through the analysis of the dimension of knowledge relevance, malicious information can also be removed from the knowledge matrix, thereby ensuring the reliability of the final local model.

[0065] The federated learning method, device and computer equipment based on knowledge sharing provided by the present application, in a communication round of a collaborative training system composed of heterogeneous devices, the assistant server determines the shared client among multiple clients, and determines the shared data set in the public data set. The shared client constructs a local knowledge tensor based on its local model and shared data set, and broadcasts the constructed local knowledge tensor to other shared clients to achieve knowledge sharing. Then, the shared client generates a knowledge matrix based on the local knowledge tensors of other shared clients, and then calculates the knowledge coefficient and data distribution difference between other shared clients based on data contribution in the knowledge matrix, and then determines the reputation weight matrix. The knowledge matrix is ​​weighted based on the reputation weight matrix, and the local model is updated with the weighted knowledge matrix. This model update method does not require global aggregation operations, and only some clients participate in federated learning each time the communication is carried out to reduce dependence on communication resources, so that devices with limited resources can also participate in federated learning. At the same time, the shared client can determine the knowledge coefficient and data distribution difference based on the shared data set for knowledge sharing. In this way, the differences in data distribution and knowledge between different clients can be fully considered, and the available knowledge of other devices can be learned to the greatest extent to improve its own local model, thereby improving the adaptability of the local model to its application client and ensuring the performance and accuracy of the local models finally obtained by different clients.

[0066] In one embodiment, the assistant server determines a shared client from among multiple clients, and determines a shared data set from among preset public data sets, including:

[0067] S1: The assistant server records the communication delay of each client when communicating with each client, and performs cluster analysis on each client according to the communication delay to obtain multiple client sets.

[0068] In this step, the assistant server will record the communication delay of each client at each communication, so that before the next communication starts, each client can be clustered and analyzed based on the recorded communication delay of each client to obtain multiple client sets with different communication delay levels.

[0069] Specifically, the number of cluster centers can be determined according to actual needs during cluster analysis. For example, when the overall training time consumption is highly required, more cluster centers can be considered, so that each cluster center contains fewer clients. When more clients are required to participate more fully, fewer clusters can be considered, so that each cluster center contains more clients.

[0070] S2: The assistant server selects a client set with the lowest communication delay from multiple client sets, and determines each client corresponding to the client set as a shared client.

[0071] In this step, since the assistant server dynamically determines multiple client sets in each communication round, and then selects the client set with the lowest communication delay from the latest determined multiple client sets, and determines each client corresponding to the client set as a shared client, this can effectively reduce the overall training time and improve the efficiency of collaborative training.

[0072] S3: The assistant server performs unbiased sampling in the preset public data set according to the preset sampling ratio, and generates a shared data set based on the sampled data samples.

[0073] In this step, the assistant server can perform unbiased sampling in the preset public data set according to the preset sampling ratio, and then use the sampled data samples to form a shared data set. Unbiased sampling can ensure the fairness and reliability of sampling as much as possible. Among them, the preset sampling ratio can be set and adjusted according to actual conditions. In an example, the preset sampling ratio can be set to 10%.

[0074] In one embodiment, the sharing client constructs a local knowledge tensor based on its local model and the shared dataset, including:

[0075] The sharing client inputs the shared data set into its local model, obtains the contribution of the sharing client to each data sample in the shared data set, and generates a local knowledge tensor based on the contribution of the sharing client to each data sample in the shared data set.

[0076] In this embodiment, the sharing client can input each data sample in the shared data set into its current local model to obtain the logits output of the local model for each data sample, that is, the contribution of the sharing client to each data sample in the shared data set. Finally, the contribution of the sharing client to each data sample in the shared data set is combined to obtain a local knowledge tensor. It can be understood that the shared data set is used to generate knowledge that is representative of the sharing client to determine the reliability of subsequent knowledge sharing.

[0077] Exemplarily, the local knowledge tensor can be expressed as:

[0078]

[0079] In the formula, Indicates a shared client In the communication round The local knowledge tensor in , Indicates a shared client For the data sample Contribution, i.e. shared client Local data for data samples The logits output.

[0080] like Figure 2 As shown, in one embodiment, the sharing client calculates the knowledge coefficient and data distribution difference with other sharing clients in the knowledge matrix according to the data contribution of each sharing client to determine the reputation weight matrix, including:

[0081] S201: The sharing client quantifies the data distribution difference between the sharing client and other sharing clients based on the data contribution of each sharing client in the knowledge matrix and using KL divergence.

[0082] In this step, the sharing client calculates the KL divergence between the sharing client and other sharing clients based on the data contribution of each sharing client measured in the knowledge matrix to quantify the data distribution difference between the sharing client and other sharing clients.

[0083] In one example, the knowledge matrix can be represented as:

[0084]

[0085] In the formula, Indicates a shared client In the communication round The knowledge matrix in Indicates a shared client In the communication round The local knowledge tensor in , Indicates the number of shared clients, Indicates a shared client For the data sample Contribution Indicates the number of data samples in the shared data set. Specifically, each row may represent the contribution of different sharing clients to the same data sample, and each column may represent the contribution of the same sharing client to different data samples.

[0086] Then, the relationship between shared client c and other shared clients i in data samples can be quantified according to the following expression: The data distribution difference on the data sample For example:

[0087]

[0088] In the formula, represents the data distribution difference between the shared client c and other shared clients i, Indicates a shared client For the data sample Contribution Representing a shared client For the data sample Contribution Indicates the number of shared clients, represents the temperature coefficient, Represents the softmax function.

[0089] S202: The sharing client quantifies the degree of knowledge difference between the sharing client and other sharing clients according to the data contribution of each sharing client in the knowledge matrix, so as to determine the knowledge coefficient between the client and other sharing clients.

[0090] In this step, the knowledge difference between a shared client and other shared clients is quantified to determine the value knowledge coefficient between the shared client and other shared clients. This fully considers the knowledge differences and correlations between different clients and minimizes the impact of irrelevant or malicious knowledge on local model updates.

[0091] In one example, the knowledge coefficient can be calculated as follows:

[0092]

[0093] In the formula, represents the knowledge coefficient, which represents the shared data sample between client c and client i The degree of knowledge difference, represents the reputation evaluation of shared client c on shared client i in communication round t, Represents shared client c and knowledge Among them, The expression can be expressed as follows:

[0094]

[0095]

[0096] In the formula, is an adjustable weight, indicating the importance of the knowledge of other shared clients. To quantify knowledge differences, Indicates the length of knowledge, Representing Knowledge The kth element in .

[0097] in, The expression can be expressed as follows:

[0098]

[0099] S203: The sharing client constructs an objective function according to the data distribution difference and the knowledge coefficient, and calculates the reputation weight matrix of the sharing client when the objective function is minimized.

[0100] Among them, the optimization goal of the objective function is to minimize the knowledge difference and data distribution difference.

[0101] In this embodiment, irrelevant or highly different knowledge or data is removed by minimizing knowledge differences and data distribution differences, and knowledge from other shared clients that is beneficial to improving its own model is fully absorbed, thereby accelerating the convergence of local model training and also improving the accuracy and personalization of the local model.

[0102] In one embodiment, the objective function is expressed as:

[0103]

[0104] In the formula, Indicates a shared client The reputation weight matrix of shared clients Indicates the current shared client, shared client Indicates other shared clients, represents the knowledge coefficient, i.e., the shared client and shared clients For the data sample knowledge differences, represents the temperature coefficient, represents the data difference distribution, Indicates a shared client For the data sample Contribution Representing a shared client For the data sample Contribution Indicates the number of shared clients, Represents the number of data samples.

[0105] In one embodiment, the sharing client weights the knowledge matrix with a reputation weight matrix, and updates the local model of the sharing client according to the weighted knowledge matrix, including:

[0106] The sharing client multiplies the reputation weight matrix and the knowledge matrix to obtain the weighted knowledge matrix, then uses the weighted knowledge matrix as a soft target and uses knowledge distillation to update the local model of the sharing client.

[0107] Among them, soft target refers to the target distribution used in the knowledge distillation process, which is used to guide the learning direction of the local model so that the shared client can absorb as much knowledge from other shared clients as possible that is beneficial to improving its own model.

[0108] In this embodiment, the sharing client first performs an element-by-element multiplication operation on the reputation weight matrix and the local knowledge matrix to generate a weighted knowledge matrix. This process filters out high-quality, highly relevant knowledge through reputation weights, while suppressing irrelevant or low-quality information. Subsequently, the sharing client uses the weighted knowledge matrix as a soft target and updates the local model using knowledge distillation technology. Specifically, the sharing client can calculate the KL divergence loss between the probability distribution of its local model output and the soft target, and optimize the model parameters through back propagation. For example, assume that sharing client A weights its knowledge matrix by the reputation weight matrix in a round of collaborative learning to obtain a soft target. Sharing client A then uses the soft target as a distillation target to adjust the parameters of its local model so that its output is closer to the weighted knowledge matrix. This process not only effectively absorbs the knowledge of other clients, but also improves the generalization ability of the model through the smoothness of the soft target.

[0109] like Figure 3 As shown, in one embodiment, the process of the client building a local model includes:

[0110] S301: The client obtains a preset initial model, and generates a plurality of mask matrices satisfying preset conditions for the initial model.

[0111] Among them, the preset condition can be expressed as:

[0112]

[0113] In the formula, represents the ratio of the computing resources currently available to client i to the computing resources required by the initial model, Indicates the proportion of computing resources that the pruned model is expected to use.

[0114]

[0115] In the formula, represents the average computing resources per unit time currently available to client i, represents the computational resources required for the initial model, represents the parameters of the pruned model, represents the parameters of the initial model, Indicates the number of floating-point operations of the initial model, Indicates the number of floating-point operations of the pruned model.

[0116] In this step, a greedy search strategy can be used to generate multiple mask matrices that meet the above preset conditions for the initial model.

[0117] S302: The client multiplies the initial model by the parameters of each mask matrix to obtain multiple candidate models, and calculates the L2 norm between each candidate model and the initial model to determine the target candidate model with the minimum L2 norm.

[0118] Among them, the L2 norm is used to describe the difference between the alternative model and the initial model. The smaller the value of the L2 norm, the smaller the difference.

[0119] In this step, the client multiplies its initial model with each mask matrix element by element, and multiple candidate models are obtained after multiplication. The L2 norm between each candidate model and the initial model is then calculated to determine the candidate model corresponding to the minimum L2 range, that is, the target candidate model.

[0120] S303: The client performs a loss value test on the target candidate model to determine whether the loss value is less than a preset threshold.

[0121] In this step, a small batch of data can be used to test the loss value of the target candidate model relative to the initial model, and then determine whether the model needs to be fine-tuned based on whether the loss value is less than a preset threshold.

[0122] S304: If the loss value is less than a preset threshold, the target candidate model is determined as the local model.

[0123] S305: If the loss value is not less than the preset threshold, fine-tune the target candidate model and determine the fine-tuned target candidate model as the local model.

[0124] In one embodiment, the preset threshold may correspond to each client on a one-to-one basis and may be set based on factors such as available resources of a specific client.

[0125] In this embodiment, personalized local models are customized for devices with different resources through adaptive pruning of each client, which can improve the adaptability and convergence speed of the local model and its application client, and ensure the performance and accuracy of the local models finally obtained by different clients.

[0126] like Figure 4 As shown, Figure 4 This is an example diagram of a client constructing a local model provided in an embodiment of the present application. Figure 4In the formula, loss check refers to judging whether the loss value is less than the preset threshold, st represents the preset condition, Th represents the preset threshold, Loss represents the loss value, the formula where min is located represents selecting the candidate model with the smallest L2 norm, the original model represents the initial model, and model pruning represents the process of determining the target candidate model.

[0127] like Figure 5 As shown, Figure 5 This is an example diagram of a federated learning method based on knowledge sharing provided in an embodiment of the present application. Figure 5 In , the selected client is the determined client set, the sampling ID can determine the public data set, the cooperative client represents the shared client, and Logits represents the output of the local model, that is, knowledge.

[0128] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0129] The following is a description of a federated learning device based on knowledge sharing provided in an embodiment of the present application. The federated learning device based on knowledge sharing described below and the federated learning method based on knowledge sharing described above can refer to each other.

[0130] like Figure 6 As shown, the present application provides a federated learning device 400 based on knowledge sharing, which is applied to a collaborative training system. The system includes multiple clients and assistant servers. The device includes:

[0131] The information determination module 401 is used to assist the server in determining a shared client among multiple clients and to determine a shared data set among a preset public data set;

[0132] A tensor construction module 402 is used for a shared client to construct a local knowledge tensor according to its local model and shared data set, and broadcast the local knowledge tensor to other shared clients;

[0133] Matrix determination module 403, used for the sharing client to generate a knowledge matrix according to the received local knowledge tensors of other sharing clients, and calculate the knowledge coefficient and data distribution difference with other sharing clients according to the data contribution of each sharing client in the knowledge matrix to determine the reputation weight matrix;

[0134] The model update module 404 is used for the shared client to weight the knowledge matrix with the reputation weight matrix, update the local model of the shared client according to the weighted knowledge matrix, and return to the assistant server to continue the step of determining the shared client among multiple clients until the preset communication conditions are met.

[0135] In one embodiment, the information determination module includes:

[0136] A set determination submodule is used to help the server record the communication delay of each client when communicating with each client, and perform cluster analysis on each client according to the communication delay to obtain multiple client sets;

[0137] The client determination submodule is used to assist the server in selecting a client set with the lowest communication delay from multiple client sets, and determine each client corresponding to the client set as a shared client;

[0138] The data sampling submodule is used to assist the server in performing unbiased sampling in a preset public data set according to a preset sampling ratio, and to generate a shared data set based on the sampled data samples.

[0139] In one embodiment, the tensor building block includes:

[0140] The tensor generation submodule is used for the shared client to input the shared data set into its local model, obtain the contribution of the shared client to each data sample in the shared data set, and generate a local knowledge tensor based on the contribution of the shared client to each data sample in the shared data set.

[0141] In one embodiment, the matrix determination module includes:

[0142] The difference calculation submodule is used for the sharing client to quantify the data distribution difference between the sharing client and other sharing clients based on the data contribution of each sharing client in the knowledge matrix and using KL divergence;

[0143] A coefficient calculation submodule, which is used for the sharing client to quantify the degree of knowledge difference between the sharing client and other sharing clients according to the data contribution of each sharing client in the knowledge matrix, so as to determine the knowledge coefficient between the client and other sharing clients;

[0144] The matrix determination submodule is used for the shared client to construct the objective function according to the data distribution difference and the knowledge coefficient, and calculate the reputation weight matrix of the shared client when the objective function is minimized.

[0145] In one embodiment, the model updating module includes:

[0146] The model update submodule is used for the shared client to multiply the reputation weight matrix and the knowledge matrix to obtain the weighted knowledge matrix, and then use the weighted knowledge matrix as a soft target and use knowledge distillation to update the local model of the shared client.

[0147] In one embodiment, the federated learning device based on knowledge sharing further includes:

[0148] A matrix generation module is used for the client to obtain a preset initial model and generate multiple mask matrices that meet preset conditions for the initial model;

[0149] The norm calculation module is used for the client to multiply the initial model with the parameters of each mask matrix respectively to obtain multiple candidate models, and calculate the L2 norm between each candidate model and the initial model to determine the target candidate model with the minimum L2 norm;

[0150] The model determination module is used by the client to test the loss value of the target candidate model. If the loss value is less than a preset threshold, the target candidate model is determined as the local model. Otherwise, the target candidate model is fine-tuned and the fine-tuned target candidate model is determined as the local model.

[0151] The division of each module in the above-mentioned federated learning device based on knowledge sharing is only for illustration. In other embodiments, the federated learning device based on knowledge sharing can be divided into different modules as needed to complete all or part of the functions of the above-mentioned federated learning device based on knowledge sharing. Each module in the above-mentioned federated learning device based on knowledge sharing can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0152] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the federated learning method based on knowledge sharing as described in any of the above embodiments.

[0153] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the federated learning method based on knowledge sharing as described in any of the above embodiments.

[0154] Indicatively, Figure 7 As shown, Figure 7 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 500 may be provided as a server. Figure 7 , the computer device 500 includes a processing component 502, which further includes one or more processors, and a memory resource represented by a memory 501, for storing instructions executable by the processing component 502, such as an application. The application stored in the memory 501 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 502 is configured to execute instructions to perform the federated learning method based on knowledge sharing of any of the above embodiments.

[0155] The computer device 500 may further include a power supply component 503 configured to perform power management of the computer device 500, a wired or wireless network interface 504 configured to connect the computer device 500 to a network, and an input / output (I / O) interface 505. The computer device 500 may operate based on an operating system stored in the memory 501, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0156] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0157] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish an entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only include those elements, but also include other elements that are not clearly listed, or also include elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, the singular "one", "one" and "described / the" may also include plural forms, unless the context clearly indicates another way. It should also be understood that the terms "include / comprise" or "have" etc. specify the existence of stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the existence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0158] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0159] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A federated learning method based on knowledge sharing, characterized in that: Applied to a collaborative training system, the system includes multiple clients and an assistant server, and the method includes: The assistant server determines a shared client from among the multiple clients, and determines a shared data set from among the preset public data sets; The shared client constructs a local knowledge tensor according to its local model and the shared data set, and broadcasts the local knowledge tensor to other shared clients; The sharing client generates a knowledge matrix according to the received local knowledge tensors of other sharing clients, and calculates the knowledge coefficient and data distribution difference with other sharing clients according to the data contribution of each sharing client in the knowledge matrix to determine the reputation weight matrix; The shared client weights the knowledge matrix with the reputation weight matrix, updates the local model of the shared client according to the weighted knowledge matrix, and returns to the assistant server to continue the step of determining the shared client among multiple clients until the preset communication conditions are met.

2. The federated learning method based on knowledge sharing according to claim 1, characterized in that: The assistant server determines a shared client from among the multiple clients, and determines a shared data set from among the preset public data sets, including: The assistant server records the communication delay of each client when communicating with each client, and performs cluster analysis on each client according to the communication delay to obtain multiple client sets; The assistant server selects a client set with the lowest communication delay from multiple client sets, and determines each client corresponding to the client set as a shared client; The assistant server performs unbiased sampling in a preset public data set according to a preset sampling ratio, and generates a shared data set based on the sampled data samples.

3. The federated learning method based on knowledge sharing according to claim 1, characterized in that: The shared client constructs a local knowledge tensor according to its local model and the shared data set, including: The sharing client inputs the shared data set into its local model, obtains the contribution of the sharing client to each data sample in the shared data set, and generates a local knowledge tensor according to the contribution of the sharing client to each data sample in the shared data set.

4. The federated learning method based on knowledge sharing according to claim 1, characterized in that: The sharing client calculates the knowledge coefficient and data distribution difference with other sharing clients in the knowledge matrix according to the data contribution of each sharing client to determine the reputation weight matrix, including: The sharing client quantifies the data distribution difference between the sharing client and other sharing clients by using KL divergence according to the data contribution of each sharing client in the knowledge matrix; The sharing client quantifies the degree of knowledge difference between the sharing client and other sharing clients according to the data contribution of each sharing client in the knowledge matrix, so as to determine the knowledge coefficient between the client and other sharing clients; The sharing client constructs an objective function according to the data distribution difference and the knowledge coefficient, and calculates a reputation weight matrix of the sharing client when the objective function is minimized.

5. The federated learning method based on knowledge sharing according to claim 4 is characterized in that: The objective function is expressed as: In the formula, Indicates a shared client The reputation weight matrix of shared clients Indicates the current shared client, shared client Indicates other shared clients, represents the knowledge coefficient, i.e., the shared client and shared clients For the data sample knowledge differences, represents the temperature coefficient, Represents the data difference distribution, Indicates a shared client For the data sample Contribution Representing a shared client For the data sample Contribution Indicates the number of shared clients, Represents the number of data samples.

6. The federated learning method based on knowledge sharing according to claim 1, characterized in that: The sharing client weights the knowledge matrix with the reputation weight matrix, and updates the local model of the sharing client according to the weighted knowledge matrix, including: The sharing client performs a multiplication operation on the reputation weight matrix and the knowledge matrix to obtain a weighted knowledge matrix, then uses the weighted knowledge matrix as a soft target, and uses knowledge distillation to update the local model of the sharing client.

7. The federated learning method based on knowledge sharing according to any one of claims 1 to 6, characterized in that: The process of the client building a local model includes: The client obtains a preset initial model, and generates a plurality of mask matrices satisfying preset conditions for the initial model; The client multiplies the initial model by the parameters of each mask matrix respectively to obtain multiple candidate models, and calculates the L2 norm between each candidate model and the initial model to determine the target candidate model with the minimum L2 norm; The client performs a loss value test on the target candidate model. If the loss value is less than a preset threshold, the target candidate model is determined as a local model. Otherwise, the target candidate model is fine-tuned and the fine-tuned target candidate model is determined as a local model.

8. A federated learning device based on knowledge sharing, characterized in that: Applied to a collaborative training system, the system includes multiple clients and an assistant server, and the device includes: An information determination module, used for the assistant server to determine a shared client from a plurality of clients, and to determine a shared data set from a preset public data set; A tensor construction module, used for the shared client to construct a local knowledge tensor according to its local model and the shared data set, and broadcast the local knowledge tensor to other shared clients; A matrix determination module, used for the sharing client to generate a knowledge matrix according to the received local knowledge tensors of other sharing clients, and calculate the knowledge coefficient and data distribution difference with other sharing clients according to the data contribution of each sharing client in the knowledge matrix to determine the reputation weight matrix; A model updating module is used for the shared client to weight the knowledge matrix with the reputation weight matrix, update the local model of the shared client according to the weighted knowledge matrix, and return to the assistant server to continue the step of determining the shared client among multiple clients until the preset communication conditions are met.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the federated learning method based on knowledge sharing as described in any one of claims 1 to 7.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the federated learning method based on knowledge sharing as described in any one of claims 1 to 7 are performed.

Images