Multi-aggregation node federated knowledge distillation learning method, system, device and medium

Abstract

The present application provides a multi-aggregation node federated knowledge distillation learning method, system, device and medium. The method includes correcting the predicted probability of the client, obtaining the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node according to the calculated corrected probability; using the data volume of the public data set of each aggregation node as the weight, calculating the weighted average of the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node to obtain the updated model parameters of the aggregation node and the aggregation node update model; according to the aggregation node update model, obtaining the predicted probability of the aggregation node update model on the public data set and correcting it; based on knowledge distillation, training the personalized model of the client connected to the aggregation node according to the calculated corrected probability. Public data sets are established separately by multiple aggregation nodes to share the communication pressure and storage pressure.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a multi-aggregation node federated knowledge distillation learning method, system, device and medium. Background Art

[0002] The accuracy of federated knowledge distillation learning models relies on the public dataset at the central node. Related technologies fail to consider the pressure that the public dataset places on the central node's communication and storage capabilities. Consequently, the central node's communication and storage capabilities limit the size of the public dataset, hindering model convergence speed and impacting model accuracy. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a multi-aggregation node federated knowledge distillation learning method, system, device and medium.

[0004] Based on the above objectives, this application provides a multi-aggregation node federated knowledge distillation learning method, including:

[0005] Obtaining model parameters of the aggregation node and model parameters of adjacent aggregation nodes of the aggregation node;

[0006] Obtaining updated model parameters of the aggregation node and an updated model of the aggregation node according to the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node;

[0007] Based on knowledge distillation, the model is updated according to the aggregation node, and the personalized model of the client connected to the aggregation node is trained.

[0008] This application also provides a multi-aggregation node federated knowledge distillation learning system, including:

[0009] An aggregation node model parameter acquisition module is used to acquire the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node;

[0010] An aggregation node update model acquisition module, configured to obtain update model parameters of the aggregation node and an aggregation node update model according to the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node;

[0011] The knowledge distillation module is used to update the model according to the aggregation node based on knowledge distillation and train the personalized model of the client connected to the aggregation node.

[0012] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the above-mentioned method is implemented when the processor executes the program.

[0013] The present application also provides a non-transitory computer-readable storage medium, which stores computer instructions for causing a computer to execute the above method.

[0014] As can be seen from the above, the multi-aggregator federated knowledge distillation learning method, system, device, and medium provided in this application establish a common dataset through multiple aggregation nodes, sharing communication and storage pressures. This reduces the size limit of the common dataset, thereby improving model convergence speed and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are merely embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 This is a flow chart of the multi-aggregation node federated knowledge distillation learning method according to an embodiment of the present application.

[0017] Figure 2 This is a structural diagram of the multi-aggregation node federated knowledge distillation learning system of an embodiment of the present application.

[0018] Figure 3 This is a schematic diagram of the hardware structure of the server according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0020] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which the present application belongs. "Include" or "comprising" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0021] Federated learning is a distributed framework that decouples data and models, addressing data silos and privacy concerns. The entire process eliminates the need to centralize data from participating parties to a central storage point, enabling joint modeling among all participants without leaving the local data store. In a federated learning framework, there is typically a central computational entity responsible for collecting model parameter information from other parties, updating the algorithm accordingly, and returning tasks to all parties. This process is repeated until convergence, ultimately constructing an effective global model. Throughout this process, neither the client nor the server has access to or control of other clients' data. The construction of the federated learning model does not affect the normal operation of client devices. Trained federated learning models can be shared and deployed among all data participants, offering broad application prospects in areas such as smart healthcare, finance and insurance, and the intelligent Internet of Things.

[0022] The accuracy of federated knowledge distillation learning models relies on the public dataset at the central node. Related technologies fail to consider the pressure that the public dataset places on the central node's communication and storage capabilities. Consequently, the central node's communication and storage capabilities limit the size of the public dataset, hindering model convergence speed and impacting model accuracy.

[0023] Based on the above-mentioned defects of the related art, the embodiments of the present application provide a multi-aggregation node federated knowledge distillation learning method, system, device and medium.

[0024] This application provides a multi-aggregator federated knowledge distillation learning method, system, device, and medium. By establishing a common dataset across multiple aggregation nodes, the system shares communication and storage pressures. This reduces restrictions on the size of the common dataset, thereby accelerating model convergence and improving model accuracy.

[0025] Figure 1 A flow chart of the multi-aggregation node federated knowledge distillation learning method provided in an embodiment of the present application is shown.

[0026] like Figure 1 As shown, the embodiment of the present application provides a multi-aggregation node federated knowledge distillation learning method, including:

[0027] Step S101: Obtain model parameters of an aggregation node and model parameters of adjacent aggregation nodes of the aggregation node.

[0028] In this embodiment, all aggregation nodes have the same model structure, so the structure of the model parameters of all aggregation nodes is also the same, and the model parameters can be transmitted between the aggregation nodes for parameter optimization.

[0029] As an optional embodiment, the federated knowledge distillation learning method includes obtaining the model parameters of the aggregation node by the following method:

[0030] Divide client data into public data and private data.

[0031] In this embodiment, the ratio of public data to private data is preset, and the client can select public data randomly or manually.

[0032] The public data set is obtained based on the public data of all clients connected to the aggregation node.

[0033] In this embodiment, the client can select the aggregator to connect to. When selecting an aggregator, the client can be presented with the connection parameters between the client and the aggregator. The client can then select the aggregator to connect to based on the connection parameters. In a specific implementation, the connection parameters may include packet loss rate, uplink and downlink speeds, and network latency.

[0034] The private data is used as a training set to train the initial model of the client to obtain the first client model.

[0035] In this embodiment, an initial model list is pre-set, and the initial model list includes multiple initial models. The model parameters of the multiple initial models have different numbers of parameters. Therefore, each initial model requires a different device computing power. The client can select the initial model to use based on its own device computing power.

[0036] The public data set is predicted using the first client model to obtain a first predicted probability distribution corresponding to each data in the public data set.

[0037] The first predicted probability distribution corresponding to each data is represented as a row vector, and then the row vectors corresponding to all data are used as elements in the column vector to obtain the first predicted probability.

[0038] The first predicted probability is corrected and calculated to obtain a first corrected probability.

[0039] The first revised probabilities of all clients connected to the aggregation node are aggregated and calculated to obtain an aggregate probability.

[0040] In this embodiment, the accuracy of the first predicted probability is improved by performing a correction calculation, resulting in a more accurate first revised probability. The aggregate probability is then calculated by averaging the first revised probabilities for all clients. This aggregate probability, which incorporates the first revised probabilities of all clients connected to the aggregate node, can be used to train the aggregate node model. The aggregate node model trained based on the aggregate probability incorporates information from the first client model.

[0041] The loss function is set according to the aggregation probability, and the public dataset is used as the training set to train the initial model of the aggregation node to obtain the aggregation node model.

[0042] In this embodiment, the loss function can be as follows:

[0043] Loss i =λ(-Y i log(p i (X i ,l,w i )))+(1-λ)KL(P avg (X i ,l)||p i (X i ,l,w i ))

[0044] Among them, i is the serial number of the aggregation node in the multi-aggregation node, Loss i is the loss function of the i-th aggregation node, λ is a parameter, which is a hyperparameter and is tuned according to the convergence of the loss function during training. label is the number of categories, l is 1×N consisting of label categories label vector, p i (X i ,l,w i ) is the predicted probability obtained by the aggregation node model of the i-th aggregation node for predicting the public data set, X i is a collection of data from public datasets, Y i is the collection of labels corresponding to the data, w i is the model parameter of the aggregation node model of the i-th aggregation node, P avg (X i ,l) is the aggregation probability, KL(P avg (X i ,l)||p i (X i ,l,w i )) is used to calculate the aggregation probability and p i (X i ,l,w i ) between the KL divergence.

[0045] When training the aggregate node model using the aforementioned loss function, knowledge distillation is applied. Knowledge distillation is a common method for model compression. It involves building a lightweight, smaller model and training it using the supervisory information from a larger, higher-performing model to achieve better performance and accuracy. This larger model is called the Teacher model, and the smaller model is called the Student model. The supervisory information output by the Teacher model is called knowledge, and the process by which the student learns to transfer this supervisory information from the teacher is called distillation.

[0046] In this way, by setting the aggregation probability in the loss function, when the loss function converges, p i (X i ,l,w i ) is close to the aggregation probability, and the knowledge in the aggregation probability is distilled, so that the trained aggregation node model is close to the first client model of all clients connected to the aggregation node.

[0047] According to the aggregation node model, the model parameters of the aggregation node are obtained.

[0048] In this embodiment, the model parameters of the aggregation node model when the loss function converges are extracted as the model parameters of the aggregation node.

[0049] Step S102: Obtain updated model parameters of the aggregate node and an updated model of the aggregate node according to the model parameters of the aggregate node and the model parameters of the adjacent aggregate nodes of the aggregate node.

[0050] In this way, updating the model parameters integrates the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes, thereby improving the fitting degree of the model parameters and further improving the performance of the aggregation node model.

[0051] As an optional embodiment, step S102 may include:

[0052] The weighted average of the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node is calculated to obtain the updated model parameters of the aggregation node; wherein the weight of each model parameter is the data volume of the public data set of the corresponding aggregation node.

[0053] In this embodiment, before calculating and updating the model parameters, a topology matrix can be established to depict the connection relationship between multiple aggregation nodes. The number of rows and columns in the topology matrix corresponds to the serial number of the aggregation node. If two aggregation nodes are connected, the intersection position of the two aggregation nodes in the topology matrix is set to 1. If the two aggregation nodes are not connected, the intersection position of the two aggregation nodes in the topology matrix is set to 0, and the intersection position of the aggregation node itself in the topology matrix is set to 1, that is, the main diagonal lines in the topology matrix are all set to 1. For example, if aggregation node 1 is connected to aggregation node 2, in the topology matrix T, T 12 =T 21 =1; aggregation node 1 and aggregation node 3 are not connected. In the topology matrix T, T 13 =T 31 =0; and, T 11 =T 22 =……=T MM =1.

[0054] When calculating and updating model parameters, the following formula can be used:

[0055]

[0056] Among them, w i ′ is the updated model parameter of the i-th aggregation node, M is the total number of aggregation nodes, N m is the amount of public data set of the mth aggregation node, t im is the value in the topology matrix, representing the connection relationship between the i-th aggregation node and the m-th aggregation node, w m is the model parameter of the mth aggregation node.

[0057] In this way, when calculating the updated model parameters, only the above general formula and topology matrix need to be used to calculate the updated model parameters of all aggregation nodes, without having to calculate each aggregation node separately, which simplifies the calculation process and improves efficiency.

[0058] Step S103: Based on knowledge distillation, update the model according to the aggregation node and train the personalized model of the client connected to the aggregation node.

[0059] As an optional embodiment, step S103 may include:

[0060] The aggregation node update model is used to predict the public data set to obtain a second prediction probability distribution corresponding to each data in the public data set.

[0061] The second predicted probability distribution corresponding to each data is represented as a row vector, and the row vectors corresponding to all data are used as elements in the column vector to obtain the second predicted probability.

[0062] The second predicted probability is corrected and calculated to obtain a second corrected probability.

[0063] A loss function is set according to the second corrected probability, and the public data set is used as a training set to train the first client model to obtain a second client model.

[0064] In this embodiment, the loss function can be as follows:

[0065] Loss ij =λ(-Y i log(p ij (X i ,l,w ij )))+(1-λ)KL(P i (X i ,l,w i )||p ij (X i ,l,w ij ))

[0066] Among them, j is the serial number of the client connected to the i-th aggregation node, Loss ij is the loss function of the jth client connected to the i-th aggregation node, N label is the number of categories, l is 1×N consisting of label categories label vector, p ij (X i ,l,w ij ) is the predicted probability obtained by the second client model of the jth client on the public dataset, w ij is the model parameter of the second client model of the jth client, P i (X i ,l,w i ) is the second modified probability, KL(P i (X i ,l,w i )||p ij (X i ,l,w ij )) is used to calculate the second modified probability and p ij (X i ,l,w ij ) between the KL divergence.

[0067] In this way, by setting the second modified probability in the loss function, when the loss function converges, p ij (X i ,l,w ij) is close to the second revised probability, and the knowledge in the second revised probability is distilled, so that the trained second client model is close to the aggregation node update model of the aggregation node connected to the client, so that the second client model learns the knowledge of the common data of multiple clients.

[0068] The private data is used as a training set to train the second client model to obtain a personalized model for the client.

[0069] In this embodiment, the loss function can be as follows:

[0070] Loss ij ′=-Y ij log(p ij (X ij ,l,w ij ′))

[0071] Among them, Loss ij ′ is the loss function of the jth client connected to the i-th aggregation node, N label is the number of categories, l is 1×N consisting of label categories label vector, p ij (X ij ,l,w ij ′) is the predicted probability obtained by the personalized model of the jth client for predicting private data, X ij is the collection of data in private data, w ij ′ is the model parameter of the personalized model of the j-th client.

[0072] In this way, the second client model is trained using private data to obtain a personalized model that better fits the client's data.

[0073] Considering that when the first client model fails to predict the public data set, the probability vector corresponding to the failed prediction will have an adverse effect on the aggregation node training model, it is necessary to correct the probability vector corresponding to the failed prediction.

[0074] As an optional embodiment, using the first client model to predict the public data set to obtain a first predicted probability; and performing a correction calculation on the first predicted probability to obtain a first corrected probability, including:

[0075] Predicting the public dataset using the first client model to obtain a first model output and a first prediction probability.

[0076] Compare the first model output with a public dataset.

[0077] In this embodiment, each of the first model outputs is compared with a corresponding label in a public dataset.

[0078] In response to determining that a label of a data in the public dataset does not match a label of the data in the first model output, a corresponding row vector in the first predicted probability is corrected using a one-hot encoding of the data to obtain a first corrected probability.

[0079] One-hot encoding, also known as single-bit encoding, uses an N-bit state register to encode N states. Each state has its own independent register bit, and at any time, only one of the bits is valid. That is, only one bit is 1, and the rest are zero.

[0080] In specific implementation, the first corrected probability can be obtained by the following formula:

[0081] P ij (X i ,l,w′ ij )=p ij (X i ,l,w′ ij )*mask ij +onehot(Y i )*(I-mask ij ),

[0082]

[0083] (k=1,2,3,…,N i )

[0084] Among them, N label is the number of categories, l is 1×N consisting of label categories label vector, P ij (X i ,l,w ij ′) is the first modified probability, p ij (X i ,l,w ij ′) is the first predicted probability, w ij ' is the model parameter of the first client model, I is a vector consisting of the identity matrix, I k =E(N label ), mask ij is the mask, onehot(Y i ) is one-hot encoding, For the kth data in the mask ij The first mask value in , is the output of the kth data in the first model output, is the kth data in the public dataset, is the label corresponding to the kth data.

[0085] In this way, by replacing the probability vector of the data that failed to be predicted with one-hot encoding, adverse effects on the training model of the aggregation node are avoided.

[0086] Considering that when the aggregation node update model fails to predict the public data set, the probability vector corresponding to the failed prediction will have an adverse effect on the second client model, it is necessary to correct the probability vector corresponding to the failed prediction.

[0087] As an optional embodiment, using the aggregation node update model to predict the public data set to obtain a second predicted probability; and performing a correction calculation on the second predicted probability to obtain a second corrected probability, including:

[0088] Use the aggregation node to update the model to predict the public data set and obtain the second model output and the second prediction probability.

[0089] Compare the second model output with a public dataset.

[0090] In this embodiment, each of the second model outputs is compared with a corresponding label in a public dataset.

[0091] In response to determining that a label of a data in the public dataset does not match a label of the data in the second model output, a corresponding row vector in the second predicted probability is corrected using a one-hot encoding of the data to obtain a second corrected probability.

[0092] In specific implementation, the second corrected probability can be obtained by the following formula:

[0093] P i (X i ,l,w′ i )=p i (X i ,l,w′ i )*mask i +onehot(Y i )*(I-mask i ),

[0094]

[0095] (k=1,2,3,…,N i )

[0096] Among them, N label is the number of categories, l is 1×N consisting of label categories label vector, P i (X i ,l,w i′) is the second modified probability, p ij (X i ,l,w i ′) is the second predicted probability, w i ′ is the updated model parameter, I is a vector composed of unit matrices, I k =E(N label ), mask i For the mask, For the kth data in the mask i The second mask value in , is the output of the kth data in the second model output.

[0097] In this way, by using one-hot encoding to replace the probability vector of the data that failed to be predicted, adverse effects on the second client model are avoided.

[0098] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a multi-aggregation node federated knowledge distillation learning system.

[0099] Figure 2 A schematic diagram of a multi-aggregation node federated knowledge distillation learning system according to an embodiment of the present application is shown.

[0100] refer to Figure 2 ,The multi-aggregation node federated knowledge distillation learning system includes:

[0101] The aggregation node model parameter acquisition module is used to acquire the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node.

[0102] The aggregation node update model acquisition module is used to obtain the update model parameters of the aggregation node and the aggregation node update model according to the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node.

[0103] Specifically, it is used to calculate the weighted average of the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes to obtain the updated model parameters of the aggregation node. The weight of each model parameter is the data volume of the public data set of the corresponding aggregation node.

[0104] The knowledge distillation module is used to update the model according to the aggregation node based on knowledge distillation and train the personalized model of the client connected to the aggregation node.

[0105] Specifically, it is used to use the aggregation node update model to predict the public dataset and obtain the second predicted probability distribution corresponding to each data in the public dataset. The second predicted probability distribution corresponding to each data is represented as a row vector, and the row vectors corresponding to all data are then used as elements in the column vector to obtain the second predicted probability. The second predicted probability is corrected and calculated to obtain a second corrected probability. A loss function is set based on the second corrected probability, and the public dataset is used as the training set to train the first client model to obtain the second client model. The private data is used as the training set to train the second client model to obtain a personalized model for the client.

[0106] As an optional embodiment, the multi-aggregation node federated knowledge distillation learning system further includes:

[0107] The aggregation node model parameter acquisition module is used to divide the client data into public data and private data. A public data set is obtained based on the public data of all clients connected to the aggregation node. The private data is used as the training set to train the initial model of the client to obtain the first client model. The first client model is used to predict the public data set to obtain the first predicted probability distribution corresponding to each data in the public data set. The first predicted probability distribution corresponding to each data is represented as a row vector, and then the row vectors corresponding to all data are used as elements in the column vector to obtain the first predicted probability. The first predicted probability is corrected and calculated to obtain the first corrected probability. The first corrected probabilities of all clients connected to the aggregation node are aggregated and calculated to obtain the aggregation probability. The loss function is set according to the aggregation probability, and the public data set is used as the training set to train the initial model of the aggregation node to obtain the aggregation node model and the model parameters of the aggregation node.

[0108] A first revised probability acquisition module is configured to use the first client model to predict the public dataset, obtain a first model output and a first predicted probability, compare the first model output with the public dataset, and, in response to determining that a data item in the public dataset does not match the output of the first model, correct the corresponding row vector in the first predicted probability using the one-hot encoding of the data item to obtain a first revised probability.

[0109] The aggregate probability acquisition module is used to calculate the average of the first revised probabilities of all clients to obtain the aggregate probability.

[0110] The second revised probability acquisition module is configured to use the aggregation node update model to predict the public dataset, obtain a second model output, and obtain a second predicted probability. The module compares the second model output with the public dataset. In response to determining that a data item in the public dataset does not match the output of the second model, the module uses a one-hot encoding of the data item to correct the corresponding row vector in the second predicted probability to obtain a second revised probability.

[0111] For the convenience of description, the above system is described as being divided into various modules according to their functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0112] The system of the above embodiment is used to implement the corresponding multi-aggregation node federated knowledge distillation learning method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0113] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the multi-aggregation node federated knowledge distillation learning method described in any of the above-mentioned embodiments is implemented.

[0114] Figure 3 A more specific server hardware structure diagram provided in this embodiment is shown. The server may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other within the device via the bus 1050.

[0115] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0116] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0117] The input / output interface 1030 is used to connect to an input / output module to implement information input and output. The input / output module can be configured as a component in the server (not shown in the figure) or can be externally connected to the server to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0118] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the server and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0119] The bus 1050 comprises a pathway for transmitting information between various components of the server (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0120] It should be noted that although the above electronic device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the electronic device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above electronic device may also include only the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figures.

[0121] The electronic device of the above embodiment is used to implement the corresponding multi-aggregation node federated knowledge distillation learning method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0122] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the multi-aggregation node federated knowledge distillation learning method as described in any of the above embodiments.

[0123] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0124] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the multi-aggregation node federated knowledge distillation learning method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0125] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0126] While the present application has been described in conjunction with specific embodiments thereof, many alternatives, modifications and variations of these embodiments will be apparent to those skilled in the art in light of the foregoing description.

[0127] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A multi-aggregation node federated knowledge distillation learning method, characterized by: include: Obtain model parameters of an aggregation node and model parameters of adjacent aggregation nodes of the aggregation node; wherein the model parameters of the aggregation node are obtained by the following method: Divide the client's data into public data and private data; Obtaining a public data set based on public data of all clients connected to the aggregation node; Using the private data as a training set to train an initial model of the client to obtain a first client model; Using the first client model to predict the public data set, obtaining a first prediction probability distribution corresponding to each data in the public data set; The first predicted probability distribution corresponding to each data is expressed as a row vector, and the row vectors corresponding to all data are used as elements in the column vector to obtain the first predicted probability; Performing a correction calculation on the first predicted probability to obtain a first corrected probability; Performing aggregation calculation on the first revised probabilities of all clients connected to the aggregation node to obtain an aggregate probability; Setting a loss function according to the aggregation probability, using the public data set as a training set to train an initial model of the aggregation node, and obtaining an aggregation node model and model parameters of the aggregation node; Obtaining updated model parameters of the aggregation node and an updated aggregation node model according to the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node; Based on knowledge distillation, a model is updated according to the aggregation node, and a personalized model of a client connected to the aggregation node is trained.

2. The multi-aggregation node federated knowledge distillation learning method according to claim 1, characterized in that: The obtaining, according to the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node, updated model parameters of the aggregation node includes: Calculate the weighted average of the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node to obtain the updated model parameters of the aggregation node. The calculation formula is as follows: Among them, w i ' is the updated model parameter of the i-th aggregation node, M is the total number of aggregation nodes, N m is the amount of public data set of the mth aggregation node, t im is the element in the i-th row and m-th column of the aggregation node connection relationship matrix, representing the connection relationship between the i-th aggregation node and the m-th aggregation node, w m is the model parameter of the mth aggregation node.

3. The multi-aggregation node federated knowledge distillation learning method according to claim 1, characterized in that: The method of updating the model based on the aggregation node and training the personalized model of the client connected to the aggregation node based on the knowledge distillation includes: Using the aggregation node update model to predict the public data set, to obtain a second prediction probability distribution corresponding to each data in the public data set; The second predicted probability distribution corresponding to each data is expressed as a row vector, and the row vectors corresponding to all data are used as elements in the column vector to obtain the second predicted probability; performing a correction calculation on the second predicted probability to obtain a second corrected probability; Setting a loss function according to the second corrected probability, and using the public data set as a training set to train the first client model to obtain a second client model; The second client model is trained using the private data as a training set to obtain a personalized model of the client.

4. The multi-aggregation node federated knowledge distillation learning method according to claim 1, characterized in that: using the first client model to predict the public dataset to obtain a first prediction probability; Performing a correction calculation on the first predicted probability to obtain a first corrected probability includes: Using the first client model to predict the public dataset, obtaining a first model output and a first prediction probability; Comparing the first model output with the public data set; in response to determining the kth data in the public data set Label With this data in the first model output The labels in the data do not match, so use the one-hot encoding of the data. Correct the corresponding row vector in the first predicted probability to obtain the first corrected probability, which can be expressed as follows: in, is the first modified probability, For data In the first corrected probability label label v The corresponding corrected probability value is, is the first predicted probability, For data Label label in the first predicted probability v The corresponding probability value, i is the serial number of the aggregation node, j is the serial number of the client connected to the i-th aggregation node, N label is the number of categories, l is 1×N composed of categories label vector, w ij ' is the model parameter of the first client model, is the first mask value of the kth data of the jth client under the i-th aggregation node in the public data set, and the first mask value is N when the data matches the output of the first model. label ×N label The identity matrix E(N label ), otherwise N label ×N label The zero matrix zeros(N label ).

5. The multi-aggregation node federated knowledge distillation learning method according to claim 1, characterized in that: The performing aggregation calculation on the first modified probabilities of all clients connected to the aggregation node to obtain the aggregated probability includes: An average of the first modified probabilities of all the clients is calculated to obtain the aggregate probability.

6. The multi-aggregation node federated knowledge distillation learning method according to claim 3, characterized in that: said using said aggregation node updating model to predict said public data set to obtain a second prediction probability; Performing a correction calculation on the second predicted probability to obtain a second corrected probability includes: Using the aggregation node update model to predict the public data set, to obtain a second model output and a second prediction probability; comparing the second model output with the public dataset; In response to determining the kth data in the public data set Label With this data the second model output The labels in the data do not match, so use the one-hot encoding of the data. Correct the corresponding row vector in the second predicted probability to obtain the second corrected probability, which can be expressed as follows: in, is the second modified probability, For data In the second modified probability label label v The corresponding corrected probability value is, is the second predicted probability, For data Label label in the second predicted probability v The corresponding probability value, i is the sequence number of the aggregation node in the multi-aggregation node, N label is the number of categories, l is 1×N consisting of label categories label vector, w i ' is the updated model parameter, is the second mask value of the kth data of the i-th aggregation node in the public data set, and the second mask value is N when the data matches the output of the second model. label ×N label The identity matrix E(N label ), otherwise N label ×N label The zero matrix zeros(N label ).

7. A multi-aggregation node federated knowledge distillation learning system, characterized by: include: The aggregation node model parameter acquisition module is used to obtain the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node; wherein the model parameters of the aggregation node are obtained by the following method: Divide the client's data into public data and private data; Obtaining a public data set based on public data of all clients connected to the aggregation node; Using the private data as a training set to train an initial model of the client to obtain a first client model; Using the first client model to predict the public data set, obtaining a first prediction probability distribution corresponding to each data in the public data set; The first predicted probability distribution corresponding to each data is expressed as a row vector, and the row vectors corresponding to all data are used as elements in the column vector to obtain the first predicted probability; Performing a correction calculation on the first predicted probability to obtain a first corrected probability; Performing aggregation calculation on the first revised probabilities of all clients connected to the aggregation node to obtain an aggregate probability; Setting a loss function according to the aggregation probability, using the public data set as a training set to train an initial model of the aggregation node, and obtaining an aggregation node model and model parameters of the aggregation node; An aggregation node update model acquisition module, configured to obtain update model parameters of the aggregation node and an aggregation node update model according to the model parameters of the aggregation node and the model parameters of the adjacent aggregation nodes of the aggregation node; The knowledge distillation module is used to update the model according to the aggregation node based on knowledge distillation and train the personalized model of the client connected to the aggregation node.

8. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the program.

9. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the method according to any one of claims 1 to 6.

Images