Government affair field-oriented model heterogeneous federal learning training method and device

Abstract

The embodiment of the invention relates to a model heterogeneous federal learning training method and device oriented to the government affair field, and relates to the technical field of privacy computing, and the method comprises the steps: each participant determines a local personalized model and a knowledge transmission model, and initializes each model parameter; each participant fixes the knowledge transmission model, trains the local personalized model based on the local holding data, fixes the local personalized model and trains the knowledge transmission model after completing the training of the local personalized model, and encrypts and sends the parameters of the knowledge transmission model to the coordinator after completing the training; after receiving the encryption parameter, the coordinator realizes aggregation calculation under the ciphertext, and issues an aggregation result to each participant to update a knowledge transfer model parameter; and repeatedly executing model training and model parameter updating until the local personalized model of each participant converges. By adopting the technical scheme, safe joint modeling in a model heterogeneous scene is realized, and the personalization of a local model of multiple participants is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of privacy computing technology, and in particular to a model heterogeneous federated learning training method and device for the government affairs field. Background Art

[0002] With the tightening of supervision on data security and personal privacy protection, and the vigorous promotion of the market-oriented allocation of data elements and the development and utilization of public data, the rapid development of privacy computing technology, which ensures that "the original data does not leave the domain and the data is available but not visible", has been promoted. Currently, privacy computing technology has been widely used in government affairs, finance, medical and other fields.

[0003] Specifically, privacy computing technology refers specifically to the technical means of implementing data analysis and computing under the premise of protecting the data itself from leakage. Among them, federated learning technology is one of the main technical paths to implement privacy computing.

[0004] In related technologies, federated learning assumes that the models of all participants in federated modeling are isomorphic. Each participant trains a local model through local aligned data, and then uploads the local model training parameters or intermediate results for aggregation. After aggregation, feedback is given to update the local model parameters to achieve multi-party joint modeling and improve the model training effect of each party. The above method requires each participant to expose the parameters or intermediate results of the local efficiency model, which poses certain security risks. In the real world, multiple participants participate in joint modeling with different goals. The existing method is difficult to meet the local personalized model training needs of all parties. If each participant gives priority to the personalization of the local model, it will destroy the generalization of the aggregated model, and it is difficult to support the needs of personalized joint modeling of multiple participants in the current public data development and utilization scenarios in the government field. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a model heterogeneous federated learning training method and device for the government affairs field, which realizes the secure joint modeling problem in model heterogeneous scenarios based on knowledge distillation and knowledge transfer model parameter security aggregation, and does not require a public data set to support knowledge transfer between heterogeneous models, thereby realizing multi-party personalized local model training and performance improvement. The present disclosure realizes a secure joint modeling method in multi-party model heterogeneity and data heterogeneity scenarios, which is suitable for public data development and utilization scenarios in the government affairs field.

[0006] The embodiment of the present disclosure provides a model heterogeneous federated learning training method for the government affairs field, including multiple participants, and any one of the multiple participants is used as a coordinator, each of the participants corresponds to a local personalized model and a knowledge transfer model, and the method includes:

[0007] Step S1: Each of the participants determines a local personalized model based on a local task goal, determines a knowledge transfer model based on the local personalized model, and determines initial values ​​of model parameters of the local personalized model and the knowledge transfer model; wherein the input data samples used to train the local personalized model and the knowledge transfer model are the same, and the model parameter dimension of the knowledge transfer model is smaller than the model parameter dimension of the local personalized model;

[0008] Step S2: Each of the participants controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on the local private data set samples of each of the participants, obtains the training loss value of the local personalized model based on a preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and a preset local training loss threshold;

[0009] Step S3, each of the participants controls the model parameters of the trained local personalized model to remain unchanged, trains the knowledge transfer model based on the local private data set sample, obtains the training loss value of the knowledge transfer model based on the preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and the preset knowledge training loss threshold;

[0010] Step S4: When the training loss value of the knowledge transfer model of each participant is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, the model parameters of the trained knowledge transfer model are encrypted based on a preset encryption algorithm and sent to the coordinator, so that the coordinator aggregates the model parameters of the knowledge transfer model sent by all the participants, obtains the aggregated model parameters and sends them to each participant;

[0011] Step S5: Each of the participants updates the model parameters of the knowledge transfer model based on the aggregation model parameters;

[0012] Step S6, repeat steps S2 to S5 until the training loss value of the local personalized model of each participant is less than the local training loss threshold or the number of local training times is equal to the preset local training times threshold, to obtain the local personalized model of each participant.

[0013] Optionally, the step S1, each of the participants determining a local personalized model based on a local task goal, determining a knowledge transfer model based on the local personalized model, and determining initial values ​​of model parameters of the local personalized model and the knowledge transfer model, includes:

[0014] Each of the participants determines a local personalized model based on the local task goal as f i (θ i ,X),i∈{1,…,n},where,θ i is the model parameter of the local personalized model of the i-th participant, X is the input data sample of the local personalized model, and n is the number of the participants;

[0015] The knowledge transfer model is determined based on the local personalized model: in, is the model parameter of the knowledge transfer model of the i-th participant, and X is the input data sample of the knowledge transfer model;

[0016] For the θ i and stated Perform parameter initialization processing to determine the θ i and stated The initial value of The dimension is smaller than the θ i .

[0017] Optionally, in step S2, each of the participants controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on the local private data set samples of each of the participants, obtains the training loss value of the local personalized model based on a preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and a preset local training loss threshold, including:

[0018] Each of the participants obtains a local private data set sample; wherein the local private data set sample of the i-th participant is represents the jth sample, m i Indicates the number of data items in the local private data set held by the i-th participant, represents the characteristics of the j-th sample data held by the i-th participant, represents the label of the jth sample data held by the i-th participant;

[0019] Each of the participants controls the model parameters of the knowledge transfer model to remain unchanged, inputs the local private data set sample into the local personalized model for processing, and obtains the training loss value of the local personalized model based on the loss function of the local personalized model; wherein the loss function of the local personalized model of the i-th participant is

[0020]

[0021] Among them, k i and They represent the weights of the local personalized model training loss function and the knowledge transfer model training loss function, respectively, and m i represents the number of data items in the local private data set of the i-th participant, Represents the characteristics of the jth sample data by the local personalized model of the i-th participant The label prediction value of Represents the characteristics of the j-sample data of the knowledge transfer model of the i-th participant The label pre-splash value, represents the loss function of the local personalized model of the participant, Represents the loss function of the knowledge transfer model;

[0022] When the training loss value of the local personalized model is greater than or equal to a local training loss threshold, a model parameter of the local personalized model is adjusted.

[0023] Optionally, in step S3, each of the participants controls the model parameters of the trained local personalized model to remain unchanged, trains the knowledge transfer model based on the local private data set sample, obtains the training loss value of the knowledge transfer model based on a preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and a preset knowledge training loss threshold, including:

[0024] Each of the participants controls the model parameters of the trained local personalized model to remain unchanged, and obtains the local private data set sample; wherein the local private data set sample of the i-th participant is represents the jth sample, mi represents the number of data in the local private dataset held by the i-th participant, represents the characteristics of the j-th sample data held by the i-th participant, represents the label of the jth sample data held by the i-th participant;

[0025] The local private data set sample is input into the knowledge transfer model for processing, and the training loss value of the knowledge transfer model is obtained based on the loss function of the knowledge transfer model; wherein the loss function of the knowledge transfer model of the i-th participant is

[0026]

[0027] in, and They represent the total loss function of the knowledge transfer model training and the weight of the knowledge distillation training loss function, respectively. i represents the number of data items in the local private data set of the i-th participant, Represents the characteristics of the jth sample data by the local personalized model of the i-th participant The label prediction value of Represents the characteristics of the j-sample data of the knowledge transfer model of the i-th participant The label prediction value of represents the loss function of the knowledge transfer model, Represents the loss function of knowledge distillation training, which is used to distill the knowledge in the local personalized model to the knowledge transfer model;

[0028] When the training loss value of the knowledge transfer model is greater than or equal to a knowledge training loss threshold, the model parameters of the knowledge transfer model are adjusted.

[0029] Optionally, in step S4, when the training loss value of the knowledge transfer model of each participant is less than the knowledge training loss threshold or the number of training times is equal to a preset knowledge training times threshold, the model parameters of the trained knowledge transfer model are encrypted based on a preset encryption algorithm and sent to the coordinator, so that the coordinator aggregates the model parameters of the knowledge transfer model sent by all the participants, obtains the aggregated model parameters and sends them to each participant, including:

[0030] When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, each of the participants encrypts the model parameters of the trained knowledge transfer model based on the preset homomorphic encryption algorithm, obtains the encrypted model parameters and sends them to the coordinator;

[0031] The coordinator aggregates all the encryption model parameters according to a preset aggregation formula, obtains the aggregated model parameters and sends them to each of the participants.

[0032] Optionally, in step S5, each of the participants updates the model parameters of the knowledge transfer model based on the aggregated model parameters, including:

[0033] Decrypting the aggregation model parameters based on a preset homomorphic decryption algorithm to obtain the decrypted aggregation model parameters;

[0034] And based on the number of all the participants and the decrypted aggregate model parameters, calculation is performed to obtain current model parameters, and the current model parameters are used as model parameters of the knowledge transfer model.

[0035] The embodiment of the present disclosure also provides a model heterogeneous federated learning training device for the government affairs field, including multiple participants, and any one of the multiple participants is used as a coordinator, each of the participants corresponds to a local personalized model and a knowledge transfer model, and the device includes:

[0036] A determination module, used to execute step S1, each of the participants determines a local personalized model based on the local task goal, determines a knowledge transfer model based on the local personalized model, and determines initial values ​​of model parameters of the local personalized model and the knowledge transfer model; wherein the input data samples used to train the local personalized model and the knowledge transfer model are the same, and the model parameter dimension of the knowledge transfer model is smaller than the model parameter dimension of the local personalized model;

[0037] A first training module, configured to execute step S2, wherein each of the participants controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on a local private data set sample of each of the participants, obtains a training loss value of the local personalized model based on a preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and a preset local training loss threshold;

[0038] A second training module, used to execute step S3, each of the participants controls the model parameters of the trained local personalized model to remain unchanged, trains the knowledge transfer model based on the local private data set sample, obtains the training loss value of the knowledge transfer model based on a preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and a preset knowledge training loss threshold;

[0039] An encryption sending module is used to execute step S4, when the training loss value of the knowledge transfer model of each participant is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, encrypt the model parameters of the trained knowledge transfer model based on a preset encryption algorithm and send them to the coordinator, so that the coordinator aggregates the model parameters of the knowledge transfer model sent by all the participants, obtains the aggregated model parameters and sends them to each participant;

[0040] An updating module, used to execute step S5, where each of the participants updates the model parameters of the knowledge transfer model based on the aggregation model parameters;

[0041] A processing module is used to execute step S6, and repeatedly execute steps S2 to S5 until the training loss value of the local personalized model of each participant is less than the local training loss threshold or the number of local training times is equal to a preset local training times threshold, to obtain a local personalized model of each participant.

[0042] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement a model heterogeneous federated learning training method for the government affairs field as provided in an embodiment of the present disclosure.

[0043] The embodiments of the present disclosure also provide a computer-readable storage medium storing a computer program for executing a model heterogeneous federated learning training method for the government affairs field as provided in the embodiments of the present disclosure.

[0044] The present disclosure embodiment also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the model heterogeneous federated learning training method for the government affairs field described in the aforementioned aspect.

[0045] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages: the model heterogeneous federated learning training solution for the government affairs field provided by the embodiments of the present disclosure includes multiple participants, and any participant among the multiple participants is used as a coordinator, each participant corresponds to a local personalized model and a knowledge transfer model, each participant determines the local personalized model based on the local task goal, and determines the knowledge transfer model based on the local personalized model, and determines the initial values ​​of the model parameters of the local personalized model and the knowledge transfer model; wherein, the input data samples used to train the local personalized model and the knowledge transfer model are the same, the model parameter dimension of the knowledge transfer model is smaller than the model parameter dimension of the local personalized model, each participant controls the model parameters of the knowledge transfer model to remain unchanged, the local personalized model is trained based on the local private data set samples of each participant, the training loss value of the local personalized model is obtained based on the preset loss function of the local personalized model, and the training loss value of the local personalized model and the preset loss function are obtained based on the training loss value of the local personalized model and the preset loss function. The local training loss threshold adjusts the model parameters of the local personalized model. Each participant controls the model parameters of the trained local personalized model to remain unchanged. The knowledge transfer model is trained based on the local private data set samples. The training loss value of the knowledge transfer model is obtained based on the preset loss function of the knowledge transfer model, and the model parameters of the knowledge transfer model are adjusted based on the training loss value of the knowledge transfer model and the preset knowledge training loss threshold. When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, each participant encrypts the model parameters of the trained knowledge transfer model based on a preset encryption algorithm and sends them to the coordinator, so that the coordinator aggregates the model parameters of the knowledge transfer model sent by all participants, obtains the aggregated model parameters and sends them to each participant. Each participant updates the model parameters of the knowledge transfer model based on the aggregated model parameters, and repeats the model training and model parameter update until the local personalized model of each participant converges. Therefore, the problem of secure joint modeling in model heterogeneous scenarios is realized based on knowledge distillation and secure aggregation of knowledge transfer model parameters. Different from the traditional implementation method based on knowledge distillation technology, the disclosed embodiment does not require a public data set to support knowledge transfer between heterogeneous models, and realizes multi-party personalized local model training and performance improvement, which is suitable for public data development and utilization scenarios in the government field. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0047] Figure 1A flowchart of a model heterogeneous federated learning training method for the government affairs field provided in an embodiment of the present disclosure;

[0048] Figure 2 A schematic diagram of the structure of a model heterogeneous federated learning training device for the government affairs field provided by an embodiment of the present disclosure;

[0049] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0051] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0052] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0053] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0054] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0055] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0056] Figure 1The present invention provides a flowchart of a model heterogeneous federated learning training method for the government affairs field, which can be performed by a model heterogeneous federated learning training device for the government affairs field, wherein the device can be implemented by software and / or hardware, and can generally be integrated in an electronic device. Figure 1 As shown, multiple participants are included, and any one of the multiple participants is used as a coordinator. Each participant corresponds to a local personalized model and a knowledge transfer model. The method includes:

[0057] Step S1, each participant determines a local personalized model based on the local task goal, determines a knowledge transfer model based on the local personalized model, and determines the initial values ​​of model parameters of the local personalized model and the knowledge transfer model; wherein the input data samples used to train the local personalized model and the knowledge transfer model are the same, and the model parameter dimension of the knowledge transfer model is smaller than the model parameter dimension of the local personalized model.

[0058] In some embodiments, each participant determines a local personalized model based on a local task goal as f i (θ i ,X), i∈{1,…,n}, where θ i is the model parameter of the local personalized model of the i-th participant, X is the input data sample of the local personalized model, and n is the number of the participants; the knowledge transfer model determined based on the local personalized model is: in, is the model parameter of the knowledge transfer model of the i-th participant, X is the input data sample of the knowledge transfer model; i and Perform parameter initialization and determine θ i and The initial value of The dimension is smaller than θ i .

[0059] For example, in a scenario in the government affairs field, the three participants are device A1 for representing organization A, device B1 for representing organization B, and device C2 for representing organization C. On device A1, a local personalized model a1 is determined based on local task objectives such as a text classification task, and a knowledge transfer model a2 is determined based on the local personalized model a1; on device B1, a local personalized model b1 is determined based on local task objectives such as an image recognition task, and a knowledge transfer model b2 is determined based on the local personalized model b1; on device C1, a local personalized model c1 is determined based on local task objectives such as a video target recognition task, and a knowledge transfer model c2 is determined based on the local personalized model c1.

[0060] Specifically, multiple participants select local personalized models based on local task objectives, and each participant selects a knowledge transfer model based on the local personalized model, and initializes model parameters of the local personalized models and knowledge transfer models of multiple participants; more specifically, for example, n participants select local personalized models f based on local task objectives, respectively. i (θ i ,X), i∈{1,…,n},θ i is the model parameter of the local personalized model of the i-th participant, X is the input of the local personalized model, and based on the local personalized model f selected by the n participants i (θ i ,X), select a knowledge transfer model suitable for knowledge transfer among multiple parties in, is the parameter of the knowledge transfer model of the ith participant, X is the input of the knowledge transfer model, The dimension is much smaller than θ i , initialize the model parameters θ of the local personalized models of n participants i , and the model parameters of the knowledge transfer model for multi-party knowledge transfer Select appropriate initial values ​​for the parameters.

[0061] Step S2: Each participant controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on the local private data set samples of each participant, obtains the training loss value of the local personalized model based on the preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and the preset local training loss threshold.

[0062] In some embodiments, each participant obtains a local private data set sample; wherein the local private data set sample of the i-th participant is represents the jth sample, m i Indicates the number of data items in the local private data set held by the i-th participant, represents the characteristics of the j-th sample data held by the i-th participant, represents the jth sample data label held by the i-th participant; each participant controls the model parameters of the knowledge transfer model to remain unchanged, inputs the local private data set samples into the local personalized model for processing, and obtains the training loss value of the local personalized model based on the loss function of the local personalized model; where the loss function of the local personalized model of the i-th participant is

[0063]

[0064] Among them, ki and They represent the weights of the local personalized model training loss function and the knowledge transfer model training loss function, respectively, and m i represents the number of data items in the local private data set of the i-th participant, Represents the characteristics of the jth sample data by the local personalized model of the i-th participant The label prediction value of Represents the characteristics of the j-sample data of the knowledge transfer model of the i-th participant The label prediction value of represents the loss function of the local personalized model of the participant, Represents the loss function of the knowledge transfer model; when the training loss value of the local personalized model is greater than or equal to the local training loss threshold, the model parameters of the local personalized model are adjusted.

[0065] In the embodiment of the present disclosure, a local training loss threshold is preset and can be selected and set according to the actual application scenario.

[0066] Specifically, multiple participants train local personalized models based on locally held data under the premise of fixing the model parameters of the knowledge transfer model. More specifically, n participants each hold a local private dataset sample, and n participants train the local personalized model f i (θ i ,X) is fixed for the knowledge transfer model Parameters

[0067] It is understandable that a local training loss threshold or a local training times threshold for each participant to train a local personalized model may be pre-set. The local training times threshold refers to the maximum number of times each participant trains a local personalized model in each round, and may be set according to actual application needs.

[0068] It can also be understood that the training loss value of the local personalized model can be calculated for each training. When the training loss value of the local personalized model is less than the local training loss threshold, it means that the local personalized model has converged and the training can be stopped. When the training loss value of the local personalized model is greater than or equal to the local training loss threshold, it is necessary to adjust the model parameters of the local personalized model and train again until the preset number of training times is reached or the model converges, thereby obtaining the local personalized model of each participant.

[0069] Step S3: Each participant controls the model parameters of the trained local personalized model to remain unchanged, trains the knowledge transfer model based on the local private data set samples, obtains the training loss value of the knowledge transfer model based on the preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and the preset knowledge training loss threshold.

[0070] In some embodiments, each participant controls the model parameters of the trained local personalized model to remain unchanged, and obtains a local private data set sample; wherein the local private data set sample of the i-th participant is represents the jth sample, m i Indicates the number of data items in the local private data set held by the i-th participant, represents the characteristics of the j-th sample data held by the i-th participant, represents the jth sample data label held by the i-th participant; the local private data set sample is input into the knowledge transfer model for processing, and the training loss value of the knowledge transfer model is obtained based on the loss function of the knowledge transfer model; where the loss function of the knowledge transfer model of the i-th participant is

[0071]

[0072] in, and They represent the total loss function of the knowledge transfer model training and the weight of the knowledge distillation training loss function, respectively. i represents the number of data items in the local private data set of the i-th participant, Represents the characteristics of the jth sample data by the local personalized model of the i-th participant The label prediction value of Represents the characteristics of the j-sample data of the knowledge transfer model of the i-th participant The label prediction value of represents the loss function of the knowledge transfer model, Represents the loss function of knowledge distillation training, which is used to distill the knowledge in the local personalized model to the knowledge transfer model; when the training loss value of the knowledge transfer model is greater than or equal to the knowledge training loss threshold, the model parameters of the knowledge transfer model are adjusted.

[0073] Specifically, after the multiple participants complete the local personalized model training, they train the knowledge transfer model. More specifically, after the multiple participants complete the local personalized model training, they fix the model parameters of the local personalized model. That is, n participants complete one or more rounds of local personalized model training based on the local private data set samples. i (θ i ,X) Fixed parameter θ after trainingi , completing one or more rounds refers to the local personalized model f i (θ i ,X) During the training process, one or more rounds of input training of private data set samples are completed; then n participants fix the model parameters of the local personalized model respectively and apply the loss function The local personalized model f i (θ i ,X) knowledge distillation to local knowledge transfer model for knowledge transfer

[0074] That is to say, after each participant determines the local personalized model and the knowledge transfer model, they initialize the model parameters of the local personalized model and the knowledge transfer model, and then keep the model parameters of the knowledge transfer model unchanged. The local personalized model is trained for a preset number of times based on the samples of the local private data set. During the training process, the loss function is used to calculate the loss function. The loss value can be calculated together with the model parameters of the knowledge transfer model, and the model parameters of the local personalized model can be continuously adjusted. After reaching the preset number of training times, the model parameters of the local personalized model are then controlled to remain unchanged, and the local private data set samples are input into the knowledge transfer model for training. During the training process, the loss function is used to calculate the loss value. The model parameters of the local personalized model and the label prediction values ​​of the local personalized model and the knowledge transfer model can be combined to input into the loss function of the knowledge distillation training to calculate the loss value, and the model parameters of the knowledge transfer model can be continuously adjusted. Subsequently, the model parameters of the knowledge transfer model are encrypted and sent to the coordinator for aggregation, and the model parameters of the knowledge transfer model of each participant are updated according to the aggregation results. Model training and model parameter update operations are performed again until the local personalized model of each participant converges.

[0075] Step S4: When the training loss value of the knowledge transfer model of each participant is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, the model parameters of the trained knowledge transfer model are encrypted based on the preset encryption algorithm and sent to the coordinator, so that the coordinator aggregates the model parameters of the knowledge transfer model sent by all participants, obtains the aggregated model parameters and sends them to each participant.

[0076] It is understandable that a knowledge training loss threshold or a training times threshold for each participant to train the knowledge transfer model may be preset. The knowledge training times threshold refers to the maximum number of times each participant trains the knowledge transfer model in each round, and may be set according to actual application needs.

[0077] It can also be understood that the training loss value of the knowledge transfer model can be calculated for each training. When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold, it means that the knowledge transfer model has converged and the training can be stopped. When the training loss value of the knowledge transfer model is greater than or equal to the knowledge training loss threshold, it is necessary to adjust the model parameters of the knowledge transfer model and train again until the preset number of training times is reached or the model converges. The model parameters of the trained knowledge transfer model are encrypted based on the preset encryption algorithm and sent to the coordinating party.

[0078] Among them, the encryption algorithm can be an additive homomorphic encryption algorithm, that is, an encryption algorithm that satisfies the homomorphic operation property of ciphertext, that is, after the data is homomorphically encrypted, a specific calculation is performed on the ciphertext, and the obtained ciphertext calculation result is equivalent to directly performing the same calculation on the plaintext data after the corresponding homomorphic decryption, thereby realizing the computable invisibility of the data. For example, the key generation process of the additive homomorphic encryption algorithm includes randomly selecting two prime numbers, calculating the public key and the private key based on the two prime numbers, the encryption process uses the public key for encryption, and the decryption process uses the private key for decryption.

[0079] In some embodiments, when the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, each participant encrypts the model parameters of the trained knowledge transfer model based on a preset homomorphic encryption algorithm, obtains the encrypted model parameters and sends them to the coordinator, wherein the coordinator aggregates all the encrypted model parameters according to a preset aggregation formula, obtains the aggregated model parameters and sends them to each participant.

[0080] Specifically, after the training is completed, the model parameters of the knowledge transfer model are encrypted and sent to the coordinator. That is, after the n participants complete the local training of the knowledge transfer model, they send the model parameters of the knowledge transfer model to the coordinator. Encrypt and send to the coordinator. To encrypt the model parameters, you can select an efficient homomorphic encryption algorithm to encrypt get and will Sent to the coordinator.

[0081] Furthermore, after receiving the encrypted model parameters, the coordinator implements the aggregate calculation under the ciphertext, and sends the aggregate model parameters to each participant to update the model parameters of the local knowledge transfer model. That is to say, after receiving the model parameters sent by n participants, the coordinator calculates the parameter aggregation: And <θ min >Sent to n participants respectively, <θ min >Represents the encrypted aggregation result of the model parameters sent by n participants.

[0082] Step S5: Each participant updates the model parameters of the knowledge transfer model based on the aggregated model parameters.

[0083] In some embodiments, the aggregate model parameters are decrypted based on a preset homomorphic decryption algorithm to obtain the decrypted aggregate model parameters, and calculations are performed based on the number of all participants and the decrypted aggregate model parameters to obtain the current model parameters, and the current model parameters are used as the model parameters of the knowledge transfer model.

[0084] Specifically, each participant updates the model parameters of the local knowledge transfer model when n participants receive <θ min >, decrypt using the secret key of the homomorphic decryption algorithm to obtain θ min and use Update the parameters of the local knowledge transfer model used for knowledge transfer

[0085] Specifically, after multiple participants complete the local personalized model training, they train the knowledge transfer model, and after the training is completed, the knowledge transfer model parameters are encrypted and sent to the coordinator. That is, after n participants complete the local training for the knowledge transfer model, the model parameters of the knowledge transfer model are encrypted and sent to the coordinator. Encrypt and send to the coordinator to encrypt the model parameters of the knowledge transfer model. You can choose to add a homomorphic cryptographic sharing protocol to implement parameter encryption and send the model parameters of the local knowledge transfer model to the coordinator. The split is performed secretly and the relevant shares are sent to the coordinator.

[0086] Specifically, after receiving the encryption parameters, the coordinator implements the aggregation calculation under the ciphertext, and sends the aggregation results to each participant to update the local knowledge transfer model parameters. The coordinator is any one of the n participants.

[0087] It should be noted that the coordinator can also average the aggregated model parameters according to the number of participants, obtain an encrypted model parameter and feed it back to the participants. After decryption, the participants directly use the decrypted model parameters as the latest model parameters of the knowledge transfer model.

[0088] Step S6, repeat steps S2 to S5 until the training loss value of the local personalized model of each participant is less than the local training loss threshold or the number of local training times is equal to the preset local training times threshold, to obtain the local personalized model of each participant.

[0089] Specifically, after completing the update of the local knowledge transfer model parameters, steps S2 to S5 are repeated until the local personalized models of multiple participants converge. That is, steps S2 to S5 are repeated until the local personalized models of n participants converge. i (θ i,X) all converge, and then stop model training, completing the multi-party joint training, and n participants obtain local personalized models.

[0090] In view of the technical problems that traditional horizontal federated learning methods are not suitable for model heterogeneity and data heterogeneity scenarios and cannot protect model parameters or intermediate calculation results, and the technical problems that heterogeneous federated learning methods based on knowledge distillation need to collect public data sets to support knowledge transfer, which is difficult to collect on the one hand and requires high training and storage resources on the other hand, the model heterogeneous federated learning training method for the government affairs field in the disclosed embodiment does not require a public data set for knowledge transfer, is suitable for model heterogeneity and data heterogeneity scenarios, and protects the intermediate results of model training.

[0091] The model heterogeneous federated learning training scheme for the government affairs field provided by the embodiment of the present disclosure includes multiple participants, and any participant among the multiple participants is used as a coordinator, each participant corresponds to a local personalized model and a knowledge transfer model, each participant determines the local personalized model based on the local task goal, determines the knowledge transfer model based on the local personalized model, and determines the initial values ​​of the model parameters of the local personalized model and the knowledge transfer model; wherein the input data samples used to train the local personalized model and the knowledge transfer model are the same, the model parameter dimension of the knowledge transfer model is smaller than the model parameter dimension of the local personalized model, each participant controls the model parameters of the knowledge transfer model to remain unchanged, the local personalized model is trained based on the local private data set samples of each participant, the training loss value of the local personalized model is obtained based on the preset loss function of the local personalized model, and the local personalized model is adjusted based on the training loss value of the local personalized model and the preset local training loss threshold. The model parameters of the local personalized model are controlled by each participant to remain unchanged after training, and the knowledge transfer model is trained based on the local private data set samples. The training loss value of the knowledge transfer model is obtained based on the preset loss function of the knowledge transfer model, and the model parameters of the knowledge transfer model are adjusted based on the training loss value of the knowledge transfer model and the preset knowledge training loss threshold. When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, each participant encrypts the model parameters of the trained knowledge transfer model based on the preset encryption algorithm and sends them to the coordinator, so that the coordinator aggregates the model parameters of the knowledge transfer model sent by all participants, obtains the aggregated model parameters and sends them to each participant, and each participant updates the model parameters of the knowledge transfer model based on the aggregated model parameters, and repeats the model training and model parameter update until the local personalized model of each participant converges. Therefore, the problem of secure joint modeling in model heterogeneous scenarios is realized based on knowledge distillation and secure aggregation of knowledge transfer model parameters. Different from the traditional implementation method based on knowledge distillation technology, the disclosed embodiment does not require a public data set to support knowledge transfer between heterogeneous models, and realizes multi-party personalized local model training and performance improvement, which is suitable for public data development and utilization scenarios in the government field.

[0092] Figure 2 This is a schematic diagram of the structure of a model heterogeneous federated learning training device for the government affairs field provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 2 As shown, it includes multiple participants and a coordinator, each of the participants corresponds to a local personalized model and a knowledge transfer model, and the device includes:

[0093] Determination module 201, used to execute step S1, each of the participants determines a local personalized model based on the local task goal, determines a knowledge transfer model based on the local personalized model, and determines initial values ​​of model parameters of the local personalized model and the knowledge transfer model; wherein the input data samples used to train the local personalized model and the knowledge transfer model are the same, and the model parameter dimension of the knowledge transfer model is smaller than the model parameter dimension of the local personalized model;

[0094] A first training module 202, configured to execute step S2, wherein each of the participants controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on a local private data set sample of each of the participants, obtains a training loss value of the local personalized model based on a preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and a preset local training loss threshold;

[0095] The second training module 203 is used to execute step S3, each of the participants controls the model parameters of the trained local personalized model to remain unchanged, trains the knowledge transfer model based on the local private data set sample, obtains the training loss value of the knowledge transfer model based on the preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and the preset knowledge training loss threshold;

[0096] The encryption sending module 204 is used to execute step S4, when the training loss value of the knowledge transfer model of each participant is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, encrypt the model parameters of the trained knowledge transfer model based on the preset encryption algorithm and send them to the coordinator, so that the coordinator aggregates the model parameters of the knowledge transfer model sent by all the participants, obtains the aggregated model parameters and sends them to each participant;

[0097] An updating module 205, configured to execute step S5, wherein each of the participants updates the model parameters of the knowledge transfer model based on the aggregated model parameters;

[0098] The processing module 206 is used to execute step S6 and repeatedly execute steps S2 to S5 until the training loss value of the local personalized model of each participant is less than the local training loss threshold or the number of local training times is equal to a preset local training times threshold, thereby obtaining a local personalized model of each participant.

[0099] Optionally, the determining module 201 is specifically configured to:

[0100] Each of the participants determines a local personalized model based on the local task goal as f i (θ i ,X),i∈{1,…,n},where,θ i is the model parameter of the local personalized model of the i-th participant, X is the input data sample of the local personalized model, and n is the number of the participants;

[0101] The knowledge transfer model is determined based on the local personalized model: in, is the model parameter of the knowledge transfer model of the i-th participant, and X is the input data sample of the knowledge transfer model;

[0102] For the θ i and stated Perform parameter initialization processing to determine the θ i and stated The initial value of The dimension is smaller than the θ i .

[0103] Optionally, the first training module 202 is specifically used to:

[0104] Each of the participants obtains a local private data set sample; wherein the local private data set sample of the i-th participant is represents the jth sample, m i Indicates the number of data items in the local private data set held by the i-th participant, represents the characteristics of the j-th sample data held by the i-th participant, represents the label of the jth sample data held by the i-th participant;

[0105] Each of the participants controls the model parameters of the knowledge transfer model to remain unchanged, inputs the local private data set sample into the local personalized model for processing, and obtains the training loss value of the local personalized model based on the loss function of the local personalized model; wherein the loss function of the local personalized model of the i-th participant is

[0106]

[0107] Among them, k i and They represent the weights of the local personalized model training loss function and the knowledge transfer model training loss function, respectively, and m i represents the number of data items in the local private data set of the i-th participant, Represents the characteristics of the jth sample data by the local personalized model of the i-th participant The label prediction value of Represents the characteristics of the j-sample data of the knowledge transfer model of the i-th participant The label prediction value of represents the loss function of the local personalized model of the participant, Represents the loss function of the knowledge transfer model;

[0108] When the training loss value of the local personalized model is greater than or equal to a local training loss threshold, a model parameter of the local personalized model is adjusted.

[0109] Optionally, the second training module 203 is specifically used to:

[0110] Each of the participants controls the model parameters of the trained local personalized model to remain unchanged, and obtains the local private data set sample; wherein the local private data set sample of the i-th participant is j∈{1,…,m i}, represents the jth sample, m i Indicates the number of data items in the local private data set held by the u-th participant, represents the characteristics of the j-th sample data held by the i-th participant, represents the label of the jth sample data held by the i-th participant;

[0111] The local private data set sample is input into the knowledge transfer model for processing, and the training loss value of the knowledge transfer model is obtained based on the loss function of the knowledge transfer model; wherein the loss function of the knowledge transfer model of the i-th participant is

[0112]

[0113] in, and They represent the total loss function of the knowledge transfer model training and the weight of the knowledge distillation training loss function, respectively. i represents the number of data items in the local private data set of the i-th participant, Represents the characteristics of the jth sample data by the local personalized model of the i-th participant The label prediction value of Represents the characteristics of the j-sample data of the knowledge transfer model of the i-th participant The label prediction value of represents the loss function of the knowledge transfer model, Represents the loss function of knowledge distillation training, which is used to distill the knowledge in the local personalized model to the knowledge transfer model;

[0114] When the training loss value of the knowledge transfer model is greater than or equal to a knowledge training loss threshold, the model parameters of the knowledge transfer model are adjusted.

[0115] Optionally, the encrypted sending module 204 is specifically used for:

[0116] When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, each of the participants encrypts the model parameters of the trained knowledge transfer model based on the preset homomorphic encryption algorithm, obtains the encrypted model parameters and sends them to the coordinator;

[0117] The coordinator aggregates all the encryption model parameters according to a preset aggregation formula, obtains the aggregated model parameters and sends them to each of the participants.

[0118] Optionally, the updating module 205 is specifically configured to:

[0119] Decrypting the aggregation model parameters based on a preset homomorphic decryption algorithm to obtain the decrypted aggregation model parameters;

[0120] And based on the number of all the participants and the decrypted aggregate model parameters, calculation is performed to obtain current model parameters, and the current model parameters are used as model parameters of the knowledge transfer model.

[0121] The model heterogeneous federated learning training device for the government affairs field provided by the embodiments of the present disclosure can execute the model heterogeneous federated learning training method for the government affairs field provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0122] The embodiments of the present disclosure also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the model heterogeneous federated learning training method for the government affairs field provided by any embodiment of the present disclosure.

[0123] Figure 3 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 3 , which shows a schematic diagram of the structure of an electronic device 300 suitable for implementing the embodiment of the present disclosure. The electronic device 300 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0124] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0125] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0126] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above functions defined in the model heterogeneous federated learning training method for the government affairs field of the embodiment of the present disclosure are executed.

[0127] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0128] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0129] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0130] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the model heterogeneous federated learning training method for the government affairs field of the aforementioned embodiment.

[0131] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0132] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0134] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0135] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0136] According to one or more embodiments of the present disclosure, the present disclosure provides an electronic device, including:

[0137] processor;

[0138] a memory for storing instructions executable by the processor;

[0139] The processor is used to read the executable instructions from the memory and execute the instructions to implement any model heterogeneous federated learning training method for the government affairs field provided in the present disclosure.

[0140] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute any model heterogeneous federated learning training method for the government affairs field as described in the present disclosure.

[0141] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0142] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0143] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A model heterogeneous federated learning training method for the government affairs field, characterized by: Including multiple participants, and taking any one of the multiple participants as a coordinator, each of the participants corresponds to a local personalized model and a knowledge transfer model, the method includes: Step S1: Each of the participants determines a local personalized model based on a local task goal, determines a knowledge transfer model based on the local personalized model, and determines initial values ​​of model parameters of the local personalized model and the knowledge transfer model; wherein the input data samples used to train the local personalized model and the knowledge transfer model are the same, and the model parameter dimension of the knowledge transfer model is smaller than the model parameter dimension of the local personalized model; Step S2: Each of the participants controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on the local private data set samples of each of the participants, obtains the training loss value of the local personalized model based on a preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and a preset local training loss threshold; Step S3, each of the participants controls the model parameters of the trained local personalized model to remain unchanged, trains the knowledge transfer model based on the local private data set sample, obtains the training loss value of the knowledge transfer model based on the preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and the preset knowledge training loss threshold; Step S4: When the training loss value of the knowledge transfer model of each participant is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, the model parameters of the trained knowledge transfer model are encrypted based on a preset encryption algorithm and sent to the coordinator, so that the coordinator aggregates the model parameters of the knowledge transfer model sent by all the participants, obtains the aggregated model parameters and sends them to each participant; Step S5: Each of the participants updates the model parameters of the knowledge transfer model based on the aggregation model parameters; Step S6, repeat steps S2 to S5 until the training loss value of the local personalized model of each participant is less than the local training loss threshold or the number of local training times is equal to the preset local training times threshold, to obtain the local personalized model of each participant.

2. The model heterogeneous federated learning training method for the government affairs field according to claim 1 is characterized in that: The step S1, each of the participants determines a local personalized model based on a local task goal, determines a knowledge transfer model based on the local personalized model, and determines initial values ​​of model parameters of the local personalized model and the knowledge transfer model, includes: Each of the participants determines a local personalized model based on the local task goal as f i (θ i ,X), i∈{1,…,n}, where θ i is the model parameter of the local personalized model of the i-th participant, X is the input data sample of the local personalized model, and n is the number of the participants; The knowledge transfer model is determined based on the local personalized model: in, is the model parameter of the knowledge transfer model of the i-th participant, and X is the input data sample of the knowledge transfer model; For the θ i and stated Perform parameter initialization processing to determine the θ i and stated The initial value of The dimension is smaller than the θ i .

3. The model heterogeneous federated learning training method for the government affairs field according to claim 1 is characterized in that: The step S2, in which each of the participants controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on the local private data set samples of each of the participants, obtains the training loss value of the local personalized model based on a preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and a preset local training loss threshold, includes: Each of the participants obtains a local private data set sample; wherein the local private data set sample of the i-th participant is represents the jth sample, m i Indicates the number of data items in the local private data set held by the i-th participant, represents the characteristics of the j-th sample data held by the i-th participant, represents the label of the jth sample data held by the i-th participant; Each of the participants controls the model parameters of the knowledge transfer model to remain unchanged, inputs the local private data set sample into the local personalized model for processing, and obtains the training loss value of the local personalized model based on the loss function of the local personalized model; wherein the loss function of the local personalized model of the i-th participant is Among them, k i and They represent the weights of the local personalized model training loss function and the knowledge transfer model training loss function, respectively, and m i represents the number of data items in the local private data set of the i-th participant, Represents the characteristics of the jth sample data by the local personalized model of the i-th participant The label prediction value of Represents the characteristics of the j-sample data of the knowledge transfer model of the i-th participant The label prediction value of represents the loss function of the local personalized model of the participant, Represents the loss function of the knowledge transfer model; When the training loss value of the local personalized model is greater than or equal to a local training loss threshold, a model parameter of the local personalized model is adjusted.

4. The model heterogeneous federated learning training method for the government affairs field according to claim 1 is characterized in that: The step S3, in which each of the participants controls the model parameters of the trained local personalized model to remain unchanged, trains the knowledge transfer model based on the local private data set sample, obtains the training loss value of the knowledge transfer model based on a preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and a preset knowledge training loss threshold, includes: Each of the participants controls the model parameters of the trained local personalized model to remain unchanged, and obtains the local private data set sample; wherein the local private data set sample of the i-th participant is represents the jth sample, m i Indicates the number of data items in the local private data set held by the i-th participant, represents the characteristics of the j-th sample data held by the i-th participant, represents the label of the jth sample data held by the i-th participant; The local private data set sample is input into the knowledge transfer model for processing, and the training loss value of the knowledge transfer model is obtained based on the loss function of the knowledge transfer model; wherein the loss function of the knowledge transfer model of the i-th participant is in, and They represent the total loss function of the knowledge transfer model training and the weight of the knowledge distillation training loss function, respectively. i represents the number of data items in the local private data set of the i-th participant, Represents the characteristics of the jth sample data by the local personalized model of the i-th participant The label prediction value of Represents the characteristics of the j-sample data of the knowledge transfer model of the i-th participant The label prediction value of represents the loss function of the knowledge transfer model, Represents the loss function of knowledge distillation training, which is used to distill the knowledge in the local personalized model to the knowledge transfer model; When the training loss value of the knowledge transfer model is greater than or equal to a knowledge training loss threshold, the model parameters of the knowledge transfer model are adjusted.

5. The model heterogeneous federated learning training method for the government affairs field according to claim 1 is characterized in that: The step S4, when the training loss value of the knowledge transfer model of each participant is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, encrypts the model parameters of the trained knowledge transfer model based on a preset encryption algorithm and sends them to the coordinator, so that the coordinator aggregates the model parameters of the knowledge transfer model sent by all the participants, obtains the aggregated model parameters and sends them to each participant, including: When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, each of the participants encrypts the model parameters of the trained knowledge transfer model based on the preset homomorphic encryption algorithm, obtains the encrypted model parameters and sends them to the coordinator; The coordinator aggregates all the encryption model parameters according to a preset aggregation formula, obtains the aggregated model parameters and sends them to each of the participants.

6. The model heterogeneous federated learning training method for the government affairs field according to claim 5 is characterized in that: In step S5, each of the participants updates the model parameters of the knowledge transfer model based on the aggregate model parameters, including: Decrypting the aggregation model parameters based on a preset homomorphic decryption algorithm to obtain the decrypted aggregation model parameters; And based on the number of all the participants and the decrypted aggregate model parameters, calculation is performed to obtain current model parameters, and the current model parameters are used as model parameters of the knowledge transfer model.

7. A model heterogeneous federated learning training device for the government affairs field, characterized in that: The device includes a plurality of participants and a coordinator, each of the participants corresponds to a local personalized model and a knowledge transfer model, and the device includes: A determination module, used to execute step S1, each of the participants determines a local personalized model based on the local task goal, determines a knowledge transfer model based on the local personalized model, and determines initial values ​​of model parameters of the local personalized model and the knowledge transfer model; wherein the input data samples used to train the local personalized model and the knowledge transfer model are the same, and the model parameter dimension of the knowledge transfer model is smaller than the model parameter dimension of the local personalized model; A first training module, configured to execute step S2, wherein each of the participants controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on a local private data set sample of each of the participants, obtains a training loss value of the local personalized model based on a preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and a preset local training loss threshold; A second training module, used to execute step S3, each of the participants controls the model parameters of the trained local personalized model to remain unchanged, trains the knowledge transfer model based on the local private data set sample, obtains the training loss value of the knowledge transfer model based on a preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and a preset knowledge training loss threshold; An encryption sending module is used to execute step S4, when the training loss value of the knowledge transfer model of each participant is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training times threshold, encrypt the model parameters of the trained knowledge transfer model based on a preset encryption algorithm and send them to the coordinator, so that the coordinator aggregates the model parameters of the knowledge transfer model sent by all the participants, obtains the aggregated model parameters and sends them to each participant; An updating module, used to execute step S5, where each of the participants updates the model parameters of the knowledge transfer model based on the aggregation model parameters; A processing module is used to execute step S6, repeatedly execute steps S2 to S5 until the training loss value of the local personalized model of each participant is less than the local training loss threshold or the number of local training times is equal to a preset local training times threshold, and obtain a local personalized model of each participant.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the model heterogeneous federated learning training method for the government affairs field as described in any one of claims 1-6 above.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is used to execute the model heterogeneous federated learning training method for the government affairs field as described in any of claims 1-6 above.

10. A computer program product, characterized in that It includes a computer program, wherein the computer program is executed by a processor for the model heterogeneous federated learning training method for the government affairs field as described in any one of claims 1 to 6.

Images