A model heterogeneous federated learning training method and device for the government affairs field

Abstract

The method comprises the following steps: each participant determines a local individualized model and a knowledge transfer model, and initializes parameters of the models; each participant fixes the knowledge transfer model and trains the local individualized model based on locally held data; after the training of the local individualized model is completed, the local individualized model is fixed and the knowledge transfer model is trained; after the training is completed, the knowledge transfer model parameters are encrypted and sent to a coordinator; after receiving the encrypted parameters, the coordinator performs aggregation calculation under the ciphertext, and sends the aggregation result to each participant to update the knowledge transfer model parameters; the model training and the model parameter updating are repeatedly performed until the local individualized model of each participant converges. By adopting the technical scheme, safe joint modeling in a model heterogeneous scene is realized, and the individualization of the local models of multiple participants is improved.

Description

Technical Field

[0001] This disclosure relates to the field of privacy computing technology, and in particular to a method and apparatus for training heterogeneous federated learning models in the field of government affairs. Background Technology

[0002] With increasingly stringent regulations on data security and personal privacy protection, and the vigorous promotion of market-based allocation of data elements and development and utilization of public data, privacy computing technology, which ensures that "raw data does not leave the domain and data is usable but not visible," has been rapidly developed and is now widely used in government, finance, and healthcare.

[0003] Specifically, privacy computing technology refers to the technical means used to perform data analysis and computation while protecting the data itself from being leaked. Among them, federated learning technology is one of the main technical paths for realizing privacy computing.

[0004] In related technologies, federated learning assumes that the models of all participating parties in federated modeling are isomorphic. Each party trains its local model using locally aligned data, and then uploads its local model training parameters or intermediate results for aggregation. After aggregation, feedback is used to update the local model parameters, thereby improving the training effect of each party's model after multi-party joint modeling. However, this approach requires each participating party to expose the parameters or intermediate results of its local efficiency model, which poses certain security risks. Moreover, in the real world, multiple participating parties have different goals in joint modeling, and the existing approach is difficult to meet the training needs of each party's local personalized model. If each participating party prioritizes the personalization of its local model, it will destroy the generalization of the aggregated model and make it difficult to support the needs of multi-party personalized joint modeling in the current scenario of public data development and utilization in the government sector. Summary of the Invention

[0005] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a method and apparatus for training heterogeneous federated learning models in the government sector. Based on knowledge distillation and secure aggregation of model parameters for knowledge transfer, it achieves secure joint modeling in heterogeneous model scenarios without requiring a public dataset to support knowledge transfer between heterogeneous models. This enables personalized local model training and performance enhancement for multiple parties. This disclosure implements a secure joint modeling method for heterogeneous models and data in multi-party scenarios, applicable to public data development and utilization scenarios in the government sector.

[0006] This disclosure provides a heterogeneous federated learning training method for models in the government domain, including multiple participants, with any one of the participants acting as a coordinator. Each participant corresponds to a local personalized model and a knowledge transfer model. The method includes:

[0007] Step S1: Each participant determines a local personalized model based on the local task objective, and determines a knowledge transfer model based on the local personalized model, and determines the initial values ​​of the model parameters of the local personalized model and the knowledge transfer model; wherein, the input data samples used to train the local personalized model and the knowledge transfer model are the same, and the model parameter dimension of the knowledge transfer model is smaller than the model parameter dimension of the local personalized model.

[0008] Step S2: Each participant controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on the local private dataset samples of each participant, obtains the training loss value of the local personalized model based on the preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and the preset local training loss threshold.

[0009] Step S3: Each participant controls the local personalized model after training to keep the model parameters unchanged, trains the knowledge transfer model based on the local private dataset samples, obtains the training loss value of the knowledge transfer model based on the preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and the preset knowledge training loss threshold.

[0010] Step S4: When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training number threshold, each participant encrypts the model parameters of the trained knowledge transfer model based on a preset encryption algorithm and sends them to the coordinator, so that the coordinator aggregates and calculates the model parameters of the knowledge transfer model sent by all the participants, obtains aggregated model parameters, and sends them to each participant.

[0011] Step S5: Each participant updates the model parameters of the knowledge transfer model based on the aggregation model parameters;

[0012] Step S6: Repeat steps S2 to S5 until the training loss value of the local personalized model of each participant is less than the local training loss threshold or the number of local training times is equal to the preset local training time threshold, so as to obtain the local personalized model of each participant.

[0013] Optionally, step S1, in which each participant determines a local personalized model based on a local task objective, determines a knowledge transfer model based on the local personalized model, and determines initial values ​​for the model parameters of the local personalized model and the knowledge transfer model, includes:

[0014] Each of the aforementioned participants determines a local personalized model as f based on its local task objective. i (θ i ,X), i∈{1,…,n}, where, θ i Let X be the model parameters of the local personalized model of the i-th participant, X be the input data sample of the local personalized model, and n be the number of participants.

[0015] Based on the local personalized model, the knowledge transfer model is determined as follows: in, Let X be the model parameters of the knowledge transfer model for the i-th participant, and let X be the input data sample for the knowledge transfer model.

[0016] For the θ i and stated Perform parameter initialization processing to determine θ i and stated The initial value; wherein, the The dimension is less than θ i .

[0017] Optionally, step S2, in which each participant controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on samples from each participant's local private dataset, obtains the training loss value of the local personalized model based on a preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and a preset local training loss threshold, includes:

[0018] Each participant obtains a local private dataset sample; wherein, the local private dataset sample of the i-th participant is... Let m represent the j-th sample. i This represents the number of data entries held by the i-th participant in their local private dataset. This represents the feature of the j-th sample data held by the i-th participant. This represents the label of the j-th sample data held by the i-th participant;

[0019] Each participant keeps the model parameters of the knowledge transfer model unchanged, inputs the local private dataset samples into the local personalized model for processing, and obtains the training loss value of the local personalized model based on its loss function; wherein, the loss function of the local personalized model of the i-th participant is...

[0020]

[0021] Where, k i and Let m represent the weights of the loss function for training the local personalized model and the weights of the loss function for training the knowledge transfer model, respectively. i This represents the number of data entries in the local private dataset of the i-th participant. This represents the feature of the local personalized model of the i-th participant on the j-th sample data. The predicted value of the label. This represents the features of the knowledge transfer model of the i-th participant on j sample data. The pre-splash value of the label, This represents the loss function of the participant's local personalized model. The loss function representing the knowledge transfer model;

[0022] When the training loss value of the local personalized model is greater than or equal to the local training loss threshold, the model parameters of the local personalized model are adjusted.

[0023] Optionally, step S3, in which each participant controls the local personalized model after training to keep its model parameters unchanged, trains the knowledge transfer model based on the local private dataset samples, obtains the training loss value of the knowledge transfer model based on a preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and a preset knowledge training loss threshold, includes:

[0024] Each participant maintains the same model parameters in its trained local personalized model and obtains a local private dataset sample; wherein, the local private dataset sample of the i-th participant is... Let represent the j-th sample, and mi represent the number of data items held by the i-th participant in their local private dataset. This represents the feature of the j-th sample data held by the i-th participant. This represents the label of the j-th sample data held by the i-th participant;

[0025] The local private dataset samples are input into the knowledge transfer model for processing, and the training loss value of the knowledge transfer model is obtained based on its loss function; wherein, the loss function of the knowledge transfer model for the i-th participant is...

[0026]

[0027] in, and Let m represent the sum of the training loss function for the knowledge transfer model and the weights of the training loss function for the knowledge distillation model, respectively. i This represents the number of data entries in the local private dataset of the i-th participant. This represents the feature of the local personalized model of the i-th participant on the j-th sample data. The predicted value of the label. This represents the features of the knowledge transfer model of the i-th participant on j sample data. The predicted value of the label. The loss function represents the knowledge transfer model. This represents the loss function used for knowledge distillation training, which is used to distill knowledge from the local personalized model into the knowledge transfer model.

[0028] When the training loss value of the knowledge transfer model is greater than or equal to the knowledge training loss threshold, the model parameters of the knowledge transfer model are adjusted.

[0029] Optionally, in step S4, when the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training iterations is equal to a preset knowledge training iterations threshold, each participant encrypts the model parameters of the trained knowledge transfer model based on a preset encryption algorithm and sends them to the coordinator. This allows the coordinator to aggregate and calculate the model parameters of the knowledge transfer models sent by all participants, obtain aggregated model parameters, and send them to each participant. This includes:

[0030] When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training iterations is equal to the preset knowledge training iterations threshold, each participating party encrypts the model parameters of the trained knowledge transfer model based on a preset homomorphic encryption algorithm, and sends the encrypted model parameters to the coordinator.

[0031] The coordinator aggregates all the encrypted model parameters according to a preset aggregation formula to obtain aggregated model parameters and sends them to each of the participating parties.

[0032] Optionally, in step S5, each participant updates the model parameters of the knowledge transfer model based on the aggregation model parameters, including:

[0033] The aggregation model parameters are decrypted based on a preset homomorphic decryption algorithm to obtain the decrypted aggregation model parameters;

[0034] The current model parameters are calculated based on the number of all participants and the decrypted aggregation model parameters, and these current model parameters are used as the model parameters of the knowledge transfer model.

[0035] This disclosure also provides a heterogeneous federated learning training device for models in the government domain, including multiple participants, with any one of the participants acting as a coordinator. Each participant corresponds to a local personalized model and a knowledge transfer model. The device includes:

[0036] The determination module is used to execute step S1, in which each participant determines a local personalized model based on the local task objective, determines a knowledge transfer model based on the local personalized model, and determines the initial values ​​of the model parameters of the local personalized model and the knowledge transfer model; wherein the input data samples used to train the local personalized model and the knowledge transfer model are the same, and the model parameter dimension of the knowledge transfer model is smaller than that of the model parameter dimension of the local personalized model.

[0037] The first training module is used to execute step S2, whereby each participant controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on the local private dataset samples of each participant, obtains the training loss value of the local personalized model based on the preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and the preset local training loss threshold.

[0038] The second training module is used to execute step S3, whereby each participant controls the model parameters of the local personalized model after training to remain unchanged, trains the knowledge transfer model based on the local private dataset samples, obtains the training loss value of the knowledge transfer model based on the preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and the preset knowledge training loss threshold.

[0039] The encrypted sending module is used to execute step S4, where each participant, when the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training number threshold, encrypts the model parameters of the trained knowledge transfer model based on a preset encryption algorithm and sends them to the coordinator, so that the coordinator aggregates and calculates the model parameters of the knowledge transfer model sent by all the participants, obtains aggregated model parameters, and sends them to each participant.

[0040] The update module is used to execute step S5, in which each participant updates the model parameters of the knowledge transfer model based on the aggregation model parameters;

[0041] The processing module is used to execute step S6 and repeat steps S2 to S5 until the training loss value of the local personalized model of each participant is less than the local training loss threshold or the number of local training times is equal to the preset local training time threshold, so as to obtain the local personalized model of each participant.

[0042] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the heterogeneous federated learning training method for government domain models provided in this disclosure.

[0043] This disclosure also provides a computer-readable storage medium storing a computer program for executing a heterogeneous federated learning training method for government affairs as provided in this disclosure.

[0044] This disclosure also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the model heterogeneous federated learning training method for the government domain described in the preceding aspect.

[0045] The technical solution provided in this disclosure has the following advantages compared with the prior art: The heterogeneous federated learning training scheme for government domain models provided in this disclosure includes multiple participants, with any one of the participants acting as a coordinator. Each participant corresponds to a local personalized model and a knowledge transfer model. Each participant determines its local personalized model based on its local task objective, determines its knowledge transfer model based on its local personalized model, and determines the initial values ​​of the model parameters for both the local personalized model and the knowledge transfer model. The input data samples used to train the local personalized model and the knowledge transfer model are the same. The model parameter dimension of the knowledge transfer model is smaller than that of the local personalized model. Each participant keeps the model parameters of the knowledge transfer model unchanged. The local personalized model is trained based on the local private dataset samples of each participant. The training loss value of the local personalized model is obtained based on a preset loss function of the local personalized model. The training loss value of the local personalized model is then compared with the preset loss function of the local personalized model. The local training loss threshold adjusts the model parameters of the local personalized model. Each participant keeps the model parameters of the local personalized model unchanged after training. The knowledge transfer model is trained based on samples from the local private dataset. The training loss value of the knowledge transfer model is obtained based on the preset loss function of the knowledge transfer model. The model parameters of the knowledge transfer model are adjusted based on the training loss value and the preset knowledge training loss threshold. When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training number threshold, each participant encrypts the model parameters of the trained knowledge transfer model based on the preset encryption algorithm and sends them to the coordinator. The coordinator then aggregates the model parameters of the knowledge transfer models sent by all participants to obtain aggregated model parameters and sends them to each participant. Each participant updates the model parameters of the knowledge transfer model based on the aggregated model parameters. The model training and model parameter update are repeated until each participant's local personalized model converges. Therefore, based on knowledge distillation and knowledge transfer model parameter secure aggregation, the problem of secure joint modeling in heterogeneous model scenarios is solved. Unlike the traditional implementation based on knowledge distillation technology, the embodiments of this disclosure do not require a public dataset to support knowledge transfer between heterogeneous models, and realize multi-party personalized local model training and performance improvement, which is suitable for public data development and utilization scenarios in the government sector. Attached Figure Description

[0046] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0047] Figure 1A flowchart illustrating a heterogeneous federated learning training method for models in the government sector, provided as an embodiment of this disclosure;

[0048] Figure 2 A schematic diagram of the structure of a heterogeneous federated learning training device for the government sector provided in this embodiment of the disclosure;

[0049] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0050] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0051] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0052] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0053] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0054] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0055] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0056] Figure 1This is a flowchart illustrating a heterogeneous federated learning training method for models in the government domain, provided as an embodiment of this disclosure. This method can be executed by a heterogeneous federated learning training device for models in the government domain, wherein the device can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method involves multiple participants, with any one of them acting as a coordinator. Each participant corresponds to a local personalized model and a knowledge transfer model. The method includes:

[0057] Step S1: Each participant determines a local personalized model based on the local task objective, and determines a knowledge transfer model based on the local personalized model, as well as the initial values ​​of the model parameters of the local personalized model and the knowledge transfer model; wherein, the input data samples used to train the local personalized model and the knowledge transfer model are the same, and the model parameter dimension of the knowledge transfer model is smaller than that of the model parameter dimension of the local personalized model.

[0058] In some embodiments, each participant determines a local personalized model f based on its local task objective. i (θ i ,X), i∈{1,…,n}, where, θ i Let X be the model parameters of the local personalized model for the i-th participant, X be the input data sample of the local personalized model, and n be the number of participants; the knowledge transfer model is determined based on the local personalized model. in, Let θ be the model parameters for the knowledge transfer model of the i-th participant, and X be the input data sample for the knowledge transfer model; for θ i and Perform parameter initialization to determine θ i and The initial value; where, The dimension is less than θ i .

[0059] For example, in a government affairs scenario, the three participants are device A1 representing organization A, device B1 representing organization B, and device C2 representing organization C. Device A1 determines a local personalized model a1 based on a local task objective, such as text classification, and determines a knowledge transfer model a2 based on the local personalized model a1. Device B1 determines a local personalized model b1 based on a local task objective, such as image recognition, and determines a knowledge transfer model b2 based on the local personalized model b1. Device C1 determines a local personalized model c1 based on a local task objective, such as video object recognition, and determines a knowledge transfer model c2 based on the local personalized model c1.

[0060] Specifically, multiple participants select local personalized models based on their local task objectives, and each participant selects a knowledge transfer model based on its local personalized model, initializing the model parameters of the local personalized models and knowledge transfer models for multiple participants; more specifically, for example, n participants each select a local personalized model f based on their local task objectives. i (θ i ,X), i∈{1,…,n},θ i Let X be the model parameters of the local personalized model for the i-th participant, and let X be the input of the local personalized model, based on the local personalized models f selected by the n participants. i (θ i Choose a knowledge transfer model suitable for multi-party knowledge transfer (X). in, Let X be the parameter of the knowledge transfer model for the i-th participant, and let X be the input of the knowledge transfer model. The dimension is much smaller than θ i Initialize the model parameters θ of the local personalized models of n participants. i And model parameters for knowledge transfer models used in multi-party knowledge transfer. Choose appropriate initial values ​​for the parameters.

[0061] Step S2: Each participant keeps the model parameters of the knowledge transfer model unchanged, trains the local personalized model based on the local private dataset samples of each participant, obtains the training loss value of the local personalized model based on the preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and the preset local training loss threshold.

[0062] In some embodiments, each participant obtains a local private dataset sample; wherein, the local private dataset sample of the i-th participant is... Let m represent the j-th sample. i This represents the number of data entries held by the i-th participant in their local private dataset. This represents the feature of the j-th sample data held by the i-th participant. Let represent the label of the j-th sample data held by the i-th participant; each participant keeps the model parameters of the knowledge transfer model unchanged, inputs the local private dataset samples into the local personalized model for processing, and obtains the training loss value of the local personalized model based on the loss function of the local personalized model; where the loss function of the local personalized model of the i-th participant is .

[0063]

[0064] Where, ki and Let m represent the weights of the loss function for training the local personalized model and the weights of the loss function for training the knowledge transfer model, respectively. i This represents the number of data entries in the local private dataset of the i-th participant. This represents the feature of the local personalized model of the i-th participant on the j-th sample data. The predicted value of the label. This represents the features of the knowledge transfer model of the i-th participant on j sample data. The predicted value of the label. This represents the loss function of the participant's local personalized model. This represents the loss function of the knowledge transfer model; the model parameters of the local personalized model are adjusted when the training loss value of the local personalized model is greater than or equal to the local training loss threshold.

[0065] In this embodiment of the disclosure, a local training loss threshold is preset, which can be selected and set according to the actual application scenario.

[0066] Specifically, multiple participants train local personalized models based on their locally held data, under the premise of fixed model parameters for the knowledge transfer model. More specifically, n participants each hold local private dataset samples, and these n participants train their local personalized models f. i (θ i When X is fixed, it is used for knowledge transfer models. parameters

[0067] Understandably, a local training loss threshold or a local training count threshold for each participant to train their local personalized model can be preset. The local training count threshold refers to the maximum number of times each participant trains their local personalized model in each round, which can be set according to the actual application needs.

[0068] It is also understandable that the training loss value of the local personalized model can be calculated in each training session. When the training loss value of the local personalized model is less than the local training loss threshold, it means that the local personalized model has converged and training can be stopped. When the training loss value of the local personalized model is greater than or equal to the local training loss threshold, the model parameters of the local personalized model need to be adjusted and training is carried out again until the preset number of training sessions is reached or the model converges, thereby obtaining the local personalized model of each participant.

[0069] Step S3: Each participant keeps the model parameters of the local personalized model unchanged after training, trains the knowledge transfer model based on the local private dataset samples, obtains the training loss value of the knowledge transfer model based on the preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and the preset knowledge training loss threshold.

[0070] In some embodiments, each participant maintains the model parameters of its trained local personalized model while acquiring a local private dataset sample; wherein, the local private dataset sample of the i-th participant is... Let m represent the j-th sample. i This represents the number of data entries held by the i-th participant in their local private dataset. This represents the feature of the j-th sample data held by the i-th participant. Let represent the label of the j-th sample data held by the i-th participant; input the local private dataset samples into the knowledge transfer model for processing, and obtain the training loss value of the knowledge transfer model based on its loss function; where the loss function of the knowledge transfer model for the i-th participant is .

[0071]

[0072] in, and Let m represent the sum of the training loss function for the knowledge transfer model and the weights of the training loss function for the knowledge distillation model, respectively. i This represents the number of data entries in the local private dataset of the i-th participant. This represents the feature of the local personalized model of the i-th participant on the j-th sample data. The predicted value of the label. This represents the features of the knowledge transfer model of the i-th participant on j sample data. The predicted value of the label. The loss function represents the knowledge transfer model. This represents the loss function for knowledge distillation training, used to distill knowledge from the local personalized model into the knowledge transfer model; the model parameters of the knowledge transfer model are adjusted when the training loss value of the knowledge transfer model is greater than or equal to the knowledge training loss threshold.

[0073] Specifically, after multiple participants complete the local personalized model training, they train the knowledge transfer model. More specifically, after completing the local personalized model training, the participants fix the model parameters of the local personalized model. In other words, n participants complete one or more rounds of local personalized model training based on local private dataset samples. i (θ i After training, the parameter θ is fixed.i Completing one or more rounds refers to the local personalized model f i (θ i During the training process, one or more rounds of training with private dataset samples are completed; then, the n participants fix the model parameters of their local personalized models and apply the loss function. Local personalized model f i (θ i Knowledge distillation in X) to local knowledge transfer models for knowledge transfer

[0074] In other words, after each participant determines its local personalized model and knowledge transfer model, and initializes the model parameters for both models, the parameters of the knowledge transfer model are kept unchanged. Then, the local personalized model is trained a pre-set number of times using samples from the local private dataset. During training, the loss function is applied... The loss value can be calculated together with the model parameters of the knowledge transfer model, and the model parameters of the local personalized model can be continuously adjusted. After reaching a preset number of training iterations, the model parameters of the local personalized model can be kept unchanged, and the local private dataset samples can be input into the knowledge transfer model for training. During the training process, the loss function can be used to calculate the loss value. The loss value can be calculated by combining the model parameters of the local personalized model and the label prediction values ​​of the local personalized model and the knowledge transfer model into the loss function of knowledge distillation training. The model parameters of the knowledge transfer model are continuously adjusted. Subsequently, the model parameters of the knowledge transfer model are encrypted and sent to the coordinator for aggregation. The model parameters of the knowledge transfer model of each participant are updated according to the aggregation results. The model training and model parameter update operations are performed again until the local personalized model of each participant converges.

[0075] Step S4: When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training number threshold, each participant encrypts the model parameters of the trained knowledge transfer model based on the preset encryption algorithm and sends them to the coordinator. This allows the coordinator to aggregate and calculate the model parameters of the knowledge transfer models sent by all participants, obtain aggregated model parameters, and send them to each participant.

[0076] Understandably, a knowledge training loss threshold or a training count threshold for each participant in training the knowledge transfer model can be preset. The knowledge training count threshold refers to the maximum number of times each participant trains the knowledge transfer model in each round, which can be set according to the actual application needs.

[0077] It is also understandable that the training loss value of the knowledge transfer model can be calculated in each training session. When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold, it means that the knowledge transfer model has converged and training can be stopped. When the training loss value of the knowledge transfer model is greater than or equal to the knowledge training loss threshold, the model parameters of the knowledge transfer model need to be adjusted and training is carried out again until the preset number of training sessions is reached or the model converges. Based on the preset encryption algorithm, the model parameters of the trained knowledge transfer model are encrypted and sent to the coordinator.

[0078] Among them, the encryption algorithm can be a homomorphic encryption algorithm, that is, an encryption algorithm that satisfies the homomorphic operation property of ciphertext. That is, after the data is homomorphically encrypted, a specific calculation is performed on the ciphertext, and the resulting ciphertext calculation result is equivalent to performing the same calculation directly on the plaintext data after homomorphic decryption. This achieves the goal of data being computable but not visible. For example, the key generation process of a homomorphic encryption algorithm includes randomly selecting two prime numbers, calculating a public key and a private key based on the two prime numbers, using the public key for encryption, and using the private key for decryption.

[0079] In some embodiments, when the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training iterations is equal to the preset knowledge training iterations threshold, each participant encrypts the model parameters of the trained knowledge transfer model based on a preset homomorphic encryption algorithm, and sends the encrypted model parameters to the coordinator. The coordinator then aggregates all the encrypted model parameters according to a preset aggregation formula to obtain aggregated model parameters, which are then sent to each participant.

[0080] Specifically, after training is complete, the model parameters of the knowledge transfer model are encrypted and sent to the coordinator. In other words, after the n participating parties have completed their local training of the knowledge transfer model, they will send the model parameters of the knowledge transfer model... The encrypted data is sent to the coordinator. Encrypting the model parameters can be achieved using an efficient homomorphic encryption algorithm. get And Send to the coordinator.

[0081] Furthermore, after receiving the encrypted model parameters, the coordinator performs aggregate calculations in ciphertext and distributes the aggregated model parameters to each participant to update the model parameters of their local knowledge transfer model. In other words, the coordinator calculates parameter aggregation after receiving model parameters from n participants. and <θ min >Send to each of the n participants,<θ min > represents the encrypted aggregation result of model parameters sent by n participating parties.

[0082] Step S5: Each participant updates the model parameters of the knowledge transfer model based on the aggregated model parameters.

[0083] In some embodiments, the aggregated model parameters are decrypted based on a preset homomorphic decryption algorithm to obtain the decrypted aggregated model parameters. The current model parameters are then calculated based on the number of all participants and the decrypted aggregated model parameters, and used as the model parameters of the knowledge transfer model.

[0084] Specifically, updating the model parameters of the local knowledge transfer model by each participant refers to the process where n participants receive <θ> min > Use the key of the homomorphic decryption algorithm to decrypt and obtain θ min and use Update the parameters of the local knowledge transfer model used for knowledge transfer.

[0085] Specifically, after multiple participants complete local personalized model training, they train the knowledge transfer model. Once training is complete, they encrypt and send the knowledge transfer model parameters to the coordinator. In other words, after n participants complete local training for their knowledge transfer models, they send the model parameters of the knowledge transfer model... The encrypted model parameters of the knowledge transfer model are sent to the coordinator. This encryption can be achieved by using a homomorphic cryptographic sharing protocol. The local knowledge transfer model parameters are then sent to the coordinator. The shares are secretly split and the relevant shares are sent to the coordinator.

[0086] Specifically, after receiving the encrypted parameters, the coordinator performs aggregate calculations under the ciphertext and sends the aggregate results to each participant to update the local knowledge transfer model parameters. The coordinator is any one of the n participants.

[0087] It should be noted that the coordinator can also average the aggregated model parameters according to the number of participants to obtain an encrypted model parameter and feed it back to the participants. After decryption, the participants can directly use the decrypted model parameter as the latest model parameter of the knowledge transfer model.

[0088] Step S6: Repeat steps S2 to S5 until the training loss value of the local personalized model of each participant is less than the local training loss threshold or the number of local training times is equal to the preset local training time threshold, so as to obtain the local personalized model of each participant.

[0089] Specifically, after updating the parameters of the local knowledge transfer model, steps S2 to S5 are repeated until the local personalized model of the multi-participant parties converges. In other words, steps S2 to S5 are repeated until the local personalized model f of the n participants is reached. i (θ iIf all n parties (x, y) converge, then model training stops, completing multi-party joint training, and each party obtains a local personalized model.

[0090] To address the technical issues that traditional horizontal federated learning methods are not suitable for heterogeneous model and data scenarios and cannot protect model parameters or intermediate computation results, as well as the technical problems that knowledge distillation-based heterogeneous federated learning methods require the collection of public datasets to support knowledge transfer, which is difficult to collect and has high requirements for training and storage resources, this disclosure provides a model heterogeneous federated learning training method for the government sector that does not require public datasets for knowledge transfer, is applicable to heterogeneous model and data scenarios, and protects intermediate results of model training.

[0091] The heterogeneous federated learning training scheme for government domain models provided in this disclosure includes multiple participants, with any one of them acting as a coordinator. Each participant corresponds to a local personalized model and a knowledge transfer model. Each participant determines its local personalized model based on its local task objective, and determines the knowledge transfer model based on its local personalized model, as well as the initial values ​​of the model parameters for both the local personalized model and the knowledge transfer model. The input data samples used to train the local personalized model and the knowledge transfer model are the same. The model parameter dimension of the knowledge transfer model is smaller than that of the local personalized model. Each participant keeps the model parameters of the knowledge transfer model constant. The local personalized model is trained based on each participant's local private dataset samples. The training loss value of the local personalized model is obtained based on a preset loss function, and the local personalized model is adjusted based on the training loss value and a preset local training loss threshold. The model parameters of the personalized model are kept unchanged by each participant after training. The knowledge transfer model is trained based on local private dataset samples. The training loss value of the knowledge transfer model is obtained based on the preset loss function of the knowledge transfer model. The model parameters of the knowledge transfer model are adjusted based on the training loss value and the preset knowledge training loss threshold. When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training number threshold, each participant encrypts the model parameters of the trained knowledge transfer model based on the preset encryption algorithm and sends them to the coordinator. The coordinator then aggregates the model parameters of the knowledge transfer models sent by all participants to obtain aggregated model parameters and sends them to each participant. Each participant updates the model parameters of the knowledge transfer model based on the aggregated model parameters. The model training and model parameter update are repeated until each participant's local personalized model converges. Therefore, based on knowledge distillation and knowledge transfer model parameter secure aggregation, the problem of secure joint modeling in heterogeneous model scenarios is solved. Unlike the traditional implementation based on knowledge distillation technology, the embodiments of this disclosure do not require a public dataset to support knowledge transfer between heterogeneous models, and realize multi-party personalized local model training and performance improvement, which is suitable for public data development and utilization scenarios in the government sector.

[0092] Figure 2 This is a schematic diagram of a heterogeneous federated learning training device for government affairs, provided as an embodiment of this disclosure. This device can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 2 As shown, the device includes multiple participants and a coordinator, with each participant corresponding to a local personalized model and a knowledge transfer model. The device includes:

[0093] The determination module 201 is used to execute step S1, in which each participant determines a local personalized model based on the local task objective, determines a knowledge transfer model based on the local personalized model, and determines the initial values ​​of the model parameters of the local personalized model and the knowledge transfer model; wherein the input data samples used to train the local personalized model and the knowledge transfer model are the same, and the model parameter dimension of the knowledge transfer model is smaller than that of the model parameter dimension of the local personalized model.

[0094] The first training module 202 is used to execute step S2, whereby each participant controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on the local private dataset samples of each participant, obtains the training loss value of the local personalized model based on the preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and the preset local training loss threshold.

[0095] The second training module 203 is used to execute step S3, whereby each participant controls the model parameters of the local personalized model after training to remain unchanged, trains the knowledge transfer model based on the local private dataset samples, obtains the training loss value of the knowledge transfer model based on the preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and the preset knowledge training loss threshold.

[0096] The encryption sending module 204 is used to execute step S4, where each participant, when the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training number threshold, encrypts the model parameters of the trained knowledge transfer model based on a preset encryption algorithm and sends them to the coordinator, so that the coordinator aggregates and calculates the model parameters of the knowledge transfer model sent by all the participants, obtains aggregated model parameters, and sends them to each participant.

[0097] Update module 205 is used to execute step S5, whereby each participant updates the model parameters of the knowledge transfer model based on the aggregation model parameters;

[0098] Processing module 206 is used to execute step S6 and repeat steps S2 to S5 until the training loss value of the local personalized model of each participant is less than the local training loss threshold or the number of local training times is equal to the preset local training time threshold, so as to obtain the local personalized model of each participant.

[0099] Optionally, the determining module 201 is specifically used for:

[0100] Each of the aforementioned participants determines a local personalized model as f based on its local task objective. i (θ i ,X), i∈{1,…,n}, where, θ i Let X be the model parameters of the local personalized model of the i-th participant, X be the input data sample of the local personalized model, and n be the number of participants.

[0101] Based on the local personalized model, the knowledge transfer model is determined as follows: in, Let X be the model parameters of the knowledge transfer model for the i-th participant, and let X be the input data sample for the knowledge transfer model.

[0102] For the θ i and stated Perform parameter initialization processing to determine θ i and stated The initial value; wherein, the The dimension is less than θ i .

[0103] Optionally, the first training module 202 is specifically used for:

[0104] Each participant obtains a local private dataset sample; wherein, the local private dataset sample of the i-th participant is... Let m represent the j-th sample. i This represents the number of data entries held by the i-th participant in their local private dataset. This represents the feature of the j-th sample data held by the i-th participant. This represents the label of the j-th sample data held by the i-th participant;

[0105] Each participant keeps the model parameters of the knowledge transfer model unchanged, inputs the local private dataset samples into the local personalized model for processing, and obtains the training loss value of the local personalized model based on its loss function; wherein, the loss function of the local personalized model of the i-th participant is...

[0106]

[0107] Where, k i and Let m represent the weights of the loss function for training the local personalized model and the weights of the loss function for training the knowledge transfer model, respectively. i This represents the number of data entries in the local private dataset of the i-th participant. This represents the feature of the local personalized model of the i-th participant on the j-th sample data. The predicted value of the label. This represents the features of the knowledge transfer model of the i-th participant on j sample data. The predicted value of the label. This represents the loss function of the participant's local personalized model. The loss function representing the knowledge transfer model;

[0108] When the training loss value of the local personalized model is greater than or equal to the local training loss threshold, the model parameters of the local personalized model are adjusted.

[0109] Optionally, the second training module 203 is specifically used for:

[0110] Each participant maintains the same model parameters in its trained local personalized model and obtains a local private dataset sample; wherein, the local private dataset sample of the i-th participant is... j∈{1,…,m i} represents the j-th sample, m i This represents the number of data entries held by the u-th participant in their local private dataset. This represents the feature of the j-th sample data held by the i-th participant. This represents the label of the j-th sample data held by the i-th participant;

[0111] The local private dataset samples are input into the knowledge transfer model for processing, and the training loss value of the knowledge transfer model is obtained based on its loss function; wherein, the loss function of the knowledge transfer model for the i-th participant is...

[0112]

[0113] in, and Let m represent the sum of the training loss function for the knowledge transfer model and the weights of the training loss function for the knowledge distillation model, respectively. i This represents the number of data entries in the local private dataset of the i-th participant. This represents the feature of the local personalized model of the i-th participant on the j-th sample data. The predicted value of the label. This represents the features of the knowledge transfer model of the i-th participant on j sample data. The predicted value of the label. The loss function represents the knowledge transfer model. This represents the loss function used for knowledge distillation training, which is used to distill knowledge from the local personalized model into the knowledge transfer model.

[0114] When the training loss value of the knowledge transfer model is greater than or equal to the knowledge training loss threshold, the model parameters of the knowledge transfer model are adjusted.

[0115] Optionally, the encrypted sending module 204 is specifically used for:

[0116] When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training iterations is equal to the preset knowledge training iterations threshold, each participating party encrypts the model parameters of the trained knowledge transfer model based on a preset homomorphic encryption algorithm, and sends the encrypted model parameters to the coordinator.

[0117] The coordinator aggregates all the encrypted model parameters according to a preset aggregation formula to obtain aggregated model parameters and sends them to each of the participating parties.

[0118] Optionally, the update module 205 is specifically used for:

[0119] The aggregation model parameters are decrypted based on a preset homomorphic decryption algorithm to obtain the decrypted aggregation model parameters;

[0120] The current model parameters are calculated based on the number of all participants and the decrypted aggregation model parameters, and these current model parameters are used as the model parameters of the knowledge transfer model.

[0121] The heterogeneous federated learning training device for the government sector provided in this disclosure can execute the heterogeneous federated learning training method for the government sector provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0122] This disclosure also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the heterogeneous federated learning training method for government domain models provided in any embodiment of this disclosure.

[0123] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present disclosure. See below for details. Figure 3 The diagram illustrates a structural schematic suitable for implementing the electronic device 300 in the embodiments of this disclosure. The electronic device 300 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0124] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0125] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0126] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the heterogeneous federated learning training method for government domain models according to embodiments of this disclosure.

[0127] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0128] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0129] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0130] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the heterogeneous federated learning training method for the government domain as described in the preceding embodiments.

[0131] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0133] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0134] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0135] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0136] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0137] processor;

[0138] Memory used to store the processor's executable instructions;

[0139] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the heterogeneous federated learning training method for government domain models as described in any of the present disclosure.

[0140] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium storing a computer program for performing a heterogeneous federated learning training method for government domain models as described in any of the present disclosure.

[0141] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0142] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0143] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A heterogeneous federated learning training method for models in the government affairs domain, characterized in that, The method includes multiple participants, with any one of the participants acting as a coordinator. Each participant corresponds to a local personalized model and a knowledge transfer model. Step S1: Each participant determines a local personalized model based on the local task objective, and determines a knowledge transfer model based on the local personalized model, and determines the initial values ​​of the model parameters of the local personalized model and the knowledge transfer model; wherein, the input data samples used to train the local personalized model and the knowledge transfer model are the same, and the model parameter dimension of the knowledge transfer model is smaller than the model parameter dimension of the local personalized model. Step S2: Each participant keeps the model parameters of the knowledge transfer model unchanged, trains the local personalized model based on the local private dataset samples of each participant, obtains the training loss value of the local personalized model based on the preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and the preset local training loss threshold; wherein, the local private dataset samples refer to the dataset samples held locally by each participant and not visible to other participants. Step S3: Each participant controls the local personalized model after training to keep the model parameters unchanged, trains the knowledge transfer model based on the local private dataset samples, obtains the training loss value of the knowledge transfer model based on the preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and the preset knowledge training loss threshold. Step S4: When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training number threshold, each participant encrypts the model parameters of the trained knowledge transfer model based on a preset encryption algorithm and sends them to the coordinator, so that the coordinator aggregates and calculates the model parameters of the knowledge transfer model sent by all the participants, obtains aggregated model parameters, and sends them to each participant. Step S5: Each participant updates the model parameters of the knowledge transfer model based on the aggregation model parameters; Step S6: Repeat steps S2 to S5 until the training loss value of the local personalized model of each participant is less than the local training loss threshold or the number of local training times is equal to the preset local training time threshold, so as to obtain the local personalized model of each participant. Specifically, step S1, in which each participant determines a local personalized model based on a local task objective, determines a knowledge transfer model based on the local personalized model, and determines initial values ​​for the model parameters of the local personalized model and the knowledge transfer model, includes: Each of the aforementioned participants determines a local personalized model as f based on its local task objective. i (θ i ,X), i∈{1,…,n}, where, θ i Let X be the model parameters of the local personalized model of the i-th participant, X be the input data sample of the local personalized model, and n be the number of participants. Based on the local personalized model, the knowledge transfer model is determined as follows: in, Let X be the model parameters of the knowledge transfer model for the i-th participant, and let X be the input data sample for the knowledge transfer model. For the θ i and stated Perform parameter initialization processing to determine θ i and stated The initial value; wherein, the The dimension is less than θ i ; Each of the participating parties is a device A1 representing institution A, a device B1 representing institution B, and a device C2 representing institution C. Device A1 determines a local personalized model a1 for a text classification task based on the local task objective, and determines a knowledge transfer model a2 based on the local personalized model a1. Device B1 determines a local personalized model b1 for an image recognition task based on the local task objective, and determines a knowledge transfer model b2 based on the local personalized model b1. Device C1 determines a local personalized model c1 for a video target recognition task based on the local task objective, and determines a knowledge transfer model c2 based on the local personalized model c1. Step S2, in which each participant controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on samples from each participant's local private dataset, obtains the training loss value of the local personalized model based on a preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and a preset local training loss threshold, includes: Each participant obtains a local private dataset sample; wherein, the local private dataset sample of the i-th participant is... Let m represent the j-th sample. i This represents the number of data entries held by the i-th participant in their local private dataset. This represents the feature of the j-th sample data held by the i-th participant. This represents the label of the j-th sample data held by the i-th participant; wherein, the local private dataset sample refers to the dataset sample held by the device corresponding to each participant locally; different local personalized models correspond to different dataset samples, including at least one of text dataset samples, image dataset samples, and video dataset samples; Each participant keeps the model parameters of the knowledge transfer model unchanged, inputs the local private dataset samples into the local personalized model for processing, and obtains the training loss value of the local personalized model based on its loss function; wherein, the loss function of the local personalized model of the i-th participant is... Where, k i and Let m represent the weights of the loss function for training the local personalized model and the weights of the loss function for training the knowledge transfer model, respectively. i This represents the number of data entries in the local private dataset of the i-th participant. This represents the feature of the local personalized model of the i-th participant on the j-th sample data. The predicted value of the label. This represents the features of the knowledge transfer model of the i-th participant on j sample data. The predicted value of the label. This represents the loss function of the participant's local personalized model. The loss function representing the knowledge transfer model; When the training loss value of the local personalized model is greater than or equal to the local training loss threshold, the model parameters of the local personalized model are adjusted.

2. The heterogeneous federated learning training method for government affairs models according to claim 1, characterized in that, Step S3, where each participant keeps the model parameters of their trained local personalized model unchanged, trains the knowledge transfer model based on the local private dataset samples, obtains the training loss value of the knowledge transfer model based on a preset loss function, and adjusts the model parameters of the knowledge transfer model based on the training loss value and a preset knowledge training loss threshold, includes: Each participant maintains the same model parameters in its trained local personalized model and obtains a local private dataset sample; wherein, the local private dataset sample of the i-th participant is... Let m represent the j-th sample. i This represents the number of data entries held by the i-th participant in their local private dataset. This represents the feature of the j-th sample data held by the i-th participant. This represents the label of the j-th sample data held by the i-th participant; The local private dataset samples are input into the knowledge transfer model for processing, and the training loss value of the knowledge transfer model is obtained based on its loss function; wherein, the loss function of the knowledge transfer model for the i-th participant is... in, and Let m represent the sum of the training loss function for the knowledge transfer model and the weights of the training loss function for the knowledge distillation model, respectively. i This represents the number of data entries in the local private dataset of the i-th participant. This represents the feature of the local personalized model of the i-th participant on the j-th sample data. The predicted value of the label. This represents the features of the knowledge transfer model of the i-th participant on j sample data. The predicted value of the label. The loss function represents the knowledge transfer model. This represents the loss function used for knowledge distillation training, which is used to distill knowledge from the local personalized model into the knowledge transfer model. When the training loss value of the knowledge transfer model is greater than or equal to the knowledge training loss threshold, the model parameters of the knowledge transfer model are adjusted.

3. The heterogeneous federated learning training method for government affairs models according to claim 1, characterized in that, Step S4, where each participant, when the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training iterations is equal to a preset knowledge training iterations threshold, encrypts the model parameters of the trained knowledge transfer model based on a preset encryption algorithm and sends them to the coordinator. This allows the coordinator to aggregate and calculate the model parameters of the knowledge transfer models sent by all participants, obtain aggregated model parameters, and send them to each participant. This includes: When the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training iterations is equal to the preset knowledge training iterations threshold, each participating party encrypts the model parameters of the trained knowledge transfer model based on a preset homomorphic encryption algorithm, and sends the encrypted model parameters to the coordinator. The coordinator aggregates all the encrypted model parameters according to a preset aggregation formula to obtain aggregated model parameters and sends them to each of the participating parties.

4. The heterogeneous federated learning training method for government affairs models according to claim 3, characterized in that, Step S5, where each participant updates the model parameters of the knowledge transfer model based on the aggregated model parameters, includes: The aggregation model parameters are decrypted based on a preset homomorphic decryption algorithm to obtain the decrypted aggregation model parameters; The current model parameters are calculated based on the number of all participants and the decrypted aggregation model parameters, and these current model parameters are used as the model parameters of the knowledge transfer model.

5. A heterogeneous federated learning training device for models in the government affairs domain, characterized in that, The device includes multiple participants and a coordinator, each participant corresponding to a local personalized model and a knowledge transfer model, and comprises: The determination module is used to execute step S1, in which each participant determines a local personalized model based on the local task objective, determines a knowledge transfer model based on the local personalized model, and determines the initial values ​​of the model parameters of the local personalized model and the knowledge transfer model; wherein the input data samples used to train the local personalized model and the knowledge transfer model are the same, and the model parameter dimension of the knowledge transfer model is smaller than that of the model parameter dimension of the local personalized model. The first training module is used to execute step S2, whereby each participant keeps the model parameters of the knowledge transfer model unchanged, trains the local personalized model based on the local private dataset samples of each participant, obtains the training loss value of the local personalized model based on a preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and a preset local training loss threshold; wherein, the local private dataset samples refer to dataset samples held locally by each participant and not visible to other participants. The second training module is used to execute step S3, whereby each participant controls the model parameters of the local personalized model after training to remain unchanged, trains the knowledge transfer model based on the local private dataset samples, obtains the training loss value of the knowledge transfer model based on the preset loss function of the knowledge transfer model, and adjusts the model parameters of the knowledge transfer model based on the training loss value of the knowledge transfer model and the preset knowledge training loss threshold. The encrypted sending module is used to execute step S4, where each participant, when the training loss value of the knowledge transfer model is less than the knowledge training loss threshold or the number of training times is equal to the preset knowledge training number threshold, encrypts the model parameters of the trained knowledge transfer model based on a preset encryption algorithm and sends them to the coordinator, so that the coordinator aggregates and calculates the model parameters of the knowledge transfer model sent by all the participants, obtains aggregated model parameters, and sends them to each participant. The update module is used to execute step S5, in which each participant updates the model parameters of the knowledge transfer model based on the aggregation model parameters; The processing module is used to execute step S6 and repeat steps S2 to S5 until the training loss value of the local personalized model of each participant is less than the local training loss threshold or the number of local training times is equal to the preset local training time threshold, so as to obtain the local personalized model of each participant. Specifically, step S1, in which each participant determines a local personalized model based on a local task objective, determines a knowledge transfer model based on the local personalized model, and determines initial values ​​for the model parameters of the local personalized model and the knowledge transfer model, includes: Each of the aforementioned participants determines a local personalized model as f based on its local task objective. i (θ i ,X), i∈{1,…,n}, where, θ i Let X be the model parameters of the local personalized model of the i-th participant, X be the input data sample of the local personalized model, and n be the number of participants. Based on the local personalized model, the knowledge transfer model is determined as follows: in, Let X be the model parameters of the knowledge transfer model for the i-th participant, and let X be the input data sample for the knowledge transfer model. For the θ i and stated Perform parameter initialization processing to determine θ i and stated The initial value; wherein, the The dimension is less than θ i ; Each of the participating parties is a device A1 representing institution A, a device B1 representing institution B, and a device C2 representing institution C. Device A1 determines a local personalized model a1 for a text classification task based on the local task objective, and determines a knowledge transfer model a2 based on the local personalized model a1. Device B1 determines a local personalized model b1 for an image recognition task based on the local task objective, and determines a knowledge transfer model b2 based on the local personalized model b1. Device C1 determines a local personalized model c1 for a video target recognition task based on the local task objective, and determines a knowledge transfer model c2 based on the local personalized model c1. Step S2, in which each participant controls the model parameters of the knowledge transfer model to remain unchanged, trains the local personalized model based on samples from each participant's local private dataset, obtains the training loss value of the local personalized model based on a preset loss function of the local personalized model, and adjusts the model parameters of the local personalized model based on the training loss value of the local personalized model and a preset local training loss threshold, includes: Each participant obtains a local private dataset sample; wherein, the local private dataset sample of the i-th participant is... Let m represent the j-th sample. i This represents the number of data entries held by the i-th participant in their local private dataset. This represents the feature of the j-th sample data held by the i-th participant. This represents the label of the j-th sample data held by the i-th participant; wherein, the local private dataset sample refers to the dataset sample held by the device corresponding to each participant locally; different local personalized models correspond to different dataset samples, including at least one of text dataset samples, image dataset samples, and video dataset samples; Each participant keeps the model parameters of the knowledge transfer model unchanged, inputs the local private dataset samples into the local personalized model for processing, and obtains the training loss value of the local personalized model based on its loss function; wherein, the loss function of the local personalized model of the i-th participant is... Where, k i and Let m represent the weights of the loss function for training the local personalized model and the weights of the loss function for training the knowledge transfer model, respectively. i This represents the number of data entries in the local private dataset of the i-th participant. This represents the feature of the local personalized model of the i-th participant on the j-th sample data. The predicted value of the label. This represents the features of the knowledge transfer model of the i-th participant on j sample data. The predicted value of the label. This represents the loss function of the participant's local personalized model. The loss function representing the knowledge transfer model; When the training loss value of the local personalized model is greater than or equal to the local training loss threshold, the model parameters of the local personalized model are adjusted.

6. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the heterogeneous federated learning training method for government domain models as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the heterogeneous federated learning training method for government domain models as described in any one of claims 1-4.

8. A computer program product, characterized in that, The system includes a computer program, wherein the computer program is executed by a processor using the heterogeneous federated learning training method for government domain models as described in any one of claims 1-4.

Images