Federal learning posterior reasoning method and system based on mutual information regularization

Abstract

The invention provides a federated learning posterior inference method and system based on mutual information regularization, and the method is applied to a client, and comprises the steps: receiving an initialized global model or an updated global model; initializing a local model according to the initialized global model or the updated global model, and determining the initialized local model and prior probability distribution of the local model; carrying out model training on the initialized local model by adopting local data, and determining a loss function of the local model; according to the loss function of the local model, performing first updating processing on the local model based on back propagation; performing second updating processing on the local model according to Gaussian noise sampled from preset normal distribution; determining a local model change amount; and sending the local model change quantity to a server side. Through the method and the device, mutual information regularization of the global model and the data is introduced, deviation in posterior estimation is relieved, and generalization ability of the global model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a federated learning posterior inference method and system based on mutual information regularization. Background Art

[0002] Federated learning is a distributed machine learning method that trains models on multiple local nodes without transferring the original data set to a central server. It is often used for privacy tasks, such as in healthcare and finance. The goal of standard federated learning is to train a global model with good generalization performance. After training, the parameters of the optimal global model are fixed values, that is, point estimates of the model parameters. However, from a probabilistic perspective, using point estimates of model parameters as model weights is prone to overfitting on the scarce training data on the client. More seriously, this method fails to evaluate the uncertainty of the model, which may lead to the model making overconfident decisions. Reliable uncertainty assessment is particularly important in some safety-critical federated learning applications, such as autonomous driving, healthcare, and finance.

[0003] Therefore, a posteriori inference of model weights is proposed, which can effectively prevent overfitting of scarce training data in theory, and provide a way to evaluate the uncertainty in weight estimation and propagate it to the model's prediction. However, in the federated learning scenario, the problem of local posteriori bias caused by data heterogeneity has not been effectively solved. Summary of the invention

[0004] In view of the defects in the prior art, the purpose of the present invention is to provide a federated learning posterior inference method and system based on mutual information regularization.

[0005] To achieve the above object, according to a first aspect of the present disclosure, a federated learning posterior inference method based on mutual information regularization is provided, which is applied to a client, wherein the client includes a local model, including:

[0006] Receive the initialized global model or updated global model sent by the server;

[0007] Initialize the local model according to the initialized global model or the updated global model sent by the server, and determine the initialized local model and a priori probability distribution of the local model;

[0008] Performing model training on the initialized local model using local data to determine a loss function of the local model, where the loss function of the local model includes a cross entropy loss between a training label and a true label of the local data and a regularization term of a prior probability distribution of the local model;

[0009] According to the loss function of the local model, performing a first update process on the local model based on back propagation to determine a local model that has undergone the first update process;

[0010] Performing a second update process on the local model that has undergone the first update process according to Gaussian noise sampled from a preset normal distribution to determine a local model that obeys a local posterior distribution;

[0011] Determine a local model change amount according to a local model obeying a local posterior distribution after the last first updating process and the last second updating process and an initial local model;

[0012] The local model change amount is sent to the server.

[0013] Optionally, performing a first update process on the local model based on back propagation according to the loss function of the local model to determine the local model that has undergone the first update process includes:

[0014] According to the loss function of the local model, determining the update gradient of the local model parameters based on back propagation;

[0015] The local model that has undergone the first update processing is determined by subtracting the product of the update gradient of the local model and the learning rate of the local model parameters.

[0016] Optionally, performing a second update process on the local model that has undergone the first update process according to Gaussian noise sampled from a preset normal distribution to determine a local model that obeys a local posterior distribution includes:

[0017] Perform noise sampling in a preset normal distribution to determine the sampled Gaussian noise;

[0018] The sampled Gaussian noise is summed with the local model that has undergone the first update process to determine the local model that obeys the local posterior distribution.

[0019] Optionally, the method further comprises:

[0020] When the update times of the first update processing and the second update processing reach a preset update threshold, the local model stops updating and determines the local model that obeys the local posterior distribution after the last first update processing and the last second update processing.

[0021] Optionally, determining the amount of change of the local model according to the local model obeying the local posterior distribution after the last first updating process and the last second updating process and the initial local model comprises:

[0022] The local model change amount is determined by subtracting the local model that obeys the local posterior distribution after the last first update processing and the last second update processing from the initial local model.

[0023] According to a second aspect of the present disclosure, a federated learning posterior inference method based on mutual information regularization is provided, which is applied to a server side and includes:

[0024] Initialize the global model and determine the initialized global model;

[0025] Sending the initialized global model to the client;

[0026] Receiving the local model change amount sent by the client;

[0027] Determining an updated global model according to the local model change amount sent by the client;

[0028] sending the updated global model to the client;

[0029] When the updated global model sampled converges to samples that obey the optimal global posterior distribution, the target global model is determined.

[0030] Optionally, determining an updated global model according to the local model change amount sent by the client includes:

[0031] Performing global aggregation processing on the local model change amount sent by the client to determine an aggregated change amount;

[0032] Performing sliding average processing on the aggregate change amount to determine the aggregate change amount after the sliding average processing;

[0033] The aggregated change amount processed by sliding average is linearly added to the original global model of this round to determine an updated global model.

[0034] According to a third aspect of the present disclosure, a federated learning posterior inference system based on mutual information regularization is provided, which is applied to a client, wherein the client includes a local model, including:

[0035] A client receiving module receives an initialized global model or an updated global model sent by a server;

[0036] A client initialization module, used to initialize the local model according to the initialized global model or the updated global model sent by the server, and determine the initialized local model and the prior probability distribution of the local model;

[0037] A client model training module, used to perform model training on the initialized local model using local data, and determine a loss function of the local model, where the loss function of the local model includes a cross entropy loss between a training label and a true label of the local data and a regularization term of a prior probability distribution of the local model;

[0038] A first updating module of the client, configured to perform a first updating process on the local model based on back propagation according to the loss function of the local model, and determine the local model that has undergone the first updating process;

[0039] A second updating module of the client is used to perform a second updating process on the local model that has undergone the first updating process according to Gaussian noise sampled from a preset normal distribution, so as to determine a local model that obeys a local posterior distribution;

[0040] A client model change amount determination module, used to determine a local model change amount according to a local model obeying a local posterior distribution after the last first update process and the last second update process and an initial local model;

[0041] The client sending module is used to send the local model change amount to the server.

[0042] According to a fourth aspect of the present disclosure, a federated learning posterior inference system based on mutual information regularization is provided, which is applied to a server side and includes:

[0043] A server-side initialization module, used to initialize the global model and determine the initialized global model;

[0044] A first sending module on the server side is used to send the initialized global model to the client;

[0045] A server-side receiving module, used for receiving the local model change amount sent by the client;

[0046] A server-side global aggregation module, used to determine an updated global model according to the local model change amount sent by the client;

[0047] A second sending module on the server side, used for sending the updated global model to the client;

[0048] The server-side target global model determination module is used to determine the target global model when the sampled updated global model converges to a sample that obeys the optimal global posterior distribution.

[0049] According to a fifth aspect of the present disclosure, a federated learning posterior inference system based on mutual information regularization is provided, comprising:

[0050] A local model updating module, configured to perform a first updating process on the local model based on back propagation according to a loss function of the local model at a local training node, and perform a second updating process on the local model after the first updating process by using Gaussian noise sampled from a preset normal distribution;

[0051] A model change amount determination module, configured to determine a local model change amount when the update times of the first update process and the second update process both reach a preset update threshold;

[0052] The global aggregation module is used to globally aggregate the local model changes sent by the client in this round and update the global model;

[0053] The target global model determination module is used to determine the target global model. When the sampled updated global model converges to a sample that obeys the optimal global posterior distribution, it is determined as the target global model.

[0054] The communication module is used to transmit the local model change of the client to the server, and transmit the initialized global model or the updated global model of the server to the client.

[0055] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:

[0056] Through the above technical solution, the client receives the initialized global model or the updated global model sent by the server, and based on the initialized global model or the updated global model, initializes the local model and obtains the prior probability distribution of the local model. The client uses local data to train the model to perform local posterior inference. During the local posterior inference process, a mutual information regularization term between the global model and the data is introduced into the loss function of the local model, that is, a regularization term of the prior probability distribution of the local model, so as to obtain a local model that obeys the local posterior distribution, alleviate the deviation in the global posterior inference, and then improve the generalization ability of the global model, and achieve better performance in uncertainty calibration.

[0057] The embodiments of the present disclosure perform local a posteriori inference based on the stochastic gradient Langevin dynamics algorithm to achieve efficient sampling of the optimal a posteriori of the local objective function with mutual information regularization. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0059] Figure 1 The present invention is a flowchart of a mutual information regularized federated learning posterior inference method applied to a client according to an exemplary embodiment.

[0060] Figure 2 The present invention is a flowchart of a mutual information regularized federated learning posterior inference method applied to a server according to an exemplary embodiment.

[0061] Figure 3 The figure is a schematic diagram of signaling interaction between a client and a server according to an exemplary embodiment.

[0062] Figure 4 It is a block diagram of a mutual information regularized federated learning posterior inference system applied to a client according to an exemplary embodiment.

[0063] Figure 5 It is a block diagram of a federated learning posterior inference system based on mutual information regularization applied to a server side according to an exemplary embodiment.

[0064] Figure 6 It is a block diagram of a federated learning posterior inference system based on mutual information regularization according to an exemplary embodiment. DETAILED DESCRIPTION

[0065] The present disclosure is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present disclosure, but are not intended to limit the present disclosure in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present disclosure. These all fall within the scope of protection of the present disclosure.

[0066] Figure 1 The present invention is a flowchart of a mutual information regularized federated learning posterior inference method applied to a client according to an exemplary embodiment.

[0067] like Figure 1 As shown, the present disclosure provides a federated learning posterior inference method based on mutual information regularization, which is applied to a client, and the client includes a local model, including S11 to S17.

[0068] S11, receiving an initialized global model or an updated global model sent by the server.

[0069] Among them, the initialized global model and the updated global model are global model samples that obey the global posterior distribution.

[0070] S12, initializing the local model according to the initialized global model or the updated global model sent by the server, and determining the initialized local model and the prior probability distribution of the local model.

[0071] The prior probability distribution of the local model is expressed as a Gaussian distribution with the initialized global model as the mean and the global uncertainty accumulated in the previous round of training as the variance.

[0072] S13, using local data to perform model training on the initialized local model to determine the loss function of the local model.

[0073] As an example, local data is input into an initialized local model, local a posteriori inference is performed based on the Stochastic Gradient Langevin Dynamics (SGLD) algorithm, training labels of the local data are output, and model training is performed on the local model.

[0074] Local posterior inference is performed based on the stochastic gradient Langevin dynamics algorithm to achieve efficient sampling of the optimal posterior of the local objective function with mutual information regularization.

[0075] Among them, the loss function of the local model includes the cross entropy loss between the training labels and the true labels of the local data and the regularization term of the prior probability distribution of the local model.

[0076] The method for determining a regularization term of a prior probability distribution of a local model comprises:

[0077] The prior probability distribution of the local model is regularized, that is, the natural logarithm of the prior probability distribution of the local model is taken, and then multiplied by a preset weight coefficient a to determine the regularization term of the prior probability distribution of the local model.

[0078] The loss function of the local model includes the regularization term of the prior probability distribution of the local model, that is, the regularization term that introduces the mutual information between the model and the data, which effectively alleviates the deviation generated in the local posterior inference process and improves the generalization ability of the local model.

[0079] S14, performing a first update process on the local model based on back propagation according to the loss function of the local model, and determining the local model that has undergone the first update process.

[0080] S15, performing a second update process on the local model that has undergone the first update process according to Gaussian noise sampled from a preset normal distribution, to determine a local model that obeys the local posterior distribution.

[0081] Among them, the default normal distribution is The normal distribution of .

[0082] In the present disclosure, steps S14 and S15 are executed in a loop until the update times of the first update process and the second update process both reach a preset update threshold, and the updating of the local model is stopped.

[0083] When the update times of the first update process and the second update process of the local model both reach a preset threshold, the local model converges to the local optimal posterior distribution, and the local model obtained that obeys the local posterior distribution is the target local model.

[0084] S16, determining a change amount of the local model according to the local model obeying the local posterior distribution after the last first updating process and the last second updating process and the initial local model.

[0085] S17, sending the local model change amount to the server.

[0086] Through the above technical solution, the client receives the initialized global model sent by the server, that is, the model sample that obeys the global posterior distribution, and based on the initialized global model, initializes the local model and obtains the prior probability distribution of the local model. The client uses local data to train the model for local posterior inference. During the local posterior inference process, the mutual information regularization term between the global model and the data is introduced into the loss function of the local model, that is, the regularization term of the prior probability distribution of the local model, so as to obtain a local model that obeys the local posterior distribution, alleviate the deviation in the global posterior inference, and then improve the generalization ability of the global model, and achieve better performance in uncertainty calibration.

[0087] In a possible embodiment, S14, performing a first update process on the local model based on back propagation according to the loss function of the local model to determine the local model that has undergone the first update process, may include S101 to S102.

[0088] S101, according to the loss function of the local model, determine the update gradient of the local model parameters based on back propagation.

[0089] S102, subtract the local model from the product of the update gradient of the local model parameters and the learning rate to determine the local model that has undergone the first update processing.

[0090] Here, the learning rate is denoted as η.

[0091] In a possible embodiment, S15, performing a second update process on the local model that has undergone the first update process according to Gaussian noise sampled from a preset normal distribution to determine a local model that obeys the local posterior distribution, may include S201 to S202.

[0092] S201, performing noise sampling in a preset normal distribution to determine the sampled Gaussian noise.

[0093] Among them, the default normal distribution is The normal distribution of .

[0094] S202, summing the sampled Gaussian noise with the local model that has undergone the first update process, to determine a local model that obeys the local posterior distribution.

[0095] As an example, the sampled Gaussian noise is added to the parameters of the local model after the first update process to obtain a local model that obeys the local posterior distribution.

[0096] In a possible embodiment, a federated learning posterior inference method based on mutual information regularization also includes step S18.

[0097] S18, when the update times of the first update process and the second update process both reach the preset update threshold, the local model stops updating, and determines the local model that obeys the local posterior distribution after the last first update process and the last second update process.

[0098] Among them, when the update times of the first update processing and the second update processing reach the preset update threshold, it indicates that the local model converges to the local optimal posterior distribution, and the local model that obeys the local posterior distribution after the last first update processing and the last second update processing is used as the target local model.

[0099] In a possible embodiment, S16, determining the amount of change of the local model according to the local model obeying the local posterior distribution after the last first update process and the last second update process and the initial local model, may include:

[0100] The local model that obeys the local posterior distribution after the last first update processing and the last second update processing is subtracted from the initial local model to determine the change amount of the local model.

[0101] Figure 2 The present invention is a flowchart of a mutual information regularized federated learning posterior inference method applied to a server according to an exemplary embodiment.

[0102] like Figure 2 As shown, the present disclosure also provides a federated learning posterior inference method based on mutual information regularization, which is applied to the server side, including S21 to S26.

[0103] S21, initializing the global model, and determining the initialized global model.

[0104] S22, sending the initialized global model to the client.

[0105] Among them, the initialized global model represents the global model sample that obeys the global posterior distribution.

[0106] S23, receiving the local model change amount sent by the client.

[0107] S24, determining an updated global model according to the local model change amount sent by the client.

[0108] S25, sending the updated global model to the client.

[0109] The updated global model is sent to the client to start a new round of updating of the local model and the global model.

[0110] S26, when the sampled updated global model converges to a sample that obeys the optimal global posterior distribution, determine the target global model.

[0111] Among them, each updated global model is sampled until the sampled updated global model samples converge to samples that obey the optimal global posterior distribution and are determined as the target global model.

[0112] In a possible embodiment, S24 may include S301 to S303.

[0113] S301, performing global aggregation processing on the local model change amount sent by the client to determine the aggregated change amount.

[0114] Among them, the global aggregation processing adopts the weighted summation method.

[0115] S302, performing sliding average processing on the aggregate change amount to determine the aggregate change amount after the sliding average processing.

[0116] S303: linearly add the aggregated change amount processed by sliding average to the original global model of this round to determine an updated global model.

[0117] The original global model of this round in step S303 is the global model that has not been updated in this round of updating.

[0118] Figure 3 The figure is a schematic diagram of signaling interaction between a client and a server according to an exemplary embodiment.

[0119] like Figure 3 As shown, the present disclosure provides a signaling interaction process between a client and a server for a federated learning posterior inference method based on mutual information regularization, including S31 to S44.

[0120] S31, the server initializes the global model and determines the initialized global model.

[0121] S32, the server sends the initialized global model to the client.

[0122] S33, the client receives the initialized global model or the updated global model sent by the server.

[0123] S34, the client initializes the local model according to the initialized global model or the updated global model sent by the server, and determines the initialized local model and the prior probability distribution of the local model.

[0124] S35, the client uses local data to perform model training on the initialized local model to determine the loss function of the local model.

[0125] S36, the client performs a first update process on the local model based on back propagation according to the loss function of the local model, and determines the local model that has undergone the first update process.

[0126] S37, the client performs a second update process on the local model that has undergone the first update process according to Gaussian noise sampled from a preset normal distribution, and determines a local model that obeys the local posterior distribution.

[0127] S38, when the number of updates of the first update process and the second update process of the client reaches the preset update threshold, the local model stops updating and determines the local model that obeys the local posterior distribution after the last first update process and the last second update process.

[0128] S39, the client determines the amount of change of the local model according to the local model that obeys the local posterior distribution after the last first update processing and the last second update processing and the initial local model.

[0129] S40, the client sends the local model change amount to the server.

[0130] S41, the server receives the local model change sent by the client.

[0131] S42: The server determines an updated global model according to the local model change amount sent by the client.

[0132] S43, sending the updated global model to the client.

[0133] S44, when the sampled updated global model converges to a sample that obeys the optimal global posterior distribution, the server determines the target global model.

[0134] In the present disclosure, after steps S31 to S35 are executed, steps S36 to S37 are executed in a loop until the update times of the first update process and the second update process both reach a preset update threshold.

[0135] After step S43 is executed, steps S33 to S43 are executed in a loop until the updated global model converges to the optimal global posterior distribution, and finally step S44 is executed to determine the target global model.

[0136] The technical solution provided by the above embodiment of the present disclosure is further described in detail below in conjunction with a specific application example.

[0137] In the medical scenario, each medical institution acts as a client and collects patient data through its internal medical equipment and systems (such as electronic medical record systems, medical imaging equipment, etc.), including patient medical records, imaging data, diagnostic records, etc. In order to protect patient privacy, these raw data will not be directly transmitted to the server, but will be trained locally through the federated learning method.

[0138] Specifically, a trusted third-party medical cooperation organization is used as the server side, and each medical institution is used as the client side to build a communication network between the server side and each medical institution, namely the communication module. In this process, the medical institutions do not need to communicate directly, and the server and the client do not transmit the original patient data or diagnosis information. Instead, the medical deep learning model is trained on the server side by sending the processed model parameters and weights.

[0139] Based on the technical solutions provided by the above embodiments of the present disclosure, a medical deep learning model with excellent generalization performance can be quickly trained. The specific implementation steps are as follows:

[0140] Step 1: The central server initializes the global medical model and randomly selects the medical clients participating in this round of training. The central server sends a unified initialized global medical model to the clients of each medical institution participating in this round of training.

[0141] Step 2: Each medical institution client participating in this round of training receives the initialized global medical model, initializes the local medical model based on the initialized global medical model, and uses the following formula to calculate the prior probability distribution of the local medical model.

[0142]

[0143] Among them, p(w) represents the prior probability distribution of the local medical model, w represents the local medical model parameters, represents the default normal distribution, w t represents the global medical model received by the clients of each medical institution participating in the tth round of training, m represents the number of nodes participating in the tth round of training, K represents the number of updates of the local medical model, η represents the learning rate, α represents the weight coefficient, and I represents the unit matrix of the same dimension as the local medical model weight.

[0144] Step 3: Each medical institution client inputs local patient data into the initialized local medical model for model training and determines the loss function of the local medical model. The loss function is as follows:

[0145]

[0146] Where L represents the loss function of the local medical model, represents the cross entropy loss between the training labels and the true labels of local patient data, represents the regularization term of the prior probability distribution of the local medical model, Represents the local medical model of the local node at the kth iteration of the tth round.

[0147] According to the loss function, the local medical model is updated based on back propagation.

[0148] Step 4: Each medical institution client samples Gaussian noise in a preset normal distribution, as follows:

[0149]

[0150] Among them, h t,k represents the sampled Gaussian noise, Represents the default normal distribution.

[0151] The Gaussian noise is added to the updated local medical model in step 3 to obtain a new local medical model.

[0152] Step 5: After repeating steps 3 to 4 K times (K ≥ 1), calculate and save the change in the local medical model:

[0153]

[0154] in, represents the local medical model of node i at the last local update (i.e., K times) in round t, represents the initial local medical model, Represents the change in the local medical model of node i in round t.

[0155] Send the changes of the local medical model in each round to the central server.

[0156] Step 6: The central server receives the local medical model change amount, and performs global aggregation processing on the local medical model change amount to determine the aggregate change amount; the aggregate change amount is calculated using the following formula:

[0157]

[0158] Among them, Δw tRepresents the aggregate change in round t.

[0159] Perform sliding average processing on the aggregate variable:

[0160]

[0161] Where Δv t represents the aggregate change after sliding average processing, β represents the momentum sliding average hyperparameter, Δv t-1 Indicates the aggregate change in the previous round after sliding average processing.

[0162] Δv0 is initially 0.

[0163] Update the global medical model:

[0164] w t+1 =w t +η g Δv t ,

[0165] Among them, w t+1 represents the updated global medical model, w t represents the original global medical model of this round, Δv t represents the aggregate change after sliding average processing, η g Represents the global learning rate.

[0166] The updated global medical model is sent to the client for the next round of updates.

[0167] Step 7: Repeat steps 2 to 6 until the global medical model converges to the optimal global posterior distribution, and then determine the target global medical model.

[0168] By introducing the mutual information regularization between the global medical model and local patient data, it not only accelerates the convergence of the global medical model, but also effectively reduces the posterior estimation bias that may occur in distributed medical scenarios. At the same time, the patient's privacy data is always retained in the local medical institution and will not be transmitted to the server, thus providing higher security for the privacy protection of patient data. It has important application value in medical data analysis, disease diagnosis, etc., especially in medical scenarios where privacy protection is crucial, and can achieve more accurate model training and output of diagnostic results.

[0169] In a possible embodiment, based on the CIFRA-10, CIFAR-100 and Shakespeare data sets, the non-iid-1 and non-iid-2 non-independent data partitioning methods are adopted to verify the generalization performance of a federated learning posterior inference method based on mutual information regularization proposed in the present invention and the methods of FedAvg, FedM, MimeLite, SCAFFOLD, FedBE, FedPA, FedEP(I) and FALD.

[0170] Among them, in the non-independent data partitioning methods non-iid-1 and non-iid-2, each local training node is sampled according to the Dirichlet distribution, and the local data of each local training node includes an unfixed number of image data of each category of image data sets, and the amount of data of each category of image data sets is different. The degree of heterogeneity of non-iid-1 is greater than that of non-iid-2.

[0171] The accuracy comparison of different methods is shown in Table 1:

[0172]

[0173] Table 1

[0174] It can be seen from Table 1 that the federated learning posterior inference method based on mutual information regularization proposed in the present disclosure can perform excellent accuracy on the non-iid-2 non-independent data partitioning method, indicating that the global model determined by the federated learning posterior inference method based on mutual information regularization proposed in the present disclosure can effectively alleviate the deviation in the posterior estimation and improve the generalization ability of the global model.

[0175] Figure 4 It is a block diagram of a mutual information regularized federated learning posterior inference system applied to a client according to an exemplary embodiment.

[0176] Based on the same concept, the present disclosure also provides a federated learning posterior inference system 100 based on mutual information regularization, which is applied to a client, wherein the client includes a local model, such as Figure 4 As shown, it includes: a client receiving module 110, a client initialization module 120, a client model training module 130, a client first updating module 140, a client second updating module 150, a client model change amount determining module 160, and a client sending module 170.

[0177] The client receiving module 110 receives the initialized global model or the updated global model sent by the server;

[0178] The client initialization module 120 is used to initialize the local model according to the initialized global model or the updated global model sent by the server, and determine the initialized local model and the prior probability distribution of the local model;

[0179] The client model training module 130 is used to perform model training on the initialized local model using local data, and determine a loss function of the local model, where the loss function of the local model includes a cross entropy loss between a training label and a true label of the local data and a regularization term of a prior probability distribution of the local model;

[0180] The first updating module 140 of the client is used to perform a first updating process on the local model based on back propagation according to the loss function of the local model, and determine the local model after the first updating process;

[0181] The second updating module 150 of the client is used to perform a second updating process on the local model that has undergone the first updating process according to Gaussian noise sampled from a preset normal distribution, so as to determine a local model that obeys a local posterior distribution;

[0182] The client model change amount determination module 160 is used to determine the local model change amount according to the local model obeying the local posterior distribution after the last first update processing and the last second update processing and the initial local model;

[0183] The client sending module 170 is used to send the local model change amount to the server.

[0184] Through the above technical solution, the client receives the initialized global model sent by the server, that is, the model sample that obeys the global posterior distribution, and based on the initialized global model, initializes the local model and obtains the prior probability distribution of the local model. The client uses local data to train the model for local posterior inference. During the local posterior inference process, the mutual information regularization term between the global model and the data is introduced into the loss function of the local model, that is, the regularization term of the prior probability distribution of the local model, so as to obtain a local model that obeys the local posterior distribution, alleviate the deviation in the global posterior inference, and then improve the generalization ability of the global model, and achieve better performance in uncertainty calibration.

[0185] Regarding the embodiment of the above system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0186] Figure 5It is a block diagram of a federated learning posterior inference system based on mutual information regularization applied to a server side according to an exemplary embodiment.

[0187] Based on the same concept, the present disclosure also provides a federated learning posterior reasoning system 200 based on mutual information regularization, which is applied to the server side, including: a server-side initialization module 210, a server-side first sending module 220, a server-side receiving module 230, a server-side global aggregation module 240, a server-side second sending module 250, and a server-side target global model determination module 260.

[0188] The server-side initialization module 210 is used to initialize the global model and determine the initialized global model;

[0189] A first sending module 220 on the server side is used to send the initialized global model to the client;

[0190] The server-side receiving module 230 is used to receive the local model change amount sent by the client;

[0191] A server-side global aggregation module 240, configured to determine an updated global model according to the local model change amount sent by the client;

[0192] A second sending module 250 on the server side is used to send the updated global model to the client;

[0193] The server-side target global model determination module 260 is used to determine the target global model when the sampled updated global model converges to a sample that obeys the optimal global posterior distribution.

[0194] Regarding the embodiment of the above system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0195] Figure 6 It is a block diagram of a federated learning posterior inference system based on mutual information regularization according to an exemplary embodiment.

[0196] In a possible embodiment, the present disclosure also provides a federated learning posterior inference system 300 based on mutual information regularization, including: a local model update module 310, a model change determination module 320, a global aggregation module 330, a target global model determination module 340 and a communication module 350.

[0197] Among them, the local model update module 310 and the model change determination module 320 are set on the client side, the global aggregation module 330 and the target global model determination module 340 are set on the server side, and the communication module 350 is used to transmit the information change amount of the client side and the global model of the server side.

[0198] A local model updating module 310 is used to perform a first updating process on the local model based on back propagation according to the loss function of the local model at the local training node, and perform a second updating process on the local model after the first updating process by using Gaussian noise sampled from a preset normal distribution;

[0199] A model change amount determination module 320, configured to determine a local model change amount when the update times of the first update process and the second update process both reach a preset update threshold;

[0200] The global aggregation module 330 is used to globally aggregate the local model changes sent by the client in this round and update the global model;

[0201] The target global model determination module 340 is used to determine the target global model. When the sampled updated global model converges to a sample that obeys the optimal global posterior distribution, it is determined as the target global model.

[0202] The communication module 350 is used to transmit the local model change of the client to the server, and transmit the initialized global model or the updated global model of the server to the client.

[0203] In one possible embodiment, in an image classification task in a user's photo album, historical input images in the user's device are analyzed, and a deep learning model is used to train the image classification task, thereby assisting the user in automatically classifying newly taken pictures.

[0204] For privacy reasons, the server cannot collect private photos on the user's device, which makes it difficult for the deep learning model to obtain sufficient training data, thereby limiting the effective training of the model. The limited amount of data can easily lead to overfitting of the model and reduce the model's generalization ability.

[0205] A federated learning posterior inference system based on mutual information regularization selects a trusted third-party organization as the server side and a user device as the client side, builds a communication network between the server side and the client side, and adopts the federated learning posterior inference method based on mutual information regularization provided by the present invention to realize federated learning.

[0206] The details are as follows:

[0207] Step 1: A third-party organization initializes the global model;

[0208] Step 2: The third-party agency sends the initialized global model to the client;

[0209] Step 3: The user device receives the initialized global model sent by the server and initializes the local model;

[0210] Step 4: The user device uses the local photo to perform model training on the initialized local model to determine the loss function of the local model;

[0211] Step 5: The user equipment performs a first update process on the local model based on back propagation according to the loss function of the local model, and determines an updated local model;

[0212] Step 6: The user equipment performs a second update process on the local model that has undergone the first update process according to Gaussian noise sampled from a preset normal distribution, and determines an updated local model;

[0213] Step 7: The user equipment determines a change amount of the local model according to the local model after the last first update process, the last second update process, and the initialized local model;

[0214] Step 8: The user device sends the local model change to a third-party organization;

[0215] Step 9: The third-party organization receives the local model change amount sent by the user device;

[0216] Step 10: The third-party agency aggregates the local model changes and updates the global model;

[0217] Step 11: The third-party organization sends the updated global model to the user device for the next round of training;

[0218] The above steps 2 to 11 are repeated until the updated global model converges to the optimal global posterior distribution.

[0219] Step 12: When the sampled and updated global model converges, the third-party agency determines it as the target global model.

[0220] Through the above steps 1 to 12, federated learning between the user device and the third-party organization is realized. In this architecture, each user device does not need to communicate directly with each other, and no original data is transmitted between the third-party organization and the user device to protect privacy data.

[0221] The above describes the specific embodiments of the present disclosure. It should be understood that the present disclosure is not limited to the above specific embodiments, and those skilled in the art can make various deformations or modifications within the scope of the claims, which does not affect the essence of the present disclosure. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A federated learning posterior inference method based on mutual information regularization, characterized in that: Applied to a client, the client includes a local model, including: Receive the initialized global model or updated global model sent by the server; Initialize the local model according to the initialized global model or the updated global model sent by the server, and determine the initialized local model and a priori probability distribution of the local model; Performing model training on the initialized local model using local data to determine a loss function of the local model, where the loss function of the local model includes a cross entropy loss between a training label and a true label of the local data and a regularization term of a prior probability distribution of the local model; According to the loss function of the local model, performing a first update process on the local model based on back propagation to determine a local model that has undergone the first update process; Performing a second update process on the local model that has undergone the first update process according to Gaussian noise sampled from a preset normal distribution, to determine a local model that obeys a local posterior distribution; Determine a local model change amount according to a local model obeying a local posterior distribution after the last first updating process and the last second updating process and an initial local model; The local model change amount is sent to the server.

2. The method according to claim 1, characterized in that The step of performing a first update process on the local model based on back propagation according to the loss function of the local model to determine the local model after the first update process comprises: According to the loss function of the local model, determining the update gradient of the local model parameters based on back propagation; The local model that has undergone the first update processing is determined by subtracting the product of the update gradient of the local model and the learning rate of the local model parameters.

3. The method according to claim 1, characterized in that The performing a second updating process on the local model that has undergone the first updating process according to Gaussian noise sampled from a preset normal distribution to determine a local model that obeys a local posterior distribution includes: Perform noise sampling in a preset normal distribution to determine the sampled Gaussian noise; The sampled Gaussian noise is summed with the local model that has undergone the first update process to determine the local model that obeys the local posterior distribution.

4. The method according to claim 1, characterized in that: The method further comprises: When the update times of the first update processing and the second update processing reach a preset update threshold, the local model stops updating and determines the local model that obeys the local posterior distribution after the last first update processing and the last second update processing.

5. The method according to claim 1, characterized in that The determining of the local model change amount according to the local model obeying the local posterior distribution after the last first updating process and the last second updating process and the initial local model comprises: The local model change amount is determined by subtracting the local model that obeys the local posterior distribution after the last first update processing and the last second update processing from the initial local model.

6. A federated learning posterior inference method based on mutual information regularization, characterized in that: Applied to the server side, including: Initialize the global model and determine the initialized global model; Sending the initialized global model to the client; Receiving the local model change amount sent by the client; Determining an updated global model according to the local model change amount sent by the client; sending the updated global model to the client; When the updated global model sampled converges to samples that obey the optimal global posterior distribution, the target global model is determined.

7. The method according to claim 6, characterized in that The step of determining an updated global model according to the local model change amount sent by the client comprises: Performing global aggregation processing on the local model change amount sent by the client to determine an aggregated change amount; Performing sliding average processing on the aggregate change amount to determine the aggregate change amount after the sliding average processing; The aggregated change amount processed by sliding average is linearly added to the original global model of this round to determine an updated global model.

8. A federated learning posterior inference system based on mutual information regularization, characterized in that: Applied to a client, the client includes a local model, including: A client receiving module receives an initialized global model or an updated global model sent by a server; A client initialization module, used to initialize the local model according to the initialized global model or the updated global model sent by the server, and determine the initialized local model and the prior probability distribution of the local model; A client model training module, used to perform model training on the initialized local model using local data, and determine a loss function of the local model, where the loss function of the local model includes a cross entropy loss between a training label and a true label of the local data and a regularization term of a prior probability distribution of the local model; A first updating module of the client, configured to perform a first updating process on the local model based on back propagation according to the loss function of the local model, and determine the local model that has undergone the first updating process; A second updating module of the client is used to perform a second updating process on the local model that has undergone the first updating process according to Gaussian noise sampled from a preset normal distribution, so as to determine a local model that obeys a local posterior distribution; A client model change amount determination module, used to determine a local model change amount according to a local model obeying a local posterior distribution after the last first update process and the last second update process and an initial local model; The client sending module is used to send the local model change amount to the server.

9. A federated learning posterior inference system based on mutual information regularization, characterized in that: Applied to the server side, including: A server-side initialization module, used to initialize the global model and determine the initialized global model; A first sending module on the server side is used to send the initialized global model to the client; A server-side receiving module, used for receiving the local model change amount sent by the client; A server-side global aggregation module, used to determine an updated global model according to the local model change amount sent by the client; A second sending module on the server side, used for sending the updated global model to the client; The server-side target global model determination module is used to determine the target global model when the sampled updated global model converges to a sample that obeys the optimal global posterior distribution.

10. A federated learning posterior inference system based on mutual information regularization, characterized in that: include: A local model updating module, configured to perform a first updating process on the local model based on back propagation according to a loss function of the local model at a local training node, and perform a second updating process on the local model after the first updating process by using Gaussian noise sampled from a preset normal distribution; A model change amount determination module, configured to determine a local model change amount when the update times of the first update process and the second update process both reach a preset update threshold; The global aggregation module is used to globally aggregate the local model changes sent by the client in this round and update the global model; The target global model determination module is used to determine the target global model. When the sampled updated global model converges to a sample that obeys the optimal global posterior distribution, it is determined as the target global model. The communication module is used to transmit the local model change of the client to the server, and transmit the initialized global model or the updated global model of the server to the client.

Images