A federated fair framework training method and device based on neuron activation difference

Abstract

The application discloses a federal fair framework training method and device based on neuron activation difference. In the process of federal learning, in addition to requiring the client to upload the local model update parameter, the server also requires the client to upload the activation frequency of neurons, and the server evaluates the contribution degree of each client by calculating the sparsity of the neuron activation frequency of the client, so as to issue different quality global models to various clients.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of federated learning, and particularly relates to a federated fair framework training method and device based on neuron activation difference. BACKGROUND

[0002] Artificial intelligence technology has been widely applied in different fields since its development. However, due to factors such as company competition, security issues, and approval processes, there are still great obstacles to achieving true interconnection between data between companies or even within a company, resulting in the phenomenon of "data silos". At the same time, data privacy protection has become a global trend, posing a great challenge to the development of artificial intelligence technology. In view of the dilemma of data silos and data privacy, Google and WeBank have proposed different "federated learning" (Federated Learning) algorithm frameworks. The AI team of WeBank has released a general federated learning solution through the open source platform Federated AI Technology Enabler (FATE), and has led the international standard IEEE Federated Learning standardization, promoting the application of federated learning technology. Federated learning is actually an encrypted distributed machine learning technology, and participants can co-build models without disclosing underlying data and encrypted (obfuscated) forms of underlying data. It can enable each enterprise to keep its own data within the local area, and through parameter exchange under the encryption mechanism, each institution can collaborate without exchanging data to improve the effect of machine learning. It collaborates with thousands of participants in a distributed manner to iteratively train a specific machine learning model. Each participant does not share data, greatly protecting privacy and security, while the training effect is better than centralized learning. Nowadays, it has been widely applied in various fields, such as machine translation, image recognition, autonomous driving, natural language processing, etc.

[0003] At present, the application field of federated learning is becoming more and more extensive, and this distributed training has a major drawback. FL uses a parameter server to aggregate model updates from individual participants. However, most existing distributed or FL frameworks ignore an important aspect of participation: collaboration fairness. In particular, all participants can receive the same or similar models regardless of their contributions. This is obviously unfair because in practice, due to various reasons, such as different quality / quantity of data owned by different participants, which is the challenge brought by the heterogeneity of data distribution. Not all participants contribute equally. Therefore, data from some participants can lead to good model updates, while updates from some other participants can even harm model performance.

[0004] In the face of the Non-IID issue, how to better ensure the fairness of federated learning clients has become a key focus of attention. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a federated fairness framework training method and apparatus based on neuronal activation differences.

[0006] The objective of this invention is achieved through the following technical solution: a federated fairness framework training method based on neuronal activation differences, comprising the following steps:

[0007] (1) Initialize the federated learning training environment;

[0008] (2) The server will initialize the global model parameters. Send to each local client p i They began federal learning and training.

[0009] (3) Perform the training in the t-th round, with each client p participating in the training. i Based on the global model parameters obtained from the server Perform local training to obtain local model parameters. And record the activation value of each neuron in this round. For each client p i Provide a matrix with all parameters initialized to 0. Each client p i Based on activation value The updated matrix is ​​obtained through the update process.

[0010] (4) Each client p i The parameters of the trained local model and the updated matrix The parameters are uploaded to the server; the server aggregates the local model parameters uploaded by all clients to obtain the aggregated model update parameters. The server calculates p for each client. i sparsity Based on sparsity Calculate p for each client i Can be obtained from the server The number of parameters, and for each client p i Distribute the global model parameters for the next round separately

[0011] (5) Repeat steps (3)-(4) until the global model converges, completing the training of the personalized federated learning model.

[0012] Further, the step (1) is specifically: setting the overall training round E, the local data D, the overall number of clients participating in the federated learning k; each client p i contains H neurons.

[0013] Further, the step (3) specifically includes the following sub-steps:

[0014] (3.1) Each client p i obtains the trained local model parameters and records the activation value of each neuron in this round Calculate the neuron threshold of the client p i in the tth round The calculation formula is as follows:

[0015]

[0016] Where subscript h represents the hth neuron in the client p i , h = 1, 2, …, h, …, H; represents the activation value of the hth neuron of the client p i in the tth round of federated learning training;

[0017] (3.2) In each round of federated learning training, a matrix with parameter initialization of 0 is provided for each client p i

[0018]

[0019] When the hth parameter in the matrix is updated:

[0020] When the hth parameter in the matrix is updated:

[0021] Repeat the above steps to update the entire matrix to obtain the updated matrix Where, represents the hth parameter in the matrix .

[0022] Further, (4.1) each client p i uploads the trained local model parameters and the updated matrix to the server;

[0023] ​(4.2) The server aggregates the local model parameters uploaded by all clients to obtain the aggregated model update parameters. The calculation formula is as follows:

[0024]

[0025] (4.3) The server calculates p for each client. i sparsity The calculation formula is as follows:

[0026]

[0027] (4.4) Based on sparsity Calculate p for each client i Can be obtained from the server The number of parameters is calculated using the following formula:

[0028]

[0029] Among them, Size t Indicates the updated parameters of the aggregated model. The total number of parameters;

[0030] (4.5) The server determines the client p based on each client p i Number of parameters For each client p i Distribute the global model parameters for the next round separately The calculation formula is as follows:

[0031]

[0032] The Random function represents updating parameters from the aggregated model. Random selection Number of parameters.

[0033] The present invention also provides a federated fairness framework training device based on neuronal activation differences, comprising one or more processors for implementing the above-described federated fairness framework training method based on neuronal activation differences.

[0034] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-described federated fairness framework training method based on neuron activation differences.

[0035] The beneficial effects of the present application are that the method can improve the fairness of federated learning under Non-IID distribution. Due to Non-IID distribution, the drift of global model training is caused. Through the fairness mechanism, the influence between Non-IID distribution is reduced, and the model quality is maximized for the client according to its contribution. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 A schematic diagram of a federated fair framework training method based on neuron activation difference;

[0037] Figure 2 A schematic diagram of a local and cloud device;

[0038] Figure 3 A specific flowchart of a federated fair framework training method based on neuron activation difference;

[0039] Figure 4 A structure diagram of a federated fair framework training device based on neuron activation difference. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present application more clear and clear, the present application is further described in detail in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, not all examples. Based on the examples in the present application, all other examples obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0041] In the training process, in addition to sending the local trained model update parameters to the server, the local client will also upload the FRE-Matrix neuron activation frequency matrix collected by it to the server. The server calculates the sparsity according to the FRE-Matrix uploaded by each client to evaluate the contribution of each client, and then the server aggregates the model parameters uploaded by the client, and distributes different number of global model parameters according to the contribution, so as to issue global models of different quality, so as to improve the fairness of federated learning.

[0042] Embodiment 1

[0043] As shown in Figures 1-3 The present application provides a federated fair framework training method based on neuron activation difference, comprising the following steps:

[0044] (1) obtaining local data D;

[0045] In this embodiment, the MNIST dataset and the ImageNet dataset are used for defense effect verification, and both of them are common datasets of image classification models. The MNIST dataset contains a total of 60,000 grayscale images, each image has a size of 28*28, and is divided into 10 categories, with a total of 6,000 samples in each category, of which 50,000 are training samples and 10,000 are test samples; the ImageNet dataset contains a total of 1,000 categories, each category contains 1,000 samples, each picture is an RGB color image, each sample has a size of 224*224, and 30% of the pictures in each category are randomly selected as local data D.

[0046] (2) Initialize the federated learning training environment: set the overall training round E, the local data D, and the total number of clients k participating in federated learning; each client p i contains H neurons;

[0047] (3) The server sends the initial global model parameters to each local client p i , and starts federated learning training;

[0048] (4) Perform the t-th round of training, and each client p i participating in the training performs local training according to the global model parameters obtained from the server, trains to obtain local model parameters and records the activation value of each neuron in this round Each client p i provides a matrix with parameter initialization of 0 Each client p i updates the updated matrix according to the activation value

[0049] The step (4) specifically comprises the following sub-steps:

[0050] (4.1) Each client p i participating in the t-th round of federated learning training obtains trained local model parameters and records the activation value of each neuron in this round The neuron threshold value of the client p i in the t-th round is calculated as follows:

[0051]

[0052] Wherein, the subscript h represents the hth neuron in the client p i , h = 1, 2, …, h, …, H; ​In the t-th round of federated learning training, client p i The activation value of the h-th neuron;

[0053] (4.2) In each round of federated learning training, for each client p i Provide a matrix with all parameters initialized to 0.

[0054]

[0055] when Then update the matrix The h-th parameter:

[0056] when Then update the matrix The h-th parameter:

[0057] Repeat the above steps to update the entire matrix. Obtain the updated matrix in, Representation matrix The h-th parameter in the equation.

[0058] (5) Each client p i The parameters of the trained local model and the updated matrix The parameters are uploaded to the server; the server aggregates the local model parameters uploaded by all clients to obtain the aggregated model update parameters. The server calculates p for each client. i sparsity Based on sparsity Calculate p for each client i Can be obtained from the server The number of parameters, and for each client p i Distribute the global model parameters for the next round separately

[0059] Step (5) specifically includes the following sub-steps:

[0060] (5.1) Each client p i The parameters of the trained local model and the updated matrix Upload to the server;

[0061] (5.2) The server aggregates the local model parameters uploaded by all clients to obtain the aggregated model update parameters. The calculation formula is as follows:

[0062]

[0063] (5.3) The server calculates the sparsity of each client p i The calculation formula is as follows:

[0064]

[0065] (5.4) According to the sparsity of each client p i The number of parameters can be obtained from the server, and the calculation formula is as follows:

[0066]

[0067] Where Size t represents the total number of parameters of the aggregated model update parameters .

[0068] (5.5) The server delivers the global model parameters of the next round to each client p i according to the number of parameters of each client p i The calculation formula is as follows:

[0069]

[0070] Where the Random function represents randomly selecting the number of parameters from the aggregated model update parameters .

[0071] (6) Repeat steps (4)-(5) until the global model converges, and complete the training of the personalized federated learning model.

[0072] In the embodiment, we consider the actual scenario under IID and Non-IID data distribution. The Non-IID data distribution adopts Dirichlet distribution for data division. The division coefficient adopts the default 0.5.

[0073] Under the IID data distribution, since the local data volume of the client is consistent, the data label has no missing, so the global model precision obtained by each client is roughly consistent. It is illustrated that our method does not implement bias to the client that makes normal contribution.

[0074] ​​​​​​Under the Non-IID data distribution, due to the large difference in the number of local clients, even the inconsistent distribution of data labels, which leads to the significant difference in the activation of local model neurons of each client during local training, resulting in the significant difference in the contribution of each client when the server evaluates. Finally, the global model accuracy obtained by each client has a big difference, which shows that our method has a fair effect, and allocates high-quality global models to clients with high contribution, and low-quality global models to clients with low contribution.

[0075] Embodiment 2

[0076] Corresponding to the foregoing embodiment of the federated fair framework training method based on the difference in neuron activation, the present application also provides an embodiment of a federated fair framework training device based on the difference in neuron activation.

[0077] Referring to Figure 4 , the embodiment of the present application provides a federated fair framework training device based on the difference in neuron activation, which comprises one or more processors for implementing the federated fair framework training method based on the difference in neuron activation in the foregoing embodiments.

[0078] The embodiment of the federated fair framework training device based on the difference in neuron activation of the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be realized by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the device with data processing capability where it is located. From the hardware level, as shown in Figure 4 , it is a hardware structure diagram of the device with data processing capability where the federated fair framework training device based on the difference in neuron activation of the present application is located. In addition to the processor, memory, network interface, and non-volatile memory shown in Figure 4 , the device with data processing capability where the device in the embodiment is located usually includes other hardware according to the actual function of the device with data processing capability, which will not be described here.

[0079] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be described here.

[0080] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be seen from the part of the method embodiment. The device embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0081] The embodiment of the present application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the training method of the federal fair framework based on the neuron activation difference in the above embodiment.

[0082] The computer readable storage medium can be an internal storage unit of any data processing device in the above-mentioned embodiments, such as a hard disk or a memory. The computer readable storage medium can also be any data processing device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit of any data processing device and an external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0083] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A federated fair framework training method based on neuron activation difference, characterized in that, The method comprises the following steps: (1) initializing a federated learning training environment; The step (1) is specifically: setting the whole training round E, the local data D, and the total number of clients k participating in the federated learning; each client p i contains H neurons; (2) The server sends the initial global model parameters to each local client p i , and starts federated learning training; (3) Perform the t-th round of training, each client p participating in the training i According to the global model parameters obtained from the server respectively Perform local training to obtain local model parameters And record the activation value of each neuron in this round ; For each client p i Provide a matrix with parameter initialization of 0 , each client p i According to the activation value Update to obtain the updated matrix ; The step (3) specifically comprises the following sub-steps: (3.1) Each client p performing the federated learning training of the tth round i Get the trained local model parameters And record the activation value of each neuron in this round Calculate the neuron threshold value of client p i In the tth round The calculation formula is as follows: ； where subscript h denotes the hth neuron of client p i in the tth round of federated learning training, ; denotes the activation value of the hth neuron of client p i in the tth round of federated learning training. (3.2) For each round of federated learning training, for each client p i provides a matrix with all parameters initialized to 0 : ; When then update the matrix of the hth parameter: ; When then update the matrix of the hth parameter: ; Repeat the above steps to update the entire matrix , resulting in the updated matrix : where denotes the hth parameter in the matrix ​ (4) Each client p i The parameters of the trained local model and the updated matrix The parameters are uploaded to the server; the server aggregates the local model parameters uploaded by all clients to obtain the aggregated model update parameters. The server calculates p for each client. i sparsity According to sparsity Calculate p for each client i Can be obtained from the server The number of parameters, and for each client p i Distribute the global model parameters for the next round separately ; The step (4) specifically comprises the following sub-steps: (4.1) Each client p i upload the trained local model parameters and the updated matrix to the server; (4.2) The server aggregates the local model parameters uploaded by all clients to obtain aggregated model update parameters The calculation formula is as follows: ； (4.3) The server computes the sparsity of each client p i , the formula is as follows:​ ； (4.4) According to sparsity Compute each client p i Can be obtained from the server The number of parameters, the formula is as follows: ； wherein, denotes the updated model parameters after the aggregation the total number of parameters of (4.5) The server distributes the global model parameters of the next round to each client p i according to the number of parameters of each client p i , and the calculation formula is as follows: ​​ ； wherein the Random function indicates randomly selecting a number of parameters from the aggregated model update parameters parameters.​ (5) repeating the step (3) to the step (4) until the global model converges, and completing the training of the personalized federated learning model.

2. A device for training a federated fair framework based on neuron activation difference, characterized in that, The method comprises one or more processors for implementing the federated fair framework training method based on neuron activation difference in claim 1.

3. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the federated fair framework training method based on neuron activation difference in claim 1.

Images