Method, device and medium for service execution based on packet generation and federated learning

Abstract

This invention discloses a business execution method, device, and medium based on group generation and federated learning, applied to a client, comprising: receiving a benchmark dataset, an initialized business execution network, and a generative adversarial network (GAN) from a server; training the business execution network using a local dataset to perform inference on the benchmark dataset, and statistically analyzing the accuracy vector of each label; each group participating in each iteration of the federated learning training process, including training the GAN; inputting Gaussian noise into several randomly selected global adversarial generative models aggregated by the server to obtain a synthetic sample set; wherein, each group is obtained by the server grouping the client based on the accuracy vector composed of threshold truncation and random flipping; training the re-initialized business execution network using the synthetic sample set and the local dataset; and receiving the global business execution network aggregated by the server to execute business tasks.

Description

Technical Field

[0001] This invention pertains to federated learning, specifically relating to a business execution method, device, and medium based on group generation and federated learning. Background Technology

[0002] To address the data silo problem in artificial intelligence, federated learning optimizes models by iteratively aggregating the parameters of models trained locally on multiple clients using a global server. Unlike centralized machine learning, clients do not expose their raw data during federated learning, thus effectively maintaining data privacy, which is why this paradigm is widely adopted.

[0003] However, in practice, client-side data is often heterogeneous, i.e., non-independent and identically distributed (Non-IID), including feature distribution, label distribution, and data skew. Training a model in a federated manner on clients with heterogeneous data will have a severe negative impact on model performance. Among these, the impact of label distribution skew is particularly significant; the fewer label categories each client holds, the slower the global model converges and the lower its accuracy.

[0004] To mitigate the impact of Non-IID on model performance, existing techniques can be categorized into distribution correction and parameter correction. Both distribution correction and parameter correction require exposing a portion of the original data (FAug) or data distribution (Astraea) to the server, which is undesirable in highly privacy-sensitive real-world scenarios.

[0005] For parameter calibration methods, although the model parameters are optimized towards each other during local training, the parameters of the models of each client are significantly different due to local training on non-homogeneous datasets. Furthermore, the global model parameters of the previous round, which serve as the benchmark, may deviate far from the global optimum in each round. Therefore, it can only slowly calibrate the parameters of the clients and cannot guarantee that the local model optimization is in the direction of the global optimum.

[0006] Furthermore, existing methods (FedProx, MOON, etc.) do not perform well in scenarios with extremely heterogeneous data. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a business execution method, device, and medium based on group generation and federated learning.

[0008] In a first aspect, embodiments of the present invention provide a business execution method based on group generation and federated learning, applied to a client, the method comprising:

[0009] Receive the baseline dataset, initialized business execution network, and generative adversarial network sent by the server;

[0010] The business execution network is trained using a local dataset, and inference is performed on a benchmark dataset using the trained business execution network. The accuracy of each label is then used to form a vector.

[0011] Each group participates in each round of federated learning's iterative training process; where each group is obtained by the server grouping and sorting clients based on a vector composed of accuracy data that has been truncated with a threshold and randomly flipped; at least the following steps are performed: training the generative adversarial network; receiving and randomly selecting several global adversarial generative models obtained by the server's aggregation; inputting Gaussian noise into the global adversarial generative model to obtain a synthetic sample set;

[0012] The re-initialized business execution network was trained using a synthetic sample set and a local dataset;

[0013] Receive the global business execution network obtained by the server aggregation, and use the global business execution network to execute business tasks.

[0014] Secondly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described business execution method based on group generation and federated learning.

[0015] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the above-described business execution method based on group generation and federated learning.

[0016] Fourthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the aforementioned business execution method based on group generation and federated learning.

[0017] Compared with the prior art, the beneficial effects of the present invention are:

[0018] This invention provides a business execution method based on group generation and federated learning. The server groups and sorts clients into groups based on a vector composed of accuracy values ​​that have been truncated by a threshold and randomly flipped. A global adversarial generative model is then obtained based on each group's participation in each round of federated learning iterations. A synthetic sample set is obtained based on the global adversarial generative model. A global business execution network is finally trained using the synthetic sample set and the local dataset. The global business execution network is used to execute business tasks, which can improve the efficiency and accuracy of business task execution even when the global data is not independent and identically distributed. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a business execution method based on group generation and federated learning provided in an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of federated learning provided for an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of client grouping provided in an embodiment of the present invention;

[0023] Figure 4 A schematic diagram illustrating intra-group client training provided in an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the global service execution network generation provided in an embodiment of the present invention;

[0025] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0028] like Figure 1 and Figure 2 As shown, this invention provides a business execution method based on group generation and federated learning, implemented using a server and several clients, with a benchmark dataset deployed on the server. We have an untrained, initialized business execution network ω and an untrained generative adversarial network θ; where all network parameters in the initialized business execution network ω are set to 0, and the global privacy parameter is set to ∈. Each client has a local dataset, denoted as θ_k ... All local datasets There are C classes of samples in the dataset.

[0029] The method specifically includes the following steps:

[0030] Step S1, the server will provide the benchmark dataset. An untrained, initialized business execution network ω and an untrained generative adversarial network θ are distributed to each client. The client receives the benchmark dataset distributed by the server. Initialize the business execution network ω and the generative adversarial network θ.

[0031] In step S2, the client trains the business execution network using the local dataset, performs inference on the benchmark dataset using the trained business execution network, and calculates the accuracy of each label to form a vector.

[0032] Specifically, taking client k as an example, client k utilizes a local dataset. The business execution network ω is trained locally on the client side, and the trained business execution network ω is then used on a benchmark dataset. Inferences are made based on the above, and the accuracy of each label is statistically analyzed to form a vector, denoted as . The element at index c is denoted as

[0033] The business execution network ω is trained using stochastic gradient descent (SGD) to minimize the cross-entropy loss.

[0034] For example, such as Figure 3 As shown, the accuracy vector obtained from client 0 statistics is (0.65, 0.71, 0.09, ..., 0.96).

[0035] Step S3: Set an accuracy threshold. Based on the accuracy threshold, truncate the elements in the accuracy vector by assigning a value of 1 to elements greater than the accuracy threshold and a value of 0 to elements less than the accuracy threshold. Then, randomly flip the truncated accuracy vector (i.e., randomly add perturbations).

[0036] The process of setting the accuracy threshold T includes:

[0037] The client composes the accuracy into a vector. Each element in the expression is summed, and noise perturbation is added before being sent to the server. The expression is as follows:

[0038]

[0039] In the formula, This indicates a value with a mean of 0 and a scale of 0. The noise is sampled from the Laplace distribution.

[0040] The server receives the accumulated value uploaded by each client, divides it by the number of sample tags C and the number of clients K, and obtains the accuracy threshold, expressed as follows:

[0041]

[0042] In the formula, C represents the number of sample labels and K represents the number of clients; at this time, the accuracy threshold T represents the mean of the global label accuracy.

[0043] The server sends the accuracy threshold T to each client. The client then truncates the elements in the accuracy vector based on the accuracy threshold, that is, assigns a value of 1 to elements in the accuracy vector that are greater than the accuracy threshold and assigns a value of 0 to elements that are less than the accuracy threshold.

[0044] For example, such as Figure 3 As shown, in this example, the accuracy threshold T = 0.6, and the client 0 accuracy vector (0.65, 0.71, 0.09, ..., 0.96) becomes (1, 1, 0, ..., 1) after threshold truncation.

[0045] The process of randomly flipping the accuracy vector after threshold truncation (i.e., randomly adding perturbations) includes:

[0046] The accuracy after threshold truncation forms a vector. Each element in by The probability of flipping the elements (i.e., 1 becomes 0, 0 becomes 1) satisfies (∈,0)-differential privacy. Let the element before flipping be denoted as . The flipped element is c∈[0,C-1], i∈{0,1], the expression is as follows:

[0047]

[0048] For example, such as Figure 3 As shown, the accuracy vector (1, 1, 0, ..., 1) after client-side 0-threshold truncation becomes (1, 0, 0, ..., 1) after random perturbation.

[0049] In step S4, the client constructs a vector from the accuracy rates that have been truncated by a threshold and randomly flipped. The data is sent to the server, which then constructs a vector based on the accuracy. Group the clients.

[0050] Specifically, in this example, the accuracy vectors corresponding to all clients, which are truncated by a threshold and randomly flipped, are concatenated to obtain an accuracy matrix.

[0051] Group the clients whose clients are equal to 1 in each column of the accuracy matrix into one group, i.e., if The server will then assign client k to the group. c middle.

[0052] For example, such as Figure 3 As shown, the accuracy vector of client 0 after threshold truncation and random flipping is (1, 0, 0, ..., 1). Therefore, client 0 is assigned to group0, group... C-1 In the middle, the accuracy vector of client k after threshold truncation and random flipping is (1, 0, 1, ..., 1). Then, client 0 is assigned to group0, group2, group3, and group4. C-1 middle.

[0053] In step S5, the server calculates the contribution vector Q based on the ratio of the total sample distribution estimate to the global label distribution. worth ; Select contribution vector Q worth The largest n in L The group corresponding to each element; the selected n L Each group participates in each round of iterative training in federated learning.

[0054] Specifically, such as Figure 4 As shown, the server calculates the contribution vector Q. worth Each element has the following value:

[0055]

[0056] In the formula, Q glob ′ It is the server's estimate of the global sample distribution. It is the global label distribution;

[0057] Global label distribution The value of each element in is:

[0058]

[0059] In the formula, C represents the number of sample labels, K represents the number of clients, and q is the flip probability.

[0060] Step S6: For each group participating in each round of federated learning's iterative training process, at least the following steps must be performed:

[0061] Step S601, group cEach client in the process trains the generative adversarial network and uploads the trained generative adversarial network to the server.

[0062] Specifically, in this example, for each non-empty group corresponding to a label, the server coordinates all clients within that group to train the generative model locally as participants; including:

[0063] The server selects label c;

[0064] When the client's local dataset contains a sample set x with local label c c At that time, for the sample set x with local label c c Noise is added to train the generative adversarial network;

[0065] When the client's local dataset does not contain a sample set x with local label c c At that time, noise is used to update the Generative Adversarial Network.

[0066] In step S602, the client uploads the trained generative adversarial network to the server. The server aggregates all the generative adversarial networks θ to obtain the global adversarial generative model; the expression is as follows:

[0067]

[0068] In the formula, N represents the participating group. c The total number of clients in this federated learning session, where k represents the k-th client.

[0069] In this context, a global adversarial generative model is obtained by the server aggregating the generative adversarial networks uploaded for each group. For example, the server selects the contribution vector Q. worth The three largest elements in the data correspond to the three groups that are selected (group1, group2, group3). c The server participates in each iteration of the federated learning training process; the server selects label 1 corresponding to group1, and the clients in group1 use the sample set x1 corresponding to this label to train the generative adversarial network; the server selects label 2 corresponding to group2, and the clients in group2 use the sample set x2 corresponding to this label to train the generative adversarial network; the server selects group... c The corresponding tag 'c' and group 'group' c The client in the middle uses the sample set x corresponding to the label. c The generative adversarial network is trained; the server aggregates the generative adversarial networks uploaded by clients in each group to obtain a corresponding global adversarial generative model, that is, finally obtains 3 global adversarial generative models.

[0070] Step S603: The client receives all the global adversarial generative models sent by the server and randomly selects n of them. G A global adversarial generative model is used; Gaussian noise is sampled as input to the adversarial generative model, and the output is a synthetic sample set. The synthetic sample set The size is the same as the original local dataset. It is α times, where α is generally 0-1.

[0071] Step S7, all clients execute network ω on the service. k After reinitialization, each iteration of the federated learning process shall perform at least the following steps:

[0072] like Figure 5 As shown, the client will synthesize the sample set. With local dataset The data is merged into a training set to train the service execution network ω. k The client will execute the trained business network ω k The parameters are uploaded to the server for aggregation, as shown in the following expression:

[0073]

[0074] In the formula, N represents the sum of the number of local samples from all clients participating in this round.

[0075] In step S8, the client receives the parameters of the aggregated global service execution network and uses the global service execution network to execute service tasks.

[0076] It should be noted that the test set on the server is denoted as... The global training dataset (i.e., the sum of the local datasets of each client) is represented as Client-side local dataset The total number of all samples with label c is denoted as c. Therefore, the local data distribution on client k can be represented by a C-dimensional array. To express.

[0077] To quantitatively compare the differences between any two distributions Q1 and Q2, this example uses the Jensen-Shannon divergence (JS divergence) to calculate the distance between them. JS divergence is a variant of the Kullback-Leibler divergence (KL divergence), eliminating the asymmetry problem of KL divergence. Its definition is as follows:

[0078]

[0079]

[0080] This example defines the following formula to measure the impact of the Non-IID degree of each client's data distribution on the performance of the global model. The first part of the formula calculates the distance between each client's training set and the global training data distribution, which represents the difference in local models during aggregation; the second part calculates the distribution distance between the global training set and the test set, which represents the final performance of the global model on the test set. The expression is as follows:

[0081]

[0082] To ensure that the input represents only the proportional difference between the number of tags on the client's local machine (i.e., the tag distribution offset) during calculation, a normalization operation is performed on it:

[0083]

[0084] Generally speaking, it can be considered Therefore, the value of the second part of the above formula is 0. At this point, as the local data label distribution offset on the client increases, The value will gradually increase from 0, and the performance of the global model will gradually decrease. Note that in order to avoid division by zero during the calculation process due to the absence of a certain label in the client (i.e., a certain item in the vector is 0), the minimum floating-point precision is added to each element in the vector before the calculation.

[0085] The goal of this example is: when performing a... For federated learning tasks where the value is greater than 0, optimize the global model ω without revealing the data distribution to the client. R exist Performance on the surface.

[0086] In summary, this invention provides a business execution method based on group generation and federated learning. The server groups and sorts clients into groups based on accuracy vectors that have been truncated by a threshold and randomly flipped. A global adversarial generative model is then obtained based on each group's participation in each round of federated learning iterations. A synthetic sample set is obtained based on the global adversarial generative model. Finally, a global business execution network is trained using the synthetic sample set and the local dataset. This global business execution network is then used to execute business tasks, improving the efficiency and accuracy of business task execution even when global data is not independently and identically distributed.

[0087] Example 1

[0088] Embodiment 1 of this invention uses lung CT image classification as a downstream task to illustrate the business execution method based on group generation and federated learning provided by this invention.

[0089] Each hospital has its own database containing lung CT images of patients during the diagnosis and treatment process. The hospitals want to use machine learning models to assist in diagnosis, using the lung CT images as input and the model outputting whether it is pneumonia. The main challenge is that each hospital has a finite amount of data, and a model trained by only one hospital will not perform well. Furthermore, due to privacy requirements, the hospital's datasets are private and cannot be sent to other parties for training; the datasets can only be used to train the model locally.

[0090] This embodiment includes a trusted platform as the server and several hospitals as clients. The trusted platform has a benchmark dataset, an untrained generative adversarial network, and an untrained pneumonia image classification network. Each hospital has several samples of two classes of images, namely normal lung images and pneumonia images, as its dataset.

[0091] (1) Each hospital receives the benchmark dataset, the initialized pneumonia image classification network, and the generative adversarial network issued by the trusted platform.

[0092] (2) Each hospital uses its local dataset to train the pneumonia image classification network, and uses the trained pneumonia image classification network to make inferences on the benchmark dataset, and calculates the classification accuracy vectors of normal lung images and pneumonia images.

[0093] (3) Each hospital accumulates the classification accuracy and adds noise before sending it to the trusted platform. The trusted platform calculates the average value as the accuracy threshold and sends it to each hospital. Each hospital assigns 1 to the elements in the classification accuracy vector that are greater than the accuracy threshold and 0 to the elements that are less than the accuracy threshold, thereby truncating the classification accuracy vector by threshold. Then the classification accuracy vector is randomly flipped.

[0094] (4) The trusted platform sorts the hospitals into groups based on the classification accuracy vectors that have been truncated by a threshold and randomly flipped. The trusted platform calculates the contribution vectors through each classification accuracy vector and considers that the distribution of pneumonia images in all hospital clients is less worth generating. It then notifies the hospitals in the pneumonia group to prepare to start the joint modeling of the pneumonia image generation adversarial network.

[0095] (5) The notified hospital clients conduct federated learning, training the adversarial generative network locally using only pneumonia images. The trained adversarial generative network is then sent to the trusted platform. The trusted platform averages the parameters of this adversarial generative network based on the number of participating hospitals to obtain a global adversarial generative model, which is then distributed to all hospitals within the group. Upon receiving the trained global adversarial generative model, the hospitals synthesize pneumonia images locally and merge the synthesized pneumonia images into their local dataset.

[0096] (6) After setting the parameters of the pneumonia image classification network to 0, all hospital clients participate in each round of federated learning iteration training, performing at least the following steps:

[0097] A pneumonia image classification network was trained using synthetic pneumonia images and a local dataset. The trained network was then uploaded to a trusted platform. The platform performed a weighted average of all pneumonia image classification networks to obtain a global pneumonia image classification network, which was then distributed to all hospitals.

[0098] (7) All hospitals used the global pneumonia image classification network to perform lung CT image classification tasks.

[0099] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the business execution method based on group generation and federated learning as described above. Figure 6 The diagram shown is a hardware structure diagram of any device with data processing capabilities where the business execution method based on group generation and federated learning provided in this embodiment of the invention is located, except... Figure 6 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0100] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the business execution method based on group generation and federated learning as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0101] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A business execution method based on group generation and federated learning, characterized in that, Applied to a client, the method includes: Receive the baseline dataset, initialized business execution network, and generative adversarial network sent by the server; The business execution network is trained using a local dataset, and inference is performed on a benchmark dataset using the trained business execution network. The accuracy of each label is then used to form a vector. Each group participates in each round of federated learning's iterative training process; where each group is obtained by the server grouping and sorting clients based on a vector composed of accuracy data that has been truncated with a threshold and randomly flipped; at least the following steps are performed: training the generative adversarial network; receiving and randomly selecting several global adversarial generative models obtained by the server's aggregation; inputting Gaussian noise into the global adversarial generative model to obtain a synthetic sample set; The re-initialized business execution network was trained using a synthetic sample set and a local dataset; Receive the global business execution network obtained by the server aggregation, and use the global business execution network to execute business tasks; The process by which the server groups and sorts clients based on a vector composed of accuracy values ​​that have been truncated by a threshold and randomly flipped includes: The accuracy vectors corresponding to all clients, after being truncated by a threshold and randomly flipped, are concatenated to obtain an accuracy matrix. Group the clients corresponding to the elements that are equal to 1 in each column of the accuracy matrix; The contribution vector is calculated based on the ratio of the total sample distribution estimate to the global label distribution. ; Select contribution vector The largest The group corresponding to each element; the filtered elements Each group participates in each round of iterative training in federated learning.

2. The business execution method based on group generation and federated learning according to claim 1, characterized in that, The process of setting an accuracy threshold includes: Let the accuracy rates corresponding to client k be denoted as a vector. Where C is the number of sample labels in the local dataset, c∈[0, C-1]; The expression for calculating the accuracy threshold T is as follows: ； ； In the formula, This indicates a value with a mean of 0 and a scale of 0. The noise sampled from the Laplace distribution, This represents the global privacy parameter, and K represents the number of clients.

3. The business execution method based on group generation and federated learning according to claim 1, characterized in that, The process of thresholding and randomly flipping the accuracy vector includes: The accuracy vector is truncated by assigning a value of 1 to elements that are greater than the accuracy threshold and a value of 0 to elements that are less than the accuracy threshold. The flip probability is set as follows: ； In the formula, Represents the natural logarithm. This represents global privacy parameters; Each element in the accuracy vector after threshold truncation is flipped with a flip probability q, as shown in the following expression: ； In the formula, The elements in the vector are the accuracy rates before the flip. The elements in the vector are the accuracy values ​​after flipping, where C is the number of sample labels in the local dataset, and c∈[0, C-1]. .

4. The business execution method based on group generation and federated learning according to claim 1, characterized in that, The expression for global label distribution is as follows: ； In the formula, C represents the number of sample tags, and K represents the number of clients. Let f be the probability of flipping. The elements in the vector are the accuracy rates before the flip.

5. The business execution method based on group generation and federated learning according to claim 1, characterized in that, Each iteration of the federated learning process for each group specifically includes: Each client in each group trains the generative adversarial network and uploads the trained generative adversarial network to the server. The server receives all aggregated global adversarial generative models and randomly selects several of them. A global adversarial generative model is obtained by the server aggregating the generative adversarial networks uploaded for each group. Sampling Gaussian noise is used as input to the global adversarial generative model, and the output is a synthetic sample set.

6. The business execution method based on group generation and federated learning according to claim 5, characterized in that, Each client in each group trains the generative adversarial network by including: When the client's local dataset contains a sample set with local label c At that time, for the sample set with local label c Noise is added to train a generative adversarial network; When the client's local dataset does not contain a sample set with local label c At that time, noise is used to update the Generative Adversarial Network.

7. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the business execution method based on group generation and federated learning as described in any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the business execution method based on group generation and federated learning as described in any one of claims 1-6.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the business execution method based on group generation and federated learning as described in any one of claims 1-6.

Images