Primary communication federal learning method and device based on statistical feature matching
By using pre-trained models to obtain global statistical information of the convolution layer during the data synthesis and matching process of federated learning, the problems of reduced performance and high communication costs caused by data heterogeneity in federated learning are solved, and efficient communication and accurate model training are achieved.
Patent Information
- Application Number
- CN202411951462.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-04-08
AI Technical Summary
In federated learning, global model performance is reduced due to client data heterogeneity, and the high communication costs brought by traditional multi-round communication methods.
A one-time communication federated learning method based on statistical feature matching is proposed. By using pre-trained models to obtain global statistical information of the convolution layer during data synthesis and matching, more statistical features are introduced to improve the quality and diversity of synthetic data.
It significantly reduces communication costs, improves the quality of synthetic data and the accuracy of global models, and is suitable for large-scale applications.
Smart Images
Figure CN119783770A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of federated learning, and in particular to a one-time communication federated learning method and device based on statistical feature matching, which is applied to solve the problems of reduced global model performance caused by client data heterogeneity in federated learning and high communication costs caused by traditional multi-round communication methods. Background Art
[0002] With the rapid development of IoT technology, data is stored and used in a large-scale, distributed manner on various terminal devices. However, due to the need for privacy protection, it is no longer feasible to upload the original data to the cloud for unified training, and federated learning technology has emerged. Federated learning trains the model locally on the client and only uploads the model parameters or gradients to the server, thereby completing the construction of a global model without directly sharing the original data, thereby effectively protecting user privacy and reducing the risk of data leakage.
[0003] However, traditional federated learning frameworks usually require multiple rounds of communication and parameter exchange between the server and each client to iteratively optimize the global model. When the number of clients is large or the network bandwidth is limited, multiple rounds of interaction will significantly increase the system burden and the difficulty of actual deployment. In addition, in practical applications, there are often significant differences in the data distribution of each client, including uneven feature distribution and differences in sample category ratios. This data heterogeneity will cause the global model to face greater challenges when aggregating, which in turn affects the convergence speed and accuracy of the model.
[0004] In this context, in order to reduce communication costs and alleviate the delays caused by multiple rounds of federated training, while reducing the problem of global model performance degradation caused by data heterogeneity, one-time communication federated learning methods have gradually attracted attention. This type of method refers to the transmission of model parameters or feature information from the client to the server only once during the federated training process, thereby greatly reducing the number of communication rounds and costs. Researchers have mainly proposed two types of methods. One is to use auxiliary data sets to integrate multiple models into a single model through knowledge distillation on the server side. However, this type of method has a high dependence on publicly available data sets that are similar to the local data distribution, which is often difficult to meet in practice. The second is to try to directly integrate models to synthesize data, thereby avoiding dependence on public data. Current related research mostly uses the statistical information of the batch normalization layer to assist in data synthesis, and the mining and utilization of other statistical features are still insufficient. Summary of the invention
[0005] The present invention aims to overcome the defects in the prior art by using a pre-trained model to obtain global statistical information of the convolutional layer during the data synthesis matching process, thereby introducing more statistical features to improve the quality and diversity of the synthesized data. To this end, the present invention proposes a one-time communication federated learning method and device based on statistical feature matching.
[0006] The one-time communication federated learning method based on statistical feature matching provided by the present invention includes:
[0007] Step 1: Assume that there are N participants collaborating to train a global model. Each participant has a local dataset, represented as i = 1, ..., N, where and denote the input data points and their corresponding labels respectively.
[0008] Step 2: Perform multiple rounds of iterative training on each client using its local dataset, update the local model parameters, and gradually adapt the model to the specific data distribution of the client.
[0009] Step 3: Initialize global model parameters θ S and the generator G parameter θ G , and all client-side pre-trained local models Upload to the server, where θ k is the pre-trained model parameter of the k-th client.
[0010] Step 4: During the global model training process, the server loops through the two phases of data generation and model training. In the data generation phase, the server samples the initial data input from the noise and label distribution. Using the generator G and the current parameter θ G Generating synthetic data Then, based on the objective function Update the optimized generator parameters. During the model training phase, the server uses the optimized generator to generate synthetic data again. And based on the global model loss function For the global model parameter θ S Perform gradient descent updates. This process is repeated for T training cycles until the global model gradually converges.
[0011] Furthermore, in step (4), the method steps for optimizing the generator and the global model are as follows:
[0012] Step 4.1: Generate initial data input by sampling a noise vector z from a standard Gaussian distribution and a random one-hot label y from a uniform distribution Where b is the batch size.
[0013] Step 4.2: Using the generator G and the current parameters θ G , the sampled noise Convert to synthetic data
[0014] Step 4.3: For the generated synthetic data Each client model f k Use the model parameters θ uploaded by it k Predict the data and output the corresponding Where k = 1, 2, ..., N. Then, the prediction results of all client models are averaged to obtain the average logits output of the synthetic data on the integrated model As the final prediction result of the integrated model for the data.
[0015] Step 4.4: Use the average of the ensemble model With label Calculating cross entropy loss
[0016] Step 4.5: Calculate the difference between the batch normalization statistics (mean and variance) of the synthetic data and the corresponding statistics of the client training data to obtain the batch normalization loss Ensure that the feature distribution of the synthetic data is as close as possible to the feature distribution of the client training data to improve the training stability of the generator.
[0017] Step 4.6: Calculate the difference between the convolutional layer statistics of the synthetic data (including channel mean, channel variance, patch mean and patch variance) and the global statistics corresponding to the client training data to obtain the convolutional layer matching loss Ensure that the convolution feature distribution of the synthetic data is consistent with the convolution feature distribution of the client training data, further improving the training stability of the generator and the quality of the synthetic data.
[0018] Step 4.7: Combining the above loss functions, calculate the total loss function of the generator Optimize generator parameters θ G , improving the performance of the generator and the quality of synthetic data.
[0019] Step 4.8: Use the optimized generator to generate synthetic data and further optimize the global model through knowledge distillation, thereby gradually improving the performance and generalization ability of the global model.
[0020] The second aspect of the present invention relates to a single communication federated learning device based on statistical feature matching, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the single communication federated learning method based on statistical feature matching of the present invention.
[0021] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the one-time communication federated learning method based on statistical feature matching of the present invention.
[0022] The present invention discloses a one-time communication federated learning method based on statistical feature matching, which aims to solve the problem of high communication cost in federated learning and decreased accuracy of global models caused by imbalanced client data. First, the clients independently complete model pre-training using their own local data, and upload the pre-trained model parameters to the server. After the server receives the models uploaded by all clients, it calculates the average logits through the model integration method to build a unified integrated model. In the data generation stage, the server uses a generator to generate synthetic data, and optimizes the generator in combination with a variety of loss functions. These loss functions ensure the label consistency and feature distribution matching of the synthetic data, thereby improving the authenticity and diversity of the synthetic data, while enhancing the knowledge distillation effect and helping to improve the performance of the global model. This method only needs to transmit the model parameters from the client to the server once, significantly reducing the communication cost. By introducing more statistical features, the quality of the synthetic data and the accuracy of the global model are effectively improved, which is suitable for large-scale applications.
[0023] The present invention proposes a one-time communication federated learning method based on statistical feature matching. First, a unified integrated model is built on the server side through pre-training of the client local model. Then, in the process of optimizing the generator, the statistical features of the batch normalization layer and the convolution layer of the matching model are used to ensure that the generated synthetic data is consistent with the features of the client training data, thereby improving the data quality and the performance of the global model. Compared with the prior art, the present invention has the following advantages:
[0024] 1. The prior art only uses the statistical features of the batch normalization layer in the data generation stage, and there is no clear solution for other layers. The present invention further strengthens the feature consistency in the matching process between the synthetic data and the client's original data features by introducing the statistical feature matching of the convolutional layer. This method significantly improves the quality of the synthetic data, thereby effectively optimizing the training effect based on knowledge distillation.
[0025] 2. Existing technologies usually rely on knowledge distillation on public data to improve the performance of server-side models, but their effectiveness is often limited by the quality of public data and the performance of integrated models. The present invention breaks through this limitation and does not need to rely on any public data set or transmit additional information other than model parameters between the client and the server, thereby further reducing communication costs and improving the practicality and security of the system.
[0026] 3. The prior art usually assumes that the client model structure is unified, which makes it difficult to deal with the problem of heterogeneity of model architecture. The present invention supports the heterogeneity of client models and allows different clients to use different model architectures. The server effectively integrates information from heterogeneous models through an integrated model, thereby better adapting to diverse practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 is a flow chart of the method of the present invention;
[0028] Figure 2 It is the basic framework of one-time communication federated learning based on statistical feature matching of the present invention. DETAILED DESCRIPTION
[0029] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.
[0030] Example 1
[0031] Reference Figure 1 The present invention provides an industrial defect image classification method based on a one-time communication federated learning method of statistical feature matching, aiming to improve the accuracy of defect detection in industrial production and achieve efficient quality control. The method comprises the following steps:
[0032] Step 1: Assume there are N factories, denoted as C1, ..., C N Each factory has its own local industrial defect image dataset and corresponding labels, represented as Where X i Represents defect images of industrial products (such as cracks, scratches, deformation, etc.), y i Indicates the classification label of the image (such as no defect, slight defect, severe defect). Due to differences in production equipment, process flow and inspection environment (such as lighting conditions, camera resolution, etc.) in different factories, the data distribution of each factory is significantly different, and the data distribution between clients does not meet the independent and identical distribution.
[0033] Step 2: Perform multiple rounds of iterative training on each client using its local dataset, update the local model parameters, and gradually adapt the model to the specific data distribution of the client.
[0034] Step 3: Initialize global model parameters θS and the generator G parameter θ G , and all client-side pre-trained local models Upload to the server, where θ k is the pre-trained model parameter of the k-th client.
[0035] Step 4: During the global model training process, the server loops through the two phases of data generation and model training. In the data generation phase, the server samples the initial data input from the noise and label distribution. Using the generator G and the current parameter θ G Generating synthetic data Then, based on the objective function Update the optimized generator parameters. During the model training phase, the server uses the optimized generator to generate synthetic data again. And based on the global model loss function For the global model parameter θ S Perform gradient descent updates. This process is repeated for T training cycles until the global model gradually converges.
[0036] Furthermore, in step (4), the method steps for optimizing the generator and the global model are as follows:
[0037] Step 4.1: From the standard Gaussian distribution The noise vector is sampled in batch size b At the same time, from the uniform distribution Sampling category index And convert it into one-hot labels
[0038]
[0039] Among them, OneHot(.) means converting the category index into a one-hot label, and C is the number of categories. Finally, the noise vector and the one-hot label are combined as the initial data input Input for the generator.
[0040] Step 4.2: Using the generator G and the current parameters θ G , the sampled noise Convert to synthetic data
[0041] Step 4.3: For the generated synthetic data Each client model f k (.) Use the model parameters θ uploaded by it k Make predictions on the synthetic data and output the corresponding logits:
[0042]
[0043] Then, the prediction results of all client models are averaged to obtain the average logits output of the synthetic data on the ensemble model:
[0044]
[0045] in Represents the ensemble model for synthetic data The final prediction result.
[0046] Step 4.4: Use the average of the ensemble model and the corresponding label y i Calculate the cross entropy loss. The specific formula is:
[0047]
[0048] Where C is the number of categories, Represents the label y i One-hot encoded value on category c, is the predicted probability of the integrated model for category c.
[0049] Step 4.5: Introduce batch normalization loss to improve the training stability of the generator By minimizing the difference in batch normalization statistics (mean and variance) between synthetic data and client training data, the feature distribution of synthetic data is ensured to be as close as possible to the feature distribution of client training data. The specific formula is:
[0050]
[0051] in, and are the batch mean and variance of the feature map of the lth layer of the generator G(.) on the synthetic data; μ k，l and The client models are k (.) Batch mean and variance of the feature maps at layer l.
[0052] Step 4.6: To ensure that the convolution feature distribution of the synthetic data is consistent with the convolution feature distribution of the client training data, first calculate the global statistics of the client training data, including the channel mean Channel variance Patch mean and patch variance The specific calculation formula is:
[0053]
[0054] in They represent the channel mean, channel variance, patch mean and patch variance of the lth layer in the kth client respectively. On this basis, the convolutional layer matching loss is introduced By minimizing the difference between synthetic data and client global statistics in these indicators, the training stability of the generator and the quality of synthetic data are further improved. The specific formula is:
[0055]
[0056] in, They are respectively the channel mean, channel variance, patch mean and patch variance of the l-th layer feature map of the generator G(.) on the synthetic data.
[0057] Step 4.7: Combining the above loss functions, the total loss function of the generator is composed of the cross entropy loss Batch Normalization Loss Matching loss with convolutional layer By optimizing the total loss function It can effectively improve the performance of the generator and the quality of generated data. The specific formula is:
[0058]
[0059] Among them, λ1 and λ2 are the weight coefficients of the loss function, which are used to balance the contribution of each part of the loss. When optimizing the total loss function, the gradient descent method is used to update the parameters θ of the generator G :
[0060]
[0061] Step 4.8: Use the updated generator to generate synthetic data, and combine the average logits of the ensemble model with the prediction results of the global model to optimize the global model through knowledge distillation to gradually improve the performance of the model.
[0062] Step 5: For newly added industrial product images, use the optimized global model to classify the images.
[0063] Furthermore, the above image classification results are used to perform defect marking, quality assessment or sorting operations on products, thereby achieving intelligent quality control and production process optimization.
[0064] Example 2
[0065] The present embodiment relates to a one-time communication federated learning device based on statistical feature matching, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the industrial image classification method of the one-time communication federated learning method based on statistical feature matching of Example 1.
[0066] Example 3
[0067] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the industrial image classification method of the one-time communication federated learning method based on statistical feature matching of Example 1.
[0068] The embodiments described above are only descriptions of the preferred modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.
Claims
1. A one-time communication federated learning method based on statistical feature matching, characterized in that: The steps include: Step 1: Assume that there are N participants collaborating to train a global model; each participant has a local dataset represented as in and Represent the input data points and their corresponding labels respectively; Step 2: Perform multiple rounds of iterative training on each client using its local data set, update the local model parameters, and gradually adapt the model to the specific data distribution of the client; Step 3: Initialize global model parameters θ S and the generator G parameter θ G , and all client-side pre-trained local models Upload to the server, where θ k is the pre-trained model parameter of the kth client; Step 4: During the global model training process, the server loops through the two phases of data generation and model training. In the data generation phase, the server samples the initial data input from the noise and label distribution. Using the generator G and the current parameter θ G Generating synthetic data Then, based on the objective function Update and optimize the generator parameters; during the model training phase, the server uses the optimized generator to generate synthetic data again And based on the global model loss function For the global model parameter θ S Perform gradient descent updates; repeat T training cycles until the global model converges.
2. The one-time communication federated learning method based on statistical feature matching according to claim 1 is characterized in that: In step (4), the method for optimizing the generator and the global model is as follows: Step 4.1: Generate initial data input by sampling a noise vector z from a standard Gaussian distribution and a random one-hot label y from a uniform distribution Where b is the batch size; Step 4.2: Using the generator G and the current parameters θ G , the sampled noise Convert to synthetic data Step 4.3: For the generated synthetic data Each client model f k Use the model parameters θ uploaded by it k Predict the data and output the corresponding Where k = 1, 2, ..., N; then, the prediction results of all client models are averaged to obtain the average logits output of the synthetic data on the integrated model As the final prediction result of the integrated model for the data; Step 4.4: Use the average of the ensemble model With label Calculating cross entropy loss Step 4.5: Calculate the difference between the batch normalization statistics (mean and variance) of the synthetic data and the corresponding statistics of the client training data to obtain the batch normalization loss Ensure that the feature distribution of synthetic data is as close as possible to the feature distribution of client training data, improving the training stability of the generator; Step 4.6: Calculate the difference between the convolutional layer statistics of the synthetic data (including channel mean, channel variance, patch mean and patch variance) and the global statistics corresponding to the client training data to obtain the convolutional layer matching loss Ensure that the convolution feature distribution of synthetic data is consistent with the convolution feature distribution of client training data, further improving the training stability of the generator and the quality of synthetic data; Step 4.7: Combining the above loss functions, calculate the total loss function of the generator Optimize generator parameters θ G , improve the performance of the generator and the quality of synthetic data; Step 4.8: Use the optimized generator to generate synthetic data and further optimize the global model through knowledge distillation, thereby gradually improving the performance and generalization ability of the global model.
3. A one-time communication federated learning device based on statistical feature matching, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the one-time communication federated learning method based on statistical feature matching as described in any one of claims 1-2.
4. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the one-time communication federated learning method based on statistical feature matching described in any one of claims 1-2 is implemented.