Non-IID federated learning method and device based on trajectory matching data set distillation

Abstract

The invention discloses a Non-IID federated learning method and device based on track matching data set distillation, and the method comprises the steps: 1, N participants exist, and the local data distribution and label of each participant and the data distribution between clients do not meet the independent identical distribution; at the clients, each client initializes a local model of the client, and replaces an original classifier of a given model architecture with the same ETF classifier; the server runs a standard federated average algorithm, the client trains a model on local data, updates model parameters and uploads the parameters to the server, the server aggregates the parameters and updates a global model, and the process is continuously repeated; the server initializes an implicit vector set of the synthetic data, and performs optimization according to an early trajectory of the global model; the server uses the optimized synthetic data to correct the aggregated global model until the number of communication rounds reaches, initializes an extended implicit vector set, and continues to perform optimization based on a later global model trajectory; further correcting the global model through synthetic data generated by the previous two times of optimization; along with continuous updating, aggregation and correction of the model, the final global model is gradually converged. According to the method, the performance of the global model can be remarkably improved, calculation and memory consumption is reduced, and the method is suitable for large-scale application.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and in particular to a Non-IID federated learning method and device based on trajectory matching data set distillation, which is applied to solve the problem of decreased global model accuracy due to uneven client data distribution in federated learning. Background Art

[0002] With the rapid development of artificial intelligence technology, its application in many fields such as medicine, finance, and transportation is becoming more and more extensive. However, with the popularization of artificial intelligence technology, the issue of privacy protection has gradually become prominent. AI systems improve their accuracy and efficiency through learning and training of large amounts of data, and these data often involve personal privacy information. Once the data is leaked or used improperly, it may lead to serious privacy risks. Therefore, how to ensure the security of data privacy while promoting the development of artificial intelligence technology has become a key issue that needs to be solved urgently.

[0003] Federated learning is an innovative machine learning method that distributes model training to various devices to avoid directly sharing raw data, thereby completing model construction while protecting privacy. Each participant only needs to upload the model parameters trained locally, rather than the data itself, effectively reducing the risk of privacy leakage. However, federated learning still faces challenges in practical applications. First, the high communication cost caused by frequent parameter transmission is more prominent when the network is complex or the device performance is limited; second, the data distribution between participants is usually uneven, which affects the model performance and generalization ability, resulting in poor federated learning results.

[0004] In order to solve the training problems caused by heterogeneous data, researchers have proposed a variety of solutions. A common method is to use data sharing methods to balance the distribution of client data. However, these methods undermine the privacy principle of federated learning. Another idea is to improve the aggregation method of global models, which usually requires a large-scale global data set on the server side to better aggregate the models of each client. However, in practical applications, such global data sets are often difficult to obtain and select.

[0005] In addition, methods such as personalized federated learning, similarity clustering, and meta-learning are also widely used. Although these methods have alleviated the impact of data heterogeneity to a certain extent, they have failed to fundamentally solve the problem of data dispersion and inability to be centralized due to privacy issues. Summary of the invention

[0006] The present invention aims to overcome the defects in the prior art and enable the server to directly obtain global data distribution to assist model training under the premise of ensuring privacy. A Non-IID federated learning method and device based on trajectory matching dataset distillation are proposed.

[0007] The Non-IID federated learning method based on trajectory matching dataset distillation provided by the present invention includes:

[0008] Step 1: Assume there are N participants, denoted as C1, ..., C N The local data distribution and labels owned by each participant are expressed as The data distribution between clients does not satisfy the independent and identically distributed (Non-IID) principle.

[0009] Step 2: On the client side, each client initializes its local model g i (θ i ), and replace the original classifier of a given model architecture with the same ETF (equilateral framework) classifier V.

[0010] Step 3: The server runs the standard federated averaging algorithm, and the client trains its model on local data and updates the model parameters θ i , and upload the parameters to the server, which aggregates the parameters and updates the global model θ g , and the process is repeated continuously.

[0011] Step 4: During the iteration of step 3 above, when the number of communication rounds reaches L (early training phase), the server initializes the latent vector set Z of the synthetic data and calculates the latent vector Z according to the early trajectory of the global model. The server uses the optimized synthetic data to correct the aggregated global model until the number of communication rounds reaches K (late training stage). At the Kth round, the extended latent vector set Z is initialized ext , and continue based on the later global model trajectory Optimize. The synthetic data generated by the first two optimizations is used to further correct the global model. With the continuous updating, aggregation and correction of the model, the global model gradually converges.

[0012] Furthermore, in step (4), the method steps for optimizing synthetic data and correcting the global model are as follows:

[0013] Step 4.1: The server initializes synthetic data using the latent vector set Z.

[0014] Step 4.2: Using the frozen generator G, these latent vectors z i Map to the image space and generate the initial image set X syn .

[0015] Step 4.3: Use composite image X syn As training data, we get the trajectory τ syn , which represents the update trajectory of the synthetic image data training process in the parameter space.

[0016] Step 4.4: Enter the iterative optimization phase. In each optimization iteration, the server minimizes the synthetic image trajectory τ syn Early trajectories with global models The server also adds some noise disturbances to the early trajectories to make the optimization process more diverse. The intensity of this noise can control the diversity of the generated images during the optimization process.

[0017] Step 4.5: The optimized latent vector Z opt1 Input the generator G again to generate new synthetic images. After each global model aggregation, the server will use these synthetic images to further correct the parameters of the global model.

[0018] Step 4.6: At the end of training, the server will initialize a new set of latent vectors Z ext , input them into the generator to generate a new batch of synthetic images X aug These images will be used to train the model again to obtain a new model update trajectory τ aug By minimizing the new trajectory and the global model's later trajectory The server can further optimize the hidden vector to get Z opt2 .

[0019] Step 4.7: The server combines the latent vectors optimized in the early and late stages to form a complete set of latent vectors. The synthetic images generated by this set will be used to continuously update and correct the global model in subsequent iterations to further improve the performance of the global model.

[0020] A second aspect of the present invention relates to a Non-IID federated learning device based on trajectory matching dataset distillation, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the Non-IID federated learning device based on trajectory matching dataset distillation of the present invention is used to implement the Non-IID federated learning device based on trajectory matching dataset distillation of the present invention. n -IID federated learning method.

[0021] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the Non-IID federated learning method based on trajectory matching dataset distillation of the present invention.

[0022] The present invention discloses a Non-IID federated learning method based on trajectory matching data set distillation, which aims to solve the problem of decreased global model accuracy caused by uneven client data distribution in federated learning. The method first uses the ETF classifier to replace the original classifier on the client to suppress the model offset, thereby improving the trajectory quality of the global model. Then, by optimizing the trajectory matching of the synthetic data in the latent space, a synthetic image with global data distribution characteristics is generated, and these images are used to further optimize the global model. In order to enhance the diversity of synthetic data, the algorithm adds perturbations to the trajectory of the global model, breaking the single trajectory limitation, thereby generating diversified synthetic data, and effectively improving the adaptability of the model in the Non-IID scenario. This method can not only significantly improve the performance of the global model, but also reduce the computational and memory consumption, and is suitable for large-scale applications.

[0023] The present invention proposes a federated learning method based on trajectory matching. First, the classifier of the original model is replaced by the ETF structure in neural collapse. Then, the latent vector is optimized in the latent space by trajectory matching, and the optimized latent vector is input into the frozen generator to generate synthetic data with global data distribution characteristics, which is finally used to correct the training of the global model. Compared with the prior art, the present invention has the following advantages:

[0024] 1. The existing technology ignores the impact of trajectory quality on the upper limit of information acquisition when mining global data distribution through trajectory matching. The present invention adopts the ETF structure in neural collapse to replace the classifier of the client model, effectively suppresses the deviation of the client model, improves the quality of the global model trajectory, and makes the global data distribution information extracted from the trajectory more accurate.

[0025] 2. When performing trajectory matching on the server, the prior art usually optimizes in the pixel space, resulting in huge consumption of memory and computing resources, making it difficult to promote in practical applications. The present invention optimizes the latent vector in the latent space, greatly reducing the consumption of computing and memory, making trajectory matching more efficient and suitable for large-scale applications.

[0026] 3. When synthesizing data, the existing technology can only match a single global model trajectory, which limits the diversity of the synthesized data. In the process of optimizing the latent vector, the present invention breaks the singleness of the trajectory by adding perturbations to the global model trajectory, thereby improving the diversity of the synthesized data and making the global model more adaptable to non-independent and identically distributed data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 It is the basic framework of Non-IID federated learning based on trajectory matching dataset distillation in this invention. DETAILED DESCRIPTION

[0029] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0030] Example 1

[0031] Reference Figure 1 The present invention provides a medical image classification method based on the Non-IID federated learning method of trajectory matching dataset distillation, aiming to improve the accuracy of medical image classification and thus achieve accurate disease diagnosis. It includes the following steps:

[0032] Step 1: Assume there are N hospitals, denoted as C1, ..., C N The local medical image dataset and its labels owned by each participant are represented as Because medical images (such as CT scans, X-rays, etc.) collected by different hospitals have different equipment models, regional medical levels and operating procedures, the distribution of image clarity, resolution and detail features is also significantly different, which makes the data distribution between clients not independent and identically distributed (Non-IID).

[0033] Step 2: On the client side, each client initializes its local model g i (θ i ), and replace the original classifier of a given model architecture with the same ETF (equilateral framework) classifier V.

[0034] In step (2), the method for replacing the original classifier with the ETF (equilateral framework) classifier V is as follows:

[0035] Step 2.1: Each client model g i (θ i ) can be expressed as:

[0036]

[0037] Among them, f i is the feature extractor, W is the original classifier, θ i are parameters of the client model.

[0038] Step 2.2: In order to improve the stability of the global model and the consistency of the output, the original classifier W is replaced by the ETF (equilateral frame) classifier V. The column vectors v1, v2, ..., v of the ETF classifier matrix V are C Meet the following equidistant distribution requirements:

[0039]

[0040] Among them, C is the number of categories, θ is each pair of vectors v i and v j The angle between them. Such an equilateral framework structure ensures the symmetry of the classifier output and suppresses the deviation caused by different data distribution in medical image classification. The replaced client model g i (θ i ) is expressed as:

[0041]

[0042] Step 3: Each client first stores data in the local database D i Train the model and update its local model parameters θ i After training is completed, the client uploads the updated model parameters to the server. The server receives the model parameters θ1, θ2, ..., θ from all clients. N , and aggregate them by weighted average to obtain the updated global model parameter θ g Among them, the weight is usually allocated according to the amount of data of the client. After the aggregation is completed, the server will update the global model parameter θ g Distribute back to each client and enter the next round of training.

[0043] Step 4: During the iteration of step 3 above, when the number of communication rounds reaches L (early training phase), the server initializes the latent vector set Z of the synthetic data and calculates the latent vector Z according to the early trajectory of the global model. The server uses the optimized synthetic data to correct the aggregated global model until the number of communication rounds reaches K (late training stage). At the Kth round, the extended latent vector set Z is initialized ext , and continue based on the later global model trajectory Optimize. The synthetic data generated by the first two optimizations is used to further correct the global model. With the continuous updating, aggregation and correction of the model, the global model gradually converges, thereby improving the accuracy of medical image classification.

[0044] Furthermore, in step (4), the method steps for optimizing synthetic data and correcting the global model are as follows:

[0045] Step 4.1: Initialize the synthetic image using the latent vector set Z

[0046] Step 4.2: Using the frozen generator G, these latent vectors z i Mapped to the image space, generating the initial medical image set X syn .

[0047] Step 4.3: Using synthetic medical image X syn As training data, we get the trajectory τsyn , which represents the update trajectory of the synthetic image data training process in the parameter space.

[0048] Step 4.4: Enter the iterative optimization phase. In each optimization iteration, minimize the synthetic image trajectory τ syn Early trajectories with global models To increase diversity, in the early trajectory Add noise disturbance ∈, the objective function is:

[0049]

[0050] in, NoiseIntensity∈ controls the diversity during the optimization process.

[0051] Step 4.5: The optimized latent vector Z opt1 Input the generator G to generate the corresponding synthetic images. These synthetic images will be used to further optimize the global model. Specifically, after each aggregation, the global model will use these images for optimization, the formula is as follows:

[0052]

[0053] Among them, θ global represents the parameters of the global model, and η is the learning rate.

[0054] Step 4.6: Initial extended latent vector set Z ext , and input it into the generator to generate the corresponding synthetic image X aug .

[0055] Step 4.7: Use the new composite image X aug As training data, we get the corresponding global model trajectory τ aug , and by minimizing the trajectory τ aug Later trajectories with the global model The error between them is , and the latent vector is optimized again to get Z opt2 The optimization goal is:

[0056]

[0057] Step 4.8: Substitute the latent vector Z opt1 and Z opt2 Combined to form a total hidden vector set Z total =Z opt ∪Z ext In subsequent iterations, based on Z total The generated image G(Z total) Update the parameters of the global model and correct the aggregation results of the global model. The optimization goal is:

[0058]

[0059] Step 5: Use the optimized global model to classify the medical imaging data of the newly added user.

[0060] Furthermore, the above image classification results are used to conduct diagnostic analysis of disease types.

[0061] Example 2

[0062] The present embodiment relates to a Non-IID federated learning device based on trajectory matching dataset distillation, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the medical image classification method of the Non-IID federated learning method based on trajectory matching dataset distillation in Example 1.

[0063] Example 3

[0064] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the medical image classification method of the Non-IID federated learning method based on trajectory matching dataset distillation of Example 1.

[0065] The embodiments described above are only descriptions of the preferred modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A Non-IID federated learning method based on trajectory matching dataset distillation, characterized by: The steps include: Step 1: Assume there are N participants, denoted as C1,...,C N ; The local data distribution and labels owned by each participant are expressed as The data distribution between clients does not satisfy the independent and identically distributed (Non-IID) principle. Step 2: On the client side, each client initializes its local model g i (θ i ), and replace the original classifier of a given model architecture with the same ETF classifier V; Step 3: The server runs the standard federated averaging algorithm, and the client trains its model on local data and updates the model parameters θ i , and upload the parameters to the server, which aggregates the parameters and updates the global model θ g , the process is repeated continuously; Step 4: During the iteration of step 3 above, when the number of communication rounds reaches L, the server initializes the latent vector set Z of the synthetic data and calculates the hidden vector set Z according to the early trajectory τ of the global model. g (1) Optimize; the server uses the optimized synthetic data to correct the aggregated global model until the number of communication rounds reaches K; In the Kth round, initialize the extended latent vector set Z ext , and continue based on the later global model trajectory τ g (2) Optimize; further correct the global model through the synthetic data generated by the first two optimizations; with the continuous updating, aggregation and correction of the model, the global model gradually converges.

2. The Non-IID federated learning method based on trajectory matching dataset distillation according to claim 1, characterized in that: In step (2), the method for replacing the original classifier with the equilateral framework ETF classifier V is as follows: Step 2.1: Each client model g i (θ i ) is represented as: Among them, f i is the feature extractor, W is the original classifier, θ i are the parameters of the client model; Step 2.2: In order to improve the stability of the global model and the consistency of the output, the original classifier W is replaced by the ETF classifier V; the column vectors v1, v2, ..., v of the ETF classifier matrix V are C Meet the following equidistant distribution requirements: Among them, C is the number of categories, θ is each pair of vectors v i and v j The angle between them; Such an equilateral frame structure ensures the symmetry of the classifier output and suppresses the deviation caused by different data distribution in medical image classification; The replaced client model g i (θ i ) is expressed as:

3. The Non-IID federated learning method based on trajectory matching dataset distillation according to claim 1, characterized in that: In step (4), the steps for synthetic data optimization and global model correction are as follows: Step 4.1: The server initializes the synthetic data using the latent vector set Z; Step 4.2: Using the frozen generator G, these latent vectors z i Map to the image space and generate the initial image set X syn ; Step 4.3: Use composite image X syn As training data, we get the trajectory τ syn , which represents the update trajectory of the synthetic image data training process in the parameter space; Step 4.4: Enter the iterative optimization phase. In each optimization iteration, the server minimizes the synthetic image trajectory τ syn With the global model early trajectory τ g (1) The server will also add some noise disturbances to the early trajectories in order to make the optimization process more diverse. The intensity of this noise can control the diversity of the images generated during the optimization process; Step 4.5: The optimized latent vector Z opt1 Input the generator G again to generate new synthetic images. After each global model aggregation, the server will use these synthetic images to further correct the parameters of the global model. Step 4.6: At the end of training, the server will initialize a new set of latent vectors Z ext , input them into the generator to generate a new batch of synthetic images X avg ; These images will be used to train the model again to get the new model update trajectory τ aug ; By minimizing the new trajectory and the global model's later trajectory τ g (2) The server can further optimize the hidden vector to get Z opt2 ; Step 4.7: The server combines the latent vectors optimized in the early and late stages to form a complete set of latent vectors. The synthetic images generated by this set will be used to continuously update and correct the global model in subsequent iterations to further improve the performance of the global model.

4. A Non-IID federated learning device based on trajectory matching dataset distillation, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the Non-IID federated learning method based on trajectory matching data set distillation according to any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the Non-IID federated learning method based on trajectory matching data set distillation described in any one of claims 1 to 3 is implemented.

Images