A data-free fusion method and system for heterogeneous networks based on federated distillation

Abstract

This invention discloses a data-free fusion method and system for heterogeneous networks based on federated distillation, belonging to the field of information processing technology. For information between different combat systems, a CGAN model is trained using federated learning and used to optimize local data, generating a training set with independent and identically distributed characteristics, thus improving model training efficiency and accuracy. The CGAN network generates a transfer set for distillation instead of transferring a small number of samples from the source data, meeting data confidentiality requirements and transferring local model knowledge in a data-free manner. The federated distillation method aggregates local models, weakening the isomorphic requirement between local and global models in traditional federated learning algorithms. This allows edge servers holding data to design local models tailored to their local data structures, mitigating information loss that may occur during standardized data preprocessing.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to a method and system for data fusion of heterogeneous networks based on federated distillation. Background Technology

[0002] With the widespread application of modern information technologies such as cloud computing, big data, the Internet of Things, and unmanned technology in the military field, equipment systems are developing rapidly, and combat equipment is constantly being upgraded, further accelerating the transformation of warfare and combat technologies. Future warfare is showing a trend towards informatization, intelligence, and collaboration. Joint operations are an essential requirement for winning modern and future wars. In the joint operations system, data, as a strategic resource supporting efficient command and decision-making, plays a crucial underlying supporting role. Proper data management and efficient utilization have become important driving forces for promoting the overall leap in combat capabilities and profound changes in combat styles. Achieving secure data interconnection between different combat systems is of great significance for further leveraging the supporting role of data resources in command and decision-making, realizing an intelligent data fusion system with high-speed computing, storage, and retrieval, constructing a big data-driven intelligent model, accelerating the construction of intelligent complex network information systems, and contributing to the development of military intelligence.

[0003] Due to the phased and independent nature of the early system construction, as well as the targeted nature of strategic objectives, the degree of isolation between different systems is relatively high, and data silos have become a key obstacle to the construction of military data. At the same time, the special strategic position of military data makes the application of big data in the military field a double-edged sword. While accelerating the modernization of national defense and the armed forces, the risk of information leakage hidden in the informatization process must be fully considered.

[0004] In the context of the rapid development of intelligent technologies, a secure and reliable method is needed to comprehensively integrate existing information resources and mitigate risks arising from information leaks, in order to better utilize existing data resources and break down data silos. Traditional machine learning techniques, with their centralized nature, may lead to serious data security issues. Federated learning, by ensuring that data does not leave its local location, allows edge servers to train and generate local models using their local data, which are then shared globally. This avoids the communication overhead and privacy risks associated with large-scale data transmission, providing a new method for integrating data resources.

[0005] However, existing technologies, due to limitations in aggregation algorithms, require all participating device models to be completely homogeneous and do not adequately consider data heterogeneity. In reality, heterogeneity is often prevalent, especially for data from local devices, where the data collected by each party exhibits significant differences. Two types of heterogeneity—distributive heterogeneity and structural heterogeneity—can lead to slow convergence or global model drift. Secondly, traditional federated learning algorithms, such as FedAvg, typically use the same model structure and training strategy. However, the extracted structures often differ significantly for different types of sample data, making uniform model training detrimental to accuracy. Furthermore, traditional federated learning techniques exchange model gradients in each training round, generating substantial communication overhead. Since training with locally distributed data may have significant biases, this affects model convergence speed and accuracy. Summary of the Invention

[0006] To address the aforementioned problems, this invention aims to provide a data-free fusion method and system for heterogeneous networks based on federated distillation. By integrating local private data information from multiple dispersed nodes through intelligent technology, a targeted network model is designed. Simultaneously, the neural network model is jointly trained without migrating the source data, effectively improving the data resource utilization rate of the federated learning system and providing technical support for the construction of intelligent information systems.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A data-free fusion method for heterogeneous networks based on federated distillation is characterized by the following steps:

[0009] S1: Establish a centralized federation architecture, deploy federation learning schemes on corresponding server nodes within the joint operations system, and coordinate the edge server nodes through the central server node.

[0010] S2: The central server initializes the global neural network model and the CGAN model;

[0011] S3: Train the CGAN network model using federated learning;

[0012] S4: The edge server uses CGAN to optimize the data distribution so that it satisfies the independent and identically distributed property, constructs a local neural network model and trains it independently to obtain a local parameterized model, and sends the local parameterized model to the central server.

[0013] S5: The central server uses sample labels and the CGAN network to generate a transfer set, and uses the local parameterized model trained on multiple edge servers as the teacher model to train the global neural network model by generating the transfer set.

[0014] Furthermore, the specific operation of step S1 includes the following steps:

[0015] S101: The training task is determined and broadcast by the central server;

[0016] S102: The edge server evaluates the training task based on local data, determines whether to participate in the current training task, and sends a response to the central server;

[0017] S103: The central server selects edge servers from the response set to participate in the training task and deploys the federated learning scheme.

[0018] Furthermore, the specific operation of step S3 includes the following steps:

[0019] S301: The central server sends the CGAN parameterized model to the edge servers participating in model training;

[0020] S302: Edge servers train CGAN networks based on local private data;

[0021] S303: The edge server calculates the model gradient and transmits the gradient encrypted to the central server;

[0022] S304: The central server receives gradient parameters and calculates the global loss for model updates using a weighted averaging method. This leads to the updating of the CGAN network;

[0023] S305: The central server sends the updated global CGAN model to all edge servers for the next round of iteration until the model converges.

[0024] S306: The central server sends the trained CGAN parameterized model and global model to the edge server.

[0025] Furthermore, the specific operations for training the CGAN network in step S302 include:

[0026]

[0027]

[0028] in, For the parameterized model of CGAN for edge servers, θ G For the generator, θ D Let α be the discriminator and α be the learning rate.

[0029] Furthermore, in step S304, the global loss is calculated for the model update using a weighted average method. The specific operation is as follows:

[0030]

[0031] in, The loss of the generative model on edge server k.

[0032] Furthermore, the specific operation of step S4 includes the following steps:

[0033] S401: Each edge server corrects its local data distribution using CGAN and generates a training set so that it meets the IID characteristics with the training sets of other edge server nodes.

[0034] S402: Each edge server designs a neural network model based on the structure of the training set samples and completes the initialization;

[0035] S403: Each edge server independently trains a locally parameterized model until the model converges;

[0036] S404: Each edge server sends a local parameterized model to the central server.

[0037] Furthermore, the specific operation of step S5 includes the following steps:

[0038] S501: The central server uses sample labels and CGAN to generate transfer sets;

[0039] S502: The central server uses local models trained on multiple edge servers as teacher models and trains the global neural network model by generating transfer sets;

[0040]

[0041] in, The knowledge distillation loss for the global model; The loss between the student model's prediction and the true label. p represents the difference between the logarithmic outputs of the teacher model and the student model. global This represents the soft decision vector for the teacher and student models.

[0042] Furthermore, an information data-free fusion system implementing the aforementioned heterogeneous network data-free fusion method based on federated distillation is characterized by comprising a central server and multiple edge servers, wherein a federated learning scheme is deployed on the server nodes, and the central server node coordinates the edge server nodes.

[0043] The beneficial effects of this invention are:

[0044] 1. This invention proposes a data-free fusion method for heterogeneous networks based on federated distillation. To achieve interconnection and interoperability of information between different combat systems, it improves upon the system security, data heterogeneity, and target difference issues inherent in traditional federated learning algorithms. Based on the traditional federated optimization algorithm, it introduces knowledge distillation to address the common data heterogeneity problem among units. Simultaneously, it uses aggregated knowledge to refine the server model instead of directly aggregating model parameters, enhancing the security of the federated learning system. By maintaining the agnosticity of the aggregation server to the local model structure, it reduces security risks and protects proxy data. It utilizes a CGAN network to integrate local information and distributed knowledge to regulate global model training, achieving knowledge distillation of the model without relying on any external data. The effectiveness of the method is verified through four sets of experimental datasets. Experimental results show that, compared to three other federated learning algorithms, this method achieves better results with fewer aggregation rounds, outperforming existing federated learning algorithms in convergence speed and model accuracy, and effectively reducing communication between edge servers and the central server.

[0045] 2. The federated data-free fusion method in this invention decouples the training of local models from the global model, allowing the training algorithm and network model structure to be adjusted according to local objectives, and enabling multiple data sources to train local models in a targeted manner; it uses a teacher-global model architecture to extract knowledge instead of directly weighting the parameters of local models, allowing a certain degree of agnosticness to the local training algorithm and model structure; it uses CGAN to achieve data augmentation, improving model training efficiency and convergence speed, reducing communication overhead; and it uses a data-free fusion method, replacing local mini-batch samples with generated data as the transfer set, ensuring the security of local data sources. Attached Figure Description

[0046] Figure 1 This is a flowchart of the fusion method in this invention;

[0047] Figure 2 This is a schematic diagram of the overall framework of the present invention;

[0048] Figure 3 This is a schematic diagram of the architecture of CGAN in this invention;

[0049] Figure 4 This is a schematic diagram of CGAN data augmentation in this invention;

[0050] Figure 5 This is a schematic diagram of the federal dataless distillation polymerization in this invention;

[0051] Figure 6 This is a comparison of the accuracy of various algorithm models on different datasets in the simulation experiment of this invention;

[0052] Figure 7This is a comparison of the loss of various algorithm models on different datasets in the simulation experiment of this invention. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0054] Example 1:

[0055] As attached Figure 1 and attached Figure 2 As shown, the heterogeneous network data-free fusion method based on federated distillation includes the following steps:

[0056] S1: Establish a centralized federated architecture, deploy federated learning schemes on corresponding server nodes within the joint operations system, and coordinate the server nodes holding data, i.e. edge server nodes, through the central server node to jointly train the CGAN model.

[0057] The architecture diagram of Conditional Generative Adversarial Network (CGAN) is attached. Figure 3 As shown, the dataset includes a generator G and a discriminator D. During training, the generator is constrained by the class label y. The generator learns from the training dataset to generate samples (x) with the corresponding label. * The discriminator learns to distinguish between real samples (x, y) and generated samples (x, y). * |y,y).

[0058] Step S1 includes the following steps:

[0059] S101: The training task is determined and broadcast by the central server;

[0060] S102: The edge server evaluates the training task based on local data, determines whether to participate in the current training task, and sends a response to the central server;

[0061] S103: The central server selects edge servers from the response set to participate in the training task and deploys the federated learning scheme;

[0062] Further, step S2: The central server initializes the global neural network model and the CGAN model;

[0063] In this invention, the neural network model employs a Convolutional Neural Network (CNN): CNN is a deep neural network with convolutional structures. It reduces the number of weights by using local connections and weight sharing, thereby lowering model complexity and mitigating overfitting. CNNs are commonly used in image processing, taking images directly as network input. Feature extraction is achieved through convolutional and pooling layers in the hidden layers, and the loss function is minimized using gradient descent. The weights in the network are then adjusted layer by layer inversely, and the network's accuracy is improved through multiple rounds of iterative training.

[0064] The hidden layers of a convolutional neural network typically consist of low-hidden layers and high-hidden layers. Low-hidden layers are composed of alternating convolutional and pooling layers, while high-hidden layers are fully connected layers, corresponding to the hidden layers and logistic regression classifiers in a traditional multilayer perceptron. The input to the first fully connected layer is the feature image obtained after feature extraction from the convolutional and pooling layers. The output layer is a classifier, typically employing logistic regression or softmax regression, used to classify the input image. Specifically, the convolutional layers use a trainable filter f(x) to perform a convolution operation with the input image, and then add a bias b. x It can enhance certain features in the original signal while reducing noise, resulting in a convolutional layer C. x Pooling layers typically employ downsampling, using various pooling functions such as max pooling to reduce the data space and prevent overfitting while maintaining feature invariance. Fully connected layers use softmax full connectivity, using image features extracted from lower hidden layers as activation values ​​for subsequent computations.

[0065] Further, step S3: train the CGAN network model using federated learning;

[0066] Specifically, S301: The central server sends the CGAN parameterized model to the edge servers participating in model training;

[0067] S302: Edge servers train CGAN networks based on local private data;

[0068]

[0069]

[0070] in, For the parameterized model of CGAN for edge servers, θ G For the generator, θ D Let α be the discriminator and α be the learning rate.

[0071] S303: The edge server calculates the model gradient and transmits the gradient encrypted to the central server;

[0072] S304: The central server receives gradient parameters and calculates the global loss for model updates using a weighted averaging method. This leads to the updating of the CGAN network;

[0073] Global loss The calculation method is as follows:

[0074]

[0075] in, The loss of the generative model on edge server k.

[0076] S305: The central server sends the updated global CGAN model to all edge servers for the next round of iteration until the model converges.

[0077] S306: The central server sends the trained CGAN parameterized model and global model to the edge server.

[0078] Further, in step S4: the edge server uses CGAN to optimize the data distribution so that it satisfies the independent and identically distributed property, constructs a local neural network model and trains it independently to obtain a local parameterized model, and sends the local parameterized model to the central server.

[0079] Specifically, S401: Each edge server corrects the local data distribution through CGAN and generates a training set so that it meets the IID characteristics with the training sets of other edge server nodes;

[0080] S402: Each edge server designs a neural network model based on the structure of the training set samples and completes the initialization;

[0081] S403: Each edge server independently trains a locally parameterized model until the model converges;

[0082] S404: Each edge server sends a local parameterized model to the central server.

[0083] Further, in step S5: the central server uses sample labels and the CGAN network to generate a transfer set, and uses the local parameterized model trained on multiple edge servers as the teacher model to train the global neural network model by generating the transfer set.

[0084] Specifically, S501: The central server uses sample labels and CGAN to generate transfer sets to achieve data augmentation;

[0085] Local devices utilize generators to aggregate knowledge and identify missing target labels in local data samples. Based on these target labels, they generate high-quality samples similar to the global data distribution to achieve sample augmentation until the IID characteristic is met, thus improving the global data distribution P. jointWith local data distribution P k Satisfy P k =P joint As attached Figure 4 As shown.

[0086] S502: The central server uses local models trained on multiple edge servers as teacher models, and trains the global neural network model by generating transition sets, as shown in the appendix. Figure 5 As shown;

[0087]

[0088] in, The knowledge distillation loss for the global model; The loss between the student model's prediction and the true label. p represents the difference between the logarithmic outputs of the teacher model and the student model. global For the soft decision vectors of the teacher and student models; and p global The Kullback-Leibler divergence is used as a metric.

[0089]

[0090] Simulation experiment:

[0091] In this simulation experiment, the source data in the edge server is assumed to have high confidentiality. Therefore, a federated learning method is adopted for training the CGAN network to protect data security. Secondly, the edge server holding the data has certain computing power, enabling it to train the CGAN parameterized model using local private data and perform encrypted calculations on the model parameters. In each training round, the edge server trains and updates the global model generated in the previous round.

[0092] In this simulation experiment, different datasets were selected to verify the effectiveness of the method, including the MNIST, EMNIST, CELEBA datasets, and the real FOQA dataset, which are widely used in machine learning task research and evaluation. The MNIST dataset contains 70,000 grayscale image samples, each with a dimension of 28x28 pixels and corresponding to 10 sample labels. The EMNIST dataset is an extension of the MNIST dataset, containing 6 categories of samples: uppercase letters, lowercase letters, numbers, and symbols. The CELEBA dataset contains 202,599 image samples representing 10,177 celebrities, all with feature labels and 40 different attribute labels for each image. The FOQA dataset is a real-world dataset open-sourced by the NASA research team, including 99,837 samples from different flight routes corresponding to 4 categories of labels, with each data sample being 160×20 dimensions.

[0093] The experiment divided the MNIST dataset into 20 non-independent, identically distributed subsets as local private data on the edge server. Each subset included a local training set, a local validation set, and a local test set. Furthermore, no two subsets had the same samples.

[0094] The experiment was conducted using a centralized system architecture, consisting of one aggregation server and 20 edge training nodes. The training and test sets were divided into 20 groups and distributed to different simulated edge training nodes to replicate the data isolation between different nodes in a real-world application scenario. The experiment consisted of 200 iterations, with the same hyperparameters set for all edge nodes. The batch size (epochs) was 32, the learning rate was 0.01, the optimizer was Adam, and the distillation temperature parameter was 10.

[0095] To demonstrate the effectiveness and usability of this invention, experiments were conducted using convolutional neural networks to compare the performance of the proposed FedND algorithm with that of FedAvg, FedProx, and FedDistill algorithms. The experimental results were evaluated using the following metrics:

[0096] (1) Accuracy: The proportion of correctly classified samples out of all test samples;

[0097] (2) Model loss: measures the degree of difference between the prediction results of the global model and the true label, and records the trend of the change of the loss function of the global model.

[0098] To verify the effectiveness of this algorithm on heterogeneous samples, comparative experiments were conducted using various datasets. For the MNIST and EMNIST datasets, the Dirichlet function was used to divide the datasets into 20 groups. The number of samples allocated to each group was controlled by adjusting the Dirichlet distribution parameters, ensuring that the sample distribution of each data subset was different, thus satisfying the data heterogeneity requirement, and this was used to train the local model. For the CELEBA dataset, images belonging to different celebrities were randomly clustered into disjoint groups to increase the heterogeneity of the data. For the FOQA dataset, different sample data were randomly divided to represent the heterogeneity of the data subsets.

[0099] Comparison of simulation results of different algorithms on four datasets (see appendix) Figure 6 As shown in the figure, the model training convergence process under different algorithms is compared. The horizontal axis represents the number of training rounds, and the vertical axis represents the model accuracy. The experimental results show that when the number of iteration rounds is less than 200, this algorithm can learn faster and achieve global model convergence. In terms of model accuracy, this method is slightly better than the other three control groups. Figure 6The results show that, under the same conditions, this method can achieve better training results with fewer communication rounds, reducing the number of interactions required for model convergence, thereby reducing communication overhead and information exposure in practical applications.

[0100] In addition, Figure 7 The graph compares the model training losses under different algorithms, with the horizontal axis representing the number of training epochs and the vertical axis reflecting the change in the global model loss function. As can be seen from the graph, FedND achieves lower losses on the experimental dataset as the number of iterations increases.

[0101] To further explore the impact of data heterogeneity on model accuracy under our proposed method, we conducted further validation using the MNIST and EMNIST datasets. We quantified the degree of heterogeneity of data subsets using the Dirichlet distribution function from the NumPy library, and controlled the shape of the distribution by setting the hyperparameter α, thereby verifying the relationship between the degree of heterogeneity of data distribution and model accuracy under our proposed method. The experimental results are shown in Table 1 below. A larger α value indicates a more concentrated probability distribution and weaker heterogeneity among data subsets, while a smaller α value indicates a more dispersed probability distribution and stronger heterogeneity among data subsets. Firstly, the experimental results show that, under the same hyperparameter conditions, the proposed method achieves slightly higher model accuracy than other control groups. Secondly, the experimental results reflect the impact of data heterogeneity on model performance: FedND is robust to heterogeneity of different magnitudes, especially when the data distribution is highly heterogeneous, where the gain effect of our method on the global model is more significant.

[0102] Table 1. Model accuracy on MNIST and EMNIST datasets.

[0103]

[0104] Example 2:

[0105] Example 2 provides an information-free data fusion system that implements the heterogeneous network-free data fusion method based on federated distillation described in Example 1. The system is characterized by including a central server and multiple edge servers. A federated learning scheme is deployed on the server nodes, and the central server node coordinates the edge server nodes.

[0106] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A data-free fusion method for heterogeneous networks based on federated distillation, characterized in that, Includes the following steps, S1: Establish a centralized federation architecture, deploy federation learning schemes on corresponding server nodes within the joint operations system, and coordinate the edge server nodes through the central server node. S2: The central server initializes the global neural network model and the CGAN model; S3: Train the CGAN network model using federated learning; S4: The edge server uses CGAN to optimize the data distribution so that it satisfies the independent and identically distributed property, constructs a local neural network model and trains it independently to obtain a local parameterized model, and sends the local parameterized model to the central server; S5: The central server uses sample labels and the CGAN network to generate a transfer set, and uses the local parameterized model trained on multiple edge servers as the teacher model to train the global neural network model by generating the transfer set. Step S5 specifically includes the following steps: S501: The central server uses sample labels and the CGAN network to generate transfer sets; S502: The central server uses local models trained on multiple edge servers as teacher models and trains the global neural network model through a transfer set; in, The knowledge distillation loss for the global model; The loss between the student model's prediction and the true label. To minimize the difference between the logarithmic outputs of the teacher model and the student model, This represents the soft decision vector for the teacher and student models.

2. The heterogeneous network data-free fusion method based on federated distillation according to claim 1, characterized in that, Step S1 includes the following steps: S101: The training task is determined and broadcast by the central server; S102: The edge server evaluates the training task based on local data, determines whether to participate in the current training task, and sends a response to the central server; S103: The central server selects edge servers from the response set to participate in the training task and deploys the federated learning scheme.

3. The heterogeneous network data-free fusion method based on federated distillation according to claim 2, characterized in that, Step S3 includes the following steps: S301: The central server sends the CGAN parameterized model to the edge servers participating in model training; S302: Edge servers train CGAN networks based on local private data; S303: The edge server calculates the model gradient and transmits the gradient encrypted to the central server; S304: The central server receives gradient parameters and calculates the global loss for model updates using a weighted averaging method. This leads to the updating of the CGAN network; S305: The central server sends the updated global CGAN model to all edge servers for the next round of iteration until the model converges. S306: The central server sends the trained CGAN parameterized model and global model to the edge server.

4. The heterogeneous network data-free fusion method based on federated distillation according to claim 3, characterized in that, The specific operations for training the CGAN network in step S302 include: in, A parameterized CGAN model for edge servers. For generator, For discriminator, This is the learning rate.

5. The heterogeneous network data-free fusion method based on federated distillation according to claim 4, characterized in that, In step S304, the global loss is calculated by weighted averaging for model updates. The specific operation is as follows: in, Generative models on edge servers k The losses.

6. The heterogeneous network data-free fusion method based on federated distillation according to claim 3, characterized in that, Step S4 includes the following steps: S401: Each edge server corrects the local data distribution through CGAN and generates a training set so that it meets the IID characteristics with the training sets of other edge server nodes; S402: Each edge server designs a neural network model based on the structure of the training set samples and completes the initialization; S403: Each edge server independently trains a locally parameterized model until the model converges; S404: Each edge server sends a local parameterized model to the central server.

7. An information-free data fusion system implementing the heterogeneous network data-free fusion method based on federated distillation as described in any one of claims 1-6, characterized in that, It includes a central server and multiple edge servers. The server nodes are equipped with a federated learning scheme, and the central server node coordinates the various edge server nodes.

Images