A Personalized Communication Method Based on Dataset Compression

Abstract

This invention discloses a personalized communication method based on dataset compression, specifically including the following steps: S1: Perform data distillation on the graph data stored by the k-th client to obtain the k-th synthetic dataset S. k S2: Based on the synthetic dataset S k The optimization objective is constructed and optimized. S3: Based on the optimization objective, the learning rate for training the global model on synthetic data is determined. S4: The central server constructs a pre-trained model based on the learning rate and distributes the pre-trained model to each client for personalized training. By introducing graph dataset extraction and replacing model parameters with synthetic datasets in a single transmission, global information collection is achieved. Next, server models are locally trained on synthetic data to compensate for model performance deficiencies caused by the lack of low-dimensional features in synthetic data. Finally, a balance is achieved between communication volume and accuracy in graph federated learning.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a personalized communication method based on dataset compression. Background Technology

[0002] By combining graph theory with deep neural networks, graph neural networks (GNNs) are widely used in research fields such as drug discovery, neuroscience, social networks, knowledge graphs, recommender systems, and traffic flow prediction due to their powerful analytical capabilities for graph data. Classifying graph data is one of the main functions of GNNs; they predict class labels or target values ​​for a set of graphs. However, some data is difficult to share publicly and transmit to a centralized location. Furthermore, even if a single entity can provide sufficient computing power, the amount of data it possesses is often insufficient to meet the training requirements of a powerful GNN.

[0003] Federated learning, as a novel distributed learning paradigm, can effectively address this problem. In a typical federated learning setup, data is distributed across a set of nodes. Due to privacy and communication constraints, these nodes often find it difficult to directly transmit real data and share it with third parties. Federated learning replaces the traditional training method of directly transmitting real data for local training by uploading and aggregating models trained on nodes, thus reducing communication costs while protecting privacy. Therefore, as a combination of these two technologies, graph federated learning has attracted widespread attention from researchers, and some studies have been conducted on this issue. However, unlike traditional federated learning training data such as images, graph data may have inherent heterogeneity in terms of structure and features, which exacerbates the non-independent identically distributed (Non-IID) problem in federated learning, leading to an increase in communication rounds and reducing the training accuracy of the final model. Although some works have noted the accuracy degradation caused by Non-IID data, they often fail to solve the problem or even exacerbate the communication burden of federated training. Furthermore, the original federated learning training process requires data and networks to remain open to the federated training plan at all times, which is often unacceptable for confidential data such as drug molecules. Summary of the Invention

[0004] To address the technical problem of numerous communication rounds in federated learning caused by data heterogeneity in existing technologies, this invention proposes a personalized communication method based on dataset compression. By extracting high-dimensional features from the acquired data and compressing it before transmitting it to a central server for training, the method reduces low-dimensional features such as semantic information in the data, lowers the data storage volume, and thus reduces the number of communication rounds.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A personalized communication method based on dataset compression specifically includes the following steps:

[0007] S1: Perform data distillation on the graph data stored by the kth client to obtain the kth synthetic dataset S. k ;

[0008] S2: Based on the synthetic dataset S k Develop and optimize optimization goals;

[0009] S3: Determine the training learning rate of the global model on synthetic data based on the optimization objective;

[0010] S4: The central server builds a pre-trained model based on the learning rate and distributes the pre-trained model to each client for personalized training.

[0011] Preferably, S1 includes:

[0012] S1-1: Organize the graph dataset G according to its data type k Rearrange to obtain G k ={A k X k , Y k ,},A k This represents the adjacency matrix obtained by concatenation. R represents the set of real numbers, N k Represents a graph dataset G k The total number of nodes in all graph data; X k This represents the corresponding node feature matrix, where d represents the dimension of the node features; |G represents the label set of all graphs. k | represents the number of graph data points used for training in the k-th client;

[0013] S1-2: By analyzing the graph dataset G k Distillation to generate synthetic datasets

[0014] Preferably, in steps S1-2, the synthetic dataset S k And graph dataset G k The performance is the same, and this performance is quantified using the one-step gradient generated from training on the real dataset and the synthetic dataset.

[0015]

[0016] In formula (1), This represents the global model f evaluated on the test dataset from the k-th client. θ The loss; D represents the distance function; This represents the gradient.

[0017] Preferably, S2 includes:

[0018] S2-1: Construct an initial optimization objective function and use gradient descent to update the synthesized dataset for optimization;

[0019] S2-2: Modeling edge weights using a reparameterization method;

[0020] S2-3: Construct the final objective function and optimize the objective.

[0021] Preferably, in step S2-1, the initial optimization objective function is:

[0022]

[0023] Preferably, in S2-2, the edge weight model is as follows:

[0024]

[0025] In formula (3), express A′ k The elements in the equation are hidden variables of the Bernoulli distribution and are also the main optimization objects; σ represents the sigmoid function; α is collected from Uniform(0,1); τ∈(0,∞) represents the continuously relaxed temperature parameter.

[0026] Preferably, in steps S2-3, the final objective function is:

[0027]

[0028] In formula (4), Ω and X′ are initialized with IPC training samples randomly selected from each category. k .

[0029] Preferably, in step S3, the training learning rate of the global model on the synthetic data is adaptively determined based on the different amounts of data in the local dataset used:

[0030]

[0031] In formula (5), θ t+1 G represents the learning rate at time t+1; K represents the total number of clients; |G k | represents the number of graph data points used for training in the k-th client; η represents the total number of training data points; η represents the learning rate.

[0032] Preferably, in step S4, after receiving the pre-trained model from the server, the client will use local data to build a personalized model for training.

[0033]

[0034] In formula (6), η represents the learning rate; the number of local training epochs is set to E. loc .

[0035] Preferably, a validation set is used to check the model's performance after 200 epochs. If the performance degrades from the beginning, local personalized model training for the client is abandoned.

[0036] In summary, by adopting the above technical solution, the present invention has at least the following beneficial effects compared with the prior art:

[0037] This invention extracts high-dimensional features from the acquired data and compresses it into a global synthetic dataset, which is then transmitted to a central server for training. This reduces low-dimensional features such as semantic information in the data, thereby reducing the amount of data storage (to only a few digits) and thus improving the number of rounds of retrieval communication.

[0038] The server model trained on high-dimensional feature data is used as a carrier of global knowledge and distributed as a pre-trained model to each client. Personalized training is then performed on each client's local data. This allows each client to obtain a model that takes into account both global perspective and local features, thereby further improving the overall training effect and learning accuracy. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of a personalized communication method based on dataset compression according to an exemplary embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram illustrating the construction of client graph data according to an exemplary embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram comparing model performance under different IPC settings according to an exemplary embodiment of the present invention. Detailed Implementation

[0042] The present invention will be further described in detail below with reference to embodiments and specific implementation methods. However, this should not be construed as limiting the scope of the above-described subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0043] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0044] Existing graph federated learning algorithms typically use models trained on clients as carriers of local knowledge and construct a global data distribution pattern by averaging model parameters across clients. However, this is often inefficient. In particular, when clients have non-IID data samples, more rounds of model parameter exchange are usually required, and the resulting models are often only suboptimal.

[0045] Unlike previous methods, our proposed FedGDC-P global information collection process uses small datasets synthesized by clients instead of model parameters uploaded to the server. Since the synthetic datasets uploaded by each client reflect the high-dimensional features of their local data, we can generate server-side models with a global perspective by performing gradient descent on the server using these datasets.

[0046] However, due to the lack of low-dimensional features and the inherent limitations of graph neural networks, trained server models often struggle to perform well in client environments with varying data conditions. Therefore, we perform a local personalized training process after the global information gathering process; that is, we distribute the server model as a pre-trained model to the client for continued local training, thereby obtaining a personalized solution suited to its environment.

[0047] like Figure 1 As shown, this invention provides a personalized communication method based on dataset compression, specifically including the following steps:

[0048] S1: Perform data distillation on the graph data stored by the kth client to obtain the kth synthetic dataset S. k .

[0049] In this embodiment, the graph dataset stored by the client can be represented as follows:

[0050] S1-1: As Figure 2 As shown, the graph dataset G is categorized by data type. k Rearrange to obtain G k ={A k X k , Y k,},A k This represents the adjacency matrix obtained by concatenation. R represents the set of real numbers, N k Represents a graph dataset G k The total number of nodes in all graph data; X k This represents the corresponding node feature matrix, where d represents the dimension of the node features; |G represents the label set of all graphs. k | indicates the number of graph data used for training in the k-th client.

[0051] S1-2: By analyzing the graph dataset G k Distillation to generate synthetic datasets

[0052] In this embodiment, the synthetic dataset S k Graph dataset (real dataset) G k The performance is the same, and this performance can be quantified using one-step gradients generated from training on real and synthetic datasets.

[0053]

[0054] In formula (1), This represents the global model f evaluated on the test dataset from the k-th client. θ The loss; D represents the distance function; This represents the gradient.

[0055] In this embodiment, gradient matching is performed within a class, meaning that gradient information is only matched between data of the same class but from different sources.

[0056] S2: Based on the synthetic dataset S k Develop optimization goals and then optimize them.

[0057] To avoid overfitting the model parameters to the synthetic data, the initial model parameters need to be continuously updated. However, considering that traditional inner-loop iterative methods are not suitable for client-side data imbalance and resource-constrained situations, a randomly generated strategy is used to initialize the model parameters before each iteration. express.

[0058] S2-1: In this way, an initial optimization objective function is constructed and gradient descent is used to update the synthesized dataset for optimization.

[0059]

[0060] S2-2: The elements in the adjacency matrix of graph data only accept binary variables, making it impossible to derive gradients. To address this issue, the elements in the adjacency matrix of synthetic data are treated as a result of a Bernoulli distribution, and the edge weights are modeled using a reparameterization method as follows:

[0061]

[0062] In formula (3), express A′ k The elements in the equation are hidden variables of the Bernoulli distribution and are also the main optimization objects; σ represents the sigmoid function; α is collected from Uniform(0,1); τ∈(0,∞) represents the continuously relaxed temperature parameter.

[0063] In this embodiment, an annealing algorithm is used to gradually reduce the value of τ during training. In this way, gradient descent can be used for optimization. Indirect optimization of adjacency matrix A′ k .

[0064] S2-3: Construct the final objective function and optimize the objective.

[0065]

[0066] In formula (4), Ω and X′ are initialized with IPC training samples randomly selected from each category. k .

[0067] S3: Determine the training learning rate of the global model on synthetic data based on the optimization objective.

[0068] In this embodiment, after the server receives all synthetic data from all clients, it can directly use this data as the real dataset to train a global model for K epochs. However, due to the heterogeneity of client data distribution, each client may have a different number of similar data points, all of which are extracted as the same number of synthetic data points, leading to a mismatch. Therefore, this invention proposes to adaptively determine the training learning rate of the global model on the synthetic data based on the different amounts of data in the local dataset used:

[0069]

[0070] In formula (5), θ t+1 G represents the learning rate at time t+1; K represents the total number of clients; |G k | represents the number of graph data points used for training in the k-th client; η represents the total number of training data points; η represents the learning rate.

[0071] S4: The central server builds a pre-trained model based on the learning rate and distributes the pre-trained model to each client for personalized training.

[0072] In this embodiment, due to the lack of low-dimensional features during training, the server model obtained by training with synthetic data typically has limited performance. Furthermore, due to the limitations of oversmoothing, the original graph neural network itself struggles to adapt to complex and ever-changing client environments.

[0073] Therefore, this invention abandons the idea of ​​generating a powerful global model on a server, and instead generates personalized models adapted to their local environments on each client. To achieve the personalization process, after receiving the pre-trained model from the server, the client utilizes local data to build a personalized model, making it perform better on local data.

[0074]

[0075] In formula (6), η represents the learning rate; the number of local training epochs is set to E. loc ;

[0076] In this embodiment, a validation set is used to check the model's performance after 200 epochs. If the performance drops from the beginning, local personalized training for the client will be abandoned.

[0077] To verify the performance of our method, we evaluated it on several datasets from different domains, specifically DD (Bioinformatics), IMDB-BINARY (Social Networks), and NCI1 (Small Molecule). Some of these datasets lacked node feature matrices for the data distillation algorithm; therefore, vectors obtained by one-hot encoding of node degrees were used as node features. Before the experiments, we also statistically analyzed some numerical features for each dataset, including the average number of nodes and average number of edges. We distributed the data from DD, IMDB-BINARY, and NCI1 across 10, 20, and 30 clients, respectively, and randomly partitioned the data on each client into local training, validation, and test sets in an 8:1:1 ratio. Detailed information about the data is shown in Table 1.

[0078] Table 1. Comparison of information from different datasets

[0079]

[0080] This invention compares itself with the following five algorithms: First, a self-trained algorithm is used as the first baseline method, where the client trains using only its own local data. This baseline allows for testing the effectiveness of various types of federated learning algorithms in improving global performance. Second, two widely used federated learning algorithms, FedAvg and FedProx, are implemented. FedProx incorporates key design features specifically designed to handle data and system heterogeneity in non-graph federated learning. Finally, a state-of-the-art graph federated learning algorithm called GCFL and its variant GCFL+ are introduced, representing a family of algorithms that can effectively handle graph data heterogeneity using clustering concepts. To be fair, all algorithms used the same GCN design in the experiments. Hyperparameters appearing in all baseline methods were set to the same values.

[0081] This invention uses a three-layer GCN network with a hidden layer size of 128. In the baseline method, the batch size is 128, the learning rate of the Adam optimizer is 0.005, the weight decay is 5e-4, and the number of local epochs is 1. μ in FedProx is set to 0.01. ε1 and ε2 are set to optimal values ​​based on the dataset. In the comparative experiments, the data synthesis process is set to 200 epochs per client. To evaluate the communication throughput, the IPC parameter of the data synthesis process is set to 1, while the learning rate of the synthesis process is adjusted appropriately based on the dataset. Table 2 compiles all parameters set differently based on the dataset in the comparative experiments. Simultaneously, parameter study experiments were conducted to investigate the performance ceiling of this invention. In this experiment, the IPC was changed to investigate whether more synthesized data would improve model accuracy. All experiments were randomly replicated five times on a server with an Nvidia RTX3080 GPU.

[0082] Table 2. All parameters set differently depending on the dataset

[0083]

[0084] Table 3 compares the experimental results, showing the performance and data transfer volume of different algorithms on different datasets. By comparing with other algorithms, it can be seen that the proposed FedGDC-P achieves state-of-the-art (SOTA) accuracy performance with very low communication costs. Furthermore, we can clearly observe a significant performance gap between traditional federated learning methods (FedAvg, FedProx) and clustering graph federated learning methods. This reflects the difference between graph federated learning and general graph federated learning; that is, heterogeneity in a graph federated learning environment exists not only between individual clients but also between individual graph data. This dual heterogeneity limits the effectiveness of some traditional heterogeneity handling methods and even hinders algorithm performance in its design.

[0085] Table 3. Comparative Experimental Results

[0086]

[0087] Table 4 shows the impact of different IPC settings on different datasets for the model. Intuitively, the more data submitted, the better the server can grasp the data distribution on the clients, and the better the model performance should be. However, experimental results show that simply increasing IPC does not improve model performance and may even lead to a slight decrease in performance. This phenomenon indicates that for data subsets scattered across clients, one synthetic dataset for each category is sufficient to accommodate its limited high-dimensional feature information to support the construction of a comprehensive picture of the data distribution. More synthetic datasets only result in more low-dimensional features. Due to the disruption of the dataset extraction process and the heterogeneity of the graphical data itself, the information carried by these low-dimensional features is mostly contradictory, leading to a slight decrease in model performance.

[0088] Table 5. Effects of different IPCs

[0089]

[0090] This invention attempts to compare the performance of the model with and without the removal of the local personalized training process, i.e., an ablation experiment. To more fully demonstrate the role and effect of the local personalized training process, the performance changes of both models under different IPC settings are also shown, and... Figure 3 The experimental results are shown in the figure. Here, we can observe that the model without the local personalized training process exhibits a significant performance degradation, even approaching the performance of the self-training algorithm that trains the model using only local data at some point. This verifies the judgment that it is difficult to train a good global model without sufficient support from low-dimensional features from local data. At the same time, it can be recognized that for different IPC settings, the performance changes of the two models in the same direction, reflecting the role of the global information aggregation process in supporting the final model's performance.

[0091] This paper proposes the FedGDC-P algorithm to address the issues of model accuracy degradation and communication pressure caused by data heterogeneity in graph federated learning. First, global information collection is achieved by introducing graph dataset extraction and replacing model parameters with a synthetic dataset in a single transmission. Next, the performance deficiency caused by the lack of low-dimensional features in the synthetic data is compensated for by locally training the server model based on the synthetic data. Finally, a balance between communication volume and accuracy in graph federated learning is achieved. Comparative experiments on multiple datasets demonstrate that this method achieves the same or even better accuracy results with very low communication cost compared to existing methods, proving its efficiency and robustness. Ablation experiments also demonstrate the necessity of the local personalization process. Future work will explore applying this method to more areas of graph federated learning algorithms and exploring its potential with more GNN models.

[0092] Those skilled in the art will understand that the above embodiments are specific examples of implementing the present invention, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of the present invention.

Claims

1. A personalized communication method based on dataset compression, characterized in that, Specifically, the following steps are included: S1: Perform data distillation on the graph data stored by the kth client to obtain the kth synthetic dataset S. k ; S2: Based on the synthetic dataset S k Develop and optimize optimization goals; S3: Determine the training learning rate of the global model on synthetic data based on the optimization objective; S4: The central server builds a pre-trained model based on the learning rate and distributes the pre-trained model to each client for personalized training; S1 includes: S1-1: Organize the graph dataset G according to its data type k Rearrange to obtain G k ={A k ,X k ,Y k ,},A k This represents the adjacency matrix obtained by concatenation. R represents the set of real numbers, N k Represents a graph dataset G k The total number of nodes in all graph data; X k This represents the corresponding node feature matrix, where d represents the dimension of the node features; |G represents the label set of all graphs. k | represents the number of graph data points used for training in the k-th client; S1-2: By analyzing the graph dataset G k Distillation to generate synthetic datasets In S1-2, the synthetic dataset S k And graph dataset G k The performance is the same, and this performance is quantified using the one-step gradient generated from training on the real dataset and the synthetic dataset. In formula (1), This represents the global model f evaluated on the test dataset from the k-th client. θ The loss; D represents the distance function; Represents the gradient; In step S3, the training learning rate of the global model on the synthetic data is adaptively determined based on the different amounts of data in the local dataset used: In formula (5), θ t+1 G represents the learning rate at time t+1; K represents the total number of clients; |G k | represents the number of graph data points used for training in the k-th client; This indicates the total number of training data points. η represents the learning rate.

2. The personalized communication method based on dataset compression as described in claim 1, characterized in that, S2 includes: S2-1: Construct an initial optimization objective function and use gradient descent to update the synthesized dataset for optimization; S2-2: Modeling edge weights using a reparameterization method; S2-3: Construct the final objective function and optimize the objective.

3. The personalized communication method based on dataset compression as described in claim 2, characterized in that, In S2-1, the initial optimization objective function is:

4. The personalized communication method based on dataset compression as described in claim 2, characterized in that, In S2-2, the edge weight model is as follows: In formula (3), express A' k The elements in the equation are hidden variables of the Bernoulli distribution and are also the main optimization objects; σ represents the sigmoid function; α is collected from Uniform(0,1); τ∈(0,∞) represents the continuously relaxed temperature parameter.

5. A personalized communication method based on dataset compression as described in claim 2, characterized in that, In S2-3, the final optimization objective function is: In formula (4), Ω and X are initialized with IPC training samples randomly selected from each category. ' k .

6. The personalized communication method based on dataset compression as described in claim 1, characterized in that, In step S4, after receiving the pre-trained model from the server, the client will use local data to build a personalized model for training. In formula (6), η represents the learning rate; the number of local training epochs is set to E. loc .

7. A personalized communication method based on dataset compression as described in claim 6, characterized in that, A validation set is used to check the model's performance after 200 epochs. If the performance degrades from the beginning, local personalized model training for the client will be abandoned.

Images