Federated learning distributed model optimization method based on symbiotic multi-agent reinforcement learning

Abstract

This paper proposes a federated learning distributed model optimization method based on symbiotic multi-agent reinforcement learning. The implementation steps are as follows: constructing a federated learning system; each client obtains a local training sample set, and the server obtains a global test sample set; the server performs adaptive symbiotic clustering on each client; the server constructs a symbiotic multi-agent reinforcement learning network and initializes the parameters; the server sends a local image classification model to each client; each client in each cluster iteratively trains the local image classification model; each client in each cluster prunes the model parameters and uploads the parameter differences; the server obtains the global model optimization results. The present invention achieves customized clustering through adaptive symbiotic clustering, improving the overall accuracy of the system; pruning model parameters through the symbiotic multi-agent reinforcement learning network allows clients in different clusters to select the optimal pruning rate, optimizes parameter calculation, and improves training efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of federated learning technology and relates to a federated training model optimization method, specifically to a federated learning distributed model optimization method based on symbiotic multi-agent reinforcement learning. Background Art

[0002] Distributed computing is a technical architecture that enables multiple computing nodes to work together to complete complex tasks. In a distributed system, each computing node can process different tasks in parallel and communicate and coordinate over a network. This architecture not only improves the system's computing power, but also provides greater fault tolerance and supports dynamic resource expansion.

[0003] With the rapid development of artificial intelligence (AI), distributed models have become an important branch of machine learning. By distributing large amounts of data and computing tasks across multiple nodes for parallel processing, distributed models significantly improve the efficiency and scalability of model training. Consequently, distributed model optimization has become an important means of improving system performance. It addresses key bottlenecks in large-scale training by improving parameter update methods and reducing communication overhead.

[0004] Federated learning, as a distributed framework, allows each computing node to independently train local data and collaboratively build a global model through parameter exchange, effectively protecting data privacy. It is suitable for privacy-critical fields such as finance and healthcare. However, as the scale of the system continues to expand, the technical challenges it faces are becoming increasingly prominent.

[0005] First, high communication overhead limits the efficiency of large-scale systems. In existing systems, frequent data exchange between clients and servers is essential. Especially during global model updates and local training, the uploading and downloading of large numbers of model parameters results in a huge communication burden, which affects overall efficiency.

[0006] Secondly, heterogeneity between clients leads to reduced training efficiency and unstable global model convergence. In a typical federated learning system, the data distribution of different clients is usually not independent and identically distributed (Non-IID). This heterogeneity leads to significant differences in model performance, which in turn affects the convergence of the global model.

[0007] To this end, research has proposed improved methods based on single-agent reinforcement learning (RL-FL). For example, Nie Yuming et al. proposed the Federated Reinforcement Learning Distributed Model Pruning (FRLP) method in Volume 33, Issue 5 of the Journal of Computer Systems and Applications in 2024. This method first abstracts the model pruning process, establishes a Markov decision process, and uses the Deep Q-Network (DQN) algorithm to construct a reinforced generalization pruning model (RGP), dynamically adjusting the pruning rate. Secondly, a sparse model aggregation method (FedSA) is designed, which only weights non-zero weights to avoid inefficient computation of zero weights. FRLP improves the model accuracy of federated learning systems in resource-constrained environments, overcoming the shortcomings of traditional pruning methods, which lack adaptive pruning. In terms of optimizing the model structure, it effectively reduces complexity and improves the model's generalization performance. However, this method relies on a single agent to make decisions, making it ineffective for large-scale heterogeneous systems. Furthermore, it only weights non-zero weights, which hinders further improvement of model accuracy and fails to achieve a balance between model accuracy and training efficiency in large-scale heterogeneous systems. Summary of the Invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a federated learning distributed model optimization method based on symbiotic multi-agent reinforcement learning, aiming to improve the accuracy of the model in large-scale heterogeneous systems while improving training efficiency.

[0009] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0010] (1) Building a federated learning system:

[0011] Build a centralized federated learning system consisting of a server and N clients, where N ≥ 5 and the nth client is C n ;

[0012] (2) Each client obtains a local training sample set, and the server obtains a global test sample set:

[0013] Each client C n Obtain M images containing Z target categories, and mark the targets in each image, and then form a local training sample set with all images and corresponding labels; at the same time, the server will contain M images of Z target categories. p The images and their corresponding labels constitute the global test sample set, where Z ≥ 2, M ≥ 3000, and M p ≥10000;

[0014] (3) The server performs adaptive symbiotic clustering on the client:

[0015] The server performs adaptive symbiotic clustering on N clients and obtains K clusters each containing I clients, where K ≥ 2 and the kth cluster is V k , I≥1, the i-th client in the k-th cluster is C k i ;

[0016] (4) The server builds a symbiotic multi-agent reinforcement learning network and initializes the parameters:

[0017] Server for each cluster V k Construct a deep Q-network agent. K deep Q-network agents form a symbiotic multi-agent reinforcement learning network. At the same time, the server initializes the global aggregation round to x, the maximum global aggregation round to X, X ≥ 10, and sets x = 0.

[0018] (5) The server sends the local image classification model to each client:

[0019] The server will use the local image classification model ω in the xth round x Sent to each client

[0020] (6) Each client in each cluster iteratively trains the local image classification model:

[0021] Each client in each cluster Local image classification model through local training sample set Perform iterative training and obtain the trained parameters as Local image classification model

[0022] (7) Each client in each cluster prunes the model parameters and uploads the parameter differences:

[0023] Each client in each cluster uses a symbiotic multi-agent reinforcement learning network to train a local image classification model Parameters Perform pruning and calculate the difference in model parameters before and after pruning Upload to the server;

[0024] (8) The server obtains the global model optimization results:

[0025] Server based on each client Uploaded Perform weighted average aggregation, update the local image classification model and the symbiotic multi-agent reinforcement learning network, and evaluate the model ω of the xth round through the global test sample set xAccuracy Acc and loss value Loss; then determine whether x=X is true. If so, get the optimized global local image classification model ω X Otherwise, the server sends ω x Go to each client in each cluster, set x=x+1, and execute step (5).

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] (1) The server of the present invention performs adaptive symbiotic clustering on all clients. Through customized clustering and dynamic adjustment, it ensures that clients in the same cluster have similar characteristics and requirements, retains the key information required by each client, makes model training and pruning strategies more targeted and effective, and improves the overall accuracy of the distributed model.

[0028] (2) Each client in each cluster of the present invention prunes the parameters of the trained local image classification model through a symbiotic multi-agent reinforcement learning network. Clients in different clusters can select their own optimal pruning rate, reduce the size of parameters during transmission, optimize parameter calculation, reduce communication time and training time, and thus improve model training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Flowchart for the implementation of the present invention;

[0030] Figure 2 This is a simulation comparison chart of the distributed model accuracy of the present invention and the prior art;

[0031] Figure 3 This is a simulation comparison chart of the distributed model training efficiency of the present invention and the prior art. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] Reference Figure 1 , the present invention comprises the following steps:

[0034] Step 1) Build a federated learning system:

[0035] Build a centralized federated learning system consisting of a server and N clients, where N ≥ 5 and the nth client is C n ;

[0036] A centralized federated learning system is a federated learning structure based on a central server. It relies on the central server to coordinate the training process of multiple clients. The central server sends a global model to all clients in each iteration. After each client trains the model on local data, it sends the updated model parameters back to the server. In this embodiment, N = 15.

[0037] Step 2) Each client obtains a local training sample set, and the server obtains a global test sample set:

[0038] Each client C n Obtain M images containing Z target categories, and mark the targets in each image, and then form a local training sample set with all images and corresponding labels; at the same time, the server will contain M images of Z target categories. p The images and their corresponding labels constitute the global test sample set, where Z ≥ 2, M ≥ 3000, and M p ≥10000;

[0039] The local training sample set used in this embodiment is the training set in the CIFAR-10 universal object color image dataset. The number of target categories corresponding to this dataset is 10, namely: airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships and trucks, and a total of 50,000 images. The pixel size of each image is 32×32, and there are 5,000 images for each target category. In this embodiment, Z=5 and M=25,000 are set to simulate the heterogeneity of the client; the global test sample set used in this embodiment is the test set of the CIFAR-10 universal object color image dataset. The number of target categories corresponding to this dataset is 10, the pixel size of each image is 32×32, and there are 1,000 images for each target category, and a total of M p = 10,000 images.

[0040] Step 3) The server performs adaptive symbiotic clustering on each client:

[0041] The server performs adaptive symbiotic clustering on N clients and obtains K clusters each containing I clients, where K ≥ 2 and the kth cluster is V k , I≥1, the i-th client in the k-th cluster is C k i ;

[0042] In this embodiment, K=5;

[0043] Adaptive symbiotic clustering is a technology that achieves better clustering effects by identifying the characteristics of different clients and adaptively adjusting them. In heterogeneous client scenarios, customized clustering and dynamic adjustment ensure that clients within the same cluster have similar characteristics and requirements, retaining the key information required by each client, making model training and pruning strategies more targeted and effective, and improving the overall accuracy of the distributed model.

[0044] The server described in this step performs adaptive symbiotic clustering on each client, and the implementation steps are as follows:

[0045] Step 3a) The server sends a n The local training samples are used to extract heterogeneous data features, and the extracted feature vector f n Normalize and get the standardized feature vector

[0046] f n =[z n ,p n ,μ n ,σ n ]

[0047]

[0048] Among them, z n It is C n The local training samples are flattened into a one-dimensional vector to calculate the data size, p i is the category distribution of local training samples, μ n and σ n It is C n The local training samples are flattened into a one-dimensional vector to calculate the mean and standard deviation, μ(f N ) and σ(f N ) are the mean and standard deviation of all client feature vectors respectively;

[0049] Step 3b) The server calculates the Euclidean distance between each two client normalized feature vectors and selects each normalized feature vector The O smallest components in the Euclidean distance Neighborhood set in, yes oth neighbor, O≥5;

[0050] Step 3c) The server initializes the number of iterations to iter, the maximum number of iterations is Iter, Iter ≥ 100, and randomly selects k standardized feature vectors from all standardized feature vectors as the initial cluster centers. The kth cluster center is And let iter=0;

[0051] In this embodiment, iter=100.

[0052] Step 3d) The server calculates the fitness distance d between each two clients n,k , and select the standardized feature vector with the minimum fitness distance for each client As the current cluster center of the client, all current cluster centers are of The corresponding clients form a cluster V kn ,in:

[0053]

[0054] Among them, α is the adjustment factor, ||.|| is the calculation of Euclidean distance;

[0055] In this embodiment, α=0.5;

[0056] Step 3e) The server calculates V for each cluster kn All normalized eigenvectors within Average value Will As the current cluster center, at the same time, judge each Neighbor set F n Whether there are neighbors in the cluster that belong to different clusters, remove the neighbors that belong to different clusters F n :

[0057]

[0058] Among them, Y kn V kn middle The collection of N Yk V kn middle The number of

[0059] Step 3f) The server determines whether iter=Iter, if so, set V k =V kn , get the clustering result V k , otherwise, let iter = iter + 1 and execute step 3d).

[0060] Step 4) The server builds a symbiotic multi-agent reinforcement learning network and initializes the parameters:

[0061] Server for each cluster V k Construct a deep Q-network agent. K deep Q-network agents form a symbiotic multi-agent reinforcement learning network. At the same time, the server initializes the global aggregation round to x, the maximum global aggregation round to X, X ≥ 10, and sets x = 0.

[0062] The symbiotic multi-agent reinforcement learning network is based on a multi-agent value decomposition network. The multi-agent value decomposition network consists of multiple deep Q-network agents. Its core idea is to decompose the global reward signal into the local value function of each agent, thereby simplifying the learning process of the multi-agent system. By linearly combining the local value functions of multiple agents, the overall value function is estimated, enabling each agent to learn and make decisions collaboratively, improving the efficiency of multi-agent learning. Based on this, the symbiotic multi-agent reinforcement learning network determines K as the number of clusters and sets the client coordinated by each deep Q-network agent as the client of the cluster.

[0063] In this embodiment, X=20;

[0064] Step 4) The deep Q network agent includes sequentially stacked layers for accepting the state space s k The input layer, one or more fully connected layers or convolutional layers pass through the state space s k , action space A k A hidden layer that calculates the Q value and an output layer that outputs the Q value, where:

[0065]

[0066] A k ={a min ,a1,a2,…,a * ,...,a max}

[0067] Q k (s k ,a * )=h1×s k +h2×a * +p

[0068] Among them, a min and a max are the minimum and maximum pruning rates respectively, h1 and h2 are the weight matrices of the deep Q network agent network, p is the bias vector of the deep Q network agent network, Q k (s k ,a * ) is the local Q value of the deep Q network agent.

[0069] In this embodiment, the structure of the DQN network is as follows: the input layer is a fully connected layer with 128 neurons, which converts the input state into 128 features. It has two hidden layers, each containing 128 neurons, using the ReLU activation function. The output layer passes through a fully connected layer to map the features to the pruning rate, with the minimum pruning rate a minand the maximum pruning rate a max Set to 0.01% and 50%, and χ are 0.1 and 0.4, respectively;

[0070] Step 5) The server sends the local image classification model to each client:

[0071] The server will use the local image classification model ω in the xth round x Sent to each client

[0072] The local image classification model ω0 described in step 5) includes an input layer stacked in sequence for receiving image data; one or more convolutional layers and pooling layers for feature extraction; one or more fully connected layers for integrating features and making final decisions; and an output layer for outputting classification results;

[0073] In this embodiment, the local image classification model is the residual neural network ResNet-18. The architecture of ResNet-18 includes an initial 7x7 convolutional layer with 64 channels and a stride of 2, followed by a 3x3 maximum pooling layer with a stride of 2. Its core consists of 4 groups of residual layers. The structure of the residual block includes two consecutive 3x3 convolutional layers, each followed by a batch normalization layer and a ReLU activation function, and the input is directly added to the output of the second convolutional layer through a jump connection; among the 4 groups of residual layers, the first group has 2 residual blocks, each with 64 filters; the second group has 2 residual blocks, each with 128 filters; the third group has 2 residual blocks, each with 256 filters; the fourth group has 2 residual blocks, each with 512 filters. Each residual block contains two 3x3 convolutional layers, which improve stability through batch normalization and ReLU activation. The global average pooling layer at the end of the network converts the feature map into a 512-dimensional vector, and finally outputs the classification result through a fully connected layer.

[0074] Step 6) Each client in each cluster iteratively trains the local image classification model:

[0075] Each client in each cluster Local image classification model through local training sample set Perform iterative training and obtain the trained parameters as Local image classification model

[0076] Step 6a) Each client in each cluster Initialize the local training iteration number to e, the maximum iteration number is E, E>1, local image classification model The model parameters are And let e=0;

[0077] In this embodiment, E=5;

[0078] Step 6b) Local Image Classification Model Perform forward propagation on each local training sample to obtain the predicted label corresponding to the mth local training sample And adopt the cross entropy loss function, through the true label of the mth local training sample and predicted labels calculate The loss value L, and then use the gradient descent method to Update to get the local image classification model for this iteration in:

[0079]

[0080] in, The actual label and predicted labels The probability of consistency, is a hyperparameter, For L with respect to the parameter gradient;

[0081] In this embodiment,

[0082] Step 6c) Each client in each cluster Determine whether e=E is true. If so, the trained model parameters are Local image classification model Otherwise, let e=e+1, And proceed to step 6b).

[0083] Step 7) Each client in each cluster prunes the model parameters and uploads the parameter differences:

[0084] Each client in each cluster uses a symbiotic multi-agent reinforcement learning network to train a local image classification model Parameters Perform pruning and calculate the difference in model parameters before and after pruning Upload to the server;

[0085] Step 7a) Each client in each cluster calculates the Q value corresponding to all possible pruning rates and selects the pruning rate that maximizes the Q value Calculate reward R k , then R k , s k , As experience data lv and stored, where:

[0086]

[0087] Among them, Acc represents the accuracy rate, and χ are hyperparameters that control the weight of the reward term, t tr x is the training time of the client in round x, t cs x is the communication time between the client and the server in round x;

[0088] Step 7b) Each client in each cluster Calculating model parameters The absolute value of each weight in the , and the absolute value is ranked first The weights of the items are reset to zero to obtain the pruned model parameters Then and The difference Upload to the server, where:

[0089]

[0090] Through symbiotic multi-agent reinforcement learning, the present invention enables clients in different clusters to select their own optimal pruning rates, reduce the size of parameters during transmission, optimize parameter calculation, shorten communication time and training time, and thus improve model training efficiency.

[0091] Step 8) The server obtains the global model optimization results:

[0092] Server based on each client Uploaded Perform weighted average aggregation, update the local image classification model and the symbiotic multi-agent reinforcement learning network, and evaluate the model ω of the xth round through the global test sample set x Accuracy Acc and loss value Loss; then determine whether x=X is true. If so, get the optimized global local image classification model ω X Otherwise, the server sends ω x Go to each client in each cluster, set x = x + 1, and execute step 5);

[0093] The server weighted aggregation, model and network update, and global model performance evaluation are implemented in the following steps:

[0094] Step 8a) The server passes To W x Update and get the model ω of the xth round x , the updated model parameters are W xThe model is then updated using the Bellman equation for symbiotic multi-agent reinforcement learning, where W x The update is:

[0095]

[0096] Among them, λ is a hyperparameter;

[0097] Use the Bellman equation to update the parameters of symbiotic multi-agent reinforcement learning. The implementation steps are as follows:

[0098] Step 8a1) The server extracts ζ empirical data lv for each cluster, ζ ≥ 5, and calculates the target Q value according to the Bellman equation

[0099]

[0100] Among them, u is a hyperparameter, R k lv is the reward corresponding to the largest Q value in the experience data extracted from each cluster, is the maximum Q value among the empirical data extracted from each cluster;

[0101] Step 8a2) The server uses the gradient descent method to update the parameter θ of the symbiotic multi-agent reinforcement learning, setting θ = θ * :

[0102]

[0103] in, is a hyperparameter;

[0104] In this embodiment, ζ=5, u is 0.91, is 0.007;

[0105] Step 8b) The server inputs the global test sample set to ω x Perform forward propagation and obtain M p Predicted labels Calculate Acc and Loss:

[0106]

[0107] in, represents the predicted label of the mth training sample, represents the actual label of the mth training sample, represent and Consistent probability.

[0108] The following is a further explanation of the technical effects of the present invention in conjunction with the simulation results:

[0109] 1. Experimental conditions and content:

[0110] The model accuracy and communication time of the present invention are simulated using PyCharm 2023.2:

[0111] Experiment 1: Comparison simulation of the accuracy of the distributed model of the present invention and the prior art is carried out. The results are as follows: Figure 2 As shown;

[0112] Experiment 2: Comparison simulation of the distributed model training efficiency of the present invention and the prior art is carried out. The results are as follows: Figure 3 shown.

[0113] 2. Analysis of experimental results:

[0114] Reference Figure 2 The MARL solution proposed in this paper was simulated and verified, and compared with the existing technology. The accuracy (%) of the two methods is shown over time (seconds). The vertical axis is accuracy, and the horizontal axis is time (seconds). The figure contains two curves, representing the change in accuracy of the FRLP and MARL-FL methods over time: the FRLP method, represented by a black solid line and circular markers. Its accuracy rises rapidly at the beginning, stabilizes around 2000 seconds, and eventually reaches an accuracy of approximately 70%; the MARL-FL method, represented by a black dashed line and square markers. The accuracy of this method rises rapidly in the initial period, reaches a stable state around 3000 seconds, and maintains a high accuracy, eventually approaching 90%.

[0115] The results show that the MARL-FL method achieves higher accuracy in a shorter time and outperforms the FRLP method. The MARL-FL method can converge stably during training and achieve optimal model accuracy after stabilization.

[0116] Reference Figure 3 , a simulation experiment was conducted to verify the MARL scheme proposed in the present invention, and it was compared with the existing technology, showing the time consumption (seconds) of the FRLP and MARL-FL methods under different training rounds. The vertical axis is time (seconds) and the horizontal axis is training rounds. The figure contains two curves, which represent the changes in the time consumption of the FRLP and MARL-FL methods with the rounds. FRLP method: represented by a black solid line and a circular mark; MARL-FL method: represented by a black dotted line and a square mark. The results show that under the same training rounds, MARL-FL takes less time, which means that the overall training efficiency is improved.

Claims

1. A federated learning distributed model optimization method based on symbiotic multi-agent reinforcement learning, characterized in that: The steps include: (1) Building a federated learning system: Build a centralized federated learning system consisting of a server and N clients, where N ≥ 5 and the nth client is C n ; (2) Each client obtains a local training sample set, and the server obtains a global test sample set: Each client C n Obtain M images containing Z target categories, label the targets in each image, and then form a local training sample set with all images and corresponding labels; At the same time, the server will contain M of Z target categories p The images and their corresponding labels constitute the global test sample set, where Z ≥ 2, M ≥ 3000, and M p ≥10000; (3) The server performs adaptive symbiotic clustering on the client: The server performs adaptive symbiotic clustering on N clients and obtains K clusters each containing I clients, where K ≥ 2 and the kth cluster is V k , I≥1, the i-th client in the k-th cluster is (4) The server builds a symbiotic multi-agent reinforcement learning network and initializes the parameters: Server for each cluster V k Construct a deep Q-network agent. K deep Q-network agents form a symbiotic multi-agent reinforcement learning network. At the same time, the server initializes the global aggregation round to x, the maximum global aggregation round to X, X ≥ 10, and sets x = 0. (5) The server sends the local image classification model to each client: The server will use the local image classification model ω in the xth round x Sent to each client (6) Each client in each cluster iteratively trains the local image classification model: Each client in each cluster Local image classification model through local training sample set Perform iterative training and obtain the trained parameters as Local image classification model (7) Each client in each cluster prunes the model parameters and uploads the parameter differences: Each client in each cluster uses a symbiotic multi-agent reinforcement learning network to train a local image classification model Parameters Perform pruning and calculate the difference in model parameters before and after pruning Upload to the server; (8) The server obtains the global model optimization results: Server based on each client Uploaded Perform weighted average aggregation, update the local image classification model and the symbiotic multi-agent reinforcement learning network, and evaluate the model ω of the xth round through the global test sample set x Accuracy Acc and loss value Loss; then determine whether x=X is true. If so, get the optimized global local image classification model ω X Otherwise, the server sends ω x Go to each client in each cluster, set x=x+1, and execute step (5).

2. The method according to claim 1, characterized in that The server described in step (3) performs adaptive symbiotic clustering on each client, and the implementation steps are as follows: (3a) The server sends a response to each client C n The local training samples are used to extract heterogeneous data features, and the extracted feature vector f n Normalize and get the standardized feature vector f n =[z n ,p n ,m n ,s n ] Among them, z n It is C n The local training samples are flattened into a one-dimensional vector to calculate the data size, p n is the category distribution of local training samples, μ n and σ n It is C n The local training samples are flattened into a one-dimensional vector to calculate the mean and standard deviation, μ(f n ) and σ(f n ) are the mean and standard deviation of all client feature vectors respectively; (3b) The server calculates the Euclidean distance between each two client's normalized feature vectors and selects each normalized feature vector The O smallest components in the Euclidean distance Neighborhood set in, yes oth neighbor, O≥5; (3c) The server initializes the number of iterations to iter, the maximum number of iterations is Iter, Iter ≥ 100, and randomly selects k standardized feature vectors from all standardized feature vectors as the initial cluster centers. The kth cluster center is And let iter=0; (3d) The server calculates the fitness distance d between each two clients n,k , and selects the normalized feature vector with the minimum fitness distance for each client As the current cluster center of the client, all current cluster centers are of The corresponding clients form a cluster V kn ,in: Among them, α is the adjustment factor, ||.|| is the calculation of Euclidean distance; (3e) The server calculates V for each cluster kn All normalized eigenvectors within Average value Will As the current cluster center, at the same time, judge each Neighbor set F n Whether there are neighbors in the cluster that belong to different clusters, remove the neighbors that belong to different clusters F n : Among them, Y kn V kn middle A collection of V kn middle The number of (3f) The server determines whether iter = Iter. If so, set V k =V kn , get the clustering result V k , otherwise, let iter = iter + 1 and execute step (3d).

3. The method according to claim 2, characterized in that The deep Q network agent described in step (4) includes sequentially stacked layers for accepting the state space s k The input layer, one or more fully connected layers or convolutional layers pass through the state space s k , action space A k A hidden layer that calculates the Q value and an output layer that outputs the Q value, where: A k ={a min ,a1,a2,…,a * ,...,a max } Q k (s k ,a * )=h1×s k +h2×a * +p Among them, a min and a max are the minimum and maximum pruning rates respectively, h1 and h2 are the weight matrices of the deep Q network agent network, p is the bias vector of the deep Q network agent network, Q k (s k ,a * ) is the local Q value of the deep Q network agent.

4. The method according to claim 3, characterized in that The local image classification model ω0 described in step (5) includes an input layer for receiving image data stacked in sequence; one or more convolutional layers and pooling layers for feature extraction; one or more fully connected layers for integrating features and making final decisions; and an output layer for outputting classification results.

5. The method according to claim 4, characterized in that Each client of each cluster described in step (6) iteratively trains the local image classification model, and the implementation steps are as follows: (6a) Each client in each cluster Initialize the local training iteration number to e, the maximum iteration number to E, E>1, local image classification model The model parameters are And let e=0; (6b) Local image classification model Perform forward propagation on each local training sample to obtain the predicted label corresponding to the mth local training sample And adopt the cross entropy loss function, through the true label of the mth local training sample and predicted labels calculate The loss value L, and then use the gradient descent method to Update to get the local image classification model for this iteration in: in, The actual label and predicted labels The probability of consistency, is a hyperparameter, For L with respect to the parameter gradient; (6c) Each client in each cluster Determine whether e=E is true. If so, the trained model parameters are Local image classification model Otherwise, let e=e+1, And execute step (6b).

6. The method according to claim 5, characterized in that Each client in each cluster described in step (7) prunes the model parameters and uploads the parameter differences. The implementation steps are: (7a) Each client in each cluster calculates the Q value corresponding to all possible pruning rates and selects the pruning rate that maximizes the Q value. Calculate reward R k , then R k , s k , As experience data lv and stored, where: Among them, Acc represents the accuracy rate, and χ are hyperparameters that control the weight of the reward term, t tr x is the training time of the client in round x, t cs x is the communication time between the client and the server in round x; (7b) Each client in each cluster Calculating model parameters The absolute value of each weight in the , and the absolute value is ranked first The weights of the items are reset to zero to obtain the pruned model parameters Then and The difference Upload to the server, where:

7. The method according to claim 6, characterized in that The server weighted aggregation and global model performance evaluation described in step (8) are implemented as follows: (8a) The server passes To W x Update and get the model ω of the xth round x , the updated model parameters are W x The Bellman equation is then used to update the parameters of the symbiotic multi-agent reinforcement learning network, where W x The update is: Among them, λ is a hyperparameter; (8b) The server inputs the global test sample set to ω x Perform forward propagation and obtain M p Predicted labels Calculate Acc and Loss: in, represents the predicted label of the mth training sample, represents the actual label of the mth training sample, represent and Consistent probability.

8. The method according to claim 7, characterized in that The Bellman equation is used to update the parameters of the symbiotic multi-agent reinforcement learning network described in step (8a). The implementation steps are: (8a1) The server extracts ζ empirical data lv for each cluster, ζ ≥ 5, and calculates the target Q value according to the Bellman equation Among them, u is a hyperparameter, R k lv is the reward corresponding to the largest Q value in the experience data extracted from each cluster, is the maximum Q value among the empirical data extracted from each cluster; (8a2) The server uses the gradient descent method to update the parameters θ of the symbiotic multi-agent reinforcement learning network, setting θ = θ * : in, is a hyperparameter.

Images