Federal learning distributed model optimization method based on symbiotic multi-agent reinforcement learning

Abstract

The invention provides a federated learning distributed model optimization method based on symbiotic multi-agent reinforcement learning. The implementation steps are as follows: constructing a federated learning system; each client obtains a local training sample set, and the server obtains a global test sample set; the server performs adaptive symbiotic clustering on each client; the server constructs a symbiotic multi-agent reinforcement learning network and initializes parameters; the server sends the local image classification model to each client; performing iterative training on a local image classification model by each client of each cluster; pruning model parameters and uploading parameter differences by each client of each cluster; and the server obtains a global model optimization result. According to the method, customized clustering is realized through adaptive symbiotic clustering, and the overall precision of the system is improved; model parameters are pruned through the symbiotic multi-agent reinforcement learning network, so that different cluster clients select optimal pruning rates, parameter calculation is optimized, and training efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of federated learning, and relates to a method for optimizing a federated training model, and specifically to a method for optimizing a federated learning distributed model based on symbiotic multi-agent reinforcement learning. Background Art

[0002] Distributed computing is a technical architecture that uses multiple computing nodes to work together to complete complex tasks. In a distributed system, each computing node can process different tasks in parallel and communicate and coordinate through the network. This architecture not only improves the computing power of the system, but also has higher fault tolerance and supports dynamic expansion of resources.

[0003] With the rapid development of artificial intelligence technology, distributed models have gradually become an important branch of machine learning. Distributed models significantly improve the efficiency and scalability of model training by distributing large amounts of data and computing tasks to multiple nodes for parallel processing. On this basis, distributed model optimization has become an important means to improve system performance. It solves the key bottleneck problem in large-scale training by improving parameter update methods and reducing communication overhead.

[0004] Federated learning is a distributed framework in which each computing node independently trains local data and collaboratively builds a global model through parameter exchange, effectively protecting data privacy. It is suitable for fields with high privacy protection requirements such as finance and medical care. However, as the scale of the system continues to expand, the technical challenges it faces are becoming increasingly prominent.

[0005] First, high communication overhead limits the efficiency of large-scale systems. In existing systems, frequent data exchange between clients and servers is indispensable, especially in the global model update and local training stages. The upload and download of a large number of model parameters leads to a huge communication burden, thus affecting the overall efficiency.

[0006] Secondly, the heterogeneity between clients leads to reduced training efficiency and unstable global model convergence. In a typical federated learning system, the data distribution of different clients is usually not independent and identically distributed (Non-IID). This heterogeneity leads to significant differences in model performance, which in turn affects the convergence of the global model.

[0007] To this end, existing studies have proposed improved methods based on single-agent reinforcement learning (RL-FL). For example, Nie Yuming et al. proposed a distributed model pruning method (FRLP) based on federated reinforcement learning in the 5th issue of Volume 33 of the Journal of Computer Systems and Applications in 2024. This method first abstracts the model pruning process, establishes a Markov decision process, uses the deep Q network (DQN) algorithm to build a reinforced generalization pruning model (RGP), and dynamically adjusts the pruning rate; secondly, a sparse model aggregation method (FedSA) is designed. This method only performs weighted averaging on non-zero weights to avoid invalid calculations on zero weights. FRLP improves the model accuracy of federated learning systems in resource-constrained environments and overcomes the shortcomings of traditional pruning that cannot be adaptively pruned; in terms of optimizing the model structure, it effectively reduces the complexity and improves the generalization performance of the model. However, this method relies on a single agent to make decisions, cannot effectively cope with large-scale heterogeneous systems, and only performs weighted averaging on non-zero weights, which affects the further improvement of model accuracy and cannot achieve a balance between model accuracy and training efficiency in large-scale heterogeneous systems. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and propose a federated learning distributed model optimization method based on symbiotic multi-agent reinforcement learning, aiming to improve the accuracy of the model under large-scale heterogeneous systems while improving the training efficiency.

[0009] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0010] (1) Building a federated learning system:

[0011] Build a centralized federated learning system consisting of a server and N clients, where N ≥ 5 and the nth client is C n ;

[0012] (2) Each client obtains a local training sample set, and the server obtains a global test sample set:

[0013] Each client C n Obtain M images containing Z target categories, and annotate the targets in each image, then form a local training sample set with all images and corresponding labels; at the same time, the server sends M images containing Z target categories to p The images and their corresponding labels constitute the global test sample set, where Z ≥ 2, M ≥ 3000, and M p ≥10000;

[0014] (3) The server performs adaptive symbiotic clustering on the client:

[0015] The server performs adaptive symbiotic clustering on N clients to obtain K clusters each containing I clients, where K ≥ 2, and the kth cluster is V k , I ≥ 1, the i-th client in the k-th cluster is C k i ;

[0016] (4) The server builds a symbiotic multi-agent reinforcement learning network and initializes the parameters:

[0017] The server is for each cluster V k Construct a deep Q network agent. K deep Q network agents form a symbiotic multi-agent reinforcement learning network. At the same time, the server initializes the global aggregation round to x, the maximum global aggregation round to X, X ≥ 10, and sets x = 0;

[0018] (5) The server sends the local image classification model to each client:

[0019] The server will use the local image classification model ω in the xth round x Send to each client

[0020] (6) Each client in each cluster iteratively trains the local image classification model:

[0021] Each client in each cluster Local image classification model through local training sample set Perform iterative training and get the trained parameters as Local image classification model

[0022] (7) Each client in each cluster prunes the model parameters and uploads the parameter differences:

[0023] Each client in each cluster uses a symbiotic multi-agent reinforcement learning network to train a local image classification model. Parameters Prune and calculate the difference between model parameters before and after pruning Upload to the server;

[0024] (8) The server obtains the global model optimization results:

[0025] Server based on each client Uploaded Perform weighted average aggregation, update the local image classification model and the symbiotic multi-agent reinforcement learning network, and evaluate the model ω of the xth round through the global test sample set xThe accuracy Acc and loss value Loss; then determine whether x = X is true, if so, get the optimized global local image classification model ω X , otherwise, the server sends ω x Go to each client in each cluster, set x=x+1, and execute step (5).

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] (1) The server of the present invention performs adaptive symbiotic clustering on all clients. Through customized clustering and dynamic adjustment, it ensures that clients in the same cluster have similar characteristics and requirements, retains the key information required by each client, makes model training and pruning strategies more targeted and effective, and improves the overall accuracy of the distributed model.

[0028] (2) Each client in each cluster of the present invention prunes the parameters of the trained local image classification model through a symbiotic multi-agent reinforcement learning network. Clients in different clusters can select their own optimal pruning rate, reduce the size of parameters during transmission, optimize parameter calculation, reduce communication time and training time, and thus improve model training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart for realizing the present invention;

[0030] Figure 2 This is a simulation comparison diagram of the distributed model accuracy of the present invention and the prior art;

[0031] Figure 3 This is a simulation comparison chart of the distributed model training efficiency of the present invention and the prior art. DETAILED DESCRIPTION

[0032] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0033] Reference Figure 1 , the present invention comprises the following steps:

[0034] Step 1) Build a federated learning system:

[0035] Build a centralized federated learning system consisting of a server and N clients, where N ≥ 5 and the nth client is C n ;

[0036] The centralized federated learning system is a federated learning structure based on a central server, which relies on the central server to coordinate the training process of multiple clients. The central server sends a global model to all clients in each iteration. After each client trains the model on local data, it sends the model update parameters back to the server. In this embodiment, N=15.

[0037] Step 2) Each client obtains a local training sample set, and the server obtains a global test sample set:

[0038] Each client C n Obtain M images containing Z target categories, and annotate the targets in each image, then form a local training sample set with all images and corresponding labels; at the same time, the server sends M images containing Z target categories to p The images and their corresponding labels constitute the global test sample set, where Z ≥ 2, M ≥ 3000, and M p ≥10000;

[0039] The local training sample set used in this embodiment is the training set in the CIFAR-10 universal object color image dataset. The number of target categories corresponding to this dataset is 10, namely: airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships and trucks, and a total of 50,000 images are included. The pixel size of each image is 32×32, and there are 5,000 images for each target category. In this embodiment, Z=5 and M=25,000 are set to simulate the heterogeneity of the client; the global test sample set used in this embodiment is the test set of the CIFAR-10 universal object color image dataset. The number of target categories corresponding to this dataset is 10, and the pixel size of each image is 32×32. There are 1,000 images for each target category, and a total of M p = 10000 images.

[0040] Step 3) The server performs adaptive symbiotic clustering on each client:

[0041] The server performs adaptive symbiotic clustering on N clients to obtain K clusters each containing I clients, where K ≥ 2, and the kth cluster is V k , I ≥ 1, the i-th client in the k-th cluster is C k i ;

[0042] In this embodiment, K=5;

[0043] Adaptive symbiotic clustering is a technology that achieves better clustering effects by identifying the characteristics of different clients and adaptively adjusting them. In heterogeneous client scenarios, customized clustering and dynamic adjustment are used to ensure that clients in the same cluster have similar characteristics and requirements, retaining the key information required by each client, making model training and pruning strategies more targeted and effective, and improving the overall accuracy of distributed models.

[0044] The server described in this step performs adaptive symbiotic clustering on each client, and the implementation steps are as follows:

[0045] Step 3a) The server sends a n The local training samples are used to extract heterogeneous data features, and the extracted feature vector f n Standardize and get the standardized feature vector

[0046] f n =[z n ,p n ,μ n ,σ n ]

[0047]

[0048] Among them, z n It is C n The local training samples are flattened into a one-dimensional vector to calculate the data size, p i is the category distribution of local training samples, μ n and σ n It is C n The local training samples are flattened into a one-dimensional vector to calculate the mean and standard deviation, μ(f N ) and σ(f N ) are the mean and standard deviation of all client feature vectors respectively;

[0049] Step 3b) The server calculates the Euclidean distance between each two client standardized feature vectors and selects each standardized feature vector The O smallest components in the Euclidean distance of Neighborhood set in, yes The oth neighbor, O≥5;

[0050] Step 3c) The server initializes the number of iterations to iter, the maximum number of iterations is Iter, Iter ≥ 100, and randomly selects k standardized feature vectors from all standardized feature vectors as the initial clustering centers. The kth clustering center is And let iter=0;

[0051] In this embodiment, iter=100.

[0052] Step 3d) The server calculates the fitness distance d between every two clients n,k , and select the standardized feature vector with the minimum fitness distance for each client As the current cluster center of the client, all current cluster centers are of The corresponding clients form a cluster V kn ,in:

[0053]

[0054] Among them, α is the adjustment factor, ||.|| is the calculation of Euclidean distance;

[0055] In this embodiment, α=0.5;

[0056] Step 3e) The server calculates each cluster V kn All normalized eigenvectors in The average Will As the current cluster center, at the same time, judge each Neighbor set F n If there are neighbors in the cluster that belong to different clusters, remove the neighbors that belong to different clusters. n :

[0057]

[0058] Among them, Y kn V kn middle The collection of N Yk V kn middle The number of

[0059] Step 3f) The server determines whether iter = Iter. If so, set V k =V kn , and get the clustering result V k , otherwise, let iter=iter+1 and execute step 3d).

[0060] Step 4) The server builds a symbiotic multi-agent reinforcement learning network and initializes the parameters:

[0061] The server is V for each cluster k Construct a deep Q network agent. K deep Q network agents form a symbiotic multi-agent reinforcement learning network. At the same time, the server initializes the global aggregation round to x, the maximum global aggregation round to X, X ≥ 10, and sets x = 0;

[0062] The symbiotic multi-agent reinforcement learning network is based on the multi-agent value decomposition network. The multi-agent value decomposition network consists of multiple deep Q network agents. Its core idea is to decompose the global reward signal into the local value function of each agent, thereby simplifying the learning process of the multi-agent system. By linearly combining the local value functions of multiple agents, the overall value function is estimated, so that each agent can learn and make decisions collaboratively, improving the efficiency of multi-agent learning. On this basis, the symbiotic multi-agent reinforcement learning network determines K as the number of clusters, and sets the client coordinated by each deep Q network agent as the client of the cluster.

[0063] In this embodiment, X=20;

[0064] Step 4) The deep Q network agent includes sequentially stacked layers for accepting the state space s k The input layer, one or more fully connected layers or convolutional layers through the state space s k , action space A k A hidden layer for calculating Q values ​​and an output layer for outputting Q values, where:

[0065]

[0066] A k ={a min ,a1,a2,…,a * ,...,a max}

[0067] Q k (s k ,a * )=h1×s k +h2×a * +p

[0068] Among them, a min and a max are the minimum and maximum pruning rates, respectively. h1 and h2 are the weight matrices of the deep Q-network agent network. p is the bias vector of the deep Q-network agent network. k (s k ,a * ) is the local Q-value of the deep Q-network agent.

[0069] In this embodiment, the structure of the DQN network is as follows: the input layer is a fully connected layer with 128 neurons, which converts the input state into 128 features. It has two hidden layers, each containing 128 neurons, and uses the ReLU activation function. The output layer passes through a fully connected layer to map the features to the pruning rate. The minimum pruning rate a minand the maximum pruning rate a max Set to 0.01% and 50%, and χ are 0.1 and 0.4, respectively;

[0070] Step 5) The server sends the local image classification model to each client:

[0071] The server will use the local image classification model ω in the xth round x Sent to each client

[0072] The local image classification model ω0 described in step 5) comprises an input layer for receiving image data, which is stacked in sequence; one or more convolutional layers and pooling layers for feature extraction; one or more fully connected layers for integrating features and making final decisions; and an output layer for outputting classification results;

[0073] In this embodiment, the local image classification model is a residual neural network ResNet-18. The architecture of ResNet-18 includes an initial 7x7 convolutional layer, 64 channels, and a stride of 2, followed by a 3x3 maximum pooling layer with a stride of 2. Its core consists of 4 groups of residual layers. The structure of the residual block includes two consecutive 3x3 convolutional layers, each of which is followed by a batch normalization layer and a ReLU activation function. The input is directly added to the output of the second convolutional layer through a jump connection; among the 4 groups of residual layers, the first group has 2 residual blocks, each with 64 filters; the second group has 2 residual blocks, each with 128 filters; the third group has 2 residual blocks, each with 256 filters; the fourth group has 2 residual blocks, each with 512 filters. Each residual block contains two 3x3 convolutional layers, which improve stability through batch normalization and ReLU activation. The global average pooling layer at the end of the network converts the feature map into a 512-dimensional vector, and finally outputs the classification result through a fully connected layer.

[0074] Step 6) Each client in each cluster iteratively trains the local image classification model:

[0075] Each client in each cluster Local image classification model through local training sample set Perform iterative training and get the trained parameters as Local image classification model

[0076] Step 6a) Each client in each cluster Initialize the number of local training iterations to e, the maximum number of iterations to E, E>1, local image classification model The model parameters are And let e = 0;

[0077] In this embodiment, E=5;

[0078] Step 6b) Local Image Classification Model Perform forward propagation on each local training sample to obtain the predicted label corresponding to the mth local training sample And adopt the cross entropy loss function, through the true label of the mth local training sample and predicted labels calculate The loss value L is then used to calculate the gradient descent method through L. Update to get the local image classification model for this iteration in:

[0079]

[0080] in, The actual label and predicted labels The probability of consistency, is a hyperparameter, For L with respect to the parameter The gradient of

[0081] In this embodiment,

[0082] Step 6c) Each client in each cluster Determine whether e=E is true. If so, the trained model parameters are Local image classification model Otherwise, let e=e+1, And execute step 6b).

[0083] Step 7) Each client in each cluster prunes the model parameters and uploads the parameter differences:

[0084] Each client in each cluster uses a symbiotic multi-agent reinforcement learning network to train a local image classification model. Parameters Prune and calculate the difference between model parameters before and after pruning Upload to the server;

[0085] Step 7a) Each client in each cluster calculates the Q values ​​corresponding to all possible pruning rates and selects the pruning rate that maximizes the Q value Calculate the reward R k , then R k , s k , As experience data lv and stored, where:

[0086]

[0087] Among them, Acc represents the accuracy, and χ are hyperparameters that control the weight of the reward term, t tr x is the training time of the client in round x, t cs x is the communication time between the client and the server in round x;

[0088] Step 7b) Each client in each cluster Calculating model parameters The absolute value of each weight in the The weights of the items are reset to zero, and the pruned model parameters are obtained. Then and The Difference Upload to the server, where:

[0089]

[0090] The present invention uses symbiotic multi-agent reinforcement learning to enable clients in different clusters to select their own optimal pruning rates, reduce the size of parameters during transmission, optimize parameter calculation, reduce communication time and training time, and thus improve model training efficiency.

[0091] Step 8) The server obtains the global model optimization results:

[0092] Server based on each client Uploaded Perform weighted average aggregation, update the local image classification model and the symbiotic multi-agent reinforcement learning network, and evaluate the model ω of the xth round through the global test sample set x The accuracy Acc and loss value Loss; then determine whether x = X is true, if so, get the optimized global local image classification model ω X , otherwise, the server sends ω x Go to each client of each cluster, set x=x+1, and execute step 5);

[0093] The server weighted aggregation, updating of models and networks, and global model performance evaluation are implemented in the following steps:

[0094] Step 8a) The server passes To W x Update and get the model ω of the xth round x , the updated model parameters are W xThe Bellman equation is then used to update the parameters of symbiotic multi-agent reinforcement learning, where W x The update is:

[0095]

[0096] Among them, λ is a hyperparameter;

[0097] Use the Bellman equation to update the parameters of symbiotic multi-agent reinforcement learning. The implementation steps are:

[0098] Step 8a1) The server extracts ζ empirical data lv for each cluster, ζ ≥ 5, and calculates the target Q value according to the Bellman equation

[0099]

[0100] Among them, u is a hyperparameter, R k lv is the reward corresponding to the largest Q value in the experience data extracted from each cluster, is the maximum Q value among the empirical data extracted from each cluster;

[0101] Step 8a2) The server uses the gradient descent method to update the parameter θ of the symbiotic multi-agent reinforcement learning, setting θ = θ * :

[0102]

[0103] in, is a hyperparameter;

[0104] In this embodiment, ζ=5, u is 0.91, is 0.007;

[0105] Step 8b) The server inputs the global test sample set to ω x Perform forward propagation and obtain M p Prediction labels Calculate Acc and Loss:

[0106]

[0107] in, represents the predicted label of the mth training sample, represents the actual label of the mth training sample, represent and Consistent probability.

[0108] The following is a further description of the technical effects of the present invention in conjunction with the simulation results:

[0109] 1. Experimental conditions and contents:

[0110] The model accuracy and communication time of the present invention are simulated using PyCharm 2023.2:

[0111] Experiment 1: Comparison simulation of the accuracy of the distributed model of the present invention and the prior art is carried out. The results are as follows: Figure 2 As shown;

[0112] Experiment 2 compares the distributed model training efficiency of the present invention and the prior art. The results are as follows: Figure 3 shown.

[0113] 2. Experimental results analysis:

[0114] Reference Figure 2 , the MARL scheme proposed in the present invention is verified by simulation experiments, and compared with the prior art, showing the accuracy (%) performance of the two methods in time (seconds), the vertical axis is the accuracy, the horizontal axis is the time (seconds), and the figure contains two curves, representing the change of the accuracy of the FRLP and MARL-FL methods over time: FRLP method: represented by a black solid line and a circular mark. Its accuracy rises rapidly at the beginning, stabilizes at about 2000 seconds, and finally reaches an accuracy of about 70%; MARL-FL method: represented by a black dotted line and a square mark. The accuracy of this method rises rapidly in the initial time period, reaches a stable state at about 3000 seconds, and maintains a high accuracy, and finally approaches 90%.

[0115] The results show that the MARL-FL method achieves higher accuracy in a shorter time and outperforms the FRLP method. The MARL-FL method can converge stably during the training process and achieve the optimal model accuracy after stabilization.

[0116] Reference Figure 3 , a simulation experiment was carried out to verify the MARL scheme proposed in the present invention, and it was compared with the prior art, showing the time consumption (seconds) of the FRLP and MARL-FL methods under different training rounds. The vertical axis is time (seconds), and the horizontal axis is training rounds. The figure contains two curves, which represent the changes in the time consumption of the FRLP and MARL-FL methods with the rounds. FRLP method: represented by a black solid line and a circular mark; MARL-FL method: represented by a black dotted line and a square mark. The results show that under the same training rounds, MARL-FL takes less time, which means that the overall training efficiency is improved.

Claims

1. A federated learning distributed model optimization method based on symbiotic multi-agent reinforcement learning, characterized in that: The steps include: (1) Building a federated learning system: Build a centralized federated learning system consisting of a server and N clients, where N ≥ 5 and the nth client is C n ; (2) Each client obtains a local training sample set, and the server obtains a global test sample set: Each client C n Obtain M images containing Z target categories, label the targets in each image, and then combine all images and corresponding labels into a local training sample set; At the same time, the server will contain M of Z target categories. p The images and their corresponding labels constitute the global test sample set, where Z ≥ 2, M ≥ 3000, and M p ≥10000; (3) The server performs adaptive symbiotic clustering on the client: The server performs adaptive symbiotic clustering on N clients to obtain K clusters each containing I clients, where K ≥ 2, and the kth cluster is V k , I ≥ 1, the i-th client in the k-th cluster is (4) The server builds a symbiotic multi-agent reinforcement learning network and initializes the parameters: The server is V for each cluster k Construct a deep Q network agent. K deep Q network agents form a symbiotic multi-agent reinforcement learning network. At the same time, the server initializes the global aggregation round to x, the maximum global aggregation round to X, X ≥ 10, and sets x = 0; (5) The server sends the local image classification model to each client: The server will use the local image classification model ω in the xth round x Send to each client (6) Each client in each cluster iteratively trains the local image classification model: Each client in each cluster Local image classification model through local training sample set Perform iterative training and get the trained parameters as Local image classification model (7) Each client in each cluster prunes the model parameters and uploads the parameter differences: Each client in each cluster uses a symbiotic multi-agent reinforcement learning network to train a local image classification model. Parameters Prune and calculate the difference between model parameters before and after pruning Upload to the server; (8) The server obtains the global model optimization results: Server based on each client Uploaded Perform weighted average aggregation, update the local image classification model and the symbiotic multi-agent reinforcement learning network, and evaluate the model ω of the xth round through the global test sample set x The accuracy Acc and loss value Loss; then determine whether x = X is true, if so, get the optimized global local image classification model ω X , otherwise, the server sends ω x Go to each client in each cluster, set x=x+1, and execute step (5).

2. The method according to claim 1, characterized in that The server described in step (3) performs adaptive symbiotic clustering on each client, and the implementation steps are as follows: (3a) The server sends a request to each client C n The local training samples are used to extract heterogeneous data features, and the extracted feature vector f n Standardize and get the standardized feature vector f n =[z n ,p n ,m n ,s n ] Among them, z n It is C n The local training samples are flattened into a one-dimensional vector to calculate the data size, p i is the category distribution of local training samples, μ n and σ n It is C n The local training samples are flattened into a one-dimensional vector to calculate the mean and standard deviation, μ(f N ) and σ(f N ) are the mean and standard deviation of all client feature vectors respectively; (3b) The server calculates the Euclidean distance between each pair of client standardized feature vectors and selects each standardized feature vector The O smallest components of the Euclidean distance Neighborhood set in, yes The oth neighbor, O≥5; (3c) The server initializes the number of iterations to iter, the maximum number of iterations is Iter, Iter ≥ 100, and randomly selects k standardized feature vectors from all standardized feature vectors as the initial clustering centers. The kth clustering center is And let iter=0; (3d) The server calculates the fitness distance d between each two clients n,k , and select the standardized feature vector with the minimum fitness distance for each client As the current cluster center of the client, all current cluster centers are of The corresponding clients form a cluster V kn ,in: Among them, α is the adjustment factor, ||.|| is the calculation of Euclidean distance; (3e) The server calculates V for each cluster kn All normalized eigenvectors in The average Will As the current cluster center, at the same time, judge each Neighbor set F n If there are neighbors in the cluster that belong to different clusters, remove the neighbors that belong to different clusters. n : Among them, Y kn V kn middle A collection of V kn middle The number of (3f) The server determines whether iter = Iter. If so, set V k =V kn , and get the clustering result V k , otherwise, let iter=iter+1 and execute step (3d).

3. The method according to claim 2, characterized in that The deep Q network agent described in step (4) includes a stack of layers for accepting the state space s k The input layer, one or more fully connected layers or convolutional layers through the state space s k , action space A k A hidden layer for calculating Q values ​​and an output layer for outputting Q values, where: A k ={a min ,a1,a2,…,a * ,...,a max } Q k (s k ,a * )=h1×s k +h2×a * +p Among them, a min and a max are the minimum and maximum pruning rates, respectively. h1 and h2 are the weight matrices of the deep Q-network agent network. p is the bias vector of the deep Q-network agent network. k (s k ,a * ) is the local Q-value of the deep Q-network agent.

4. The method according to claim 3, characterized in that The local image classification model ω0 described in step (5) includes an input layer for receiving image data stacked in sequence; one or more convolutional layers and pooling layers for feature extraction; one or more fully connected layers for integrating features and making final decisions; and an output layer for outputting classification results.

5. The method according to claim 4, characterized in that Each client of each cluster described in step (6) iteratively trains the local image classification model, and the implementation steps are as follows: (6a) Each client in each cluster Initialize the number of local training iterations to e, the maximum number of iterations to E, E>1, local image classification model The model parameters are And let e = 0; (6b) Local Image Classification Model Perform forward propagation on each local training sample to obtain the predicted label corresponding to the mth local training sample And adopt the cross entropy loss function, through the true label of the mth local training sample and predicted labels calculate The loss value L is then used to calculate the gradient descent method through L. Update to get the local image classification model for this iteration in: in, The actual label and predicted labels The probability of consistency, is a hyperparameter, ▽L is L with respect to the parameter The gradient of (6c) Each client in each cluster Determine whether e=E is true. If so, the trained model parameters are Local image classification model Otherwise, let e=e+1, And execute step (6b).

6. The method according to claim 5, characterized in that Each client of each cluster described in step (7) prunes the model parameters and uploads the parameter differences. The implementation steps are: (7a) Each client in each cluster calculates the Q values ​​corresponding to all possible pruning rates and selects the pruning rate that maximizes the Q value. Calculate the reward R k , then R k , s k , As experience data lv and stored, where: Among them, Acc represents the accuracy, and χ are hyperparameters that control the weight of the reward term, t tr x is the training time of the client in round x, t cs x is the communication time between the client and the server in round x; (7b) Each client in each cluster Calculating model parameters The absolute value of each weight in the The weights of the items are reset to zero, and the pruned model parameters are obtained. Then and The Difference Upload to the server, where:

7. The method according to claim 6, characterized in that The server weighted aggregation and global model performance evaluation described in step (8) is implemented as follows: (8a) The server passes To W x Update and get the model ω of the xth round x , the updated model parameters are W x The Bellman equation is then used to update the parameters of the symbiotic multi-agent reinforcement learning network, where W x The update is: Among them, λ is a hyperparameter; (8b) The server inputs the global test sample set to ω x Perform forward propagation and obtain M p Prediction labels Calculate Acc and Loss: in, represents the predicted label of the mth training sample, represents the actual label of the mth training sample, represent and Consistent probability.

8. The method according to claim 7, characterized in that The Bellman equation described in step (8a) is used to update the parameters of the symbiotic multi-agent reinforcement learning network. The implementation steps are: (8a1) The server extracts ζ empirical data lv for each cluster, ζ ≥ 5, and calculates the target Q value according to the Bellman equation Among them, u is a hyperparameter, R k lv is the reward corresponding to the largest Q value in the experience data extracted from each cluster, is the maximum Q value among the empirical data extracted from each cluster; (8a2) The server uses the gradient descent method to update the parameters θ of the symbiotic multi-agent reinforcement learning network, setting θ = θ * : in, is a hyperparameter.

Images