A federated learning method and system based on deep reinforcement learning
By using deep reinforcement learning to assist federated learning in selecting devices, the communication and energy consumption problems caused by device heterogeneity and channel instability in the Industrial Internet of Things are solved, achieving efficient federated learning, reducing costs and improving performance.
Patent Information
- Application Number
- CN202310223773.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-09
- Publication Date
- 2026-02-17
- Estimated Expiration
- 2043-03-09
AI Technical Summary
Traditional centralized machine learning frameworks face challenges in the Industrial Internet of Things (IIoT) such as data silos, device heterogeneity, high communication costs, and excessive energy consumption. Existing federated learning methods have failed to effectively address the communication and energy consumption issues caused by device heterogeneity and channel instability.
We employ a federated learning approach based on deep reinforcement learning. By modeling and analyzing latency and energy consumption, we construct a Markov decision process, use the DDQN reinforcement learning algorithm to select suitable edge devices for training, adjust the weight coefficients to balance communication latency and energy consumption, and build a reinforcement learning agent network that runs in the cloud to select the optimal subset of devices.
It reduces communication and energy costs in the Industrial Internet of Things (IIoT), improves the efficiency and performance of federated learning, has high accuracy and robustness, and avoids additional computational burden on edge devices.
Smart Images
Figure CN116486192B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of industrial Internet of Things, in particular, to a federated learning method and system based on deep reinforcement learning, which is used to reduce the communication and energy consumption cost in industrial Internet of Things. BACKGROUND
[0002] Industrial Internet of Things introduces a large number of artificial intelligence technologies, aiming to realize data-driven machine learning solutions in heterogeneous and large-scale networks. However, the traditional cloud computing-based centralized machine learning framework focuses on the processing of data by the central cloud server, which usually requires widely distributed Internet of Things devices, wearable devices and other mobile devices to unload their training data to the remote cloud server for centralized learning. The centralized learning scheme faces several serious challenges:
[0003] (1) In actual production process, data is usually scattered on multiple mobile devices, and it is difficult for the cloud server to collect these scattered data;
[0004] (2) For edge devices with limited computing and communication resources, it is impractical to transfer a large amount of local data to the remote cloud center through wireless network;
[0005] (3) The training data on the edge device may carry private information.
[0006] With the rapid development of digital technology and the increasing awareness of self-protection, people pay more and more attention to personal privacy protection. This makes the centralized training mode with the central cloud server as the core greatly limited in many application scenarios (such as clinical diagnosis). This also brings new challenges to industrial Internet of Things: data island problem. In order to protect the data privacy of mobile devices, break the data island, Google first proposed the concept of federated learning in 2016. Federated learning is a distributed machine learning framework that protects user privacy and completes joint modeling without sharing data. The main idea is to train local models on edge devices, perform federated averaging algorithm on the central cloud server to complete model aggregation, and distribute the aggregated model to all participating federated learning edge devices for the next round of local training. The edge device and the cloud server repeat the above process until the global model reaches the target accuracy. However, the traditional federated learning faces the following challenges:
[0007] (1) Device heterogeneity: different edge devices have different computing power, storage space, battery capacity, etc.
[0008] (2) High communication cost: edge devices send and receive a large amount of model parameters to the cloud server through wireless network
[0009] (3) Network instability: the communication resources of edge devices are limited, and the data transmission rate is affected by the environment, which fluctuates greatly, which may increase the communication cost and transmission power consumption of the federated learning process.
[0010] Especially in the field of IIoT, the computing and communication resources of devices are limited, and many IIoT devices are powered by batteries, so the communication and energy consumption costs of devices need to be considered. In addition, due to the unstable communication quality of IIoT devices, devices may need to spend a lot of time on the transmission of local models, reducing the efficiency of federated learning and increasing the transmission energy consumption of devices. There are currently three main methods to reduce the communication and energy consumption costs of federated learning: 1) Random selection strategy. Federated learning randomly selects a subset of devices for local model updates in each communication round, but the random selection strategy is blind and may select nodes with good channel quality and low energy consumption, but also may select nodes with poor channel quality or low energy utilization efficiency, which will increase the communication and energy consumption costs of the federated learning process. 2) Model compression. Based on coding technology, the local model is compressed and encoded, which can effectively reduce the communication cost of uplink, but compression encoding will increase the additional computational overhead, and the random selection strategy will select high-energy consumption devices. 3) Adaptive control of device CPU frequency. Through intelligent algorithms to automatically control the CPU frequency of each device, balance the delay and energy consumption in the training process. This algorithm control authority is too high, introduces an unsafe factor for IIoT devices, affecting the user experience of terminal devices.
[0011] WANG et al. in their published paper "Optimizing Federated Learning on Non-IID Data with Reinforcement Learning" (IEEE Conference on Computer Communications. 2020) proposed an experience-driven control framework FAVOR, which can intelligently select client devices to participate in each round of federated learning to offset the bias introduced by non-independent and identically distributed data and accelerate convergence. FAVOR pays more attention to the convergence speed of the global model, and does not consider the heterogeneity of terminal devices and the long-tail effect caused by the instability of channel state.
[0012] In their published paper "Homogeneous Learning: Self-Attention Decentralized Deep Learning" (IEEE Access, 2022), SUN et al. proposed a self-attention distributed learning model called Homogeneous Learning (HL), which performs local model training on selected nodes and uses reinforcement learning algorithms to decide the next selected node at the end of each training round and sends the trained model to the selected node. HL can effectively reduce the impact of Non-IID data on the accuracy of the global model. However, since HL is a decentralized distributed deep learning framework, the training of the reinforcement learning agent network and the training of the local model both need to be performed on terminal devices, which puts high requirements on the computing power of terminal devices and adds additional burden to energy-constrained mobile terminals.
[0013] In their published paper "Global Aggregation Node Selection Scheme in Federated Learning for Vehicular Ad Hoc Networks." (IEEE International Conference on Omni-layer Intelligent Systems, 2022), TRABELSI et al. used a variable global aggregation node to replace the fixed global aggregator and proposed a client selection technique based on attributes such as minimum residual energy, minimum available memory, and minimum data records. This algorithm only selects nodes that meet certain criteria to participate in federated learning, which helps to improve the overall learning process and eliminate unnecessary delays. However, the authors did not consider the impact of network quality instability on the communication and energy consumption costs of the federated learning process. SUMMARY
[0014] To address the shortcomings of existing solutions, the present invention provides a scheme for federated learning node selection assisted by deep reinforcement learning to reduce communication and energy consumption costs in industrial Internet of Things. The technical solutions of the present invention are as follows:
[0015] A federated learning method based on deep reinforcement learning includes the following steps:
[0016] Modeling and analyzing the time delay of the federated learning system;
[0017] Modeling and analyzing the energy consumption of the federated learning system;
[0018] Determine the optimization goal;
[0019] Constructing an image classification network and a reinforcement learning agent network;
[0020] Convert the node selection problem into a Markov decision process;
[0021] Training the reinforcement learning agent network;
[0022] Using a Q network for federated learning;
[0023] The reinforcement learning agent network adopts a DDQN reinforcement learning algorithm; the reinforcement learning agent network includes the Q network and a Target Q network, and the Q network and the Target Q network adopt the same network structure.
[0024] Further, the network structure of the Q network and the Target Q network is composed of two linear networks; the input dimension of the first linear network is the state dimension of the federated learning system, the output dimension is set to 128, and the output of the first linear network is connected to a ReLU activation function; the input dimension of the second linear network is 128, and the output dimension is equal to the number of edge IIoT devices.
[0025] Further, the time delay of the federated learning system is modeled and analyzed, specifically including:
[0026] The time consumed by the computing device i for model training in the kth communication round:
[0027]
[0028] Wherein, i and k are positive integers, d i is the number of samples of device i, c i is the number of cpu cycles required to train a sample, f i is the working frequency, and τ is the number of local iterations.
[0029] The transmission time of the model is:
[0030]
[0031] Wherein, represents the absolute value of the local model network parameter, and the unit is Byte, represents the data transmission rate of device i in the kth communication round.
[0032] Combining the time consumed by the model training and the transmission time of the model, the local training delay of device i in the kth communication round is:
[0033]
[0034] According to the local training delay, a training delay T of the kth communication process is obtained k Training:
[0035]
[0036] Further, the energy consumption of the federated learning system is modeled and analyzed, specifically including:
[0037] The energy consumption of the computing device i performing local training in the kth communication round;
[0038] The communication power consumption of the computing device i performing model transmission;
[0039] The idle energy consumption of the device i is obtained by combining the energy consumption and the communication power consumption;
[0040] The total energy consumption of the computing device is calculated according to the idle energy consumption.
[0041] Further, the optimization target is represented as:
[0042] Wherein, f min ≤f i ≤f max , λ is a non-negative hyperparameter for connecting the delay and the energy consumption index, represents the decision variable, represents that the device i is selected to participate in federated learning in the kth communication round, represents that the device i is not selected to participate in federated learning in the kth communication round, represents the communication power consumption of the device i, f min is the minimum value of the device operating frequency, f max is the maximum value of the device operating frequency, B is the server bandwidth, and N is the total amount of devices.
[0043] Further, the training reinforcement learning agent network specifically includes:
[0044] The time series difference target is calculated;
[0045] The loss function is defined according to the time series difference target;
[0046] The network parameters of the Q network are updated by minimizing the loss function through gradient descent method.
[0047] Further, the image classification network is a two-layer MLP network; the image classification network includes an image input layer, a first linear network, a first activation function layer, a second linear network, a second activation function layer, and a fully connected network.
[0048] Further, the Markov decision process comprises a system state, an action space, a policy, a reward function and a neighboring state; wherein the policy represents a mapping from the state space to the action space; and the reward function is set as a weighted sum of the minimized time delay and energy consumption.
[0049] Further, the federated learning using the Q network specifically comprises:
[0050] The cloud server sends global model parameters to all edge devices participating in federated learning and collects state information of the federated learning system;
[0051] The state information is input into the Q network, and the Q network outputs a Q value, wherein the Q value represents the value of each action, and the top-k Q values are used to determine the best device subset in the current state;
[0052] The best device subset uses local data to train the image classification network to update the local model, and then uploads the local model to the cloud server;
[0053] The cloud server executes a model aggregation algorithm to update the global model;
[0054] The above process is continuously executed until the global model reaches the target accuracy.
[0055] The application also provides a federated learning system based on deep reinforcement learning, which is used to execute the federated learning method; the system comprises a cloud server, a plurality of edge devices and a wireless network environment;
[0056] The cloud server is used to store and update global model parameters, receive and send messages, and run a reinforcement learning agent network and execute a model aggregation algorithm; the edge device is used to store and process local data, calculate node scores, perform local training, receive and send messages; and the wireless network environment is used to connect the cloud server and the edge device.
[0057] Compared with the prior art, the application has the following beneficial effects:
[0058] The application uses a reinforcement learning agent to assist the federated learning system in selecting appropriate devices for training, and adjusts the weight coefficient to balance the relationship between communication time delay and energy consumption, thereby reducing the cost of communication and energy consumption, to solve the problems of device heterogeneity and channel instability in industrial Internet of Things, and improve the efficiency and performance of federated learning;
[0059] The reinforcement learning agent constructed in the application is continuously improved in node selection strategy through a large amount of interaction data, so that the method has high accuracy and high robustness;
[0060] The reinforcement learning intelligent agent constructed in the application runs on a cloud server, and does not introduce additional computing burden to an edge device. BRIEF DESCRIPTION OF DRAWINGS
[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.
[0062] Figure 1 The flow chart of the federated learning method based on deep reinforcement learning of the present application;
[0063] Figure 2 The structure schematic diagram of the image classifier of the present application;
[0064] Figure 3 The structure schematic diagram of the Q network in the present application;
[0065] Figure 4 The curve diagram of the change of the reward obtained by the reinforcement learning intelligent agent in an embodiment of the present application with the iteration rounds in the training stage;
[0066] Figure 5 The curve diagram of the change of the loss function of the reinforcement learning intelligent agent network in an embodiment of the present application with the iteration rounds in the training stage;
[0067] Figure 6 The time delay comparison curve diagram of the simulation experiment in an embodiment of the present application;
[0068] Figure 7 The energy consumption comparison curve diagram of the simulation experiment in an embodiment of the present application;
[0069] Figure 8 The training cost comparison curve diagram of the simulation experiment in an embodiment of the present application. DETAILED DESCRIPTION
[0070] The technical solutions of the present application will be described clearly and completely below in combination with the embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.
[0071] The present application will be further described below in combination with the drawings, but the scope of the present application is not limited in any way by the drawings.
[0072] Please refer to Figure 1A federated learning method based on deep reinforcement learning, specifically comprising the following steps:
[0073] Step 1. Model analysis of the latency of the federated learning system.
[0074] In one communication round, the training latency of the device is composed of the computation latency during local model training and the communication latency during model transmission. Assuming that device i has d i samples, the number of cpu cycles required to train one sample is c i , the working frequency is f i , and τ local iterations are performed, then the time consumed by device i to perform model training in the kth communication round is:
[0075]
[0076] The local model network parameters trained by device i using local data are uploaded to the parameter server Es, and the transmission time of the model is calculated using the following formula:
[0077]
[0078] where, represents the size of the local model network parameters, in bytes, represents the data transmission rate of device i in the kth communication round, The calculation of is obtained by the following Shannon formula.
[0079]
[0080] According to the above content, the data transmission rate of device i is related to the bandwidth B i,k allocated to device i, the transmission power p i,k of device i, the channel gain g i,k , and the noise power N0. Changes in environmental conditions will affect the data transmission rate and exacerbate the uncertainty of communication latency.
[0081] Combining the above content, the local training latency of device i in the kth communication round is:
[0082]
[0083] In the federated synchronous learning algorithm, the local training time of each communication round is determined by the slowest device, so the training latency of the kth communication process is calculated by the following formula.
[0084]
[0085] Step 2. Model analysis of the energy consumption of the federated learning system.
[0086] The energy consumption of device i performing local training at the kth communication round is:
[0087]
[0088] σ represents the effective capacitance coefficient, which is related to the properties of the chip itself.
[0089] When transmitting the model, device i uses the power value p i,k to transmit the model, then the communication power consumption of device i is:
[0090]
[0091] The device that first completes local model training and model transmission needs to wait for other unfinished devices, and the device with fast execution speed has idle waiting time. The energy consumed in the idle waiting time is called idle energy consumption. Then the idle energy consumption of device i is equal to the idle waiting time multiplied by the unit energy consumption in the idle state, and the calculation formula is as follows:
[0092]
[0093] represents the energy consumed by device i per unit time when waiting. In summary, the energy consumed by device i at the kth communication round is
[0094]
[0095] Then the total energy consumed by all devices at the kth communication round is
[0096]
[0097] Step 3. Determine the optimization target.
[0098] The goal is to select a group of edge devices in each communication round to minimize the latency and energy consumption in the dynamic network scenario considering the heterogeneity of IIoT (Industrial Internet of Things) devices. To balance the latency and energy consumption indicators, a non-negative hyperparameter λ is defined to connect the latency and energy consumption indicators. The optimization target is described using the following formula:
[0099]
[0100] f min ≤f i ≤f max (12)
[0101]
[0102]
[0103] wherein, denotes the decision variable, denotes that device i is selected to participate in federated learning in the kth communication round, denotes that device i is not selected, f min is the minimum value of the device working frequency, f max is the maximum value of the device working frequency, B is the server bandwidth, and N is the total number of devices. Equation (1-12) represents the constraint on the device working frequency, equation (1-13) represents that the total bandwidth of the selected devices is not greater than the server bandwidth B, and equation (1-14) represents that at least one device is selected to participate in federated learning in one communication round, and the maximum number does not exceed the total number of devices N. λ shows the preference for the optimization objective. If λ takes a relatively large value, it means that the edge server focuses more on energy consumption; if λ takes a relatively small value, then the server focuses more on the delay index.
[0104] Step 4. Constructing an image classification network and a reinforcement learning agent network.
[0105] Referring to the accompanying drawings Figure 2 , the specific structure of the image classification network is further described as follows:
[0106] A two-layer MLP (Multi-Layer Perceptron) network is constructed as the local image classification network of the IIoT device, and the structure is in turn: image input layer, first linear network, first activation function layer, second linear network, second activation function layer, and fully connected network; the input dimension of the first linear network is set to 784, the output layer dimension is 200, the input dimension of the second linear network is 200, the output dimension is 200, the input dimension of the fully connected layer is 200, and the output dimension is 10; the first and second activation function layers both use ReLU function to realize.
[0107] The reinforcement learning agent network adopts DDQN (Double Deep Q Network) reinforcement learning algorithm, including a Q network and a Target Q network, and the Q network and the Target Q network adopt the same network structure. Among them, the Q network is also called reinforcement learning agent or agent.
[0108] Referring to the accompanying drawings Figure 3 , the network structure of the Q network and the Target Q network is further described as follows:
[0109] The network structure of the Q network and the Target Q network is composed of two linear networks, the input dimension of the first linear network is the state dimension of the federated learning system, the output dimension is set to 128, and the output of the first linear network is connected to a ReLU activation function. The input dimension of the second linear network is 128, and the output dimension is equal to the number of edge IIoT devices. In the experiment, 50 IIoT devices are used, and the data transmission rate, working frequency, signal transmission power and sample number of each device at the current time are used as state information, so the state dimension of the federated learning system is 200.
[0110] Step 5. Convert the node selection problem into a Markov decision process.
[0111] The method of using reinforcement learning to solve the node selection problem of federated learning first needs to abstract this problem into a Markov decision process. A Markov decision process includes system state S(t), action space A(t), policy π, reward function r and adjacent state S(t+1), and the detailed parameter description is as follows.
[0112] The system state S(t) is composed of the data transmission rate β(t) between the device and the server, the working frequency ζ(t) of the device, the signal transmission power T p (t) of the device and the number of samples ψ(t) owned by the device. At time slot t, the state of the system can be defined as:
[0113] S(t) = {β(t), ζ(t), T p (t), ψ(t)} (1-15)
[0114] The action space A(t) is a vector composed of discrete variables 0-1, which represents the selected state of device i at t. which represents that the device is selected to participate in the federated learning process at t; otherwise, it represents that device i is in an unselected state, and device i can perform other tasks at this time.
[0115] The policy π represents the mapping from the state space S(t) to the action space A(t), that is, A(t) = π(S(t)). The goal of DRL (deep reinforcement learning) is to learn an optimal policy π, so that the action made by the agent according to the current state can obtain the maximum expected reward.
[0116] The reward function is set in accordance with the optimization goal, that is, to minimize the weighted sum of the delay and energy consumption, therefore, the reward function r is expressed by using the following formula:
[0117]
[0118] The critical state S(t+1) is determined by the current state S(t) and the policy π, and the specific expression is as follows:
[0119] S(t+1) = S(t) + π(S(t)) (1-17)
[0120] Step 6. Train the reinforcement learning agent network.
[0121] The DDQN reinforcement learning algorithm updates the Q network parameters during training, and the Target Q network parameters remain unchanged. After a certain number of iterations, the parameters of the Q network are copied to the Target Q network, thereby avoiding estimation errors and overestimation problems. Assuming that the network parameters of the Q network are denoted as θ, and the network parameters of the target network are denoted as θ - , the calculation method of the time difference target is as follows:
[0122]
[0123] In the above formula, r represents the immediate reward returned by the system, s' represents the system state at the next time, a' represents the action taken, A represents the action space, represents the action taken to obtain the maximum Q value using the Q network for the next state, represents the value estimated by the Target Q network according to the state at the next time and the action estimated by the Q network.
[0124] Define the following loss function, update θ by backpropagation of the loss function, s and a represent the current time and the action taken at the current time.
[0125] l(θ) = (Y target -Q(s, a; θ)) 2 (1-19)
[0126] The updating process of θ is as follows by minimizing the loss function l(θ) through gradient descent method:
[0127]
[0128] where θ t+1 represents the updated Q network parameters, θ t represents the parameters before updating, represents the gradient of the parameter θ t , s t and a t represent the state and the action taken at time t. α represents the step size of the update. In an embodiment, α is set to 0.001, the update frequency of the Target Q network is 20 times of the Q network, and the Target Q network updates the parameters once using the Q network.
[0129] The DDQN reinforcement learning algorithm updates the parameters of the Q network according to the loss function constructed by the interaction data of the federated learning system and the Target Q network, so that the obtained expected reward reaches convergence.
[0130] Step 7. Perform federated learning using the Q network.
[0131] The cloud server sends global model parameters to all edge devices participating in federated learning and collects state information of the federated learning system; the state information is input into the trained Q network, and the Q network outputs the value of each action, that is, the Q value. The top-k Q value corresponding devices are used as the best device subset in the current state; the best device subset uses local data to train the image classification network to update the local model, and then uploads the latest local model to the server, and the server executes the model aggregation algorithm to update the global model; the above process is continuously performed until the global model reaches the target accuracy.
[0132] The application also provides a federated learning system based on deep reinforcement learning, which is used to execute the above-mentioned federated learning method based on deep reinforcement learning. The system includes a cloud server, multiple edge devices and a wireless network environment. The cloud server is used to store and update global model parameters, receive and send messages, run a reinforcement learning agent network and execute a model aggregation algorithm. The edge devices are used to store and process local data, calculate node scores, perform local training, receive and send messages. The wireless network environment is used to connect the cloud server and the edge devices.
[0133] The effect of the application will be further described below in combination with a simulation experiment:
[0134] 1. Simulation experiment conditions:
[0135] The hardware platform of the simulation experiment of the application is: the processor is Intel(R) Core i7-12700H CPU, the memory is 16GB, and the graphics card is NVIDIA GeForce RTX 3060.
[0136] The software platform of the simulation experiment of the application is: win11 operating system, python 3.9.12, PyTorch1.12.1.
[0137] 2. Simulation content and simulation result analysis:
[0138] The MNIST dataset is used in the simulation experiment of the application, of which 60000 sample data are used to construct a local dataset for each IIoT device, and another 10000 samples are used to construct a test set. Figure 4The change curve of the reward obtained by the reinforcement learning agent in the training stage with the iteration round is shown, and with the increase of the iteration number, the reward obtained by the DDQN agent in the training process gradually rises after the trough of the early exploration stage, and tends to be convergent after 200 iterations.
[0139] Figure 5 The change curve of the loss function of the reinforcement learning agent network in the training stage with the iteration round is shown, the loss curve represents the loss function of each training round, and the ma loss curve is the result of smoothing the loss curve. In the early stage of training, the loss function during training is large due to the lack of information of the federated learning environment by the agent. With the increase of the interaction times of the Q network and the environment, the Q network and the Target Q network update their parameters by using the interaction data, so that the training loss rapidly decreases, and after 200 iterations, the training loss becomes stable.
[0140] The node selection strategy proposed by the federated learning method of the application is to guide the node selection of federated learning through the Q network. In the following description combined with the simulation experiment, the node selection strategy is called LCNSFL algorithm, and the two algorithms used for comparison are random selection (random selection strategy) and Bqn selection (node selection algorithm) based on the best channel quality. The node selection strategy based on the best channel quality selects only the node with the highest output transmission rate for local training each time. The following compares the delay, energy consumption, training cost and the like of each communication round. Please refer to Figure 6 to Figure 8 It can be seen that the LCNSFL algorithm is close to the Bnq selection strategy in terms of delay, and is superior to the random selection strategy, and in terms of energy consumption and training cost of each communication round, the LCNSFL algorithm is superior to the other two strategies.
[0141] In summary, the LCNSFL algorithm can reduce the delay and energy consumption of each communication round in the dynamic network scenario without reducing the global model accuracy, and has excellent performance.
[0142] The above is only an example and description of the structure of the application, and those skilled in the art can make various modifications or supplements to the described specific embodiments or use similar ways to replace, as long as the modifications or supplements do not deviate from the structure of the application or exceed the scope defined by the claims, and should belong to the protection scope of the application.
Claims
1. A federated learning method based on deep reinforcement learning, characterized in that: comprising the following steps: modeling and analyzing the latency of the federated learning system; modeling and analyzing the energy consumption of the federated learning system; determining an optimization goal; constructing an image classification network and a reinforcement learning agent network; transforming the node selection problem into a Markov decision process; training the reinforcement learning agent network; using the Q network for federated learning; wherein the reinforcement learning agent network adopts a DDQN reinforcement learning algorithm; the reinforcement learning agent network comprises the Q network and a Target Q network, and the Q network and the Target Q network adopt the same network structure; the modeling and analyzing the latency of the federated learning system specifically comprises: Computing device Time taken to perform model training in the kth communication round: ； where i and k are positive integers, is the number of samples for the device , is the number of samples for the device , is the number of CPU cycles required for one sample in the training device is the number of iterations of the local iteration; calculating the transmission time of the model; ； wherein, represents an absolute value of the local model network parameter, in Byte, represents a device a data transmission rate in the kth communication round; In combination with the time consumed for training the model and the transmission time of the model, a device is obtained The local training delay in the kth communication round is: ； According to the local training delay, a training delay of the kth communication process is obtained : ； the modeling and analyzing the energy consumption of the federated learning system specifically comprises: calculating the energy consumption of the computing device i for performing local training in the kth communication round; calculating the communication power consumption of the computing device i for model transmission; combining the energy consumption with the communication power consumption to obtain an idle energy consumption of the device ; calculating the total energy consumed by the computing device according to the idle energy consumption; The optimization objective is expressed as: wherein, , , ; is a non-negative hyper-parameter for connecting latency and energy consumption metrics, denotes a decision variable, ; denotes that the device is selected to participate in federated learning in the kth communication round, denotes that the device is not selected to participate in federated learning in the kth communication round, denotes the communication power consumption of the device , is a minimum value of the device operating frequency, is a maximum value of the device operating frequency, is a server bandwidth, is a total amount of devices; the training the reinforcement learning agent network specifically comprises: calculating a time-difference target; defining a loss function according to the time-difference target; updating the network parameters of the Q network by minimizing the loss function through gradient descent method; the image classification network is a two-layer MLP network; the image classification network comprises an image input layer, a first linear network, a first activation function layer, a second linear network, a second activation function layer, and a fully connected network; the using the Q network for federated learning specifically comprises: the cloud server sends global model parameters to all edge devices participating in federated learning and collects state information of the federated learning system; the state information is input into the Q network, the Q network outputs Q values, the Q values represent the value of each action, and the devices corresponding to the top k Q values are used as the best device subset in the current state; the best device subset trains the image classification network using local data to update the local model, and then uploads the local model to the cloud server; the cloud server executes a model aggregation algorithm to update the global model; the above process is continuously executed until the global model reaches the target accuracy.
2. The method of claim 1, wherein: the network structure of the Q network and the Target Q network is composed of two linear networks; wherein the input dimension of the first linear network is the state dimension of the federated learning system, the output dimension is set to 128, and the output of the first linear network is connected to the ReLU activation function; the input dimension of the second linear network is 128, and the output dimension is equal to the number of edge IIoT devices.
3. The method of claim 1, wherein: the Markov decision process comprises system state, action space, policy, reward function, and adjacent state; wherein the policy represents the mapping from the state space to the action space; the reward function is set to minimize the weighted sum of latency and energy consumption.
4. A federated learning system based on deep reinforcement learning, characterized in that: the federated learning system is configured to perform the method of any one of claims 1-3; the system comprises a cloud server, a plurality of edge devices, and a wireless network environment; A cloud server for storing and updating global model parameters, receiving and sending messages, and running a reinforcement learning agent network and executing a model aggregation algorithm; an edge device for storing and processing local data, computing node scores, performing local training, receiving and sending messages; a wireless network environment for connecting the cloud server and the edge device.