A federated learning resource allocation optimization system and method for deep reinforcement learning

Abstract

The application provides a federated learning resource allocation optimization system and method based on deep reinforcement learning. A base station is connected with a plurality of mobile devices, and a global federated learning model is sent to each mobile device. The mobile devices perform local training and generate training delay and training energy consumption. The mobile devices upload the trained model to the base station and generate transmission delay and transmission energy consumption; the maximum energy constraint and the maximum asynchronous communication time delay constraint are obtained by combining the training delay and the training energy consumption, the transmission delay and the transmission energy consumption. In combination with the constraint conditions, a federated learning resource allocation optimization target model is constructed; the optimization strategy of the mobile device is obtained by solving the federated learning resource allocation optimization target model with the minimum utility as the optimization target through deep reinforcement learning. The federated learning resource allocation optimization strategy based on safe deep reinforcement learning, in combination with the maximum energy constraint and the maximum asynchronous communication time delay constraint, can achieve the purpose of obtaining the minimum utility of the system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of wireless communication, and particularly relates to a federated learning resource allocation optimization system and method based on deep reinforcement learning. BACKGROUND

[0002] In future wireless systems, due to the limitation of privacy and limited communication resources for data transmission, all wireless devices no longer need to transmit the collected data to a data center, but cooperatively train a shared machine learning model in a privacy-protected manner. To this end, a distributed learning framework is needed to enable wireless devices to collaboratively establish a shared learning model by training the data they collect locally. Federated learning is one of the most popular directions of distributed machine learning. Unlike traditional machine learning, each mobile device trains a separate deep learning model provided by the cloud server on its own data set and uploads the trained model parameters to the cloud server, which greatly protects the privacy of mobile devices.

[0003] In order to implement federated learning on a wireless network, wireless devices must transmit their local training results over wireless links. Due to resource limitations, the total bandwidth of all wireless devices is limited, which results in some devices possibly losing data packets when communicating with the base station under poor channel conditions. In addition, the increasing number of model parameters induced by deep neural networks leads to increased training and communication delays. As the model complexity increases, the training delay of federated learning becomes critical for many time-sensitive scenarios such as autonomous driving and industrial control. In order to make large-size models compatible with resource-constrained devices, it is necessary to compress the model. Due to the limitations of computing power, storage and energy, each device has an energy constraint for learning and transmission on all rounds, which results in not every device can participate in the federated learning training process. Therefore, how to balance the delay and workload in a federated learning system supported by adaptive network pruning is still a challenge. SUMMARY

[0004] The purpose of the present application is to introduce the ideas of deep reinforcement learning and federated learning into the algorithm of multi-device resource allocation, and to propose a federated learning resource allocation method based on secure deep reinforcement learning on the premise of protecting the privacy and security of mobile devices. The resource allocation algorithm maps the resource allocation problem to a reinforcement learning and federated learning problem, establishes a joint optimization pruning rate, device selection, and resource management system model under the premise of meeting the delay and accuracy of the wireless federated system, and obtains the minimum utility of the system through the resource allocation optimization strategy of deep reinforcement learning and federated learning combined with the maximum energy constraint and the maximum asynchronous communication delay constraint.

[0005] The technical scheme of the system of the application is a federated learning resource allocation optimization system based on deep reinforcement learning, comprising a base station and a plurality of mobile devices.

[0006] The base station is wirelessly connected with the plurality of mobile devices in turn.

[0007] The base station generates an initialization parameter vector of a federated learning model, and wirelessly sends the initialization parameter vector of the federated learning model to each mobile device.

[0008] The technical scheme of the method of the application is a federated learning resource allocation optimization method based on deep reinforcement learning, specifically comprising the following steps:

[0009] Step 1: Each mobile device initializes a local federated learning model by using the initialization parameter vector of the federated learning model to obtain an initialized local federated learning model of each mobile device. Each mobile device uses historical traffic data as a training set and combines a gradient algorithm to perform federated learning model training to obtain model parameters of a trained federated learning model of each mobile device, and generates training delay of each mobile device and training energy consumption of each mobile device. Each mobile device wirelessly uploads the model parameters of the trained federated learning model, the training delay, and the training energy consumption to the base station. During the wireless uploading process, each mobile device generates transmission delay and transmission energy consumption, and each mobile device wirelessly uploads the generated transmission delay and transmission energy consumption to the base station.

[0010] Step 2: A constraint condition is constructed in combination with a maximum energy constraint, a maximum asynchronous communication delay constraint, a bandwidth allocation constraint, and a pruning rate constraint. A federated learning resource allocation optimization objective model is constructed in combination with the training delay of each mobile device, the training energy consumption of each mobile device, the transmission delay generated by each mobile device, and the transmission energy consumption generated by each mobile device.

[0011] Step 3: In combination with the constraint condition of step 2, a minimum federated learning resource allocation optimization objective model is taken as an optimization objective, and an optimization strategy of each mobile device is obtained by deep reinforcement learning optimization solution.

[0012] As a preferred, the training delay of each mobile device in step 1 is specifically defined as follows:

[0013]

[0014] i∈[1,M]

[0015] wherein, represents the training delay of the mobile device i, u i represents a binary decision variable of the mobile device i, u i u =1 represents that the mobile device i chooses to participate in the current federated learning model training, otherwisei = 0; C is the amount of computational resources required by the device to process the data bits, D i is the historical traffic data set of mobile device i, CD i is the number of CPU cycles for federated learning model training, f i represents the computing capability of mobile device i, M represents the number of mobile devices;

[0016] The training energy consumption of each mobile device described in step 1 is specifically defined as follows:

[0017]

[0018] wherein, represents the training energy consumption of mobile device i, κ represents the power consumption of the CPU;

[0019] The transmission delay generated by each mobile device described in step 1 is specifically defined as follows:

[0020]

[0021] wherein, represents the transmission delay generated by mobile device i, m i represents the amount of data occupied by the local federated learning model of mobile device i, u i is a binary decision variable of mobile device i, u i = 1 indicates that device i chooses to participate in the current federated learning model training, otherwise u i = 0; r i is the transmission rate of mobile device i; v i is the model pruning rate of mobile device i, and satisfies 0 < v i ≤ v th , v th is the maximum value of the set pruning rate;

[0022] The transmission energy consumption generated by each mobile device described in step 1 is specifically defined as follows:

[0023]

[0024] wherein, represents the transmission energy consumption generated by mobile device i, represents the transmission delay generated by mobile device i when uploading the model, p i is the transmission power of mobile device i;

[0025] As preferred, the maximum energy constraint described in step 2 is specifically as follows:

[0026] E ≤ E max

[0027]

[0028] i∈[1,M]

[0029] where E denotes the total energy consumption of mobile devices in the federated learning process, E max is the maximum energy, is the training energy consumption of mobile device i, is the transmission energy consumption generated by mobile device i, and M denotes the number of mobile devices.

[0030] The maximum asynchronous communication latency constraint in step 2 is specifically defined as follows:

[0031] T≤τ0

[0032]

[0033] where T denotes the total delay of mobile devices in the federated learning process, τ0 is a fixed delay value, is the training delay of mobile device i, is the transmission delay generated by mobile device i.

[0034] The bandwidth allocation constraint in step 2 is specifically defined as follows:

[0035]

[0036] where w i is the bandwidth of mobile device i, u i is a binary decision variable of mobile device i, u i = 1 indicates that device i chooses to participate in the current federated learning model training, otherwise u i = 0, and W max is the available bandwidth of the base station.

[0037] The pruning rate constraint in step 2 is specifically defined as follows:

[0038] 0<v i ≤v th

[0039] where v i is the model pruning rate of mobile device i, and v th is the maximum value of the set pruning rate.

[0040] The federated learning resource allocation optimization objective model in step 2 is constructed as follows:

[0041]

[0042] where U denotes the federated learning resource allocation optimization objective model, Min is the minimization, β is the weight factor, and mi denotes the data amount occupied by the local federated learning model of the mobile device i, u i is a binary decision variable of the mobile device i, u i = 1 indicates that the device i chooses to participate in the current federated learning model training, otherwise u i = 0.

[0043] As preferred, the optimization strategy of each mobile device is obtained by solving the optimization problem in step 3 through deep reinforcement learning, and the specific process is as follows:

[0044] In combination with the federated learning resource allocation optimization objective model U, a deep reinforcement learning resource allocation optimization objective model is constructed through a deep reinforcement learning method, and the specific definition is as follows:

[0045] (S k,t ,A k,t ,R k,t ,P Sk,t(x),Sk,t(y) ,γ)

[0046] k∈[1,M],t∈[1,iter],x∈[1,k+1],y∈[1,k+1]

[0047] Wherein, M represents the number of mobile devices, iter represents the maximum iteration, k+1 represents the number of states in the state space, S k,t is the state space of the mobile device k at the tth iteration, A k is the action space of the mobile device k at the tth iteration, R k,t is the reward function of the mobile device k at the tth iteration, is the state transition probability of the mobile device k from state S k,t (x) to state S k,t (y) at the tth iteration, γ is a discount factor, S k,t (x) is the xth state in the state space of the mobile device k at the tth iteration, S k,t (y) is the yth state in the state space of the mobile device k at the tth iteration.

[0048] State space, according to the resource allocation optimization problem, the connection state of the mobile device is defined, which is as follows:

[0049] S k,t ={S E ,α k,t}

[0050] α k,t ={u 1,t ,u 2,t ,...,u k,t}

[0051] where S k,t represents the state space of the mobile device k at the tth iteration, a k,t represents the connection state set of the mobile device 1, the mobile device 2, …, the mobile device k participating in the federated learning process at the tth iteration, u k,t represents the binary decision variable of the mobile device k at the tth iteration, u k,t = 1 represents that the mobile device k at the tth iteration selects to participate in the federated learning training process, otherwise u k,t = 0; S E ∈ {0, 1}, when the energy consumption E satisfies the maximum energy constraint E max , S E = 1, and if not, S E = 0;

[0052] Action space, according to the observed environmental state of all mobile devices, specifically as follows:

[0053] A k = {a 1,t , a 2,t , …, a k,t}

[0054] a k,t = {p k,t , u k,t , v k,t}

[0055] where A k,t represents the action space set of the mobile device 1, the mobile device 2, …, the mobile device k at the tth iteration, a k,t is the environmental state of the mobile device k at the tth iteration, p k,t is the transmission power of the mobile device k at the tth iteration, u k,t represents the binary decision variable of the mobile device k at the tth iteration, u k,t = 1 represents that the mobile device k at the tth iteration selects to participate in the current federated learning training process, otherwise u k,t = 0; v k,t represents the pruning rate of the mobile device k at the tth iteration;

[0056] Reward function, under the condition of satisfying the constraint described in step 2, the reward function of the federated learning is defined, which is specifically represented as:

[0057] R k,t = U k,t

[0058]

[0059] where R k,tdenotes the reward function of mobile device k at the tth iteration, U k,t is the federated learning resource allocation optimization objective model of mobile device k at the tth iteration, β is the weight factor, j ∈ [1, k] denotes the jth mobile device, m j,t denotes the amount of data occupied by the local federated learning model of mobile device j at the tth iteration, u j,t is the binary decision variable of mobile device j at the tth iteration, u j,t = 1 indicates that mobile device j chooses to participate in the federated learning training process, otherwise u j,t = 0; v j,t denotes the pruning rate of mobile device j at the tth iteration.

[0060] If the condition of the constraint in step 2 is not met, the reward function of federated learning is represented as:

[0061] R k,t = U k,t - ξ

[0062] wherein ξ is a penalty factor;

[0063] Based on the above deep reinforcement learning resource allocation optimization objective model, combined with the Markov decision process, the cumulative reward function model of mobile device k at the tth iteration is defined as:

[0064]

[0065] wherein, denotes the cumulative reward function model of mobile device k at the tth iteration, F(S k,t ,a k,t ) denotes the connection state of mobile device k at the tth iteration based on S k,t , a k,t , F(S k,t , a k,t ) = 1 indicates that the mobile device chooses to participate in the current federated learning process, otherwise F(S k,t , a k,t ) = 0, and γ is a discount factor;

[0066] is the state transition probability of mobile device k at the tth iteration from state S k,t (x) to state S k,t (y), is the value function of mobile device k at the tth iteration when the state of mobile device k is the yth state in the state space;

[0067] Based on the Bellman criterion, the optimal strategy of mobile device k is finally obtained by solving the cumulative reward function

[0068] Compared with the prior art, the beneficial effects of the present application are: under the premise of mobile device security and privacy, a joint optimization pruning rate is established, a device selection, resource management system model is established, a federated learning resource allocation optimization strategy of safe deep reinforcement learning is used, combined with maximum energy constraints and minimum asynchronous communication time delay constraints, and the purpose of obtaining system minimum utility is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 Method flowchart of the embodiment of the present application. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0071] In specific implementation, the method proposed by the technical solutions of the present application can be automatically run by computer software technology, and the system device of the method, such as a computer readable storage medium storing the corresponding computer program of the technical solutions of the present application and a computer device including the running of the corresponding computer program, should also be within the protection scope of the present application.

[0072] The technical solutions of the embodiments of the present application will be described below. Figure 1 The technical solutions of the embodiments of the present application are a deep reinforcement learning federated learning resource allocation optimization system and method, which are as follows:

[0073] The technical solutions of the system of the embodiments of the present application are a deep reinforcement learning federated learning resource allocation optimization system, which includes:

[0074] a base station and a plurality of mobile devices;

[0075] The base station and the plurality of mobile devices are sequentially wirelessly connected;

[0076] The base station generates an initialization parameter vector of a federated learning model, and wirelessly sends the initialization parameter vector of the federated learning model to each mobile device;

[0077] The base station is a macro base station;

[0078] The mobile device is a smart phone.

[0079] The technical solutions of the method of the embodiments of the present application are a deep reinforcement learning federated learning resource allocation optimization method, which are as follows:

[0080] Step 1: Each mobile device initializes a local federated learning model with the initialization parameter vector of the federated learning model to obtain an initialized local federated learning model of each mobile device, each mobile device uses historical traffic data as a training set to combine a gradient algorithm for federated learning model training to obtain a trained federated learning model parameter of each mobile device, and generates a training delay of each mobile device, an energy consumption of each mobile device for training, each mobile device wirelessly uploads the trained federated learning model parameter, the training delay, and the energy consumption for training to the base station, each mobile device generates transmission delay and transmission energy consumption during wireless uploading, and each mobile device wirelessly uploads the generated transmission delay and transmission energy consumption to the base station;

[0081] The training delay of each mobile device in step 1 is specifically defined as follows:

[0082]

[0083] i∈[1,M]

[0084] Wherein, represents the training delay of mobile device i, u i represents a binary decision variable of mobile device i, u i =1 indicates that mobile device i chooses to participate in the current federated learning model training, otherwise u i =0; C=15 is the amount of computing resources required for device processing data bits, D i =20 is the historical traffic data set of mobile device i, CD i is the number of CPU cycles of federated learning model training, f i =1 indicates the computing power of mobile device i, and M=10 indicates the number of mobile devices.

[0085] The training energy consumption of each mobile device in step 1 is specifically defined as follows:

[0086]

[0087] Wherein, represents the training energy consumption of mobile device i, and κ represents the power consumption of CPU.

[0088] The transmission delay generated by each mobile device in step 1 is specifically defined as follows:

[0089]

[0090] Wherein, represents the transmission delay generated by mobile device i, m i =1 indicates the amount of data occupied by the local federated learning model of mobile device i, u iis a binary decision variable of mobile device i, u i = 1 indicates that device i chooses to participate in the current federated learning model training, otherwise u i = 0; r i is the transmission rate of mobile device i; v i is the model pruning rate of mobile device i, and satisfies 0 < v i ≤ v th , v th is the maximum value of the set pruning rate;

[0091] The transmission energy consumption generated by each mobile device in step 1 is defined as follows:

[0092]

[0093] wherein, represents the transmission energy consumption generated by mobile device i, represents the transmission delay generated by mobile device i when uploading the model, p i is the transmission power of mobile device i;

[0094] Step 2: Combine the maximum energy constraint, the maximum asynchronous communication delay constraint, the bandwidth allocation constraint, and the pruning rate constraint to construct the constraint condition, and combine the training delay of each mobile device, the training energy consumption of each mobile device, the transmission delay generated by each mobile device, and the transmission energy consumption generated by each mobile device to construct the federated learning resource allocation optimization objective model;

[0095] The maximum energy constraint in step 2 is as follows:

[0096] E ≤ E max

[0097]

[0098] i ∈ [1, M]

[0099] wherein E represents the total energy consumption generated by the mobile device in the federated learning process, E max = 1 is the maximum energy, is the training energy consumption of mobile device i, is the transmission energy consumption generated by mobile device i, and M = 10 represents the number of mobile devices;

[0100] The maximum asynchronous communication delay constraint in step 2 is as follows:

[0101] T ≤ τ0

[0102]

[0103] where T represents the total delay generated by the mobile device in the federated learning process, τ0= 0.5 is a fixed delay value, denotes the training delay of the mobile device i, denotes the transmission delay generated by the mobile device i;

[0104] The bandwidth allocation constraint in step 2 is defined as follows:

[0105]

[0106] where w i denotes the bandwidth of the mobile device i, u i is a binary decision variable of the mobile device i, u i = 1 indicates that the device i chooses to participate in the current federated learning model training, otherwise u i = 0, W max = 6 is the available bandwidth of the base station;

[0107] The pruning rate constraint in step 2 is defined as follows:

[0108] 0 < v i ≤ v th

[0109] where v i is the model pruning rate of the mobile device i, v th is the maximum value of the set pruning rate;

[0110] Step 2 constructs a federated learning resource allocation optimization objective model, and the specific process is as follows:

[0111]

[0112] where U represents the federated learning resource allocation optimization objective model, Min is the minimization, β is the weight factor, m i = 1 indicates the data amount occupied by the local federated learning model of the mobile device i, u i is a binary decision variable of the mobile device i, u i = 1 indicates that the device i chooses to participate in the current federated learning model training, otherwise u i = 0, M = 10 indicates the number of mobile devices;

[0113] Step 3: combining the constraint conditions in step 2, taking the minimization of the federated learning resource allocation optimization objective model as the optimization objective, and obtaining the optimization strategy of each mobile device through deep reinforcement learning optimization solution;

[0114] Step 3: obtaining the optimization strategy of each mobile device through deep reinforcement learning optimization solution, and the specific process is as follows:

[0115] In combination with the federated learning resource allocation optimization target model U, a deep reinforcement learning resource allocation optimization target model is constructed by a deep reinforcement learning method, and is specifically defined as follows:

[0116] (S k,t ,A k,t ,R k,t ,P Sk,t(x),Sk,t(y) ,γ)

[0117] k∈[1,M],t∈[1,iter],x∈[1,k+1],y∈[1,k+1]

[0118] Wherein, M = 10 represents the number of mobile devices, iter represents the maximum iteration, k + 1 represents the number of states in the state space, S k,t is the state space of mobile device k at the tth iteration, A k is the action space of mobile device k at the tth iteration, R k,t is the reward function of mobile device k at the tth iteration, is the state transition probability of mobile device k from state S k,t (x) to state S k,t (y) at the tth iteration, γ is the discount factor, S k,t (x) is the xth state in the state space of mobile device k at the tth iteration, S k,t (y) is the yth state in the state space of mobile device k at the tth iteration;

[0119] State space, according to the resource allocation optimization problem, the connection state of mobile device is defined, and is specifically as follows:

[0120] S k,t ={S E ,α k,t}

[0121] α k,t ={u 1,t ,u 2,t ,...,u k,t}

[0122] Wherein, S k,t represents the state space of mobile device k at the tth iteration, α k,t represents the connection state set of mobile device 1, mobile device 2,..., mobile device k participating in the federated learning process at the tth iteration, u k,t represents the binary decision variable of mobile device k at the tth iteration, u k,t = 1 represents that mobile device k at the tth iteration selects to participate in the federated learning training process, otherwise u k,t = 0; S E∈ {0,1}, S = 1 when the energy consumption E satisfies the maximum energy constraint E max E = 0 otherwise. E

[0123] The action space, according to the observed environmental state of all mobile devices, is specifically as follows:

[0124] A k = {a 1,t ,a 2,t ...,a k,t}

[0125] a k,t = {p k,t ,u k,t ,v k,t}

[0126] Wherein, A k,t represents the action space set of mobile device 1, mobile device 2,..., mobile device k at the tth iteration, a k,t is the environmental state of mobile device k at the tth iteration, p k,t is the transmission power of the mobile device of mobile device k at the tth iteration, u k,t represents the binary decision variable of mobile device k at the tth iteration, u k,t = 1 indicates that mobile device k at the tth iteration selects to participate in the current federated learning training process, otherwise u k,t = 0; v k,t represents the pruning rate of mobile device k at the tth iteration;

[0127] The reward function, under the condition of satisfying the constraint described in step 2, defines the reward function of federated learning, which is specifically represented as:

[0128] R k,t = U k,t

[0129]

[0130] Wherein, R k,t represents the reward function of mobile device k at the tth iteration, U k,t is the federated learning resource allocation optimization target model of mobile device k at the tth iteration, β is a weight factor, j ∈ [1, k] represents the jth mobile device, m j,t represents the data volume occupied by the local federated learning model of mobile device j at the tth iteration, u j,t is the binary decision variable of mobile device j at the tth iteration, u j,t = 1 indicates that mobile device j selects to participate in the federated learning training process, otherwise u​​j,t = 0; v j,t denotes the pruning rate of mobile device j at the tth iteration.

[0131] If the condition of the constraint in step 2 is not met, the reward function of federated learning is represented as:

[0132] R k,t = U k,t - ξ

[0133] wherein, ξ is a penalty factor;

[0134] Based on the above deep reinforcement learning resource allocation optimization objective model, combined with Markov decision process, the cumulative reward function model of mobile device k at the tth iteration is defined as:

[0135]

[0136] wherein, denotes the cumulative reward function model of mobile device k at the tth iteration, F(S k,t , a k,t ) denotes the connection state of mobile device k at the tth iteration based on S k,t , a k,t , F(S k,t , a k,t ) = 1 indicates that the mobile device chooses to participate in the current federated learning process, otherwise F(S k,t , a k,t ) = 0, and γ is a discount factor;

[0137] is the state transition probability of mobile device k from state S k,t (x) to state S k,t (y) at the tth iteration, is the value function of mobile device k when the state of mobile device k at the tth iteration is the yth state in the state space;

[0138] Based on the Bellman criterion, the optimal strategy of mobile device k is finally obtained by solving the cumulative reward function

[0139] It should be understood that parts not elaborated in the specification are all prior art.

[0140] It should be understood that the above description of the preferred embodiments is more detailed and should not be considered as limiting the scope of patent protection of the present application. Ordinary skilled in the art can make substitutions or modifications without departing from the scope of the claims, which fall within the scope of protection of the present application. The scope of protection of the present application should be subject to the appended claims.

Claims

1. A federated learning resource allocation optimization method for deep reinforcement learning, characterized in that, The method comprises the following steps: Step 1: Each mobile device obtains the trained federated learning model parameters of each mobile device through federated learning model training, and generates the training delay of each mobile device, the training energy consumption of each mobile device, and each mobile device wirelessly uploads the trained federated learning model parameters, the training delay, and the training energy consumption to the base station, and each mobile device generates the transmission delay and the transmission energy consumption during the wireless uploading process, and each mobile device wirelessly uploads the generated transmission delay and transmission energy consumption to the base station; Step 2: A constraint condition is constructed in combination with the maximum energy constraint, the maximum asynchronous communication delay constraint, the bandwidth allocation constraint, and the pruning rate constraint, and a federated learning resource allocation optimization target model is constructed in combination with the training delay, the training energy consumption, the transmission delay, and the transmission energy consumption of each mobile device; Step 3: In combination with the constraint condition of step 2, the optimization target model of the federated learning resource allocation optimization target model is minimized as the optimization target, and the optimization strategy of each mobile device is obtained by deep reinforcement learning optimization solution; The federated learning resource allocation optimization target model constructed in step 2 has the following specific process: wherein U denotes a federated learning resource allocation optimization objective model, is a weight factor, denotes the amount of data occupied by the local federated learning model of the mobile device , is a binary decision variable of the mobile device , = 1 indicates that the device chooses to participate in the current federated learning model training, otherwise = 0, denotes the total delay generated by the mobile device in the federated learning process, is the model pruning rate of the mobile device , denotes the number of mobile devices; The optimization strategy of each mobile device obtained by deep reinforcement learning optimization solution in step 3 has the following specific process: In combination with the federated learning resource allocation optimization target model, a deep reinforcement learning resource allocation optimization target model is constructed by a deep reinforcement learning method; The deep reinforcement learning resource allocation optimization target model includes a state space, an action space, and a reward function; The deep reinforcement learning resource allocation optimization target model is specifically defined as follows: in, Indicates the number of mobile devices. This represents the maximum self-emission during the iteration. This represents the number of states in the state space. For the first Mobile devices at the next iteration The state space, For the first Mobile devices during the next iteration The space of motion For the first Mobile devices at the next iteration The reward function, For the first Mobile devices during the next iteration From state To state The state transition probability of the transition. As a discount factor, For the first Mobile devices at the next iteration The x-th state in the state space. For the first Mobile devices during the next iteration The y-th state in the state space; based on the first iteration of the mobile device federated learning resource allocation optimization objective model, incorporating a Markov decision process, the iteration of the mobile device cumulative reward function model is defined as follows: in, Indicates the first Mobile devices during the next iteration The cumulative reward function model, Indicates mobile device In the In the next iteration, based on , The connection status at that time, This indicates that the mobile device has chosen to participate in the current federated learning process; otherwise... , It is a discount factor; For the first Mobile devices during the next iteration From state To state The state transition probability of the transition. For the first Mobile devices during the next iteration The state is the value function of the y-th state in the state space; Based on the Bellman principle, by solving the cumulative reward function, the optimal strategy of the mobile device is finally obtained .​ 2. The federated learning resource allocation optimization method of deep reinforcement learning according to claim 1, wherein: In step 1, each mobile device obtains the trained federated learning model parameters of each mobile device through federated learning model training, and generates the training delay of each mobile device, the training energy consumption of each mobile device, and each mobile device wirelessly uploads the trained federated learning model parameters, the training delay, and the training energy consumption to the base station, and each mobile device generates the transmission delay and the transmission energy consumption during the wireless uploading process, and each mobile device wirelessly uploads the generated transmission delay and transmission energy consumption to the base station; The training delay of each mobile device in step 1 is specifically defined as follows: i∈[1,M] The training energy consumption of each mobile device in step 1 is specifically defined as follows: wherein, denotes a training delay of a mobile device , denotes a binary decision variable of a mobile device , =1 denotes that a mobile device chooses to participate in the current federated learning model training, otherwise =0; is the amount of computational resources required for a device to process a bit of data, is a historical traffic dataset of a mobile device , is the number of CPU cycles for federated learning model training, denotes the computing capability of a mobile device , and M denotes the number of mobile devices. The transmission delay generated by each mobile device in step 1 is specifically defined as follows: wherein, represents the training energy consumption of a mobile device represents the power consumption of a CPU;​ The transmission energy consumption generated by each mobile device in step 1 is specifically defined as follows: wherein, represents a mobile device the generated transmission delay, represents a mobile device the data volume occupied by the local federated learning model, is a binary decision variable of the mobile device = 1 indicates that the device chooses to participate in the current federated learning model training, otherwise = 0; is the transmission rate of the mobile device ; is the model pruning rate of the mobile device , and satisfies , is the maximum value of the set pruning rate;​ 3. The federated learning resource allocation optimization method of deep reinforcement learning according to claim 2, wherein: wherein, representing the mobile device generated transmission energy consumption, representing the mobile device transmission delay resulting from uploading the model, for the mobile device transmission power. The maximum energy constraint in step 2 is specifically as follows: i∈[1,M] The maximum asynchronous communication delay constraint in step 2 is specifically as follows: wherein, denotes the total energy consumption generated by the mobile devices in the federated learning process, is the maximum energy, is the training energy consumption of the mobile device , is the transmission energy consumption generated by the mobile device , and M denotes the number of mobile devices; The bandwidth allocation constraint in step 2 is specifically defined as follows: wherein, denotes the total delay generated by the mobile device during the federated learning process, is a fixed delay value, denotes the training delay of the mobile device , denotes the transmission delay generated by the mobile device ; The pruning rate constraint in step 2 is specifically defined as follows: wherein, denotes a mobile device bandwidth, is a binary decision variable for a mobile device = 1 indicates that the device chooses to participate in the current federated learning model training, otherwise = 0, is the available bandwidth for a base station;​ 4. The federated learning resource allocation optimization method of deep reinforcement learning according to claim 3, wherein, wherein, is a model pruning rate of a mobile device , is a maximum value of the set pruning rate. ​ The state space defines the connection state of the mobile device according to the resource allocation optimization problem, and is specifically as follows: wherein, denotes the state space of the mobile device at the i-th iteration, denotes the set of connection states of the mobile device 1, mobile device 2,..., mobile device participating in the federated learning process at the i-th iteration, denotes the binary decision variable of the mobile device at the i-th iteration, denotes the selection of the mobile device to participate in the federated learning training process at the i-th iteration, otherwise; when the energy consumption satisfies the maximum energy constraint , and otherwise .​​​​ 5. The federated learning resource allocation optimization method of deep reinforcement learning according to claim 4, characterized in that: The action space is defined according to the observed environment state of all mobile devices, and is specifically as follows: in, This refers to mobile device 1, mobile device 2, ..., mobile devices. A collection of motion spaces, For the first Mobile devices during the next iteration The environmental conditions, For the first Mobile devices during the next iteration The transmission power of mobile devices Indicates the first Mobile devices during the next iteration Binary decision variables, Indicates the first Mobile devices during the next iteration Your choice to participate in the current federal learning training process, otherwise ; Indicates the first Mobile devices during the next iteration The pruning rate.

6. The federated learning resource allocation optimization method of deep reinforcement learning according to claim 5, characterized in that: The reward function defines the reward function of federated learning under the constraint condition in step 2, and is specifically represented as: in, Indicates the first Mobile devices during the next iteration The reward function, For the first Mobile devices during the next iteration The objective model for optimizing the allocation of federated learning resources. As a weighting factor, Indicates the first A mobile device Indicates the first Mobile devices during the next iteration The amount of data required for local federated learning models. It is the first Mobile devices during the next iteration Binary decision variables, Indicates mobile device Choose to participate in the federal learning training process, otherwise ; Indicates mobile device In the Pruning rate at the next iteration; If the constraint condition in step 2 is not met, the reward function of federated learning is represented as: wherein is a penalty factor.

Images