A federated learning method, system and medium based on reinforcement learning in a mobile edge computing network

Abstract

The present invention discloses a federated learning method, system and medium based on reinforcement learning in a mobile edge computing network. The method steps are as follows: the edge server downloads the machine learning model to be trained to the user equipment through the base station; each user equipment trains the machine learning model using local data to obtain the machine learning model parameters w i (k), and upload it to the edge server through the base station; according to the amount of local data of the device to be aggregated, the edge server aggregates the machine learning model parameters of all the devices to be aggregated, obtains the machine learning model parameter aggregation value #imgabs0# and downloads it to the user device that joins the federated learning through the base station; the system includes an edge server and a user device. The medium stores a computer program. The present invention comprehensively considers the energy consumption in the federated learning process and the loss function value of the task model to optimize the federated aggregation strategy, thereby reducing energy consumption while ensuring the accuracy of the task model.

Description

Technical Field

[0001] The present invention relates to the technical fields of mobile edge computing, reinforcement learning and federated learning, and specifically to a federated learning method, system and medium based on reinforcement learning in a mobile edge computing network. Background Art

[0002] In recent years, with the continuous emergence of many new technologies such as computer vision, natural language processing, and recommendation systems, artificial intelligence has entered a period of vigorous development. However, due to problems such as data silos and green communications, the traditional way of concentrating all data on one device to train artificial intelligence models is difficult to process and train data distributed on various mobile devices.

[0003] Mobile edge computing is an emerging technology with great potential. It can process data locally and then offload computing tasks to the edge of the network. By deploying a federated learning framework in a mobile edge computing network, data distributed in various devices can be efficiently trained in a decentralized manner to obtain a fusion model.

[0004] Federated learning is proposed to build distributed machine learning models based on multi-party data. Usually, a federated learning system contains at least one parameter server and many working devices. Each working device and parameter server are responsible for updating the model locally and aggregating the model respectively. Specifically, each working device trains the model locally and then uploads the model to the parameter server. The parameter server aggregates the received model according to a certain strategy weight, and then sends the aggregated model to each working device. The content transmitted between each working device and the parameter server only contains model parameters, without specific data, so that the model can be trained in a decentralized manner, which greatly improves the training efficiency and protects the privacy of all devices.

[0005] However, there are many devices with different computing resources in the mobile edge computing network, and these devices usually have great uncertainty, such as offline, power outages, network congestion, etc. The data volume distribution in different devices is uneven and will change over time. The computing power and battery life of different devices are also different. These will lead to slow model convergence and high training energy consumption. Summary of the invention

[0006] The purpose of the present invention is to provide a federated learning method based on reinforcement learning in a mobile edge computing network, comprising the following steps:

[0007] 1) Determine the user devices currently participating in federated learning;

[0008] The edge server downloads the machine learning model to be trained to the user equipment through the base station;

[0009] 2) Each user device uses local data to train the machine learning model and obtain the machine learning model parameters w i (k) and uploaded to the edge server through the base station;

[0010] 3) The edge server judges the received machine learning model parameters and the locally preset convergence conditions. If there are machine learning model parameters that do not meet the convergence conditions, it proceeds to step 4). If all machine learning model parameters meet the convergence conditions, the machine learning model training is completed;

[0011] 4) The edge server selects n t A user device is used as a device to be aggregated;

[0012] According to the amount of local data of the device to be aggregated, the edge server aggregates the machine learning model parameters of all the devices to be aggregated to obtain the aggregated value of the machine learning model parameters. And download it to the user equipment participating in the federated learning through the base station;

[0013] 5) The user device participating in the federated learning aggregates the machine learning model parameters As new machine learning model parameters, update the machine learning model, set the number of iterations k = k + 1, and return to step 2) until a trained machine learning model is obtained.

[0014] Furthermore, the machine learning model parameter w i (k) is as follows:

[0015]

[0016] In the formula, w i (k-1) is the machine learning model parameter updated in the k-1th iteration; is the first-order gradient of the machine learning model parameters updated at the k-1th iteration; α is the learning rate.

[0017] Furthermore, the machine learning model parameter aggregation value As shown below:

[0018]

[0019] In the formula, |D i | is the local data volume of the i-th user device; w i (k t，i ) is the machine learning model parameter of the i-th user device; x t，i ∈{0, 1} indicates whether device i participates in the tth round of aggregation. N is the number of user devices.

[0020] Furthermore, based on the dynamic asynchronous federated aggregation algorithm, the edge server selects n t The user equipment is used as the device to be aggregated.

[0021] Furthermore, the number of devices to be aggregated n t Determined by a dynamic asynchronous federation aggregation algorithm.

[0022] Further, determine the number of devices to be aggregated n t The steps include:

[0023] s1) The edge server is used as an intelligent agent, which obtains feedback information from the user device to establish a perception state t is the number of polymerization rounds; ΔF t is the difference of the global loss function between two adjacent aggregations;

[0024] Among them, the time required to complete the aggregation of machine learning model parameters is E t , the energy H required to complete the aggregation of machine learning model parameters t , global loss function value F t As shown below:

[0025]

[0026] In the formula, is the loss function value corresponding to the i-th user equipment;

[0027] The i-th user device updates the learning model parameters w i (k) Time required Energy consumed As shown below:

[0028]

[0029]

[0030] In the formula, K, C, f i They are the effective switching capacitance of the device chip architecture, the number of CPU rounds required for training a single piece of data, the amount of data per batch on the i-th user device, and the device CPU frequency.

[0031] The i-th user device sets the machine learning model parameters w i (k) Time required to upload to edge server Energy consumed As shown below:

[0032]

[0033]

[0034] In the formula, s, b i 、p i , g i , N0 are the model size, the bandwidth occupied by the i-th user equipment, the average transmission power of the i-th user equipment, the channel gain between the i-th user equipment and the edge server, and the power spectral density of Gaussian noise.

[0035] s2) The edge server will perceive the state s t As input data, it is input into the pre-stored deep neural network to obtain the maximum reward value r t Action t , and the action a t As the number of devices to be aggregated.

[0036] Furthermore, the loss function Loss(θ) of the deep neural network is as follows:

[0037]

[0038] In the formula, Q(s j , a; θ) is the value of executing action a; For expectations;

[0039] Target value y j As shown below:

[0040]

[0041] In the formula, r j is the reward for executing action aj; s j+1 is the perception state; γ is the attenuation factor; θ is the deep neural network parameter; a′ is s j+1 action space;

[0042] Furthermore, the loss function gradient of the deep neural network As shown below:

[0043]

[0044] In the formula, is the reward gradient.

[0045] A system for applying the federated learning method based on reinforcement learning under the mobile edge computing network, wherein the system is used to complete the training of the machine learning model to obtain a machine learning model that meets preset requirements;

[0046] The system includes an edge server and a number of user devices;

[0047] When working, the edge server downloads the machine learning model to be trained to the user device through the base station;

[0048] Each user device uses local data to train the machine learning model and obtains the machine learning model parameters w i (k) and uploaded to the edge server through the base station;

[0049] The edge server judges the received machine learning model parameters and the locally preset convergence conditions. If all machine learning model parameters meet the convergence conditions, the machine learning model training is completed;

[0050] If there are machine learning model parameters that do not meet the convergence conditions, select n t The user devices are used as the devices to be aggregated, and the machine learning model parameters of all the devices to be aggregated are aggregated to obtain the aggregated value of the machine learning model parameters. And download it to the user equipment participating in the federated learning through the base station;

[0051] The user device aggregates the machine learning model parameters As new machine learning model parameters, the machine learning model is updated, and the machine learning model continues to be trained using local data.

[0052] A computer-readable storage medium having a computer program stored thereon;

[0053] When the computer program is called, the steps of the above method are executed.

[0054] The technical effect of the present invention is unquestionable. The present invention provides a federated learning method based on reinforcement learning in a mobile edge computing network, which has the following beneficial effects:

[0055] The dynamics and uncertainty of the network are taken into account when optimizing the federation aggregation strategy, so that the system can operate normally and stably in most network environments.

[0056] Furthermore, the present invention comprehensively considers the energy consumption in the federated learning process and the loss function value of the task model to optimize the federated aggregation strategy, thereby reducing energy consumption while ensuring the accuracy of the task model.

[0057] Furthermore, the federated aggregation strategy used in the present invention is based on a reinforcement learning algorithm, which can meet the needs of different networks and users, and can simultaneously optimize the algorithm network during use, so that the system can achieve better results. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is the system model diagram;

[0059] Figure 2This is a structural diagram for reinforcement learning;

[0060] Figure 3 This is a flowchart of federated learning based on reinforcement learning;

[0061] Figure 4 This is the flow chart of the reinforcement learning algorithm. DETAILED DESCRIPTION

[0062] The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.

[0063] Embodiment 1:

[0064] See also Figures 1 to 4 , a federated learning method based on reinforcement learning in a mobile edge computing network, comprising the following steps:

[0065] 1) Determine the user devices currently participating in federated learning;

[0066] The edge server downloads the machine learning model to be trained to the user equipment through the base station;

[0067] 2) Each user device uses local data to train the machine learning model and obtain the machine learning model parameters w i (k) and uploaded to the edge server through the base station;

[0068] 3) The edge server judges the received machine learning model parameters and the locally preset convergence conditions. If there are machine learning model parameters that do not meet the convergence conditions, it proceeds to step 4). If all machine learning model parameters meet the convergence conditions, the machine learning model training is completed;

[0069] 4) The edge server selects n t A user device is used as a device to be aggregated;

[0070] According to the amount of local data of the device to be aggregated, the edge server aggregates the machine learning model parameters of all the devices to be aggregated to obtain the aggregated value of the machine learning model parameters. And download it to the user equipment participating in the federated learning through the base station;

[0071] 5) The user device participating in the federated learning aggregates the machine learning model parameters As new machine learning model parameters, update the machine learning model, set the number of iterations k = k + 1, and return to step 2) until a trained machine learning model is obtained.

[0072] Embodiment 2:

[0073] See also Figures 1 to 4 , a federated learning method based on reinforcement learning in a mobile edge computing network, the technical content is the same as that of Example 1, further, the machine learning model parameter w i (k) is as follows:

[0074]

[0075] In the formula, w i (k-1) is the machine learning model parameter updated in the k-1th iteration; is the first-order gradient of the machine learning model parameters updated at the k-1th iteration; α is the learning rate.

[0076] Embodiment 3:

[0077] See also Figures 1 to 4 , a federated learning method based on reinforcement learning in a mobile edge computing network, the technical content is the same as any one of embodiments 1-2, further, the machine learning model parameter aggregation value As shown below:

[0078]

[0079] In the formula, |D i | is the local data volume of the i-th user device; w i (k t，i ) is the machine learning model parameter of the i-th user device; x t，i ∈{0, 1} indicates whether device i participates in the tth round of aggregation. N is the number of user devices.

[0080] Embodiment 4:

[0081] See also Figures 1 to 4 , a federated learning method based on reinforcement learning in a mobile edge computing network, the technical content is the same as any one of embodiments 1-3, further, based on a dynamic asynchronous federated aggregation algorithm, the edge server selects n according to the time sequence of receiving the machine learning model parameters t The user equipment is used as the device to be aggregated.

[0082] Embodiment 5:

[0083] See also Figures 1 to 4 , a federated learning method based on reinforcement learning in a mobile edge computing network, the technical content is the same as any one of embodiments 1-4, further, the number of devices to be aggregated n t Determined by a dynamic asynchronous federation aggregation algorithm.

[0084] Embodiment 6:

[0085] See also Figures 1 to 4 , a federated learning method based on reinforcement learning in a mobile edge computing network, the technical content is the same as any one of embodiments 1-5, further, determining the number of devices to be aggregated n t The steps include:

[0086] s1) The edge server is used as an intelligent agent, which obtains feedback information from the user device to establish a perception state t is the number of polymerization rounds; ΔF t is the difference of the global loss function between two adjacent aggregations;

[0087] Among them, the time required to complete the aggregation of machine learning model parameters is E t , the energy H required to complete the aggregation of machine learning model parameters t , global loss function value F t As shown below:

[0088]

[0089]

[0090]

[0091] In the formula, is the loss function value corresponding to the i-th user equipment;

[0092] The i-th user device updates the learning model parameters w i (k) Time required Energy consumed As shown below:

[0093]

[0094]

[0095] In the formula, K, C, f i They are the effective switching capacitance of the device chip architecture, the number of CPU rounds required for training a single piece of data, the amount of data per batch on the i-th user device, and the device CPU frequency.

[0096] The i-th user device sets the machine learning model parameters w i (k) Time required to upload to edge server Energy consumed As shown below:

[0097]

[0098]

[0099] In the formula, s, b i 、p i , g i , N0 are the model size, the bandwidth occupied by the i-th user equipment, the average transmission power of the i-th user equipment, the channel gain between the i-th user equipment and the edge server, and the power spectral density of Gaussian noise.

[0100] s2) The edge server will perceive the state s t As input data, it is input into the pre-stored deep neural network to obtain the maximum reward value r t Action t , and the action a t As the number of devices to be aggregated.

[0101] Embodiment 7:

[0102] See also Figures 1 to 4 , a federated learning method based on reinforcement learning in a mobile edge computing network, the technical content is the same as any one of Examples 1-6, further, the loss function Loss(θ) of the deep neural network is as follows:

[0103]

[0104] In the formula, Q(s j , a; θ) is the value of executing action a; For expectations;

[0105] Target value y j As shown below:

[0106]

[0107] In the formula, r j is the reward for executing action aj; s j+1 is the perception state; γ is the attenuation factor; θ is the deep neural network parameter; a′ is s j+1 action space;

[0108] Embodiment 8:

[0109] See also Figures 1 to 4 , a federated learning method based on reinforcement learning in a mobile edge computing network, the technical content is the same as any one of Examples 1-7, further, the loss function gradient of the deep neural network As shown below:

[0110]

[0111] In the formula, is the reward gradient.

[0112] Embodiment 9:

[0113] A system for a federated learning method based on reinforcement learning in a mobile edge computing network according to any one of Application Examples 1-8, wherein the system is used to complete the training of a machine learning model to obtain a machine learning model that meets preset requirements;

[0114] The system includes an edge server and a number of user devices;

[0115] When working, the edge server downloads the machine learning model to be trained to the user device through the base station;

[0116] Each user device uses local data to train the machine learning model and obtains the machine learning model parameters w i (k) and uploaded to the edge server through the base station;

[0117] The edge server judges the received machine learning model parameters and the locally preset convergence conditions. If all machine learning model parameters meet the convergence conditions, the machine learning model training is completed;

[0118] If there are machine learning model parameters that do not meet the convergence conditions, select n t The user devices are used as the devices to be aggregated, and the machine learning model parameters of all the devices to be aggregated are aggregated to obtain the aggregated value of the machine learning model parameters. And download it to the user equipment participating in the federated learning through the base station;

[0119] The user device aggregates the machine learning model parameters As new machine learning model parameters, the machine learning model is updated, and the machine learning model continues to be trained using local data.

[0120] Embodiment 10:

[0121] A computer-readable storage medium having a computer program stored thereon;

[0122] When the computer program is called, the steps of the method described in any one of Examples 1-8 are executed.

[0123] Embodiment 11:

[0124] A federated learning method based on reinforcement learning in a mobile edge computing network mainly includes the following steps:

[0125] 1) At the current time t, federated learning begins, and N devices to be federated learning within the signal range of the edge base station are read from the network.

[0126] 2) Each device participating in federated learning trains and updates the model parameters w locallyi (k), the specific update rules are as follows:

[0127]

[0128] Update the learning model parameters w i (k) The time required can be calculated using CPU cycles:

[0129]

[0130] Also calculate the energy consumed by each device:

[0131]

[0132] The updated parameters are then uploaded to the edge server through the base station, and the time and energy consumed by the upload are calculated based on the information transmission model:

[0133]

[0134]

[0135] 3) According to the dynamic asynchronous federation aggregation algorithm, select n according to the order in which the model parameters uploaded by each device are received. t The parameters uploaded by each device are processed in the edge server according to the data volume of the corresponding device. i |Perform weighted aggregation:

[0136]

[0137] The edge server then sends the updated model parameters to each device that joins the federated learning. At the same time, the global loss function value is obtained:

[0138]

[0139] At the same time, the time and energy required for each round of federation aggregation can be calculated based on the specific devices participating in each round of aggregation:

[0140]

[0141]

[0142] 4) When performing model aggregation, n is determined based on the reinforcement learning algorithm DQN training t strategy, determine n t Specific value.

[0143] 4.1) The edge server is regarded as an intelligent agent, and the mobile edge computing network where the device is located is regarded as the environment. The intelligent agent perceives the state from the feedback message from the device. It includes the number of aggregations, energy and time consumption, and the loss function value of the model, and then outputs the value of each action in the corresponding state, that is, the number of devices participating in the federation aggregation in this round, and selects an action a with the maximum value. t To execute and get rewards t In state s t Execute a t The actual value after

[0144] 4.2) Use a deep neural network to develop a strategy π, which outputs the action with the greatest value when the current state is input. When this action is chosen, the agent will receive a reward:

[0145]

[0146] Minimizing the energy consumption of federated learning by maximizing rewards.

[0147] 4.3) The agent randomly selects actions in the corresponding state through strategy π and returns rewards. After completing this round of aggregation, it enters the next round of aggregation and repeats this step.

[0148] 4.4) After the agent has collected a certain amount of experience, the agent's strategy network is trained:

[0149]

[0150] The target value is updated by the value function:

[0151]

[0152] The agent updates the parameters of the network according to the street gradient descent algorithm:

[0153]

[0154] 5) According to the above reinforcement learning algorithm, the federation aggregation strategy is dynamically updated and the strategy is used for federation aggregation.

[0155] 5.1) In the edge server, when a device uploads an aggregation request, the agent selects the number of devices participating in the aggregation by predicting the value function through the above-mentioned training and updating network to perform federated aggregation.

[0156] 5.2) After executing the action, update the current federated learning environment.

[0157] 5.3) The edge server broadcasts the aggregated parameters to each device participating in federated learning.

Claims

1. A federated learning method based on reinforcement learning in a mobile edge computing network, characterized in that: The following steps are involved: 1) Determine the user devices currently participating in federated learning; The edge server downloads the machine learning model to be trained to the user device through the base station; 2) Each user device uses local data to train the machine learning model and obtain the machine learning model parameters w i (k) and uploaded to the edge server through the base station; 3) The edge server judges the received machine learning model parameters and the locally preset convergence conditions. If there are machine learning model parameters that do not meet the convergence conditions, it proceeds to step 4). If all machine learning model parameters meet the convergence conditions, the machine learning model training is completed; 4) The edge server selects n t A user device is used as a device to be aggregated; According to the amount of local data of the device to be aggregated, the edge server aggregates the machine learning model parameters of all the devices to be aggregated to obtain the aggregated value of the machine learning model parameters. And download it to the user equipment participating in the federated learning through the base station; 5) The user device participating in the federated learning aggregates the machine learning model parameters As new machine learning model parameters, update the machine learning model, set the number of iterations k = k + 1, and return to step 2) until a trained machine learning model is obtained; Determine the number of devices to be aggregated n t The steps include: s1) The edge server is used as an intelligent agent, which obtains feedback information from the user device to establish a perception state t is the number of polymerization rounds; ΔF t is the difference of the global loss function between two adjacent aggregations; is the energy aggregation value; Among them, the time required to complete the aggregation of machine learning model parameters is E t , the energy H required to complete the aggregation of machine learning model parameters t , global loss function value F t As shown below: In the formula, is the loss function value corresponding to the i-th user equipment; |D i | is the local data volume of the i-th user device; x t,i ∈{0,1} indicates whether device i participates in the tth round of aggregation; N is the number of user devices; The i-th user device updates the learning model parameters w i (k) Time required Energy consumed As shown below: In the formula, κ, C, f i are the effective switch capacitance of the device chip architecture, the number of CPU rounds required for training a single piece of data, the amount of data per batch on the i-th user device, and the device CPU frequency; The i-th user device sets the machine learning model parameters w i (k) Time required to upload to edge server Energy consumed As shown below: In the formula, s, b i 、p i , g i , N0 are the model size, the bandwidth occupied by the i-th user equipment, the average transmission power of the i-th user equipment, the channel gain between the i-th user equipment and the edge server, and the power spectral density of Gaussian noise respectively; s2) The edge server will perceive the state s t As input data, it is input into the pre-stored deep neural network to obtain the maximum reward value r t Action t , and the action a t As the number of devices to be aggregated; The loss function Loss(θ) of the deep neural network is as follows: In the formula, is the value of executing action a; For expectations; Target value y j As shown below: In the formula, r j is the reward for executing action aj; s j+1 is the perception state; γ is the attenuation factor; θ is the deep neural network parameter; a ′ For j+1 action space; The gradient of the loss function of the deep neural network As shown below: In the formula, is the reward gradient; The value of executing action a.

2. According to claim 1, a federated learning method based on reinforcement learning in a mobile edge computing network is characterized in that: The machine learning model parameter w i (k) is as follows: In the formula, w i (k-1) is the machine learning model parameter updated in the k-1th iteration; is the first-order gradient of the machine learning model parameters updated at the k-1th iteration; α is the learning rate.

3. According to claim 1, a federated learning method based on reinforcement learning in a mobile edge computing network is characterized in that: Machine learning model parameter aggregation values As shown below: In the formula, |D i | is the local data volume of the i-th user device; w i (k t,i ) is the machine learning model parameter of the i-th user device; x t,i ∈{0,1} indicates whether device i participates in the tth round of aggregation; N is the number of user devices.

4. According to claim 1, a federated learning method based on reinforcement learning in a mobile edge computing network is characterized in that: Based on the dynamic asynchronous federated aggregation algorithm, the edge server selects n t The user equipment is used as the device to be aggregated.

5. The method for federated learning based on reinforcement learning in a mobile edge computing network according to claim 1, characterized in that: Number of devices to be aggregated n t Determined by a dynamic asynchronous federation aggregation algorithm.

6. A system for applying the federated learning method based on reinforcement learning in a mobile edge computing network according to any one of claims 1 to 5, characterized in that: The system is used to complete the training of the machine learning model to obtain a machine learning model that meets preset requirements; The system includes an edge server and a number of user devices; When working, the edge server downloads the machine learning model to be trained to the user device through the base station; Each user device uses local data to train the machine learning model and obtains the machine learning model parameters w i (k) and uploaded to the edge server through the base station; The edge server judges the received machine learning model parameters and the locally preset convergence conditions. If all machine learning model parameters meet the convergence conditions, the machine learning model training is completed; If there are machine learning model parameters that do not meet the convergence conditions, select n t The user devices are used as the devices to be aggregated, and the machine learning model parameters of all the devices to be aggregated are aggregated to obtain the aggregated value of the machine learning model parameters. And download it to the user equipment participating in the federated learning through the base station; The user device aggregates the machine learning model parameters As new machine learning model parameters, the machine learning model is updated and the local data is continued to be used to train the machine learning model.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon; When the computer program is called, the steps of the method according to any one of claims 1 to 5 are executed.

Images