An intelligent resource allocation method for air federated learning based on wireless energy communication

Abstract

An airborne federated learning intelligent resource allocation method based on wireless power communication belongs to the field of wireless communication. In the present invention, the base station obtains the total energy information of each user and determines the resource allocation plan. The base station uses wireless power communication technology to broadcast the global model and sends radio frequency energy to charge the user at the same time. After receiving the global model, the user uses the collected environmental data to perform multiple rounds of iterative local learning and obtains the updated local model using the stochastic gradient descent method. Using air computing technology, all users upload the updated local model at the same frequency at the same time to achieve model transmission and aggregation in the air. The present invention supports charging users when the user's battery capacity is insufficient and the distribution location is unreachable. It can obtain the approximately optimal resource allocation and beamforming plan under the condition of ensuring the convergence of federated learning, thereby maximizing the energy efficiency of the system. The present invention can effectively reduce communication delays and improve spectrum efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of wireless communications, and in particular relates to an air federated learning intelligent resource allocation method based on wireless energy-carrying communication. Background Art

[0002] As a typical distributed learning, federated learning has been widely used in various IoT application scenarios such as smart cities, medical monitoring, and smart factories. Over-the-air federated learning applies over-the-air computing technology to the model aggregation process of federated learning to reduce the communication delay of federated learning and improve spectrum utilization.

[0003] In actual federated learning scenarios, the battery capacity of IoT users is often limited, and they may be distributed in geographical locations such as underwater, high in buildings, and on drones. This makes it impossible to frequently charge users manually, making it difficult for some users to continue to participate in the federated learning process, which in turn affects the accuracy and convergence of federated learning. In order to achieve sustainable aerial federated learning under limited energy conditions, it is necessary to explore new federated learning methods, efficiently allocate resources such as energy and time, and design reasonable beamforming schemes to optimize the energy efficiency of the federated learning system.

[0004] Currently, existing federated learning has the following technical problems:

[0005] (1) Insufficient energy of IoT users leads to unsustainable system operation. The battery capacity of federated learning users in IoT scenarios is often limited, and federated learning requires periodic local large-scale neural network training and high-dimensional model parameter uploads. This process consumes a lot of energy, so the energy of local users often cannot meet the demand. The current method of frequently charging a large number of users is too costly, especially since federated learning users are distributed in the wild, toxic areas, drones and other geographical locations, making battery replacement or charging more difficult. Insufficient energy of local users will result in insufficient transmission power for the uploaded model, and even cause some local users to be unable to participate in federated learning due to lack of energy, which in turn leads to low accuracy and poor convergence of federated learning.

[0006] (2) Existing technologies, such as federated learning based on wireless energy-carrying communication and over-the-air federated learning, often sacrifice the transmission power of the base station to ensure error-free reception of downlink signals, resulting in excessive total system power consumption and low energy efficiency.

[0007] (3) The joint optimization problem of energy, beam, and time resources is too complex. Since the joint optimization problem of energy, beam, time, and other resource allocation is a long-term non-convex optimization problem with coupled variables, the time complexity of traditional optimization methods is too high and it is difficult to obtain the optimal solution.

[0008] In summary, there is an urgent need for an intelligent resource allocation method for over-the-air federated learning to meet the huge energy needs of federated learning users, while ensuring efficient communication and reliable information transmission, achieving efficient allocation of resources such as energy, beams, and time, improving energy efficiency, and efficiently optimizing high-dimensional continuous variables to accelerate learning convergence. Summary of the invention

[0009] The present invention provides an air federated learning intelligent resource allocation method based on wireless energy-carrying communication to solve the problems existing in the prior art.

[0010] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0011] An air federated learning intelligent resource allocation method based on wireless energy communication of the present invention comprises the following steps:

[0012] Step S1: In the first stage, the base station obtains the total energy information of each user and determines the resource allocation plan. Then the base station uses wireless energy communication technology to broadcast the global model and send radio frequency energy to charge the user.

[0013] Step S2: In the second stage, after receiving the global model, the user uses the collected environmental data to perform multiple rounds of iterative local learning, and then uses the stochastic gradient descent method to obtain the updated local model;

[0014] Step S3: In the third stage, using air computing technology, all users upload the updated local model at the same time and frequency to achieve model transmission and aggregation in the air.

[0015] Furthermore, the specific operation process of step S1 is as follows:

[0016] Assume that the channel between the base station and user k is h k,t ∈￡ M×1 , M is the number of base station antennas. The base station obtains the total energy information of each user by receiving the pilot signal. In the tth communication cycle, the base station uses wireless energy communication technology to send a signal s carrying global model information and RF energy. 0,t , the signal received by user k is expressed as: v j,t ∈￡ M×1 Send beamforming vectors to the base station, The variance for user k is Gaussian white noise, K is the number of users; first, power division is performed, that is, user k is allocated according to the power allocation ratio λ k,t The λ in the received signal k,t Part of it is used as useful information for information decoding, and we get: is the useful signal carrying the global model received by user k, Gaussian white noise introduced for information decoding, is the variance of Gaussian white noise; energy is collected at the same time, that is, user k will receive 1-λ in the signal k,t Part of it is converted into the collected energy, which is expressed as: τ 1,t is the time of wireless energy communication, For energy conversion efficiency.

[0017] Furthermore, the specific operation process of step S2 is as follows:

[0018] After receiving the global model, the user uses the collected environmental data to perform local learning for τ rounds of iterations, and then uses the stochastic gradient descent method to obtain the updated local model. The updated local model is expressed as: is the global model received by user k, is the local loss function, is the local model obtained by the l∈{0,…,τ-1}th round of iteration, is the sampling set of the local data set, η is the learning rate, and τ is the number of iterations of local learning;

[0019] The energy consumed by local learning is expressed as: C k The number of CPU cycles for processing one bit of data for user k, Z k is the number of bits in the sampling set, f k is the CPU frequency of user k, κ is the effective capacitance coefficient;

[0020] The time consumed by local learning is expressed as: τ 2,k =τC k Z k / f k ; Due to the need to meet the time synchronization characteristics of air computing, the time consumed by local learning is the longest time spent by all users, which is τ2=max k (τ 2,k ).

[0021] Furthermore, the specific operation process of step S3 is as follows:

[0022] Using over-the-air computing technology, the user transforms the normalized signal s k,t With power p k,t Upload, the air computing technology uses the superposition characteristics of the wireless channel to perform model aggregation while communicating; the signal received by the base station is: a t ∈￡ M×1is the base station receive beamforming vector, n 0,t ~CN(0,σ 2 I) is the Gaussian white noise of the base station, σ 2 is the variance of the base station Gaussian white noise, I is the M×M dimensional identity matrix;

[0023] The aggregate model mean square error is expressed as:

[0024] The air calculation rate is expressed as:

[0025] in, is a perfectly aggregated signal, σ 2 is the variance of the base station Gaussian white noise, is the total power of the superimposed signal, B is the bandwidth;

[0026] The energy consumed by user k uploading the local model is expressed as: is the air computing time, W is the number of bits of the model uploaded by user k; after the tth communication cycle, the total amount of energy remaining for user k is expressed as: is the energy harvested by user k in the i-th communication cycle, ) is the energy consumed by user k in uploading the local model in the i-th communication cycle, E max is the battery capacity of user k.

[0027] Furthermore, in step S1, the method for determining the resource allocation solution targets the following optimization problem:

[0028] The energy efficiency of the system in the tth communication cycle is defined as the ratio of the air computing rate to the total power consumed by the system, that is, The total power consumed by the system in the tth communication cycle is defined as: P c is the power consumed by the base station signal processing circuit, v k,t is the transmission beamforming vector of the base station to user k;

[0029] The system energy efficiency is summed and averaged over time to obtain the optimization equation of the system and the complete optimization problem as follows:

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] Among them, ε t ≤ε, is the convergence guarantee equation for federated learning, that is, the constraint condition to ensure that the designed air federated learning intelligent resource allocation method meets the convergence requirements, ε t is the error term, ε is the upper bound of the error, For any t∈{1,…,T}, For any k∈{1,…,K}, in is the global model received by user k Relative to the initial global model w 0,t The deviation is the aggregation model w received by the base station t+1 Relative to the ideal aggregation model Deviation; is the QoS guarantee equation, is the downlink wireless energy communication rate, is the signal-to-noise ratio of user k; τ 1,t +τ2+τ 3,t ≤τ max , is the time constraint equation, τ max is the maximum time constraint for a communication cycle; 0≤λ k,t ≤1, , is the power split ratio constraint equation; is the power constraint equation of the base station and user k, and The maximum power constraints for base stations and users; is the energy constraint equation of user k.

[0039] Furthermore, the method for determining the resource allocation scheme has the following specific operation process:

[0040] Step 1: Divide the target task into two subtasks, namely, minimizing total power consumption and maximizing computing rate. Two agents are deployed at the base station to complete the two subtasks respectively. The goal of agent 1 is to minimize total power consumption, and the goal of agent 2 is to maximize computing rate.

[0041] Step 2: Model the system's long-term energy efficiency optimization problem as a Markov decision process to obtain the state space, action space, and reward function required for Markov decision making;

[0042] Step 3: Use deep reinforcement learning methods to maximize the long-term energy efficiency of the system, determine the base station transmit beamforming, time allocation and user power partitioning scheme for the first phase, and the user transmit power and base station receive beamforming scheme for the third phase.

[0043] Furthermore, the specific operation process of step 2 is as follows:

[0044] (1) Define the state space, action space and reward function of agent 1: In the tth communication cycle, the state space of agent 1 is expressed as: The action space is represented as: a 1,t =(v t ,τ 1,t ,λ t ,p t ), the reward function is expressed as:

[0045] (2) Define the state space, action space and reward function of agent 2: In the tth communication cycle, the state space of agent 2 is expressed as: The action space is represented as: a 2,t =a t , the reward function is expressed as:

[0046] Among them, the condition ifε t ≤ε reflects the convergence constraint of federated learning; when the error term ε t Less than the upper bound of error ε, r 1,t (r 2,t ) as a positive reward helps the agent get a larger reward; when the error term ε t Exceeding the given upper bound ε, r 1,t (r 2,t ) acts as a penalty to guide the agent to reach the goal.

[0047] Furthermore, the specific operation process of step three is as follows:

[0048] Deep reinforcement learning trains the action value network by optimizing the action value function. The iterative expression of the action value function is: is the optimal strategy, s t is the state of agent 1 or agent 2 in the t-th step training, r t is the reward for training agent 1 or agent 2 at step t, s t+1The next state of agent 1 or agent 2 is trained for the tth step, γ is the discount factor, and θ is the model parameter of the action value network; each agent obtains state information by observing the environment, performs actions based on the state information, and forms experience by combining the reward and the next state. The mean square error method is used to train the action value network, and the loss function is expressed as: L(θ)=E[(Φ t -Q(s t ,a t ;θ)) 2 ],Q(s t ,a t ; θ) is the output value of the action value network after observing the state and executing the action, Φ t As the goal, if the training does not reach the required number of steps, then Φ t =r t +γ(Q(s t+1 ,π(s t+1 ;φ');θ'),π(s t+1 ; φ') is the target policy network, φ' is the model parameter of the target policy network, Q(s t+1 ,π(s t+1 ; φ'); θ') is the target action value network, θ' is the model parameter of the target action value network, γ is the discount factor; if the training reaches the required number of steps, then Φ t =r t ; Update the model parameters θ of the action value network by executing the stochastic gradient descent method to minimize the loss function. The θ update process is η θ ∈[0,1] is the learning rate; the model parameter φ of the policy network is updated by determining the policy gradient method to maximize the target, and the φ update process is: in is the performance target of the policy network, and λ is the normalization factor.

[0049] The beneficial effects of the present invention are:

[0050] In the method of intelligent resource allocation based on wireless power communication in the air federated learning of the present invention, the base station broadcasts the global model to the user through the wireless power communication technology and wirelessly charges the user. After receiving the global model, the local user performs multiple rounds of local learning to obtain an updated local model. Finally, the model is aggregated using air computing technology and uploaded to the base station. Compared with the prior art, the advantages of the present invention are:

[0051] (1) The present invention can support wireless charging for users when the user's battery capacity is insufficient or the distributed location is unreachable, which is conducive to realizing a sustainable aerial federated learning system.

[0052] (2) The present invention can quickly grasp the surrounding environment information without prior knowledge of the channel state, and use dual agents to improve the training efficiency of reinforcement learning. It can obtain approximately optimal resource allocation and beamforming solutions while ensuring the convergence of federated learning, thereby maximizing the long-term energy efficiency of the system.

[0053] (3) The present invention uses air computing technology to upload the updated local model. All users upload the updated local model using the same frequency band at the same time. The superposition characteristics of the wireless channel are used to integrate the communication process and the model, which can effectively reduce the communication delay and improve the spectrum efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 The present invention is a flowchart of an air federated learning intelligent resource allocation method based on wireless energy-carrying communication.

[0056] Figure 2 The present invention is a scenario diagram of an air federated learning intelligent resource allocation method based on wireless energy-carrying communication. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The invention proposes an air federated learning intelligent resource allocation method based on wireless energy-carrying communication, which can be applied to any federated learning system with limited user energy, and the system at least includes: a base station and a user.

[0059] Among them, each user is located in a designated federated learning system, and uploads the local model obtained by multiple iterations to the base station. It should be noted that the local model parameters uploaded by each user can be: data of model information such as the weight of the local model and the gradient of the local model. In this regard, the local model parameters uploaded by each user are not restricted in this embodiment. It should be noted that each user has his own local original data set, and the local original data sets of different users may have overlapping parts or be completely different. In addition, the local original data sets can be images, text, audio, etc. In this regard, the content of the local original data sets of each user is not limited in this embodiment.

[0060] Among them, each user needs to pre-process the local model parameters before uploading them through wireless signals; after the user uploads the local model and aggregates the model in the air, the base station needs to post-process the aggregated model to obtain the aggregated global model. It should be noted that the method for pre-processing the user's local model parameters can be: encoding, modulation and other methods. In this regard, the pre-processing method of the user's local model parameters is not limited in this embodiment. At the same time, the method for the base station to post-process the aggregated model can be: demodulation, decoding and other methods. In this regard, the method for the base station to post-process the aggregated model is not limited in this embodiment.

[0061] In addition, the execution subject of the method for intelligent resource allocation of air federated learning based on wireless energy communication proposed by the present invention can be a control device in the federated learning system, and the control device can be installed in the base station or can be an independent device deployed outside the base station. Further, the control device can be any type of electronic device, such as a microcomputer, a single-chip microcomputer, etc., and in this regard, the installation location and device type of the control device are not limited in this embodiment.

[0062] The present invention discloses an airborne federated learning intelligent resource allocation method based on wireless power communication, wherein each communication cycle mainly includes three stages, wherein the first stage is ① a wireless power communication stage, in which the base station obtains the total energy information of each user and determines the resource allocation plan, and then the base station broadcasts the global model using the wireless power communication technology and simultaneously sends radio frequency energy to charge the user; the second stage is ② a local learning stage, in which after the user receives the global model, the user performs multiple rounds of iterative local learning using the collected environmental data, and then obtains the updated local model using the stochastic gradient descent method; the third stage is ③ an airborne computing stage, in which all users simultaneously and simultaneously upload the updated local model using the airborne computing technology to realize model transmission and aggregation in the air.

[0063] The following is a specific description of an air federated learning intelligent resource allocation method based on wireless energy-carrying communication of the present invention.

[0064] See also Figure 1 and Figure 2 To illustrate, the present invention is an air federated learning intelligent resource allocation method based on wireless energy communication, and the specific operation process is as follows:

[0065] Step S1: Figure 2 As shown, an air federated learning intelligent resource allocation method based on wireless energy communication of the present invention, when applied to a federated learning system, includes: a base station and K users, wherein the base station has M antennas, serving as a federated learning server for calculating a global model; each user is a user with a single antenna and limited battery capacity, and each user carries a local original data set. It should be noted that this embodiment does not limit the number of antennas of the base station and the user.

[0066] Assume that there are T federated learning communication cycles. When entering the tth communication cycle, in the first stage, the base station obtains the total energy information of each user and determines the resource allocation plan. Then the base station uses wireless energy communication technology to broadcast the global model and send radio frequency energy to charge the user. The specific operation process is as follows:

[0067] Assume that the channel between the base station and user k is h k,t ∈￡ M×1 , M is the number of base station antennas. It should be noted that the channel model between the base station and the user can be a Rayleigh channel, a Rice channel, etc. This embodiment does not limit the channel model between the base station and the user. The base station obtains the total energy information of each user by receiving the pilot.

[0068] In the tth communication cycle, the base station uses wireless energy communication technology to send a signal s carrying global model information and radio frequency energy. 0,t , the signal received by user k is expressed as: The superscript H represents the conjugate transpose, v j,t ∈￡ M×1 Send beamforming vectors to the base station, The variance for user k is Gaussian white noise, K is the number of users. First, power division is performed, that is, user k is allocated according to the power ratio λ k,t The λ in the received signal k,t Part of it is used as useful information for information decoding, and we get: in, is the useful signal carrying the global model received by user k, Gaussian white noise introduced for information decoding, is the variance of Gaussian white noise.

[0069] Then energy is harvested simultaneously, that is, user k will receive 1-λ of the signal k,t Part of it is converted into the collected energy, which is expressed as: Among them, τ 1,t is the time of wireless energy communication, For energy conversion efficiency.

[0070] The present invention can support wireless charging for users when the user's battery capacity is insufficient and the distribution location is unreachable, which is conducive to realizing a sustainable aerial federated learning system.

[0071] Step S2: In the second stage, after receiving the global model, the user uses the collected environmental data to perform multiple rounds of iterative local learning, and then uses the stochastic gradient descent method to obtain the updated local model. The specific operation process is as follows:

[0072] After receiving the global model, the user uses the collected environmental data to perform local learning for τ rounds of iterations, and then uses the stochastic gradient descent method to obtain the updated local model. The updated local model is expressed as: in, is the global model received by user k, is the local loss function, is the local model obtained by the l∈{0,…,τ-1}th round of iteration, is the sampling set of the local data set, η is the learning rate, and τ is the number of iterations of local learning.

[0073] The energy consumed by local learning is expressed as: Among them C k The number of CPU cycles for processing one bit of data for user k, Z k is the number of bits in the sampling set, f k is the CPU frequency of user k, and κ is the effective capacitance coefficient.

[0074] The time consumed by local learning is expressed as: τ 2,k =τC k Z k / f k Due to the need to meet the time synchronization characteristics of air computing, the time consumed by local learning is the longest time spent by all users, which is τ2 = max k (τ 2,k ).

[0075] Step S3: In the third stage, using air computing technology, all users upload the updated local model at the same time and frequency, and realize model transmission and aggregation in the air. The specific operation process is as follows:

[0076] Using over-the-air computing technology, the user transforms the normalized signal sk,t With power p k,t The air computing technology used in the upload uses the superposition characteristics of the wireless channel to aggregate the model while communicating. The signal received by the base station is: The superscript H represents the conjugate transpose, a t ∈￡ M×1 is the base station receive beamforming vector, n 0,t ~CN(0,σ 2 I) is the Gaussian white noise of the base station, σ 2 is the variance of the base station Gaussian white noise, and I is the M×M dimensional unit matrix.

[0077] The aggregate model mean square error is expressed as:

[0078] The air calculation rate is expressed as:

[0079] in, is a perfectly aggregated signal, σ 2 is the variance of the base station Gaussian white noise, is the total power of the superimposed signal, and B is the bandwidth.

[0080] The energy consumed by user k uploading the local model is expressed as: in, is the air computing time, and W is the number of bits of the model uploaded by user k. After the tth communication cycle ends, the total amount of energy remaining for user k is expressed as: in, is the energy harvested by user k in the i-th communication cycle, ) is the energy consumed by user k in uploading the local model in the i-th communication cycle, E max is the battery capacity of user k.

[0081] The present invention uses air computing technology to upload updated local models. All users use the same frequency band to upload updated local models at the same time. The superposition characteristics of wireless channels are used to integrate the communication process and model aggregation, which can effectively reduce communication delays and improve spectrum efficiency.

[0082] Among them, in step S1, the optimization problem targeted by the method for determining the resource allocation scheme is: based on the principle of maximizing the long-term energy efficiency of the system, based on the total amount of user energy obtained, federated learning convergence guarantee, QoS (Quality of Service) guarantee, power constraint, time constraint and other constraints, the resource allocation scheme within the current communication cycle is determined using a deep reinforcement learning method. The resource allocation scheme mainly includes: base station transmit beamforming, time allocation, and user power partitioning scheme for the first stage, and user transmit power and base station receive beamforming scheme for the third stage.

[0083] Among them, the present invention adopts a deep reinforcement learning method to formulate a reasonable resource allocation plan. It should be noted that the deep reinforcement learning model in the deep reinforcement learning method can be: deep Q learning, actor-critic network, deep deterministic policy gradient and other models. In this regard, the deep reinforcement learning model is not limited in this embodiment.

[0084] The energy efficiency of the system in the tth communication cycle can be defined as the ratio of the air computing rate to the total power consumed by the system, that is, Specifically, the total power consumed by the system in the tth communication cycle is defined as: Among them, P c is the power consumed by the base station signal processing circuit, v k,t is the transmit beamforming vector of the base station to user k.

[0085] The system energy efficiency is summed and averaged over time to obtain the optimization equation of the system and the complete optimization problem as follows:

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] Among them, ε t ≤ε, is the convergence guarantee equation for federated learning, that is, the constraint condition to ensure that the designed air federated learning intelligent resource allocation method meets the convergence requirements, where ε t is the error term, ε is the upper bound of the error, , for any t∈{1,…,T}, For any k∈{1,…,K}, in is the global model received by user k Relative to the initial global model w 0,t The deviation is the aggregation model w received by the base station t+1 Relative to the ideal aggregation model Deviation; is the QoS guarantee equation, is the downlink wireless energy communication rate, is the signal-to-noise ratio of user k; τ 1,t +τ2+τ 3,t ≤τ max , is the time constraint equation, τ max is the maximum time constraint for a communication cycle; 0≤λ k,t ≤1, is the power split ratio constraint equation; is the power constraint equation of the base station and user k, and The maximum power constraints for base stations and users; is the energy constraint equation of user k.

[0095] In summary, in this implementation, the specific operation process of the method for determining the resource allocation scheme is as follows:

[0096] 1. Divide the target task into two subtasks, namely, minimizing total power consumption and maximizing computing rate. Deploy two agents at the base station to complete the two subtasks respectively. The goal of agent 1 is to minimize total power, and the goal of agent 2 is to maximize computing rate.

[0097] 2. Model the long-term energy efficiency optimization problem of the system as a Markov decision process to obtain the state space, action space and reward function required for Markov decision making. The specific operation process is as follows:

[0098] (1) Define the state space, action space and reward function of agent 1: In the tth communication cycle, the state space of agent 1 is expressed as: The action space is represented as: a 1,t =(v t ,τ 1,t ,λ t ,p t), the reward function is expressed as:

[0099] (2) Define the state space, action space and reward function of agent 2: In the tth communication cycle, the state space of agent 2 is expressed as: The action space is represented as: a 2,t =a t , the reward function is expressed as:

[0100] Among them, the condition ifε t ≤ε reflects the convergence constraint of federated learning. The error term ε t Less than the upper bound of error ε, r 1,t (r 2,t ) as a positive reward helps the agent get a larger reward; the error term ε t Exceeding the given upper bound ε, r 1,t (r 2,t ) acts as a penalty to guide the agent to reach the goal.

[0101] 3. Use deep reinforcement learning methods to maximize the long-term energy efficiency of the system, determine the base station transmit beamforming, time allocation and user power division scheme for the first phase, and the user transmit power and base station receive beamforming scheme for the third phase. The specific operation process is as follows:

[0102] Deep reinforcement learning trains the action value network by optimizing the action value function. The iterative expression of the action value function is:

[0103] in is the optimal strategy, s t is the state of agent 1 or agent 2 in the t-th step training, r t is the reward for training agent 1 or agent 2 at step t, s t+1 is the next state of agent 1 or agent 2 in the t-th step training, γ is the discount factor, and θ is the model parameter of the action value network. Each agent obtains state information by observing the environment, performs actions based on the state information, and forms experience by combining the reward and the next state. The mean square error method is used to train the action value network, and the loss function is expressed as: L(θ)=E[(Φ t -Q(s t ,a t ;θ)) 2 ], where Q(s t ,a t ; θ) is the output value of the action value network after observing the state and executing the action, Φ t As the goal, if the training does not reach the required number of steps, then Φ t =rt +γ(Q(s t+1 ,π(s t+1 ;φ');θ'), where π(s t+1 ; φ') is the target policy network, φ' is the model parameter of the target policy network, Q(s t+1 ,π(s t+1 ; φ'); θ') is the target action value network, θ' is the model parameter of the target action value network, γ is the discount factor; if the training reaches the required number of steps, then Φ t =r t The model parameters θ of the action value network are updated by executing the stochastic gradient descent method to minimize the loss function. The θ update process is where η θ ∈[0,1] is the learning rate. The policy gradient method is used to update the model parameter φ of the policy network to maximize the target. The φ update process is: in is the performance target of the policy network, and λ is the normalization factor.

[0104] The present invention is based on the efficient allocation of resources such as energy and time, and designs a reasonable beamforming scheme to optimize the energy efficiency of the federated learning system and ensure the convergence and accuracy of federated learning.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein, but these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An air federated learning intelligent resource allocation method based on wireless energy communication, characterized in that: The following steps are involved: Step S1: In the first stage, the base station obtains the total energy information of each user and determines the resource allocation plan. Then the base station uses wireless energy communication technology to broadcast the global model and send radio frequency energy to charge the user. Assume that the channel between the base station and user k is M is the number of base station antennas. The base station obtains the total energy information of each user by receiving pilot signals; In the tth communication cycle, the base station uses wireless energy communication technology to send a signal s carrying global model information and radio frequency energy. 0,t , the signal received by user k is expressed as: Send beamforming vectors to the base station, The variance for user k is Gaussian white noise, K is the number of users; first, power division is performed, that is, user k is allocated according to the power allocation ratio λ k,t The λ in the received signal k,t Part of it is used as useful information for information decoding, and we get: is the useful signal carrying the global model received by user k, Gaussian white noise introduced for information decoding, is the variance of Gaussian white noise; energy is collected at the same time, that is, user k will receive 1-λ in the signal k,t Part of it is converted into the collected energy, which is expressed as: τ 1,t is the time of wireless energy communication, is the energy conversion efficiency; The optimization problem targeted by the method for determining the resource allocation scheme is as follows: The energy efficiency of the system in the tth communication cycle is defined as the ratio of the air computing rate to the total power consumed by the system, that is, The total power consumed by the system in the tth communication cycle is defined as: P c is the power consumed by the base station signal processing circuit, v k,t is the transmission beamforming vector of the base station to user k; The system energy efficiency is summed and averaged over time to obtain the optimization equation of the system and the complete optimization problem as follows: Among them, ε t ≤ε, is the convergence guarantee equation for federated learning, that is, the constraint condition to ensure that the designed air federated learning intelligent resource allocation method meets the convergence requirements, ε t is the error term, ε is the upper bound of the error, For any t∈{1,…,T}, For any k∈{1,…,K}, in is the global model received by user k Relative to the initial global model w 0,t The deviation, is the aggregation model w received by the base station t+1 Relative to the ideal aggregation model Deviation; is the QoS guarantee equation, is the rate of downlink wireless energy communication, is the signal-to-noise ratio of user k; τ 1,t +τ2+τ 3,t ≤τ max , is the time constraint equation, τ max is the maximum time constraint for a communication cycle; 0≤λ k,t ≤1, t, is the power split ratio constraint equation; t is the power constraint equation of the base station and user k, and The maximum power constraints for base stations and users; t is the energy constraint equation of user k; The method for determining the resource allocation scheme has the following specific operation process: Step 1: Deploy two agents at the base station to complete two subtasks, namely, minimizing total power and maximizing computing rate. The goal of agent 1 is to minimize total power, and the goal of agent 2 is to maximize computing rate. Step 2: Model the system's long-term energy efficiency optimization problem as a Markov decision process to obtain the state space, action space, and reward function required for Markov decision making; (1) Define the state space, action space and reward function of agent 1: In the tth communication cycle, the state space of agent 1 is expressed as: The action space is represented as: a 1,t =(v t ,τ 1,t ,λ t ,p t ), the reward function is expressed as: (2) Define the state space, action space and reward function of agent 2: In the tth communication cycle, the state space of agent 2 is expressed as: The action space is represented as: a 2,t =a t , the reward function is expressed as: Among them, the condition ifε t ≤ε reflects the convergence constraint of federated learning; when the error term ε t Less than the upper bound of error ε, r 1,t (r 2,t ) as a positive reward helps the agent get a larger reward; when the error term ε t Exceeding the given upper bound ε, r 1,t (r 2,t ) acts as a penalty to guide the agent to reach the goal; Step 3: Use deep reinforcement learning methods to maximize the long-term energy efficiency of the system, determine the base station transmit beamforming, time allocation and user power partitioning scheme for the first phase, and the user transmit power and base station receive beamforming scheme for the third phase; Deep reinforcement learning trains the action value network by optimizing the action value function. The iterative expression of the action value function is: is the optimal strategy, s t is the state of agent 1 or agent 2 in the t-th step training, r t is the reward for training agent 1 or agent 2 at step t, s t+1 The next state of agent 1 or agent 2 is trained for the tth step, γ is the discount factor, and θ is the model parameter of the action value network; each agent obtains state information by observing the environment, performs actions based on the state information, and forms experience by combining the reward and the next state. The mean square error method is used to train the action value network, and the loss function is expressed as: Q(s t ,a t ; θ) is the output value of the action value network after observing the state and executing the action, Φ t As the goal, if the training does not reach the required number of steps, then Φ t =r t +γ(Q(s t+1 ,π(s t+1 ;φ');θ'),π(s t+1 ; φ') is the target policy network, φ' is the model parameter of the target policy network, Q(s t+1 ,π(s t+1 ; φ'); θ') is the target action value network, θ' is the model parameter of the target action value network, γ is the discount factor; if the training reaches the required number of steps, then Φ t =r t ; Update the model parameters θ of the action value network by executing the stochastic gradient descent method to minimize the loss function. The θ update process is θ←θ-η θ ▽L(θ),η θ ∈[0,1] is the learning rate; the model parameter φ of the policy network is updated by determining the policy gradient method to maximize the target, and the φ update process is φ←φ+η φ ▽J(φ), where is the performance target of the policy network, λ is the normalization factor; Step S2: In the second stage, after receiving the global model, the user uses the collected environmental data to perform multiple rounds of iterative local learning, and then uses the stochastic gradient descent method to obtain the updated local model; After receiving the global model, the user uses the collected environmental data to perform local learning for τ rounds of iterations, and then uses the stochastic gradient descent method to obtain the updated local model. The updated local model is expressed as: is the global model received by user k, is the local loss function, is the local model obtained by the l∈{0,…,τ-1}th round of iteration, is the sampling set of the local data set, η is the learning rate, and τ is the number of iterations of local learning; The energy consumed by local learning is expressed as: C k The number of CPU cycles for processing one bit of data for user k, Z k is the number of bits in the sampling set, f k is the CPU frequency of user k, κ is the effective capacitance coefficient; The time consumed by local learning is expressed as: τ 2,k =τC k Z k / f k ; Due to the need to meet the time synchronization characteristics of air computing, the time consumed by local learning is the longest time spent by all users, which is τ2=max k (τ 2,k ); Step S3: In the third stage, using air computing technology, all users upload the updated local model at the same time and frequency, and realize model transmission and aggregation in the air; Using over-the-air computing technology, the user transforms the normalized signal s k,t With power p k,t Upload, the air computing technology uses the superposition characteristics of the wireless channel to perform model aggregation while communicating; the signal received by the base station is: is the base station receive beamforming vector, is the Gaussian white noise of the base station, σ 2 is the variance of the base station Gaussian white noise, I is the M×M dimensional identity matrix; The aggregate model mean square error is expressed as: The air calculation rate is expressed as: in, is a perfectly aggregated signal, σ 2 is the variance of the base station Gaussian white noise, is the total power of the superimposed signal, B is the bandwidth; The energy consumed by user k uploading the local model is expressed as: is the air computing time, W is the number of bits of the model uploaded by user k; after the tth communication cycle, the total amount of energy remaining for user k is expressed as: is the energy harvested by user k in the i-th communication cycle, is the energy consumed by user k in uploading the local model in the i-th communication cycle, E max is the battery capacity of user k.

Images